Drupal's optimize CSS and JavaScript setting causing problems in multiple server environments

There's a common Drupal setting that allows you to optimize and JavaScript or CSS in the Site Configuration > Performance menu. Turning this setting on will minify and concatenate all of these used files into a single JavaScript and single CSS file. Doing this lessens the total number of server requests and total size of those requests, resulting in a faster user experience.

An interesting issue arises when using Drupal in a multi web server environment. In some multi server installations, Drupal shares the same exact /sites/site-name/files/ directory across all the web servers. This can be accomplished through a mounted share to all web heads. This would be ideal, and I've used this in the past. However with a new group of sites I'm working on, this mounted share is not in place and the decision to implement it is out of my hands. To make Drupal uploads work, they've implemented a solution that monitors the /files folders for uploads and then mirrors across the web servers using rsync when it finds a new asset.

This works seemingly well, but consider what happens on a high traffic site receiving many requests per second - there's the possibility of a user seeing a page with a missing image in the time it takes rsync to catch up. Okay... not the end of the world and very transient.

While the temporary missing image bug is minor, turning on optimize CSS and JS creates a big issue as it puts its created compressed files into the /files directory. We have a single *shared* memcached instance that is being used for all Drupal cache tables through the popular Drupal Memcache module. This combination creates an series of events that continually breaks the pages:

  1. The Drupal cache (a shared memcached instance in our case) is cleared.
  2. Drupal page Z is loaded and a compressed copy of the CSS and JS is put into the /files directory on web server A with a unique hashed name which is stored in the database for future use.
  3. The above page's HTML is cached in memcached, including the path to the compressed CSS and JS files.
  4. Servers A, B, C, and D load new pages in varying orders, but sometimes are missing the compressed JS and CSS in /files and go about creating new compressed files and cleaning out outdated compressed files.
  5. Rsync continues to spin, catching up and getting out of date.
  6. Page Z is loaded again with the HTML stored in memory, but the page breaks because the cached .css and .js file paths that are in its HTML are long gone. This page will continue to break until the cache's ttl runs out or the cache is manually flushed.

Imagine the above sequence in a high traffic site repeating itself over and over again in slightly different ways, causing transient page breakages.

Given the decision to not mount a share on the /files directories, the easiest solution to the problem was to turn off CSS and JavaScript compression and live without the potential performance benefits. It's not an ideal solution, but it's better than a intermittently broken site.