extdir feed performance issues

We had an oom last night (maybe unrelated) and then today the feed started causing problems this morning (EST time), eventually causing the server to be unresponsive.

Logs had some very weird symptoms, such as:

PHP Warning:  fopen(wordpress/sites/civicrm.com/wp-content/plugins/yooessentials/checksums.pass): Failed to open stream: Permission denied in wordpress/sites/civicrm.com/wp-content/plugins/yooessentials/classes/Helper.php on line 136

which seemed like it was an open-file-limit issue, because the permissions were fine, and eventually the problem went away on its own:

# ls -la wordpress/sites/civicrm.com/wp-content/plugins/yooessentials/checksums.pass
-rw-rw-r-- 1 aegir www-data 2 Dec 19 15:34 wordpress/sites/civicrm.com/wp-content/plugins/yooessentials/checksums.pass

# ls -lad wordpress/sites/civicrm.com/wp-content/plugins/yooessentials/
drwxrwsr-x 4 aegir www-data 4096 Dec 19 15:34 wordpress/sites/civicrm.com/wp-content/plugins/yooessentials/

I was also seeing a lot of errors with the extdir feed, so I tested disabling that as a way to reduce the load. I made extdir/src/Controller/SingleExtController.php throw an Exception. The server became responsive again pretty much immediately. Uncommenting it crashed the server again.

After testing whether it was #1037 (closed) that was causing problem (it didn't change anything), I commented out the filter for listing all extensions (not just reviewed). That flag was known to significantly slow down the feed, but it gets heavily cached. Performance came back to normal. Uncommenting slowed performance again.

Looking at the history for slow queries:

Last 7 days:

It could be a cache/timing issue, or for some reason, that feed now pushed the server resources over the edge.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information