SearchKit can't export more than a couple thousand rows
Overview
When attempting to "Download Spreadsheet" in SK, it times out with more than a couple thousand rows. It's very PHP-intensive to do the pseudoconstant lookups, Smarty calculations, etc. That's not noticeable when it's calculating a page of 50, but doesn't scale.
Reproduction steps
- Create an SK query that yields a large number of contacts. Ensure you have 2-3 pseudoconstant fields in your
SELECT
. - Attempt to export with "Download Spreadsheet"
Current behaviour
Times out, but appears to still be downloading.
Expected behaviour
Finishes successfully, or at least reports back to the user that the export won't finish.
Comments
Most systems that offer exports of large data sets - including our competitors like NationBuilder etc., but also many apps like PayPal - do not attempt to offer the export in real-time. They queue the export, then you visit a queue page to download the completed exports (most systems will email you when it's ready).
@eileen @totten I know you've both been working on queuing mechanisms. I can drum up some funding (maybe $750 USD? Need to check) to support this. I'm not sure if it's a heavy lift or a light one though, so I don't know if that's a substantial amount of funding.
Ideally, this would work like imports or CiviMail, where the queue job can build the export over multiple cron runs. The client who would fund this is on Pantheon, which doesn't have an unlimited cron job run time.