Replace `civicrm-packages.git` with micro repos and composer-downloads-plugin
Background
CiviCRM stores a number of dependencies in the civicrm-packages.git
repo. This has allowed those dependencies to retain their historical file-loading-mechanisms/file-organization, and it has accommodated (for good and ill) adhoc forking.
However, it also adds special steps during day-to-day build/development tasks -- e.g. when creating a clean local build from source, when switching between versions, when bisecting to track a regression, when tagging/branching releases, or when running distmaker
.
A desirable end-state would be to make civicrm-packages.git
obsolete -- by updating composer.json
to require
equivalent dependencies. This is a known albeit cumbersome process:
- (1) determine the old version of code;
- (2) determine which (if any) patches are applied;
- (3) load an equivalent package from
composer.json
(requires:
); - (4) validate/port/discard any patches
- (5) find and update anything that relies on the old location;
- (6) remove the old code from
civicrm-packages.git
- (7) test relevant use-cases; and
- (8) coordinate testing/merging of two interdependent PRs (
civicrm-core.git
+civicrm-packages.git
)
Because the process is somewhat cumbersome and brings risks, many packages are still waiting around for migration.
Goal
Enable composer
to download packages
-- simplifying some dev tasks (triage/bisection/git-pull) and simplifying the Civi-D8 install mechanics (s.t. it's easier to have a clean build). Do this by default, without requiring any special configuration or add-ons. Tangentially, make incremental progress toward de-forking packages
.
Approach
-
Split
civicrm-packages.git
. Copy each subdir (e.g.civicrm-packages.git/foobar
) into a new, separate git repo (e.g.https://lab.civicrm.org/dev/packages-foobar.git
). In each repo, add a README to link back to some docs aboutpackages
maintenance. Use a quickie script to automate the split. -
Update
civicrm-core.git:composer.json
. Under extra.downloads, add a step to download/extractpackages/foobar
. (*The download URL should be pinned to a specific tag/commit forpackages-foobar.git
.) - Update
civibuild
,gitify
, etc to not clonecivicrm-packages.git
.composer
can now download that stuff.
Benefits
- When you get a copy of
civicrm-core.git
and runcomposer install
, it downloads all thepackages
. - When you download
civicrm-core
as a dependency (as with Civi-D8), it will putpackages
in their normal spot ($civicrm_root/packages
) without requiring any special top-level configuration. - If you're upgrading/downgrading/bisecting/triaging, the steps for switching code are simplified...
- Old:
git checkout <foo> && composer install && cd packages && git checkout <foo> && cd ..
. - New:
git checkout <foo> && composer install
- Old:
- When bisecting (trying out commits from the alpha/beta stages), you will get the mix of dependencies appropriate to the specific revision, rather than an approximation from some whole-numbered release.
- When switching versions, it will only download packages if they have changed - and such downloads will be small/focused. (And if you juggle several builds/versions,
composer
will cache the downloads.) - The de-forking process becomes easier. Before, steps 6+8 required you to juggle interdependent patches in two repos (which doesn't jive well with test/review processes). Now, de-forking can be done atomically within
civicrm-core.git:composer.json
. (To remove the forked package, remove it fromextra.downloads
. To add the canonical package, add it torequires
.) The test/review process should feel more normal.
Caveats
- If you actually intend to modify a forked copy of a dependency, that will become more cumbersome.
- The new process for patching a forked dependency (such as PEAR
Mail
) would be:- Do a pull request for
packages-mail.git
(updating the forked code) - Make a tag in
packages-mail.git
(or note the specific commit) - Do a pull request for
civircm-core.git:composer.json
(updatingextra.downloads
to use the new tag/commit)
- Do a pull request for
-
This doesn't bother me too much - in practice, we don't change
packages
often. Also, it's not bad to tilt the balance away from "easier to stay forked" and toward "easier to de-fork".
- The new process for patching a forked dependency (such as PEAR
- Overall, this issue describes an incremental change. It makes some things better, but it doesn't make any one thing great.
Decision-making
I'm having trouble qualifying how much buy-in to seek/build for this -- because the proposal is sort of a contradiction:
- It's a major change which puts a bunch of files in new homes and affects developer workflows
- It's a trivial change for which we can demonstrate (via
diff -r
) that on-disk file-content is the same before+after. - It's an improvement that should simplify various developer tasks
- It's not an improvement because users wind up with the same code/functionality/features/performance/bugs.
- For custom build systems, it should be a non-issue (because you're already required to run
composer install
). - For custom build systems, it could be a breaking change (because you need to not checkout
civicrm-packages.git
)