Improve administrative experience for managing/upgrading bundled extensions

Background

The canonical release of CiviCRM now includes a couple extensions (e.g. org.civicrm.api4 and com.iatspayments.civicrm). The business-need for distributing bundled extensions will continue to grow as we progress on "leap by extension".

Problem

Suppose we have a process like this:

1. Administrator Alice downloads/installs CiviCRM 5.0 with a bundled copy of an extension, APIv4 (4.0.0).
2. Developer Bob publishes a new version of APIv4 (4.0.1).
3. Alice sees APIv4 (4.0.1) and upgrades it.
4. Bob publishes another version of APIv4 (4.1.0).
5. The next version of CiviCRM (5.1.0) is released. It bundles APIv4 (4.1.0).
6. Alice installs CiviCRM (5.1.0). It includes APIv4 (4.1.0), but Alice's system still uses APIv4 (4.0.1).

The situation should be resolvable by either removing or upgrading Alice's extra copy of APIv4 (4.0.1) stored in the site's customizable extensionsDir (e.g. $WEBROOT/sites/default/files/civicrm/ext).

However, the situation can be confusing (leading to support requests). This particularly true if there are any intertwined dependencies.

Analysis

Why does this happen? Well, CiviCRM loads extensions from a list of prioritized layers:

• (E1) Extensions installed by the web-based administrator in the customizable extensionsDir (e.g. $WEBROOT/sites/default/files/civicrm/ext).
• (E2) Extensions installed with the CiviCRM bundle (e.g. $WEBROOT/sites/all/modules/civicrm).
• (E3) Extensions installed in site's master composer project (e.g. $WEBROOT/vendor).

When Alice (step 3) downloads any extension manually (whether upgrade or anew), it goes into the first layer E1. The version she specifically downloads will always take precedence over any versions in layer E2/E3.

Generally, this prioritization is a result of two opinions that:

• Web-administrators tend to be "closer to the ground" on the goal/situation for a particular site -- their upstream providers (e.g. civicrm.org and/or saas/partner) are more disconnected. Therefore, if the admin forms an opinion about what version to run, then it should take precedence.
• Only one of those layers (the first) tends to be writable at run-time. This may be a general security prophylactic, or it may specifically be a protection in multitenant deployments.

Those opinions are valid in many cases, but they are certainly not universal. For example:

• Many web-based administrators are only casually engaged (e.g. they're not tracking the nitty-gritty of each release every day), and many don't really want to think about the versions of each extension. They may not realize their upgrade today causes them to take greater ownership over future upgrades.
• Many WordPress and Joomla builds are usually web-writeable, which means that they are not confined to the customizable extensionsDir -- one can put upgraded code in any location desired/appropriate/necessary.

Possible Resolutions

There are several possible ways to resolve this. The choice is an ambiguous question -- none is stand-out obvious winner; each option involves a trade-off which has some impact on the developer-experience/administrator-experience and which will likely be good and bad for different groups of people. However, to resolve the ambiguous question, it may help to list the possible resolutions:

• REORDER: Change the order of the layers -- put the custom folder (E1) below the others (E2/E3).
  • Strengths: It is probably the simplest change (i.e. the explanation and implementation are both short).
  • Challenges: It inhibits the ability of site-owners to act as testers and early-adopters on those extensions. Also, when it goes live, it may cause unexpected changes for sites who have tried to manage these extensions on their own.
• BLOCK: In step 3, block Alice from upgrading the extension. She may only upgrade an extension if it's stored in E1. If an extension is provided by another layer (E2/E3), then prohibit downloads/upgrades that would go into E1. Provide messaging to indicate why upgrades aren't allowed.
  • Strength: This tightens the general shape of deployments across the ecosystem, which would be an overall simplification that makes it easier to reason about the system.
  • Challenges: It inhibits the ability of site-owners to act as testers and early-adopters on those extensions. It also puts a greater responsibility on upstream bundler to issue a new bundle whenever one if its constituent parts has a critical fix.
• ADVISE: After step 3, the "Manage Extensions" screen (and possibly the status-check screen) should display a notice indicating that an extension is overridden. The administrator still has the prerogative here, but the app should try harder to educate/inform them about the consequences/responsibilities.
  • Strength: This is the most backward-compatible resolution -- it doesn't change the real behavior, so no one can accuse of breaking anything.
  • Challenges: The underlying situation (multiple copies of the same extension) can be confusing, and we're giving more of a blessing. That may lead to questions like "what order should I apply upgrades in?" for which there may not be a simple+generic answers.
• UPGRADE-IN-PLACE: In step 3, allow Alice to upgrade the extension. However, the upgrade should not go into the extension folder -- it should replace the existing code (wherever that may be). If the folder is not writeable, display a warning with advice on how to make it writeable/upgradeable.
  • Strength: Most people reading the user-interface (but without any expertise in CMS+Civi file-structures) would probably expect this behavior.
  • Challenges: The extensions can live in spaces which we don't "own" (like E3 - $WEBROOT/vendor). It's a bit scary to automatically delete a whole file-tree if we haven't traditionally "owned" that part of the filesystem. Also, if Alice in step 3 had upgraded really far ahead (APIv4 4.3.beta1), then the upgrade in step 6 would revert her system to 4.2.0.
• DEFER => COMPOSER: This issue reflects only one of several edge-cases in managing dependencies for a build. Don't spend any more time trying to solve this one scenario -- instead, focus energy on adopting a more powerful build system like composer, where their documentation/workflows/UIs address the various edge-cases.
  • Strength: More comprehensive approach. More "industry standard".
  • Challenges: Generally, longest process of any of this list. Most extensions aren't currently accessible via composer (although there's a proof-of-concept bridge). composer is primarily a CLI tool, and it's not currently widely deployed among Civi sites, so adoption will be an undertaking.
• VERSION-PRECEDENCE: Continue to allow extension code to be stored in all these locations. However, change prioritization -- it doesn't matter where the extension is stored. It only matters what the version number is. Highest local version wins.
  • Strength: This also feels intuitive to me.
  • Challenges: During the life of an extension, the absolute URL+path of its assets may fluctuate. For some use-cases (where hyperlinks and paths are integrated with another system or stored as content), this could cause problems.