Upgrader - Apply extension updates after core updates

Overview

Functionality in civicrm-core is increasingly organized with internal extensions (civicrm-core:ext/*). The main upgrade workflow should incorporate the upgrade-steps for extensions.

Example use-case

1. Extract tarball with a new version of civicrm-core, including updated ext/*
2. Run the main upgrader (civicrm/upgrade?reset=1, cv core:upgrade, drush civicrm-upgrade-db, or wp civicrm upgrade-db)

Current behaviour

It updates the schema for things like civicrm_contact (eg CRM/Upgrade/*) but not things like civicrm_search_segment (eg ext/search_kit/CRM/Search/Upgrader.php).

As a sysadmin, you must run the upgrades separately.

Most are not habituated to running these together. This leads to confusing user-stories like https://civicrm.stackexchange.com/questions/42094/civimail-issue-db-error-no-such-table-when-sending-mailing (where the first upgrade seemed to work - but you get random errors because the second upgrade hasn't run).

Proposed behaviour

Run both.

The core-upgrade should delegate/incorporate the ext-upgrade, and it should preserve essential elements of the ext-upgrade contract. Specifically: ext-upgrades can use features/services/APIs from core. This implies that (1) ext-upgrades run after core-upgrades and (2) when it gets to ext-upgrades, it should follow a normal/open dispatch-policy.

Comments

Filing issue after some Mattermost discussion with rudanpal, @colemanw, @bgm, @totten: https://chat.civicrm.org/civicrm/pl/b76y4gy517yxdet331ueqrxgrw

I posted a draft at https://github.com/civicrm/civicrm-core/compare/5.50...totten:run-ext-upgrades?expand=1

As mentioned in PHP comments+Mattermost, there's an issue with the current draft -- when it gets to ext-upgrade tasks, it would (unexpectedly) continue to apply the limited dispatch-policy used for core-upgrades (upgrade.main vs upgrade.finish). Some ways to address that:

• Runner state: While the upgrader goes through phases (ex: 1-core upgrade, 2-ext upgrade, 3-new exts), keep a stored value naming the phase. this stored value determines the active dispatch-policy
  • ex: Civi::settings()->set('upgrade_phase', '...')
• Task wrapping/task tagging: When taking ext-upgrade tasks and putting them into the core-upgrade queue, put some wrapper or tag on each one to indicate the phase/policy that it relies on
  • wrapper example: `CRM_Upgrade_Form::wrappedTask(string $dispatchPolicy, array $realCallback)
  • tag example: $queue->createItem($task, ['dispatchPolicy' => string])