Meta - token usage 5.43 standardisation effort

We are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can be around the one release - documentation is here

The goal is to consolidate such that

1. All token rendering is done through the token processor, token hooks are redundant and deprecated
2. All functions on CRM_Utils_Token are unused in core and deprecated except for those in the legacy BAO mailing classes.
3. We freeze, deprecate and phase out in legacy BAO classes in favour of flexmailer
4. All core classes that render tokens do so using consistent token naming & output conventions (rather than dates being formatted in various ways and fields sometimes outputting labels and other times values or name fields #2650 (closed))
5. As much as possible we render via tokens rather than smarty in workflow message templates (that couples them with the entity rather than any form flow)
6. pdf & email & scheduled reminder tasks consistently offer the same entity-specific, contact & domain tokens
7. We can start to define obvious missing tokens such as {domain.now} (done), {contribution.balance} and formatted addresses

Task list / streams - this gives some idea of where things are progressing or blocked

• standardise contact tokens to support/ advertise label style (e.g {contact.prefix_id:label}

  • make Contact tokens work with partial contact inputs - currently they only work with no contact array passed in, or an array with all possible values - currently blocked on a caching bug https://github.com/civicrm/civicrm-core/pull/21790
  • split contact tokens into own class & extend entity tokens like the other classes(blocked on stdise contact tokens)
  • switch contact tokens to use apiv4
  • work through standardising location token loading (this could go further - ie I had thought about adding loading for billing

• date formatting standardisation documentation #2746 (closed)

• locale sensitive standardised money formatting #2638 (closed) & pr

• switch badge tokens to use token processor

• event tokens cleanup & loading fixes + remove unindexed query, move the fee_amount and balance tokens to the participant tokens class, fix event class to listen to either 'participantId' or 'eventId'. Participant class should run first & load eventId if required. Events should be cached in a static cache as it would be very rare to message thousands of events in one run.

• Recurring contribution tokens work (currently availble in only the recurring_edit template with a pr to extend to cancelled recurring https://test.civicrm.org/job/CiviCRM-Core-PR/44632/

• pdf letter is sane (#2790 (closed)) enough to extend to deliver tokens for other entities (participant #780 (closed))

• email letter extend to deliver tokens for other entities (membership & participant) (#2862 (closed))

• scheduled reminders supports participant tokens #779 (closed)

• domain tokens are consistently available & render #2838 (closed) - done but a gap on case tasks as they prioritise being able to filter tokens by case type

• install flexmailer on new installs #2836 (closed)

• fix remaining places that process tokens other than via the payment processor

  • eliminate replaceGreeting - currently blocked on apiv4 caching bug
  • fix CRM_Contact_BAO_Contact_Utils::updateGreeting to not call getTokenDetails
  • confirm civicrm_api3_mailing_preview is replaced by flexmailer
  • remove getTokenDetails call from CRM_Contribute_Form_Task_PDFLetter - note this appears to be an expensive way to call the contact api & tokens are unused https://github.com/civicrm/civicrm-core/pull/21805
  • remove extra rendering in BAO_Pledge https://github.com/civicrm/civicrm-core/pull/21789
  • removed getTokenDetails from updatePledgeStatus https://github.com/civicrm/civicrm-core/pull/21847
  • remove getTokenDetails from transitionParticipants
  • remove getTokenDetails from sendCancellation & cancelParticipant in selfsvc flow
  • deliverGroup - replaced by flexmailer?
  • MailingPage::preview - replaced by flexmailer?
  • remove hook:;tokenValues from label task
  • getAnonymousTokenDetails ???
  • confirm BAO_Mailing::compose is only via flexmailer

• communicate rc status via dev list

• communicate changes via blog

• upgrade message #2871 (closed)

• docs update to these instructions / check it can still all be done https://docs.civicrm.org/dev/en/latest/step-by-step/create-custom-case-token/

Currenly out of scope

• Token.getlist api - until we have this we have an issue with audience-filtering and with switching tokens depending on the schema. Also case tasks still have to render tokens weirdly without this because they currently filter by case type with partial success #2788 (closed)

• Clarify how extensions should render tokens #2863 (closed)

• encourage flexmailer with status checks on existing installs #2836 (closed)

Analysis of current token rendering it core:

Method	Used in	Status
Use the token processor	scheduled reminders, civimail with flexmailer in use, activity pdf task	This is the method we are consolidating all paths to use.
Use the functions in CRM_Utils_Token	PDF tasks other than activity	Migrated (PRs on membership and contribution. Also work on fixing up the pdf task class supportingadd participant tokens (once we’ve finished the standardisation of those)
	Email tasks	Migrated to token processor. Added tokens for other entities (participant, membership #1396 (closed)) also see
	Export tasks (when merging contacts in export)	Migrated to token processor
	Profile link rendering	Migrated to token processor
	Storing contact greetings (eg. email_greeting_display)	In progress. Currently resolving inconsistent token syntax and load issues as this needs to be able to use partial pre-loads
Use random ad hoc code	Event badges	migrated - this is the only place outside of scheduled reminders where event or participant tokens were used in core
	Send SMS task (BAO_Activity::sendSMS)	migrated
	Labels task CRM_Contact_Form_Task_Label	argh
	CRM_Event_Form_SelfSvcTransfer	Still pretty argh - but this is a workflow message template so the fix is a bit different
	Transition participants	Also a workflow - possibly redundant as seems to be mostly about contact tokens which are resolved in the send function anyway
	Pledge acknowledgements	Also appear to be redundant
	CiviMail with flexmailer not used (many functions & classes)	Push the switch to flexmailer.
	Various unsubscribe / resubscribe type actions	Probably out of scope for this round