URL Tokens - CiviMail BAO and FlexMailer disagree over duplicate `http://`

Overview

CiviMail BAO and Flexmailer provide two mechanisms to rendering/delivering mail-blasts. To phase out BAO-rendering, we should eventually migrate existing sites to Flexmailer. However, small discrepancies between them can be an obstacle. This issue asks how to deal with a discrepancy when handling certain URL tokens.

(The issue is an off-shoot from the review discussion on https://github.com/civicrm/civicrm-core/pull/21522. However, they are somewhat distinct as 21522 applies to new-sites and this problem is more pressing on upgrade-sites.)

Background

Suppose you have an email token which generates a URL (eg {action.forward} or {action.unsubscribeUrl}) . The most correct way to use this token in an HTML-email is to place it in a hyperlink (<a href="{action.forward}>).

However, if you use CKEditor to compose a message, the hyperlink dialog strongly encourages you to put http:// at the front of any hyperlink. Thus, you can organically produce expressions like <a href="http://{action.forward}">.

CkeditorHttpToken

CiviMail BAO has a feature which mitigates this - in effect, both notations give the same output. However, Flexmailer (TokenProcessor) does not have a similar mitigation. Thus:

HTML Email Expression	BAO Output	Flexmailer Output
<a href="{action.forward}>	<a href="https://example.com/civicrm/mailing/forward?...	<a href="https://example.com/civicrm/mailing/forward?...
<a href="http://{action.forward}>	<a href="https://example.com/civicrm/mailing/forward?...	<a href="http://https://example.com/civicrm/mailing/forward?...

Note the duplicate URL scheme (http://http://).

Scenarios: Good

There are several scenarios wherein this discrepancy does not matter:

• You use CKEditor + BAO-renderer. (The BAO renderer mitigates extraneous http://.)
• You use Mosaico + Flexmailer. (Mosaico's UI doesn't create extraneous http://.)

(I suspect these two cohorts are the largest.)

Scenarios: Bad

The potential for difficulty arises when mixing CKEditor with Flexmailer -- you take some HTML that was composed with CKEditor, run it through Flexmailer, and now you have invalid URLs with http://https://.

If we flip over more existing sites to Flexmailer, then it is foreseeable that more sites will be in this mixed scenario.

There's a secondary consideration in who may experience problems -- templates. Most users are not in the habit of typing {action.forward} or {action.unsubscribeUrl} regularly. Anecdotally, the typical practice is to put these kinds of tokens into a template, eg

• (A) Add a "User-Defined Template" (edited w/CKEditor in web; stored in civicrm_msg_template), or...
• (B) Reuse/copy a previous "Mailing" (edited w/CKEditor in web; stored in civicrm_mailing), or...
• (C) Configure default header/footer (no CKEditor; stored in civicrm_mailing_component)

Practical recap: today, if you enable Flexmailer on a site that uses CKEditor for mailings, then either:

• Your mailings and templates will have broken links -- because they use <a href="http://{action.unsubscribeUrl}">.
  • This seems more likely if you use (A)/(B) - because the web UI uses CKEditor.
• Your mailings and templates will be cleanly interoperable -- because they use <a href="{action.unsubscribeUrl}">.
  • This seems more likely if you use (C) header/footer because the web UI uses textarea.

Approaches

1. Add a mitigation to Flexmailer or TokenProcessor which removes extraneous schema during the rendering phase.
   • Strength: Best interoperability
   • Criticism: Adds quirky bits to rendering.
   • Variations: Patch Flexmailer vs patch TokenProcessor.
   • Variations: Apply cleanup before rendering (http://{action.*} => {action.*}) or after rendering (http://https?:// => https?://).
2. Add an upgrade-task to remove extra http:// in front of tokens.
   • Strength: No quirky bits in rendering.
   • Criticism: CKEditor is still there. Can only upgrade core tables. Rewrites history (civicrm_mailing.body_html of past mailings).
3. Do nothing - Downstream users should manually convert their mailings/templates.
   • Strength: Look, ma, no hands!
   • Criticism: We cannot measure how many users will be impacted by break. For users with a "Reuse/Copy" workflow, they cannot fix historical mailings in UI.