Improve process for creating plain text version of mailing

Overview

Convert any html contained in tokens to plain text for plain text emails

Before

HTML in tokens that contained HTML was being outputted to plain text versions of emails.

After

HTML in tokens is converted to plain text.

Technical Details

This issue is NOT about generating plain text and HTML versions of tokens. It is about converting an entire email from HTML to plain text when no plain text version is given. This conversion currently proceeds as follows:

1. Check to see if a plain text version has been submitted. If not
2. Convert the HTML to plain text
3. Do token substitution

The problem is that if the token contains html, that HTML persists in the email. The proposed solution is to do the conversion from HTML to plain text after token substitution, i.e.

1. Check to see if a plain text version has been submitted. If not
2. Convert the HTML to plain text
3. Do token substitution

This patch is split between PR 12061 on core and PR 20 on flexmailer.

The flexmailer patch will continue to report a single failure until the core patch is merged, at which point we can hopefully run the tests against flexmailer again and merge this patch.

@totten suggested defining tokens with TokenProcessor, whose tokens auto-convert from HTML to plain text and visa versa. However, I am not sure this is going to work for the following reason. When you convert from HTML to plain text (at least with the HTML to plain text converter that we are using) the structure of the text is altered and links appear at the bottom of the converted text. Therefore, if you run multiple conversions, you will get multiple lists of links scattered throughout the email. Best to do html to plain text conversion at the end of the process.

All this is not to say that one shouldn't define plain text and HTML versions of tokens. I think it is fine to do that if you want to. I just think that automatic conversion of HTML to plain text should happen one time only, after the entire email has been assembled.

Note that PR 12061 on core removes the check for nofollow in the email html output of the test CRM_Mailing_BaseMailingSystemTest::testHtmlWithOpenAndUrlTracking. The intent of https://issues.civicrm.org/jira/browse/CRM-21768 is not affected. nofollow still appears in the public view (e.g. http://example.org/civicrm/mailing/view) but it is not tested.

I would have added it to a test of civicrm/mailing/view but I could not find any tests there.

This issue was originally discussed here: https://issues.civicrm.org/jira/browse/CRM-21197 and a PR submitted here: https://github.com/civicrm/civicrm-core/pull/10998. But the PR was rejected because it caused issues with Flexmailer.

Also, on the seperate issue of click tracking plain text emails. @totten - you said here that you thought it was a significant policy change. It isn't as significant as you think. Links that are tracked in HTML continue to be tracked in plain text. Links that are not tracked in HTML (because they were not in an <a> tag) are not tracked in plain text either. i.e. there is no significant loss of functionality.