Mailing issueshttps://lab.civicrm.org/dev/mail/-/issues2022-06-11T14:55:17Zhttps://lab.civicrm.org/dev/mail/-/issues/106Proposal: Scheduled Reminders should not (by default) use Smarty on new installs2022-06-11T14:55:17ZJonGoldProposal: Scheduled Reminders should not (by default) use Smarty on new installsWhen token rendering is enabled, and the `TokenProcessor` context says to use Smarty, the TokenRow rendering barfs on any curly brace that isn't a token or Smarty. Which includes, for instance, [inline CSS](https://civicrm.stackexchange...When token rendering is enabled, and the `TokenProcessor` context says to use Smarty, the TokenRow rendering barfs on any curly brace that isn't a token or Smarty. Which includes, for instance, [inline CSS](https://civicrm.stackexchange.com/q/17643/12).
At the time that SE post was made, Smarty stripped out the CSS. It no longer does, and instead generates large numbers of errors. Rather than fight that battle, I'm curious whether Smarty on scheduled reminders is necessary and serves a purpose in a post-5.43 TokenProcessor world.
If we decided that Scheduled Reminder TokenProcessors had [Smarty set to false](https://github.com/civicrm/civicrm-core/blob/ce0d67d6a686fcfae1d7046e615107f25cec501a/CRM/Core/BAO/ActionSchedule.php#L647) this problem would go away.
Does anyone use Smarty here? Should they?JonGoldJonGoldhttps://lab.civicrm.org/dev/mail/-/issues/98Set time limit for bounce types with hold_threshold > 12022-06-11T14:50:53ZMichael McAndrewSet time limit for bounce types with hold_threshold > 1Bounce types can be split into those with a hold_threshold = 1 where a single bounce should cause this contact to be put on hold (e.g. mailbox non existent), and those where more than one bounce is required (mailbox full).
We don't curr...Bounce types can be split into those with a hold_threshold = 1 where a single bounce should cause this contact to be put on hold (e.g. mailbox non existent), and those where more than one bounce is required (mailbox full).
We don't currently set a time limit on those where more than one bounce is required, which is an oversight. For example, the bounce threshold for Mailbox full is 3. This means that as soon as we have received more than 3 mailbox full messages for a contact, their email will be put on hold. This might make sense if it was 3 in the last year, but doesn't really make sense if it was three in the last 10 years.
Similar arguments can be made for other bounce types, like Away, which is set to 30. If I send a fair amount of email to people that are out of the office quite a lot, over time a significant chunk of these will send me > 30 out of the office messages.
Hence we should only consider messages received in a certain time period. I am going to suggest one year. I would be up for making it user configurable (number of days) if people feel strongly about it.
Let me know what you think.https://lab.civicrm.org/dev/mail/-/issues/97Send order when A/B testing2022-06-11T14:54:42ZufundoSend order when A/B testingAs far as I can see current behaviour when processing an A/B test mailing is that CiviMail sends out to all A recipients then all B recipients.
If you have rate limiting on outbound email that is significant relative to your list size, ...As far as I can see current behaviour when processing an A/B test mailing is that CiviMail sends out to all A recipients then all B recipients.
If you have rate limiting on outbound email that is significant relative to your list size, this can result in a significant difference in (average) send time between the A group and the B group, which may then be confounding your A/B test results.
I wonder how hard it would be to implement some kind of interpolation (?) for the sends for an A/B mailing -- something like firing off equal numbers of As and Bs in each mailing batch as they are processed?https://lab.civicrm.org/dev/mail/-/issues/95<link> URLs are tracked and shouldn't be2022-06-11T14:56:58Zlarsssandergreen<link> URLs are tracked and shouldn't beIf you use <link href=URL> in a mailing template, for example to load a webfont, that URL is converted to a tracked URL. Presumably it shouldn't be as this makes a mess of the clickthrough statistics (every one of these links will be cou...If you use <link href=URL> in a mailing template, for example to load a webfont, that URL is converted to a tracked URL. Presumably it shouldn't be as this makes a mess of the clickthrough statistics (every one of these links will be counted as clicked as long as the user's mail client fetches the resource). In other words, a link being fetched in a not a click and shouldn't be counted as one.
Do we need to track any URLs that aren't inside an <a> tag? It might be easier to limit to <a... rather than trying to exclude <link.... The only valid way to write an <a> is with the tag right after the <, so I think we could just look for <a instead of <.
I can provide a patch if this approach is supported.https://lab.civicrm.org/dev/mail/-/issues/94Should tracking pixel be placed at start of email for when Gmail clips messages?2022-06-11T14:57:01ZlarsssandergreenShould tracking pixel be placed at start of email for when Gmail clips messages?With Mosaico templates, it isn't that difficult to reach Gmail's limit of 96-102kB with an email newsletter, after which your email message is clipped and not displayed in full. This is unfortunate in itself, but it also breaks open trac...With Mosaico templates, it isn't that difficult to reach Gmail's limit of 96-102kB with an email newsletter, after which your email message is clipped and not displayed in full. This is unfortunate in itself, but it also breaks open tracking as the tracking pixel is only embedded at the very end of the message and thus isn't fetched unless the reader clicks "View entire message".
Would it be worthwhile to consider putting the tracking pixel at the start of the message to avoid this issue? Obviously, less than ideal for those who don't display images, but I'm not sure how these two might balance out or if there is a clever way to have the best of both worlds.
I'd be happy to provide a patch, but not sure what the best way forward is here.
(Was a former Flexmailer issue, reposted here as I think this is still important.)https://lab.civicrm.org/dev/mail/-/issues/93Include optional unsubscribe group field on regular mailings2022-06-11T14:54:10ZlarsssandergreenInclude optional unsubscribe group field on regular mailingsWhen you create a mailing from search results, you are required to add an unsubscribe group. When you create a regular mailing and add groups to the recipients, you can't specify the unsubscribe group. Sometimes, it would be helpful to b...When you create a mailing from search results, you are required to add an unsubscribe group. When you create a regular mailing and add groups to the recipients, you can't specify the unsubscribe group. Sometimes, it would be helpful to be able to specify an unsubscribe group that is different than the group the contact is a member of. For instance, if you send a mailing to a few groups based on your main mailing list, perhaps a set of smart group that are subsets of the main list, you may want contacts to unsubscribe from your main list instead of the smart group.
Proposal: Add an optional unsubscribe group to all mailings. If it is blank, unsubscribes work as they do now. If it is populated, all recipients are unsubscribed from the unsubscribe group (exactly as this works for a mailing based on a search now).
EDIT: On @JoeMurray's suggestion, contacts would be unsubscribed from both the original group that are recipients of the mailing and the specified unsubscribe group.
I can give this a go if there is support for the concept.https://lab.civicrm.org/dev/mail/-/issues/90Make it possible to make a copy of a draft mailing2022-06-11T14:50:38ZlarsssandergreenMake it possible to make a copy of a draft mailingIt would be quite useful to be able to make a copy of a draft mailing. As it stands currently, if you want to duplicate a draft email (for instance, if you want to send two slightly different mailings), you have to send it first and then...It would be quite useful to be able to make a copy of a draft mailing. As it stands currently, if you want to duplicate a draft email (for instance, if you want to send two slightly different mailings), you have to send it first and then reuse it.
I'm not sure if this would be complicated or simple to implement (essentially the same as re-using a mailing or something more complex). Does anyone have thoughts?5.38.0https://lab.civicrm.org/dev/mail/-/issues/81Support tracking URLs with tokens in query strings2022-06-11T14:58:13ZlarsssandergreenSupport tracking URLs with tokens in query stringsURLs with tokens in mailings do not have click tracking. This is an important metric for fundraising (e.g. to see how many and even which contacts click through to contribution pages). @artfulrobot put together some code that worked well...URLs with tokens in mailings do not have click tracking. This is an important metric for fundraising (e.g. to see how many and even which contacts click through to contribution pages). @artfulrobot put together some code that worked well to track these clicks in Flexmailer, but that code was not merged before Flexmailer was merged into core.
https://github.com/civicrm/org.civicrm.flexmailer/issues/30
We used this without issue for four months and found it worked very well. @artfulrobot was using it since March. Would it be possible to merge this into core. Rich, would you be willing to copy your code to a patch against core?RichRichhttps://lab.civicrm.org/dev/mail/-/issues/67Mailing priority2022-06-11T14:55:13ZmfbMailing priorityOrganizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simult...Organizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simultaneously a press release is sent to press contacts. The latter would have higher priority.
CiviMail would send messages for mailings with a high priority before those with a lower priority.https://lab.civicrm.org/dev/mail/-/issues/66Core should provide bounce handling (and queueing) for transactional email2022-06-11T14:55:11ZmfbCore should provide bounce handling (and queueing) for transactional emailCurrently CiviCRM core does not provide bounce handling for transactional email. This can result in automated processes sending out a large volume of scheduled reminders, etc. (as well as user-triggered forms sending out subscription co...Currently CiviCRM core does not provide bounce handling for transactional email. This can result in automated processes sending out a large volume of scheduled reminders, etc. (as well as user-triggered forms sending out subscription confirmations, etc.) with only manual bounce handling by humans.
An extension exists to provide bounce handling and other features for transactional emails: https://civicrm.org/extensions/transactional-emails
Arguably, however, this feature should be provided by core itself, so all civi instances have bounce handling for any email message they send.
Assuming we continue the architecture where it is the CiviMail component that provides bounce handling, a solution could be that enabling CiviMail overrides how transactional email is sent, adding the extra bounce processing and other features.
Ideally, transactional email could also be queued by CiviMail, so failures are not fatal and delivery attempts can be retried. Currently, if you are over-quota or your mail delivery provider temporarily suspended your account, then you have real problems when doing some critical thing that sends a transactional email message.https://lab.civicrm.org/dev/mail/-/issues/64"View in Browser" does not feed mailing statistics2022-06-11T14:57:07Ztotten"View in Browser" does not feed mailing statisticsI just want to log the existence of the issue generally... suppose you send a newsletter and it has a "View in Browser" link.
Suppose the recipient uses that link. The view the mailing and then drill-down, clicking through some more lin...I just want to log the existence of the issue generally... suppose you send a newsletter and it has a "View in Browser" link.
Suppose the recipient uses that link. The view the mailing and then drill-down, clicking through some more links.
What happens to the stats?
* The web-based view does not count as an "Open". The MUA may or may not report back the "Open" on the original message.
* You will not have any "Click Through" events logged.
This is because the "View in browser" variant of the newsletter is not instrumented with tracking codes (while the MUA variant of the newsletter does have tracking codes).https://lab.civicrm.org/dev/mail/-/issues/59Eventually google will require OAUTH for bounce processing and may require it...2022-06-11T14:58:10ZDaveDEventually google will require OAUTH for bounce processing and may require it for outbound SMTP through gmail serversAs noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue supp...As noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue support for just username/password.
The way I read it, this is mostly applying to POP/IMAP clients for reading mail, and they are *hinting* that someday they might require it for outbound SMTP, i.e. at Administer - System Settings - Outbound Mail you choose SMTP and have it set to use gmail servers. The relevant quote is
> No change is required for scanners or other devices using simple mail transfer protocol (SMTP) or LSAs to send emails. If you replace your device, look for one that sends email using OAuth.
(LSA means less secure apps, which is google-speak for username/password)
They are targeting June 2020 - Feb 2021 for IMAP etc, but as noted above they have not specifically set a date for changes regarding sending outbound emails.
It occurs to me though that bounce processing and the email processor use POP/IMAP. For the email processor you can work around it by just forwarding it to a non-gmail address and polling that, but bounce processing might be harder depending on setup.https://lab.civicrm.org/dev/mail/-/issues/88SMS limit is hardcoded at 460 but should be changeable somewhere2022-06-11T14:58:16ZjasonobrownSMS limit is hardcoded at 460 but should be changeable somewhereTwillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the sys...Twillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the system now accepts (and sends) messages up to 1600 characters. It seems logical that the character limit should able to be adjusted and stored somewhere rather than set as a static value in the code. We are currently using the larger limit, and would really like to see this implemented asap so that we can stop manually updating the code each time we update civi.https://lab.civicrm.org/dev/mail/-/issues/49Password for mail accounts should not be stored in plain text2022-06-11T14:54:35ZscardiniusPassword for mail accounts should not be stored in plain textMySQL field civicrm_mail_settings.password contains password in plain text. Should it? I think not
This issue could by linked with https://lab.civicrm.org/dev/core/issues/236MySQL field civicrm_mail_settings.password contains password in plain text. Should it? I think not
This issue could by linked with https://lab.civicrm.org/dev/core/issues/236https://lab.civicrm.org/dev/mail/-/issues/42Provide a fullPage property for message templates2022-06-11T14:57:53ZmfbProvide a fullPage property for message templatesIn various parts of the UI, CiviCRM allows users to send a message template either on its own, or wrapped in a header and footer.
e.g. you can send a message template to a contact.
Or, you can create a mailing based on a message templa...In various parts of the UI, CiviCRM allows users to send a message template either on its own, or wrapped in a header and footer.
e.g. you can send a message template to a contact.
Or, you can create a mailing based on a message template, with additional header and footer.
CKEditor needs different settings at civicrm/admin/ckeditor?preset=civimail re: allowed HTML tags to support these two scenarios. With fullPage = true, the HTML is forcibly wrapped with html/head/body tags, and so a message template that includes its header and footer can be composed. With fullPage = false, html/head/body are silently stripped, so this is useful for message templates that should be used with a header and footer.
However, there is only one CKEditor setting for both of these scenarios.
I would propose a new database column where we can indicate if a message template is "fullPage" - i.e. includes header and footer, or not fullPage, i.e. designed to be used with header and footer.
This would allow us to toggle the fullPage setting appropriately. It would also allow us to prevent a user from sending a message template that needs a header and footer without one, or wrapping a message template that includes its header and footer with an additional header and footer.https://lab.civicrm.org/dev/mail/-/issues/41Preview as HTML doesn't open window at all2022-06-11T14:50:56ZscardiniusPreview as HTML doesn't open window at allCRM_Mailing_BAO_TrackableURL::getTrackerURL() requires id of mailing but after https://lab.civicrm.org/dev/mail/issues/20 mailing_id is removed from params to Mailing.preview
https://github.com/civicrm/civicrm-core/commit/0075b0cb75e542...CRM_Mailing_BAO_TrackableURL::getTrackerURL() requires id of mailing but after https://lab.civicrm.org/dev/mail/issues/20 mailing_id is removed from params to Mailing.preview
https://github.com/civicrm/civicrm-core/commit/0075b0cb75e5422660366acf280538167369d8ba#diff-d08e5d3ceca1972df6d467b2824ffab5R279
errors from api rest
```
{"error_code":"unknown error","sql":"INSERT INTO civicrm_mailing_trackable_url (url ) VALUES ('https:\/\/onet.pl' )
[nativecode=1364 ** Field 'mailing_id' doesn't have a default value]",
"tip":"add debug=1 to your API call to have more info about the error",
"is_error":1,
"error_message":"DB Error: unknown error",
"debug_information":"INSERT INTO civicrm_mailing_trackable_url (url ) VALUES ('https:\/\/onet.pl' ) [nativecode=1364 ** Field 'mailing_id' doesn't have a default value]"}
```5.12.3https://lab.civicrm.org/dev/mail/-/issues/34Recipients in smart group are less than actual in group2022-06-11T14:50:39ZGhost UserRecipients in smart group are less than actual in groupFor some reason smart group is only sending to five people even though group has 47 and none have opted out of bulk mailing or email so not sure why the mailing interface is only calculating 5 recipients.
![Screen_Shot_2018-12-20_at_1.0...For some reason smart group is only sending to five people even though group has 47 and none have opted out of bulk mailing or email so not sure why the mailing interface is only calculating 5 recipients.
![Screen_Shot_2018-12-20_at_1.05.06_PM](/uploads/2e3a12989c82f5549079a674b79a8001/Screen_Shot_2018-12-20_at_1.05.06_PM.png)
![Screen_Shot_2018-12-20_at_1.05.35_PM](/uploads/46574d53f23d23c2a1e0a28e780791d7/Screen_Shot_2018-12-20_at_1.05.35_PM.png)https://lab.civicrm.org/dev/mail/-/issues/31CiviCRM, Find Mailings. If there are no results, the same message "There are ...2022-06-11T14:57:50Zjustinfreeman (Agileware)CiviCRM, Find Mailings. If there are no results, the same message "There are no Unscheduled Mailings" is shownCiviCRM, Find Mailings. If there are no results, the same message "There are no Unscheduled Mailings" is shown regardless of the search criteria. If a different mailing status was used for search, it does not make sense to refer to "Unsc...CiviCRM, Find Mailings. If there are no results, the same message "There are no Unscheduled Mailings" is shown regardless of the search criteria. If a different mailing status was used for search, it does not make sense to refer to "Unscheduled Mailings".
This text should be changed to something else like: "There are no mailings matching this criteria".
Using CiviCRM 5.7.2
Agileware Ref: CIVICRM-1121
![Find_Mailings-2](/uploads/2abe0973a4395a1e1fb50d2d4f71e1eb/Find_Mailings-2.png)https://lab.civicrm.org/dev/mail/-/issues/29Emails marked as bounce when quota is full2022-06-11T14:57:49ZEdselopezEmails marked as bounce when quota is fullThere are cases (usually when using Gmail SMTP) when the Outbound email fails to deliver a mailing due to the user's daily quota being exhausted. In this case, if the mailing is re-used unknowingly, attempts to re-send the mailing adds t...There are cases (usually when using Gmail SMTP) when the Outbound email fails to deliver a mailing due to the user's daily quota being exhausted. In this case, if the mailing is re-used unknowingly, attempts to re-send the mailing adds to the bounce threshold, finally surpassing the limit of three bounces, marking the email as "on hold". To solve this, I'm thinking bounce type of "Quota" should be excluded from the calculation of marking the email as on hold. Please let me know if this is reasonable and I will provide a patch.https://lab.civicrm.org/dev/mail/-/issues/28When using a smart group as a mailing list, users who unsubscribe from the sm...2022-06-11T16:01:12Ztom.mWhen using a smart group as a mailing list, users who unsubscribe from the smart group are still included in the mailing.It looks like in CRM/MailingBAO/Mailing.php, is not taking into account those who have been removed from the smart group. Civi version 5.3.
```
if (count($includeSmartGroupIDs)) {
$query = CRM_Utils_SQL_Select::from($contact)
...It looks like in CRM/MailingBAO/Mailing.php, is not taking into account those who have been removed from the smart group. Civi version 5.3.
```
if (count($includeSmartGroupIDs)) {
$query = CRM_Utils_SQL_Select::from($contact)
->select("$contact.id as contact_id, $entityTable.id as $entityColumn")
->join($entityTable, " INNER JOIN $entityTable ON $entityTable.contact_id = $contact.id ")
->join('gc', " INNER JOIN civicrm_group_contact_cache gc ON $contact.id = gc.contact_id ")
->join('mg', " INNER JOIN civicrm_mailing_group mg ON gc.group_id = mg.entity_id AND mg.search_id IS NULL ")
->join('temp', " LEFT JOIN $excludeTempTablename temp ON $contact.id = temp.contact_id ")
->where('gc.group_id IN (#groups)')
->merge($criteria)
->replaceInto($includedTempTablename, array('contact_id', $entityColumn))
->param('#groups', $includeSmartGroupIDs)
->param('#mailingID', $mailingID)
->execute();
}
```