Development issueshttps://lab.civicrm.org/groups/dev/-/issues2024-02-18T05:03:26Zhttps://lab.civicrm.org/dev/core/-/issues/3616Should tracking pixel be placed at start of email for when Gmail clips messages?2024-02-18T05:03:26ZlarsssandergreenShould tracking pixel be placed at start of email for when Gmail clips messages?With Mosaico templates, it isn't that difficult to reach Gmail's limit of 96-102kB with an email newsletter, after which your email message is clipped and not displayed in full. This is unfortunate in itself, but it also breaks open trac...With Mosaico templates, it isn't that difficult to reach Gmail's limit of 96-102kB with an email newsletter, after which your email message is clipped and not displayed in full. This is unfortunate in itself, but it also breaks open tracking as the tracking pixel is only embedded at the very end of the message and thus isn't fetched unless the reader clicks "View entire message".
Would it be worthwhile to consider putting the tracking pixel at the start of the message to avoid this issue? Obviously, less than ideal for those who don't display images, but I'm not sure how these two might balance out or if there is a clever way to have the best of both worlds.
I'd be happy to provide a patch, but not sure what the best way forward is here.
(Was a former Flexmailer issue, reposted here as I think this is still important.)https://lab.civicrm.org/dev/core/-/issues/3615<link> URLs are tracked and shouldn't be2022-06-13T09:15:05Zlarsssandergreen<link> URLs are tracked and shouldn't beIf you use <link href=URL> in a mailing template, for example to load a webfont, that URL is converted to a tracked URL. Presumably it shouldn't be as this makes a mess of the clickthrough statistics (every one of these links will be cou...If you use <link href=URL> in a mailing template, for example to load a webfont, that URL is converted to a tracked URL. Presumably it shouldn't be as this makes a mess of the clickthrough statistics (every one of these links will be counted as clicked as long as the user's mail client fetches the resource). In other words, a link being fetched in a not a click and shouldn't be counted as one.
Do we need to track any URLs that aren't inside an <a> tag? It might be easier to limit to <a... rather than trying to exclude <link.... The only valid way to write an <a> is with the tag right after the <, so I think we could just look for <a instead of <.
I can provide a patch if this approach is supported.https://lab.civicrm.org/dev/core/-/issues/3614Civi not automatically imploding a dropdown Array for use as a token in a mes...2022-06-13T12:03:17ZswebervnaCivi not automatically imploding a dropdown Array for use as a token in a message templateHello!
I have a custom field, which is a multi-select dropdown, and I've inserted a token for that field to be used in a Message Template. Unfortunately, anytime I do the mailing using that Message Template, instead of the token inserti...Hello!
I have a custom field, which is a multi-select dropdown, and I've inserted a token for that field to be used in a Message Template. Unfortunately, anytime I do the mailing using that Message Template, instead of the token inserting what you'd expect (Option1,Option2, etc as selected in the dropdown), it just says "Array". I've found that it's just referencing the underlying infrastructure, in this case it's the Array entity *(Drupal 7)* for the dropdown value, but Civi won't implode it and list them like it should for that token:
**Mailing Result:**
![image](/uploads/7843e1a6df4105fea0ce1dcef982ac44/image.png)
**Message Template:**
![image](/uploads/3e863bdebab4a99dd00236fe36402df7/image.png)
**Custom Field:**
![image](/uploads/8ee568e35f5d7faa020cc74bf1393edf/image.png)https://lab.civicrm.org/dev/core/-/issues/3613System Workflow Messages - Improve localization experience2024-01-14T14:55:57ZtottenSystem Workflow Messages - Improve localization experience## Background
Several CiviCRM components - such as CiviContribute, CiviCase, and CiviEvent - send automated email notifications. Many organizations find it useful to customize/tune these emails, and the emails may be administered in the...## Background
Several CiviCRM components - such as CiviContribute, CiviCase, and CiviEvent - send automated email notifications. Many organizations find it useful to customize/tune these emails, and the emails may be administered in the web UI ("Administer: Message Templates: System Workflow Messages").
In principle, this is a powerful tool in which an administrator may use Smarty templating to produce highly tuned messages -- and it requires only the ability to understand template notations (e.g. `{$contact.first_name}` or `{$event.start_date}`). However, in practice, this is often difficult -- real-world templates are long and complex, and they may be used with data of varying quality/character, and the workflow to test a template can be tedious.
## Goals
This epic (*overarching issue*) aims to improve the experience of maintaining system workflow templates. We are specifically funded to work on it from the following perspective: You have an organization which maintains multiple templates for use in multiple countries/locales -- periodically, as campaigns/operations are revised, you may need to engage with a few people to update+retest a few templates. We have three main themes to improve upon:
* __Drafting/Workflow__: Currently, Civi has exactly one variant/revision of a template, and it is always live. There is no way to develop a draft (new revisions) or to keep a record of past revisions.
* __~~Translation~~ Localization__: For any given message, you may need to compose different versions for different locales.
* __Previews/Samples__: Many templates are sensitive to fine-grained details. This is true in multiple ways, e.g. (1) the template adjusts based on *available data* ("Do we have a mailing address for this person? Do we have a full-name, or just an email? Is this one-off contribution or a recurring contribution?"), and (2) the template includes *precise codes* that must be well-formed (e.g. matching tags, matching quotes, using an email-friendly dialect of HTML/CSS).
## Related work
* https://lab.civicrm.org/extensions/msgtpltools
* https://lab.civicrm.org/community/feature-request/-/issues/26
* https://github.com/eileenmcnaughton/civi-data-translate
* https://lab.civicrm.org/extensions/l10nx
## Subtopics / Brainstorming
(Some of the descriptions here are terse - but they may get more explanation in the wireframe.)
* __Sample data presentation__: While composing a message, how do you show a preview based on sample data? Some ideas:
* __Edit-Preview Split-Panes__
* __Pro__: very amenable to live-update
* __Con__: needs wide screen (prob dialog). agreeable with O(10) sample records - but disagreeable with O(1000) records
* __Edit-Preview Accordion__
* __Pro__: fairly amenable to live-update. full width.
* __Con__: agreeable with O(10) sample records - but disagreeable with O(1000) records
* __Explore-Preview Panes in dialog__
* __Pro__: agreeable with O(1000) records
* __Con__: requires more navigating.
* __Comment__: live-update might be possible with separate popup - but more difficult
* __Comment__: might mitigate navigation work with a hotkey
* __Sample data source__: When viewing a preview based on some example record(s), where does the example come from?
* __Basic/prepackaged__: Define some JSON files, which are linked to the workflows.
* __Comment__: Requires metadata to say, "workflows $a+$b can use sample data $c+$d"
* __Pro__: Some templates have very nuanced purposes and may have special vars. This allows you to mock data that's attuned to edge-cases, and you can use these samples in any environment (dev-site/public-demo-site/live-site). No mixing-up real records and sample records. Can simulate complex data (e.g. "tpl for use-case with 2 contribution records")
* __Con__: More upfront work in making sample data. User cannot test against real data.
* __Comment__: Should have unit-test to ensure sample-data-structure stays in sync with real-data-structure
* __Data search__: Search for a live record (eg `Contribution` or `Membership`) to use as a sample.
* __Comment__: Requires metadata to say, "workflows $a+$b need data from entity $c"
* __Pro__: You can use real data, which can be subjectively attuned.
* __Con__: You have to find real data that's *useful*. May be awkward if you're trying to test edge-cases, unless you're willing to add mock/synthetic records to the DB. All mail-merge data must be selectable from one record (eg "Contribution #123"; eg no other inputs besides DB content).
* __Variants__: The UI for this can take on a few variations, eg
* In "Edit Msg Tpl", show a searchable tree. Filter by contact name and select the target record.
* In "Edit Msg Tpl", ask the user to enter a numeric ID.
* In "View Contact", add links/actions for generating previews.
* __Flagged records__: As an adminsitrator, find real records and save them in a list (e.g. "Record #$x is a sample for use in workflows $a, $b")
* __Comment__: Requires somewhere to store the list (e.g. custom-field or new-field or new-table)
* __Pro__: You can use real data, which can be subjectively attuned. The list of samples is curated/abbreviated.
* __Con__: The individual composing the template may not know how to flag/unflag sample records. Requires configuration (and extant samples) before you can use email preview.
* __Grain of ~~translation~~ localization__
* __Record level__: For each `(locale,workflow)`, create another `civicrm_msg_template` record.
* __Field level__: Each `(workflow)` only has one `civicrm_msg_template` record. Attached to that record are multiple translations (ex: `field=subject,locale=fr_FR,text="Bonjour"`).
* __String level__: Each `(workflow)` only has one `civicrm_msg_template` record, and that record has one `body_text`. However, the `body_text` uses translatable strings (ex: 'body_text={cts id=greeting}Greetings,{/cts}`)
* Trade-off
| Record-level | Field-level | String-level |
| -- | -- | -- |
| Each translation goes through a separate, well-delineated drafting workflow. | Translation workflow is batched (per template). All translations must be ready to activate together. | Orthogonal workflows for templates vs strings. |
| Message structure (paras/sections) is localized. | Message structure (paras/sections) is localized. | Message structure is uniform across locales. (Realistic? What abouts `<table>` LTR/RTL? |
| Cannot re-use translated snippets. | Cannot re-use translated snippets. | Re-use translated snippets. |
## Wireframes
There are a few different ways to approach this. To get a sense for the usability and work, we can post a few alternative wireframes. (*This section may be updated as more are added.*)
* __(0) Status quo__: [_0__Sys_Wf_Msgs](/uploads/ad96003d71133d298f95ce79902a7583/_0__Sys_Wf_Msgs.png), [_0__Edit_Msg_Tpl](/uploads/81bc234651182b65993bc28c8052b68b/_0__Edit_Msg_Tpl.png)
* __(1) Record-level translations w/previews and drafting workflow__: [_1Rec__Sys_Wf_Msgs](/uploads/532b842406c17c31be5ae9195ceb6aa8/_1Rec__Sys_Wf_Msgs.png), [_1Rec__Edit_Msg_Tpl](/uploads/13a1f717e829e3de480750c4893969a7/_1Rec__Edit_Msg_Tpl.png) (*Note: The blue-arcs indicate a couple variations on how to include the preview/sample functionality. This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(2) Field-level translations__: [_2Fld__Sys_Wf_Msgs](/uploads/7e1f46e722d8afc706a92c56e295c55e/_2Fld__Sys_Wf_Msgs.png), [_2Fld__Edit_Msg_Tpl](/uploads/4bc932b7bb3f304853b0c107b98d41b5/_2Fld__Edit_Msg_Tpl.png) (*Note: This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(3) String-level translations__: [_3Subfld__Sys_Wf_Msgs](/uploads/f52ffbb764917954790fbbd3ed807acf/_3Subfld__Sys_Wf_Msgs.png), [_3Subfld__Edit_Msg_Tpl](/uploads/b20bc83d302d55a485d975aff6ca70e1/_3Subfld__Edit_Msg_Tpl.png) (*Note: This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(4) Record-level translations w/browse-edit tree + accordion-preview__: [_4Tree__Browse-Edit](/uploads/8cc47238a9d597f80950c1d98c0ea43a/_4Tree__Browse-Edit.png)
* __(5) Field-level translations w/browse-edit tree + accordion-preview__ [_5Tree__Browse-Edit](/uploads/aab231199d0b4217a2f44f9ad45aecb1/_5Tree__Browse-Edit.png)
* __(6) Field-level translations w/matrix, preview-dialog, and drafting workflow__: [_6Mat__Sys_Wf_Msgs](/uploads/3a32390f36f50d8e158e1155dc586969/_6Mat__Sys_Wf_Msgs.png), [_6Mat__Edit_Msg_Tpl](/uploads/353f97846fd9d2b1e037ec92e60101fc/_6Mat__Edit_Msg_Tpl.png) (*Note: This assumes that there is a separate string-storage service like [civi-data-translate](https://github.com/eileenmcnaughton/civi-data-translate), and it assumes that the `String` record has some workflow property like like `is_draft` or `revision`. So each string might have the properties `entity`,`entity_id`,`field`,`locale`,`text`,`is_draft`.*)
* __(7) As with 6, but with various refinements__: [_7Mat__Sys_Wf_Msgs](/uploads/73d37acab4307054b4fe4f3291ae7dcd/_7Mat__Sys_Wf_Msgs.png),
[_7Mat__Edit_Msg_Tpl](/uploads/4807652bd20bfb5bf0912bf24f05d444/_7Mat__Edit_Msg_Tpl.png)https://lab.civicrm.org/dev/core/-/issues/3612Tokens broken2022-06-11T14:56:45ZJoeMurrayTokens brokenThe contact source and created date tokens are exposed in CiviMail/Mosaico. They do work when doing one off emails and print/merge.The contact source and created date tokens are exposed in CiviMail/Mosaico. They do work when doing one off emails and print/merge.https://lab.civicrm.org/dev/core/-/issues/3611High Database Load INSERT IGNORE INTO `civicrm_tmp_e_gccache_2024-02-18T05:03:25ZMickCHigh Database Load INSERT IGNORE INTO `civicrm_tmp_e_gccache_I thought I'd bring this to attention of the Civi Mail team as it's causing high load.
This has happened on more than 1 occasion, running many of these queries for about 3 hours.
![Screen_Shot_2021-09-17_at_3.44.33_pm](/uploads/c7f78b94b...I thought I'd bring this to attention of the Civi Mail team as it's causing high load.
This has happened on more than 1 occasion, running many of these queries for about 3 hours.
![Screen_Shot_2021-09-17_at_3.44.33_pm](/uploads/c7f78b94b5e0fddd7e7d75f074c06525/Screen_Shot_2021-09-17_at_3.44.33_pm.png)
I get notified of it by an AWS Cloudwatch alarm on RDS.
The database was not running a mailing at the time.
Contacts approx 100,000 so quite large.
However database load is usually very low, even when running a mailing.
I haven't been able to identify what this process is - if anyone knows, then is there something that can be done to address the load?
The database joins in the query make is a large query - is there another way to do this? e.g. break the query up into temp tables, fewer joins, use indexes etc
Thankshttps://lab.civicrm.org/dev/core/-/issues/3610Jira Issue CRM-21666 still exists in 5.8.12022-06-11T14:56:07ZjeffmikelsJira Issue CRM-21666 still exists in 5.8.1https://issues.civicrm.org/jira/browse/CRM-21666
CRM/Mailing/DAO/Mailing.php
and
CRM/Mailing/BAO/Mailing.php
still have a discrepancy regarding the naming of the mailing_date field
messageNotice: Undefined property: CRM_Mailing_BAO...https://issues.civicrm.org/jira/browse/CRM-21666
CRM/Mailing/DAO/Mailing.php
and
CRM/Mailing/BAO/Mailing.php
still have a discrepancy regarding the naming of the mailing_date field
messageNotice: Undefined property: CRM_Mailing_BAO_Mailing::$mailing_modified_date in CRM_Mailing_BAO_Mailing::report()5.9https://lab.civicrm.org/dev/core/-/issues/3609CiviCRM, Find Mailings by Mailing Status, select Draft/Unscheduled will show ...2022-06-11T14:55:57Zjustinfreeman (Agileware)CiviCRM, Find Mailings by Mailing Status, select Draft/Unscheduled will show "Complete" and "Not scheduled" mailings in the Mailing resultsCiviCRM, Find Mailings by Mailing Status, select Draft/Unscheduled will show "Complete" and "Not scheduled" mailings in the Mailing results.
Using CiviCRM 5.7.2
![find-mailings](/uploads/88cf1c56b62d0bb8c5cb5bca4be042a2/find-mailings.p...CiviCRM, Find Mailings by Mailing Status, select Draft/Unscheduled will show "Complete" and "Not scheduled" mailings in the Mailing results.
Using CiviCRM 5.7.2
![find-mailings](/uploads/88cf1c56b62d0bb8c5cb5bca4be042a2/find-mailings.png)
Agileware Ref: CIVICRM-1120https://lab.civicrm.org/dev/core/-/issues/3607Feature: Provide an administrative option to make unsubscribe affect all matc...2024-02-17T05:03:32ZlolcodeFeature: Provide an administrative option to make unsubscribe affect all matching email addressesOur organization would like the unsubscribe action to unsubscribe all contacts with the same email from the group. In cases of duplicate contacts they unsubscribe once but one of the duplicates will still get an email and then we get CAS...Our organization would like the unsubscribe action to unsubscribe all contacts with the same email from the group. In cases of duplicate contacts they unsubscribe once but one of the duplicates will still get an email and then we get CASL complaints. Due to CASL we are trying to be cautious by default here.
We suggest an administrative setting for Civimail: "Unsubscribe contact (radio) or unsubscribe email address (radio)"
There was some support for this option expressed in a discussion here: https://chat.civicrm.org/civicrm/pl/sxd4jpwxpbynx8y15994ua7jea
A good point is that this needs to be guarded by token access to stop attackers being able to unsubscribe prominent emails from organisational lists (for example).
A further suggestion was made to make this a user selection option on the unsubscribe page. I feel that this would be a good but separate feature request.https://lab.civicrm.org/dev/core/-/issues/3606send test email may create duplicate contacts2022-06-11T14:55:49Zlcdwebsend test email may create duplicate contactsto reproduce:
1. construct a new email
2. in the "send test email to:" field list two email addresses that match existing contacts. separate them with a comma and space
3. trigger the test email
the first email will match the existi...to reproduce:
1. construct a new email
2. in the "send test email to:" field list two email addresses that match existing contacts. separate them with a comma and space
3. trigger the test email
the first email will match the existing contact. the second email will result in a new duplicate contact getting created. the issue is if you have a space after the comma separating the email addresses. the values are not trimmed when parsed, causing a failed match and the creation of a duplicate.5.6lcdweblcdwebhttps://lab.civicrm.org/dev/core/-/issues/3605"Tracking Click-Throughs" option in mailings generates 404 links when using m...2024-02-17T05:03:31Zjensschuppe"Tracking Click-Throughs" option in mailings generates 404 links when using multi-language with path prefixThis occurred in a Drupal environment with multiple languages and the following configuration:
- Languages in Drupal:
- English (activated)
- German (activated, default)
- Languages in CiviCRM:
- Default language: English
- avai...This occurred in a Drupal environment with multiple languages and the following configuration:
- Languages in Drupal:
- English (activated)
- German (activated, default)
- Languages in CiviCRM:
- Default language: English
- available: German, English
- Inherit CMS language: yes
Conditions under which the erroneous behavior can be reproduced:
- Mailing language: English
- UI language: German
- language prefix in the URL: none or "de"
or:
- Mailing language: German
- UI language: English
- language prefix in the URL: none or "en"
ergo: Using a UI language different from the mailing language and having path prefixes for language detection.
The result is links to the tracking script being prefixed with the language code of the UI language, like so:
https://example.org/de/sites/all/modules/civicrm/extern/url.php?u=123&qid=12345
which is producing a 404 as obviously the path to the script file is not valid due to the language prefix.
Apparently, this code in CRM/Mailing/BAO/TrackableURL.php:93 is where the URL is constructed, but with `userFrameworkResourceURL` being prefixed with the language code:
```
$redirect = $config->userFrameworkResourceURL . "extern/url.php?u=$id";
```https://lab.civicrm.org/dev/core/-/issues/3604Can't save Message Template - authorization failed2022-06-11T14:55:44ZkenCan't save Message Template - authorization failedThis problem has been [reported on StackExchange](https://civicrm.stackexchange.com/questions/36233/problem-with-permissions-to-edit-message-templates) with version 5.27.2 and I have experienced it with 5.28.3.
Unless you have permissio...This problem has been [reported on StackExchange](https://civicrm.stackexchange.com/questions/36233/problem-with-permissions-to-edit-message-templates) with version 5.27.2 and I have experienced it with 5.28.3.
Unless you have permission "Administer CiviCRM" you can't edit a Message Template. This fails with an "Authorization failure" message.
I don't understand the code well enough to suggest a fix, but here's a hack to get around it.
The saving of the Message Template is done by this code on line 311 of CRM/Admin/Form/MessageTemplate.php
```
$messageTemplate = MessageTemplate::save()->setDefaults($params)->setRecords([['id' => $this->_id]])->execute()->first();
```
At the point when it comes to checking permissions, there are permissions on 'message_template' for 'get', 'create' and 'update' but not for 'save'. So the default permission is used (Administer CiviCRM).
The following patch to CRM/Core/Permission.php works around the problem ...
```
@@ -1483,6 +1483,7 @@ class CRM_Core_Permission {
$permissions['message_template'] = [
'get' => ['access CiviCRM'],
+ 'save' => [['edit message templates', 'edit user-driven message templates', 'edit system workflow message templates']],
'create' => [['edit message templates', 'edit user-driven message templates', 'edit system workflow message templates']],
'update' => [['edit message templates', 'edit user-driven message templates', 'edit system workflow message templates']],
];
```https://lab.civicrm.org/dev/core/-/issues/3603Undefined index: error-codes in recaptcha_check_answer()2024-02-16T05:03:24Zchris_bluejacUndefined index: error-codes in recaptcha_check_answer()## Overview
When non-authenticated user subscribes to mailing list at /civicrm/mailing/subscribe, checks the "I am not a robot" reCAPTCHA, and clicks subscribe. The subscription request is processed properly, but an error is thrown:
Not...## Overview
When non-authenticated user subscribes to mailing list at /civicrm/mailing/subscribe, checks the "I am not a robot" reCAPTCHA, and clicks subscribe. The subscription request is processed properly, but an error is thrown:
Notice: Undefined index: error-codes in recaptcha_check_answer() (line 159 of /bitnami/drupal/modules/contrib/civicrm/ext/recaptcha/lib/recaptcha/recaptchalib.php).
## Reproduction steps
1. /civicrm/mailing/subscribe
2. complete email address, check relevant mailing list (in this case, "General Newsletter")
3. Check "I'm not a robot" in reCAPTCHA
4. Click "Subscribe"
5. Subscription is processed properly.
6. Error reported on confirmation page.
## Current behaviour
Error message presented.
Functionality of reCAPTCHA and subscription is okay.
~~~
Notice: Undefined index: error-codes in recaptcha_check_answer() (line 159 of /bitnami/drupal/modules/contrib/civicrm/ext/recaptcha/lib/recaptcha/recaptchalib.php).
~~~
## Expected behaviour
No error message
## Environment Information
- CiviCRM: 5.40.2
- PHP: 7.3.27
- CMS: Drupal 7.82
- Database: MySQL 5.7.33
- Web Server: Apache 2.4.46
- reCAPTCHA: Version 5.40.2
![Screenshot_2021-10-31_084608](/uploads/286470768de6dd8bc6a632827856eafd/Screenshot_2021-10-31_084608.png)https://lab.civicrm.org/dev/core/-/issues/3601MS Exchange - IMAP-XOAuth2 authentication fails2022-09-12T18:25:55ZtottenMS Exchange - IMAP-XOAuth2 authentication failsOverview
----------------------------------------
https://lab.civicrm.org/dev/core/-/issues/2141 introduced support for (1) obtaining OAuth2 tokens and (2) using the OAuth2 tokens for IMAP-XOAuth2.
In my test setup, I am able to obtai...Overview
----------------------------------------
https://lab.civicrm.org/dev/core/-/issues/2141 introduced support for (1) obtaining OAuth2 tokens and (2) using the OAuth2 tokens for IMAP-XOAuth2.
In my test setup, I am able to obtain a token from Microsoft which appears valid -- but their IMAP server is not accepting the token.
The "Critical Evaluation" below include a more thorough review of theories/data, but here's a high-level summary of debugging efforts:
| Provider | OAuth2 initiation | Use token for IMAP/XOAuth2 | Use token for HTTP/REST |
| -- | -- | -- | -- |
| Microsoft Exchange Online | :white_check_mark: | :stop_sign: fails, tho JWT has IMAP claims | :white_check_mark: |
| Google Mail | :white_check_mark: | :white_check_mark: | (not tested) |
Reproduction steps
----------------------------------------
Configure Civi's OAuth client as described in the draft documentation: https://lab.civicrm.org/documentation/docs/sysadmin/-/merge_requests/278
Recap:
1. Enable `oauth-client`
2. Register the OAuth details in Civi and in Azure Portal.
3. In Civi's "Admin => CiviMail => Mail Accounts", add a mail account
4. For the newly created account, choose "Save & Test".
Current behavior
----------------------------------------
The IMAP server responds with `NO AUTHENTICATE failed.` and `* BAD Command Error. 12`
In the GUI, it appears as:
![Screen_Shot_2020-11-06_at_5.10.49_PM](/uploads/dfa7fd59bd5be48743ef1be7132778c8/Screen_Shot_2020-11-06_at_5.10.49_PM.png)
On a network level, the interaction is:
```
<< * OK The Microsoft Exchange IMAP4 service is ready. [{_LONG_RANDOM_CODE_}]
>> A0001 AUTHENTICATE XOAUTH2 {_THE_XOAUTH2_TOKEN_}
<< A0001 NO AUTHENTICATE failed.
>>
<< * BAD Command Error. 12
>> A0002 LOGOUT
<< * BAD Command Error. 12
```
(To see this, I applied [a hack to log network I/O](https://gist.github.com/totten/05de1aaf6948513792a40fb6f1eafb26) and re-ran the check via CLI.)
Expected behavior
----------------------------------------
The IMAP test should succeed.
Environment information
----------------------------------------
* __Browser:__ _Firefox_
* __CiviCRM:__ _5.32.beta1_
* __PHP:__ _7.1_
* __CMS:__ _Drupal 7_
* __Database:__ _MySQL 5.6_
* __Web Server:__ _Apache 2.4_
Critical evaluation
----------------------------------------
Here are a few theories for where the problem may reside.
1. The problem is in the OAuth2 client.
1. The OAuth2 client is broken - it does not obtain a real token from Microsoft.
2. The OAuth2 client obtains a real token - but the token does not have the proper scopes.
3. The OAuth2 client obtains a valid token with proper scopes - but it doesn't speak IMAP-XOAuth2 correctly.
2. The problem is in the OAuth2 service provider (either the authorization-server or the resource-server/IMAP-server).
1. There is a problem in the account flags/configuration.
2. There is a bug in the OAuth2 service provider.
Let's consider each in turn to see if we can prove/disprove the theory:
## (Theory 1.1) The OAuth2 client is broken - it does not obtain a real token from Microsoft.
This explanation doesn't hold -- because the same token works for other scenarios.
* Navigate to "Admin => System Settings => OAuth => ms-exchange"
![Screen_Shot_2020-11-06_at_5.41.30_PM](/uploads/c7cd3286bdebf306c40466d3b22b7555/Screen_Shot_2020-11-06_at_5.41.30_PM.png)
* Click "Inspect"
* Find "Access Token: Raw". Copy the value.
* In CLI, make an HTTP request to MS with the same token:
```
$ curl 'https://graph.microsoft.com/v1.0/me' \
-H 'Accept: application/json' \
-H 'Authorization: Bearer <TOKEN_VALUE>'
{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users/$entity","businessPhones":["XXX-XXX-XXXX"],"displayName":"My Name","givenName":"My","jobTitle":null,"mail":"myname@wariomail.onmicrosoft.com","mobilePhone":null,"officeLocation":null,"preferredLanguage":"en-US","surname":"Name","userPrincipalName":"myname@wariomail.onmicrosoft.com","id":"XXXXX-XXXX-XXXXX"}
```
* The response is well-formed and has the correct information.
## (Theory 1.2) The OAuth2 client obtains a real token - but the token does not have the proper scopes.
I can show that the client works according to spec.
* In the Microsoft documentation [Authenticate an IMAP, POP or SMTP connection using OAuth](https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#configure-your-application), we can see the required scope is `IMAP.AccessAsUser.All`.
* In Azure Portal, navigate to "Azure Active Directory => App registrations => (my-app) => API permissions". Note the list of permissions includes IMAP:
![Screen_Shot_2020-11-06_at_6.06.48_PM](/uploads/739adc203b05b30fdeb3efd152b340f7/Screen_Shot_2020-11-06_at_6.06.48_PM.png)
* In CiviCRM, navigate to "Admin => System Settings => OAuth => ms-exchange"
* In the "Details" tab, we can see the list of scopes that requested during setup:
![Screen_Shot_2020-11-06_at_5.56.41_PM](/uploads/9f23d454175a2ed8d8b16da55b5a4bd1/Screen_Shot_2020-11-06_at_5.56.41_PM.png)
* In the "Client #X" tab, we can "Inspect" the token.
* Observe: The "Token Record" shows that the `raw` response from the OAuth2 service. It indicates approval for the necessary scopes:
![Screen_Shot_2020-11-06_at_5.58.47_PM](/uploads/3ebf06fe493d8aa83d22c25fc3ce3380/Screen_Shot_2020-11-06_at_5.58.47_PM.png)
* Observe: The "Access Token: JWT Payload" also reports these scopes:
```
"scp": "IMAP.AccessAsUser.All POP.AccessAsUser.All SMTP.Send User.Read profile openid email",
```
So, it appears that every component (from Azure app registration, to the client's request configuration, to the received token data, to the decoded JWT token data) reports `IMAP.AccessAsUser.All` as an included scope.
There is a subtle discrepancy in how the scope is labeled - sometimes, it's presented as the shorter `IMAP.AccessAsUser.All`; other times, it's the longer `https://outlook.office.com/IMAP.AccessAsUser.All`. To be sure, I've requested tokens with both notations -- and in both cases, the outcome is the same. (To wit: the JWT token has `IMAP.AccessAsUser.All` - but it does not work for IMAP access.)
## (Theory 1.3) The OAuth2 client obtains a valid token with proper scopes - but it doesn't speak IMAP-XOAuth2 correctly.
I've tested this theory a few ways:
1. Connect to another service (Google Mail) which uses the same protocol (IMAP-XOauth2). It works. :white_check_mark:
2. Hack the PHP IMAP library (`imap_transport.php`) to inspect the wire-communication. Manually decode the XOAuth2 statement to ensure that it follows [the XOAuth2 formula](https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#sasl-xoauth2). :white_check_mark:
3. Write a new script with a different PHP IMAP library to try the same token. The alternate library works with Google Mail but not Microsoft Exchange.
## (Theory 2.1) There is a problem in the account flags/configuration.
As I mentioned above (theory 1.2), the "App registration" has all the scopes described in the docs
![Screen_Shot_2020-11-06_at_6.06.48_PM](/uploads/afe29d69f170150a89b98802fd5262b8/Screen_Shot_2020-11-06_at_6.06.48_PM.png)
In the "Exchange Administration Center", the account has IMAP enabled:
![Screen_Shot_2020-11-06_at_6.32.17_PM](/uploads/a5980fff21dcf85add1df96d03e1c9b2/Screen_Shot_2020-11-06_at_6.32.17_PM.png)
In the Exchange webmail UI, there is a settings page with a search-box. Search for "IMAP", and it shows:
![Screen_Shot_2020-11-06_at_6.27.06_PM](/uploads/a4ae9d1aebae308df98a41b3ac348379/Screen_Shot_2020-11-06_at_6.27.06_PM.png)
It's peculiar that searching for "IMAP" presents an inert radio-button for POP, but it does show connection details for IMAP.
TBH, Microsoft's backend here is labyrinthine, so it's hard to be sure that I've checked every possible permission. But I can't find anything else that would prevent IMAP access.
Perhaps someone with more experience in Microsoft's backend should try to run on a different account? Perhaps an account that has been demonstrated to work with IMAP?
## (Theory 2.2) There is a bug in the OAuth2 service provider.
I can't really prove or disprove this theory. I don't really want to believe this is the problem (*surely this stuff works for other people...*). OTOH, maybe it's actually quite rare for people to use IMAP+XOAuth2 with Exchange Online...5.32.0https://lab.civicrm.org/dev/core/-/issues/3599Test mailings create new contacts even when "Add Contacts" permission is not ...2022-06-11T14:55:17ZJonGoldTest mailings create new contacts even when "Add Contacts" permission is not present.### Overview
When sending a test mail, CiviCRM will check if the email matches an existing contact in the database. If it does, it uses that contact ID; if not, it creates a contact. However, it creates a contact without regard for wh...### Overview
When sending a test mail, CiviCRM will check if the email matches an existing contact in the database. If it does, it uses that contact ID; if not, it creates a contact. However, it creates a contact without regard for whether a user has permission to add contacts.
### Steps to replicate
* Create a user that does not have the "Add Contacts" permission.
* With that user, create a new mailing (traditional or Mosaico, doesn't matter).
* Send a test ("draft") mail to an email address that doesn't exist in the database.
### Expected behavior
No new contact is created.
### Actual behavior
A new contact is created.5.29.0JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3598Proposal: Scheduled Reminders should not (by default) use Smarty on new installs2023-01-27T15:27:33ZJonGoldProposal: Scheduled Reminders should not (by default) use Smarty on new installsWhen token rendering is enabled, and the `TokenProcessor` context says to use Smarty, the TokenRow rendering barfs on any curly brace that isn't a token or Smarty. Which includes, for instance, [inline CSS](https://civicrm.stackexchange...When token rendering is enabled, and the `TokenProcessor` context says to use Smarty, the TokenRow rendering barfs on any curly brace that isn't a token or Smarty. Which includes, for instance, [inline CSS](https://civicrm.stackexchange.com/q/17643/12).
At the time that SE post was made, Smarty stripped out the CSS. It no longer does, and instead generates large numbers of errors. Rather than fight that battle, I'm curious whether Smarty on scheduled reminders is necessary and serves a purpose in a post-5.43 TokenProcessor world.
If we decided that Scheduled Reminder TokenProcessors had [Smarty set to false](https://github.com/civicrm/civicrm-core/blob/ce0d67d6a686fcfae1d7046e615107f25cec501a/CRM/Core/BAO/ActionSchedule.php#L647) this problem would go away.
Does anyone use Smarty here? Should they?JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3597Contact with non-primary on-hold email addresses prevent delivery2022-06-11T14:55:15ZAlanDixonContact with non-primary on-hold email addresses prevent deliveryHere's the detailed origin of the problem. I have a client who complained of his list being too small. Normally I just explain the usual reasons a group size is smaller than what gets sent, but he was adamant that something else was goin...Here's the detailed origin of the problem. I have a client who complained of his list being too small. Normally I just explain the usual reasons a group size is smaller than what gets sent, but he was adamant that something else was going on.
So, I created a mailing to that group and it calculated and found 9560 recipients, same as what it reported on screen.
Next I generated a list of who I think should get the mailing, using this sql:
`select count(distinct(gc.contact_id)), gc.status, e.on_hold, c.do_not_email, c.is_opt_out from civicrm_group_contact gc inner join civicrm_email e on gc.contact_id = e.contact_id inner join civicrm_contact c on gc.contact_id = c.id where group_id = 19 and e.is_primary = 1 group by status, on_hold, do_not_email, is_opt_out;`
which gave me:
```
+--------------------------------+---------+---------+--------------+------------+
| count(distinct(gc.contact_id)) | status | on_hold | do_not_email | is_opt_out |
+--------------------------------+---------+---------+--------------+------------+
| 10374 | Added | 0 | 0 | 0 |
| 1094 | Added | 0 | 0 | 1 |
| 2 | Added | 0 | 1 | 0 |
| 2 | Added | 0 | 1 | 1 |
| 3792 | Added | 1 | 0 | 0 |
| 4 | Added | 1 | 0 | 1 |
| 1 | Added | 2 | 0 | 0 |
| 1 | Pending | 0 | 0 | 0 |
| 2 | Pending | 1 | 0 | 0 |
| 2547 | Removed | 0 | 0 | 0 |
| 312 | Removed | 0 | 0 | 1 |
| 2 | Removed | 0 | 1 | 0 |
| 2 | Removed | 0 | 1 | 1 |
| 67 | Removed | 1 | 0 | 0 |
| 1 | Removed | 1 | 1 | 1 |
+--------------------------------+---------+---------+--------------+------------+
```
So that left me with about 800 contacts that were not going to be mailed for unaccounted reasons.
I looked up the first one that differed and discovered that it had a non-primary email address on hold (but also a primary one that was not on hold).
My conclusion is that this claim: https://civicrm.stackexchange.com/questions/11517/on-hold-what-happens-with-mailings
is false.https://lab.civicrm.org/dev/core/-/issues/3596Manual Recipients in Scheduled Reminders2022-06-11T14:55:13ZguyiacManual Recipients in Scheduled RemindersI can't test this on the dev, but in our Civi (Drupal 7 / Civi 5.24.5) when you save a scheduled reminder for an event and add manual recipients (per the attached image), the reminder gets sent to the manual recipients right away when th...I can't test this on the dev, but in our Civi (Drupal 7 / Civi 5.24.5) when you save a scheduled reminder for an event and add manual recipients (per the attached image), the reminder gets sent to the manual recipients right away when the reminder is saved (the others don't get it until the scheduled time). Am I misreading the help bullets and this is expected functionality, or is this a bug?
![image](/uploads/05ab358390d318a31a161fb42806464c/image.png)https://lab.civicrm.org/dev/core/-/issues/3595Mailing priority2024-02-16T05:03:23ZmfbMailing priorityOrganizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simult...Organizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simultaneously a press release is sent to press contacts. The latter would have higher priority.
CiviMail would send messages for mailings with a high priority before those with a lower priority.https://lab.civicrm.org/dev/core/-/issues/3594Core should provide bounce handling (and queueing) for transactional email2024-02-15T05:03:25ZmfbCore should provide bounce handling (and queueing) for transactional emailCurrently CiviCRM core does not provide bounce handling for transactional email. This can result in automated processes sending out a large volume of scheduled reminders, etc. (as well as user-triggered forms sending out subscription co...Currently CiviCRM core does not provide bounce handling for transactional email. This can result in automated processes sending out a large volume of scheduled reminders, etc. (as well as user-triggered forms sending out subscription confirmations, etc.) with only manual bounce handling by humans.
An extension exists to provide bounce handling and other features for transactional emails: https://civicrm.org/extensions/transactional-emails
Arguably, however, this feature should be provided by core itself, so all civi instances have bounce handling for any email message they send.
Assuming we continue the architecture where it is the CiviMail component that provides bounce handling, a solution could be that enabling CiviMail overrides how transactional email is sent, adding the extra bounce processing and other features.
Ideally, transactional email could also be queued by CiviMail, so failures are not fatal and delivery attempts can be retried. Currently, if you are over-quota or your mail delivery provider temporarily suspended your account, then you have real problems when doing some critical thing that sends a transactional email message.