Development issueshttps://lab.civicrm.org/groups/dev/-/issues2022-06-11T14:55:55Zhttps://lab.civicrm.org/dev/core/-/issues/3608getRecipients() appears to use a WHERE clause as an ORDER BY clause2022-06-11T14:55:55ZkengetRecipients() appears to use a WHERE clause as an ORDER BY clauseSee https://github.com/civicrm/civicrm-core/commit/906298d3f63825adbee0e85963d89354edf8acf1#r30370554See https://github.com/civicrm/civicrm-core/commit/906298d3f63825adbee0e85963d89354edf8acf1#r303705545.8https://lab.civicrm.org/dev/core/-/issues/3607Feature: Provide an administrative option to make unsubscribe affect all matc...2024-02-17T05:03:32ZlolcodeFeature: Provide an administrative option to make unsubscribe affect all matching email addressesOur organization would like the unsubscribe action to unsubscribe all contacts with the same email from the group. In cases of duplicate contacts they unsubscribe once but one of the duplicates will still get an email and then we get CAS...Our organization would like the unsubscribe action to unsubscribe all contacts with the same email from the group. In cases of duplicate contacts they unsubscribe once but one of the duplicates will still get an email and then we get CASL complaints. Due to CASL we are trying to be cautious by default here.
We suggest an administrative setting for Civimail: "Unsubscribe contact (radio) or unsubscribe email address (radio)"
There was some support for this option expressed in a discussion here: https://chat.civicrm.org/civicrm/pl/sxd4jpwxpbynx8y15994ua7jea
A good point is that this needs to be guarded by token access to stop attackers being able to unsubscribe prominent emails from organisational lists (for example).
A further suggestion was made to make this a user selection option on the unsubscribe page. I feel that this would be a good but separate feature request.https://lab.civicrm.org/dev/core/-/issues/3606send test email may create duplicate contacts2022-06-11T14:55:49Zlcdwebsend test email may create duplicate contactsto reproduce:
1. construct a new email
2. in the "send test email to:" field list two email addresses that match existing contacts. separate them with a comma and space
3. trigger the test email
the first email will match the existi...to reproduce:
1. construct a new email
2. in the "send test email to:" field list two email addresses that match existing contacts. separate them with a comma and space
3. trigger the test email
the first email will match the existing contact. the second email will result in a new duplicate contact getting created. the issue is if you have a space after the comma separating the email addresses. the values are not trimmed when parsed, causing a failed match and the creation of a duplicate.5.6lcdweblcdwebhttps://lab.civicrm.org/dev/core/-/issues/3605"Tracking Click-Throughs" option in mailings generates 404 links when using m...2024-02-17T05:03:31Zjensschuppe"Tracking Click-Throughs" option in mailings generates 404 links when using multi-language with path prefixThis occurred in a Drupal environment with multiple languages and the following configuration:
- Languages in Drupal:
- English (activated)
- German (activated, default)
- Languages in CiviCRM:
- Default language: English
- avai...This occurred in a Drupal environment with multiple languages and the following configuration:
- Languages in Drupal:
- English (activated)
- German (activated, default)
- Languages in CiviCRM:
- Default language: English
- available: German, English
- Inherit CMS language: yes
Conditions under which the erroneous behavior can be reproduced:
- Mailing language: English
- UI language: German
- language prefix in the URL: none or "de"
or:
- Mailing language: German
- UI language: English
- language prefix in the URL: none or "en"
ergo: Using a UI language different from the mailing language and having path prefixes for language detection.
The result is links to the tracking script being prefixed with the language code of the UI language, like so:
https://example.org/de/sites/all/modules/civicrm/extern/url.php?u=123&qid=12345
which is producing a 404 as obviously the path to the script file is not valid due to the language prefix.
Apparently, this code in CRM/Mailing/BAO/TrackableURL.php:93 is where the URL is constructed, but with `userFrameworkResourceURL` being prefixed with the language code:
```
$redirect = $config->userFrameworkResourceURL . "extern/url.php?u=$id";
```https://lab.civicrm.org/dev/core/-/issues/3604Can't save Message Template - authorization failed2022-06-11T14:55:44ZkenCan't save Message Template - authorization failedThis problem has been [reported on StackExchange](https://civicrm.stackexchange.com/questions/36233/problem-with-permissions-to-edit-message-templates) with version 5.27.2 and I have experienced it with 5.28.3.
Unless you have permissio...This problem has been [reported on StackExchange](https://civicrm.stackexchange.com/questions/36233/problem-with-permissions-to-edit-message-templates) with version 5.27.2 and I have experienced it with 5.28.3.
Unless you have permission "Administer CiviCRM" you can't edit a Message Template. This fails with an "Authorization failure" message.
I don't understand the code well enough to suggest a fix, but here's a hack to get around it.
The saving of the Message Template is done by this code on line 311 of CRM/Admin/Form/MessageTemplate.php
```
$messageTemplate = MessageTemplate::save()->setDefaults($params)->setRecords([['id' => $this->_id]])->execute()->first();
```
At the point when it comes to checking permissions, there are permissions on 'message_template' for 'get', 'create' and 'update' but not for 'save'. So the default permission is used (Administer CiviCRM).
The following patch to CRM/Core/Permission.php works around the problem ...
```
@@ -1483,6 +1483,7 @@ class CRM_Core_Permission {
$permissions['message_template'] = [
'get' => ['access CiviCRM'],
+ 'save' => [['edit message templates', 'edit user-driven message templates', 'edit system workflow message templates']],
'create' => [['edit message templates', 'edit user-driven message templates', 'edit system workflow message templates']],
'update' => [['edit message templates', 'edit user-driven message templates', 'edit system workflow message templates']],
];
```https://lab.civicrm.org/dev/core/-/issues/3603Undefined index: error-codes in recaptcha_check_answer()2024-02-16T05:03:24Zchris_bluejacUndefined index: error-codes in recaptcha_check_answer()## Overview
When non-authenticated user subscribes to mailing list at /civicrm/mailing/subscribe, checks the "I am not a robot" reCAPTCHA, and clicks subscribe. The subscription request is processed properly, but an error is thrown:
Not...## Overview
When non-authenticated user subscribes to mailing list at /civicrm/mailing/subscribe, checks the "I am not a robot" reCAPTCHA, and clicks subscribe. The subscription request is processed properly, but an error is thrown:
Notice: Undefined index: error-codes in recaptcha_check_answer() (line 159 of /bitnami/drupal/modules/contrib/civicrm/ext/recaptcha/lib/recaptcha/recaptchalib.php).
## Reproduction steps
1. /civicrm/mailing/subscribe
2. complete email address, check relevant mailing list (in this case, "General Newsletter")
3. Check "I'm not a robot" in reCAPTCHA
4. Click "Subscribe"
5. Subscription is processed properly.
6. Error reported on confirmation page.
## Current behaviour
Error message presented.
Functionality of reCAPTCHA and subscription is okay.
~~~
Notice: Undefined index: error-codes in recaptcha_check_answer() (line 159 of /bitnami/drupal/modules/contrib/civicrm/ext/recaptcha/lib/recaptcha/recaptchalib.php).
~~~
## Expected behaviour
No error message
## Environment Information
- CiviCRM: 5.40.2
- PHP: 7.3.27
- CMS: Drupal 7.82
- Database: MySQL 5.7.33
- Web Server: Apache 2.4.46
- reCAPTCHA: Version 5.40.2
![Screenshot_2021-10-31_084608](/uploads/286470768de6dd8bc6a632827856eafd/Screenshot_2021-10-31_084608.png)https://lab.civicrm.org/dev/core/-/issues/3602Bounce processing fails for invalid unicode characters2022-06-11T14:55:40ZtschuettlerBounce processing fails for invalid unicode charactersWe received some bounces that where containing invalid unicode characters. Due to MySQL not accepting to write those incorrect string values to the database, the bounces are moved to the processed folder although they are not actually re...We received some bounces that where containing invalid unicode characters. Due to MySQL not accepting to write those incorrect string values to the database, the bounces are moved to the processed folder although they are not actually recorded.
Error Log:
```
[info] $backTrace = #0 /var/www/drupal/sites/all/modules/civicrm/CRM/Core/Error.php(912): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 [internal function](): CRM_Core_Error::exceptionHandler(Object(DB_Error))
#2 /var/www/drupal/sites/all/modules/civicrm/packages/PEAR.php(931): call_user_func((Array:2), Object(DB_Error))
#3 /var/www/drupal/sites/all/modules/civicrm/packages/DB.php(976): PEAR_Error->PEAR_Error("DB Error: unknown error", -1, 16, (Array:2), "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#4 /var/www/drupal/sites/all/modules/civicrm/packages/PEAR.php(564): DB_Error->DB_Error(-1, 16, (Array:2), "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#5 /var/www/drupal/sites/all/modules/civicrm/packages/DB/common.php(1905): PEAR->raiseError(NULL, -1, NULL, NULL, "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...", "DB_Error", TRUE)
#6 /var/www/drupal/sites/all/modules/civicrm/packages/DB/mysql.php(895): DB_common->raiseError(-1, NULL, NULL, NULL, "1366 ** Incorrect string value: '\x81hrt ' for column 'bounce_reason' at row 1")
#7 /var/www/drupal/sites/all/modules/civicrm/packages/DB/mysql.php(328): DB_mysql->mysqlRaiseError()
#8 /var/www/drupal/sites/all/modules/civicrm/packages/DB/common.php(1216): DB_mysql->simpleQuery("INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#9 /var/www/drupal/sites/all/modules/civicrm/packages/DB/DataObject.php(2438): DB_common->query("INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#10 /var/www/drupal/sites/all/modules/civicrm/packages/DB/DataObject.php(1060): DB_DataObject->_query("INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#11 /var/www/drupal/sites/all/modules/civicrm/CRM/Core/DAO.php(468): DB_DataObject->insert()
#12 /var/www/drupal/sites/all/modules/civicrm/CRM/Mailing/Event/BAO/Bounce.php(81): CRM_Core_DAO->save()
#13 /var/www/drupal/sites/all/modules/civicrm/api/v3/Mailing.php(316): CRM_Mailing_Event_BAO_Bounce::create((Array:6))
#14 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_mailing_event_bounce((Array:5))
#15 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Kernel.php(96): Civi\API\Provider\MagicFunctionProvider->invoke((Array:9))
#16 /var/www/drupal/sites/all/modules/civicrm/api/api.php(23): Civi\API\Kernel->run("Mailing", "event_bounce", (Array:5), NULL)
#17 /var/www/drupal/sites/all/modules/civicrm/CRM/Utils/Mail/EmailProcessor.php(381): civicrm_api("Mailing", "event_bounce", (Array:5))
#18 /var/www/drupal/sites/all/modules/civicrm/CRM/Utils/Mail/EmailProcessor.php(58): CRM_Utils_Mail_EmailProcessor::_process(TRUE, Object(CRM_Core_DAO_MailSettings))
#19 /var/www/drupal/sites/all/modules/civicrm/api/v3/Job.php(361): CRM_Utils_Mail_EmailProcessor::processBounces()
#20 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_job_fetch_bounces((Array:1))
#21 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Kernel.php(96): Civi\API\Provider\MagicFunctionProvider->invoke((Array:9))
#22 /var/www/drupal/sites/all/modules/civicrm/api/api.php(23): Civi\API\Kernel->run("Job", "fetch_bounces", (Array:1), NULL)
#23 /var/www/drupal/sites/all/modules/civicrm/CRM/Core/JobManager.php(134): civicrm_api("Job", "fetch_bounces", (Array:1))
#24 /var/www/drupal/sites/all/modules/civicrm/CRM/Core/JobManager.php(79): CRM_Core_JobManager->executeJob(Object(CRM_Core_ScheduledJob))
#25 /var/www/drupal/sites/all/modules/civicrm/api/v3/Job.php(99): CRM_Core_JobManager->execute(FALSE)
#26 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_job_execute((Array:1))
#27 /var/www/drupal/sites/all/modules/civicrm/Civi/API/Kernel.php(96): Civi\API\Provider\MagicFunctionProvider->invoke((Array:9))
#28 /var/www/drupal/sites/all/modules/civicrm/api/api.php(23): Civi\API\Kernel->run("job", "execute", (Array:1), NULL)
#29 /var/www/drupal/sites/all/modules/civicrm/drupal/drush/civicrm.drush.inc(1560): civicrm_api("job", "execute", (Array:1))
#30 [internal function](): drush_civicrm_api("job.execute")
#31 /usr/share/php/drush/includes/command.inc(334): call_user_func_array("drush_civicrm_api", (Array:1))
#32 /usr/share/php/drush/includes/command.inc(208): _drush_invoke_hooks("civicrm-api", (Array:1), "civicrm")
#33 [internal function](): drush_command("job.execute")
#34 /usr/share/php/drush/includes/command.inc(175): call_user_func_array("drush_command", (Array:1))
#35 /usr/share/php/drush/drush.php(92): drush_dispatch((Array:27))
#36 /usr/share/php/drush/drush.php(61): _drush_bootstrap_and_dispatch()
#37 /usr/share/php/drush/drush.php(16): drush_main()
#38 {main}
```
I will create PR with a unit test.5.11https://lab.civicrm.org/dev/core/-/issues/3601MS Exchange - IMAP-XOAuth2 authentication fails2022-09-12T18:25:55ZtottenMS Exchange - IMAP-XOAuth2 authentication failsOverview
----------------------------------------
https://lab.civicrm.org/dev/core/-/issues/2141 introduced support for (1) obtaining OAuth2 tokens and (2) using the OAuth2 tokens for IMAP-XOAuth2.
In my test setup, I am able to obtai...Overview
----------------------------------------
https://lab.civicrm.org/dev/core/-/issues/2141 introduced support for (1) obtaining OAuth2 tokens and (2) using the OAuth2 tokens for IMAP-XOAuth2.
In my test setup, I am able to obtain a token from Microsoft which appears valid -- but their IMAP server is not accepting the token.
The "Critical Evaluation" below include a more thorough review of theories/data, but here's a high-level summary of debugging efforts:
| Provider | OAuth2 initiation | Use token for IMAP/XOAuth2 | Use token for HTTP/REST |
| -- | -- | -- | -- |
| Microsoft Exchange Online | :white_check_mark: | :stop_sign: fails, tho JWT has IMAP claims | :white_check_mark: |
| Google Mail | :white_check_mark: | :white_check_mark: | (not tested) |
Reproduction steps
----------------------------------------
Configure Civi's OAuth client as described in the draft documentation: https://lab.civicrm.org/documentation/docs/sysadmin/-/merge_requests/278
Recap:
1. Enable `oauth-client`
2. Register the OAuth details in Civi and in Azure Portal.
3. In Civi's "Admin => CiviMail => Mail Accounts", add a mail account
4. For the newly created account, choose "Save & Test".
Current behavior
----------------------------------------
The IMAP server responds with `NO AUTHENTICATE failed.` and `* BAD Command Error. 12`
In the GUI, it appears as:
![Screen_Shot_2020-11-06_at_5.10.49_PM](/uploads/dfa7fd59bd5be48743ef1be7132778c8/Screen_Shot_2020-11-06_at_5.10.49_PM.png)
On a network level, the interaction is:
```
<< * OK The Microsoft Exchange IMAP4 service is ready. [{_LONG_RANDOM_CODE_}]
>> A0001 AUTHENTICATE XOAUTH2 {_THE_XOAUTH2_TOKEN_}
<< A0001 NO AUTHENTICATE failed.
>>
<< * BAD Command Error. 12
>> A0002 LOGOUT
<< * BAD Command Error. 12
```
(To see this, I applied [a hack to log network I/O](https://gist.github.com/totten/05de1aaf6948513792a40fb6f1eafb26) and re-ran the check via CLI.)
Expected behavior
----------------------------------------
The IMAP test should succeed.
Environment information
----------------------------------------
* __Browser:__ _Firefox_
* __CiviCRM:__ _5.32.beta1_
* __PHP:__ _7.1_
* __CMS:__ _Drupal 7_
* __Database:__ _MySQL 5.6_
* __Web Server:__ _Apache 2.4_
Critical evaluation
----------------------------------------
Here are a few theories for where the problem may reside.
1. The problem is in the OAuth2 client.
1. The OAuth2 client is broken - it does not obtain a real token from Microsoft.
2. The OAuth2 client obtains a real token - but the token does not have the proper scopes.
3. The OAuth2 client obtains a valid token with proper scopes - but it doesn't speak IMAP-XOAuth2 correctly.
2. The problem is in the OAuth2 service provider (either the authorization-server or the resource-server/IMAP-server).
1. There is a problem in the account flags/configuration.
2. There is a bug in the OAuth2 service provider.
Let's consider each in turn to see if we can prove/disprove the theory:
## (Theory 1.1) The OAuth2 client is broken - it does not obtain a real token from Microsoft.
This explanation doesn't hold -- because the same token works for other scenarios.
* Navigate to "Admin => System Settings => OAuth => ms-exchange"
![Screen_Shot_2020-11-06_at_5.41.30_PM](/uploads/c7cd3286bdebf306c40466d3b22b7555/Screen_Shot_2020-11-06_at_5.41.30_PM.png)
* Click "Inspect"
* Find "Access Token: Raw". Copy the value.
* In CLI, make an HTTP request to MS with the same token:
```
$ curl 'https://graph.microsoft.com/v1.0/me' \
-H 'Accept: application/json' \
-H 'Authorization: Bearer <TOKEN_VALUE>'
{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users/$entity","businessPhones":["XXX-XXX-XXXX"],"displayName":"My Name","givenName":"My","jobTitle":null,"mail":"myname@wariomail.onmicrosoft.com","mobilePhone":null,"officeLocation":null,"preferredLanguage":"en-US","surname":"Name","userPrincipalName":"myname@wariomail.onmicrosoft.com","id":"XXXXX-XXXX-XXXXX"}
```
* The response is well-formed and has the correct information.
## (Theory 1.2) The OAuth2 client obtains a real token - but the token does not have the proper scopes.
I can show that the client works according to spec.
* In the Microsoft documentation [Authenticate an IMAP, POP or SMTP connection using OAuth](https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#configure-your-application), we can see the required scope is `IMAP.AccessAsUser.All`.
* In Azure Portal, navigate to "Azure Active Directory => App registrations => (my-app) => API permissions". Note the list of permissions includes IMAP:
![Screen_Shot_2020-11-06_at_6.06.48_PM](/uploads/739adc203b05b30fdeb3efd152b340f7/Screen_Shot_2020-11-06_at_6.06.48_PM.png)
* In CiviCRM, navigate to "Admin => System Settings => OAuth => ms-exchange"
* In the "Details" tab, we can see the list of scopes that requested during setup:
![Screen_Shot_2020-11-06_at_5.56.41_PM](/uploads/9f23d454175a2ed8d8b16da55b5a4bd1/Screen_Shot_2020-11-06_at_5.56.41_PM.png)
* In the "Client #X" tab, we can "Inspect" the token.
* Observe: The "Token Record" shows that the `raw` response from the OAuth2 service. It indicates approval for the necessary scopes:
![Screen_Shot_2020-11-06_at_5.58.47_PM](/uploads/3ebf06fe493d8aa83d22c25fc3ce3380/Screen_Shot_2020-11-06_at_5.58.47_PM.png)
* Observe: The "Access Token: JWT Payload" also reports these scopes:
```
"scp": "IMAP.AccessAsUser.All POP.AccessAsUser.All SMTP.Send User.Read profile openid email",
```
So, it appears that every component (from Azure app registration, to the client's request configuration, to the received token data, to the decoded JWT token data) reports `IMAP.AccessAsUser.All` as an included scope.
There is a subtle discrepancy in how the scope is labeled - sometimes, it's presented as the shorter `IMAP.AccessAsUser.All`; other times, it's the longer `https://outlook.office.com/IMAP.AccessAsUser.All`. To be sure, I've requested tokens with both notations -- and in both cases, the outcome is the same. (To wit: the JWT token has `IMAP.AccessAsUser.All` - but it does not work for IMAP access.)
## (Theory 1.3) The OAuth2 client obtains a valid token with proper scopes - but it doesn't speak IMAP-XOAuth2 correctly.
I've tested this theory a few ways:
1. Connect to another service (Google Mail) which uses the same protocol (IMAP-XOauth2). It works. :white_check_mark:
2. Hack the PHP IMAP library (`imap_transport.php`) to inspect the wire-communication. Manually decode the XOAuth2 statement to ensure that it follows [the XOAuth2 formula](https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#sasl-xoauth2). :white_check_mark:
3. Write a new script with a different PHP IMAP library to try the same token. The alternate library works with Google Mail but not Microsoft Exchange.
## (Theory 2.1) There is a problem in the account flags/configuration.
As I mentioned above (theory 1.2), the "App registration" has all the scopes described in the docs
![Screen_Shot_2020-11-06_at_6.06.48_PM](/uploads/afe29d69f170150a89b98802fd5262b8/Screen_Shot_2020-11-06_at_6.06.48_PM.png)
In the "Exchange Administration Center", the account has IMAP enabled:
![Screen_Shot_2020-11-06_at_6.32.17_PM](/uploads/a5980fff21dcf85add1df96d03e1c9b2/Screen_Shot_2020-11-06_at_6.32.17_PM.png)
In the Exchange webmail UI, there is a settings page with a search-box. Search for "IMAP", and it shows:
![Screen_Shot_2020-11-06_at_6.27.06_PM](/uploads/a4ae9d1aebae308df98a41b3ac348379/Screen_Shot_2020-11-06_at_6.27.06_PM.png)
It's peculiar that searching for "IMAP" presents an inert radio-button for POP, but it does show connection details for IMAP.
TBH, Microsoft's backend here is labyrinthine, so it's hard to be sure that I've checked every possible permission. But I can't find anything else that would prevent IMAP access.
Perhaps someone with more experience in Microsoft's backend should try to run on a different account? Perhaps an account that has been demonstrated to work with IMAP?
## (Theory 2.2) There is a bug in the OAuth2 service provider.
I can't really prove or disprove this theory. I don't really want to believe this is the problem (*surely this stuff works for other people...*). OTOH, maybe it's actually quite rare for people to use IMAP+XOAuth2 with Exchange Online...5.32.0https://lab.civicrm.org/dev/core/-/issues/3600Bounce processing fails for 4-byte unicode characters2022-06-11T14:55:27ZtschuettlerBounce processing fails for 4-byte unicode charactersWhen the bounced message contains any 4-byte unicode character, it cannot be saved to the database due to the currently missing support for utf8mb4 in CiviCRM.
Error Log:
```
[info] $backTrace = #0 /opt/buildkit/build/dmaster/sites/all...When the bounced message contains any 4-byte unicode character, it cannot be saved to the database due to the currently missing support for utf8mb4 in CiviCRM.
Error Log:
```
[info] $backTrace = #0 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/CRM/Core/Error.php(948): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/PEAR.php(921): CRM_Core_Error::exceptionHandler(Object(DB_Error))
#2 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/DB.php(985): PEAR_Error->__construct("DB Error: unknown error", -1, 16, (Array:2), "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#3 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/PEAR.php(575): DB_Error->__construct(-1, 16, (Array:2), "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#4 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/PEAR.php(223): PEAR->_raiseError(Object(DB_mysqli), NULL, -1, 16, (Array:2), "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...", "DB_Error", TRUE)
#5 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/DB/common.php(1907): PEAR->__call("raiseError", (Array:7))
#6 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/DB/mysqli.php(933): DB_common->raiseError(-1, NULL, NULL, "INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...", "1366 ** Incorrect string value: '\xF0\x9F\x8C\xB4' for column 'bounce_reason'...")
#7 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/DB/mysqli.php(403): DB_mysqli->mysqliRaiseError()
#8 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/DB/common.php(1216): DB_mysqli->simpleQuery("INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#9 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/DB/DataObject.php(2415): DB_common->query("INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#10 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/packages/DB/DataObject.php(1040): DB_DataObject->_query("INSERT INTO civicrm_mailing_event_bounce (event_queue_id , bounce_type_id , b...")
#11 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/CRM/Core/DAO.php(571): DB_DataObject->insert()
#12 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/CRM/Mailing/Event/BAO/Bounce.php(87): CRM_Core_DAO->save()
#13 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/api/v3/Mailing.php(349): CRM_Mailing_Event_BAO_Bounce::create((Array:7))
#14 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(100): civicrm_api3_mailing_event_bounce((Array:7))
#15 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/Civi/API/Kernel.php(169): Civi\API\Provider\MagicFunctionProvider->invoke((Array:9))
#16 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/Civi/API/Kernel.php(100): Civi\API\Kernel->runRequest((Array:9))
#17 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/api/api.php(23): Civi\API\Kernel->runSafe("Mailing", "event_bounce", (Array:6), NULL)
#18 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/CRM/Utils/Mail/EmailProcessor.php(341): civicrm_api("Mailing", "event_bounce", (Array:6))
#19 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/CRM/Utils/Mail/EmailProcessor.php(59): CRM_Utils_Mail_EmailProcessor::_process(TRUE, Object(CRM_Core_DAO_MailSettings), 0)
#20 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/api/v3/Job.php(403): CRM_Utils_Mail_EmailProcessor::processBounces(0)
#21 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(100): civicrm_api3_job_fetch_bounces((Array:3))
#22 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/Civi/API/Kernel.php(169): Civi\API\Provider\MagicFunctionProvider->invoke((Array:9))
#23 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/Civi/API/Kernel.php(100): Civi\API\Kernel->runRequest((Array:9))
#24 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/api/api.php(23): Civi\API\Kernel->runSafe("job", "fetch_bounces", (Array:2), NULL)
#25 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/Civi/Test/Api3TestTrait.php(262): civicrm_api("job", "fetch_bounces", (Array:2))
#26 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/Civi/Test/Api3TestTrait.php(151): CiviUnitTestCase->civicrm_api("job", "fetch_bounces", (Array:2))
#27 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/tests/phpunit/CRM/Utils/Mail/EmailProcessorTest.php(78): CiviUnitTestCase->callAPISuccess("job", "fetch_bounces", (Array:2))
#28 [internal function](): CRM_Utils_Mail_EmailProcessorTest->testBounceProcessingUTF8mb4()
#29 phar:///opt/buildkit/extern/phpunit5/phpunit5.phar/phpunit/Framework/TestCase.php(1062): ReflectionMethod->invokeArgs(Object(CRM_Utils_Mail_EmailProcessorTest), (Array:0))
#30 /opt/buildkit/build/dmaster/sites/all/modules/civicrm/tests/phpunit/CiviTest/CiviUnitTestCase.php(196): PHPUnit_Framework_TestCase->runTest()
#31 phar:///opt/buildkit/extern/phpunit5/phpunit5.phar/phpunit/Framework/TestCase.php(913): CiviUnitTestCase->runTest()
#32 phar:///opt/buildkit/extern/phpunit5/phpunit5.phar/phpunit/Framework/TestResult.php(686): PHPUnit_Framework_TestCase->runBare()
#33 phar:///opt/buildkit/extern/phpunit5/phpunit5.phar/phpunit/Framework/TestCase.php(868): PHPUnit_Framework_TestResult->run(Object(CRM_Utils_Mail_EmailProcessorTest))
#34 phar:///opt/buildkit/extern/phpunit5/phpunit5.phar/phpunit/Framework/TestSuite.php(733): PHPUnit_Framework_TestCase->run(Object(PHPUnit_Framework_TestResult))
#35 phar:///opt/buildkit/extern/phpunit5/phpunit5.phar/phpunit/TextUI/TestRunner.php(517): PHPUnit_Framework_TestSuite->run(Object(PHPUnit_Framework_TestResult))
#36 phar:///opt/buildkit/extern/phpunit5/phpunit5.phar/phpunit/TextUI/Command.php(186): PHPUnit_TextUI_TestRunner->doRun(Object(PHPUnit_Framework_TestSuite), (Array:54), TRUE)
#37 phar:///opt/buildkit/extern/phpunit5/phpunit5.phar/phpunit/TextUI/Command.php(116): PHPUnit_TextUI_Command->run((Array:11), TRUE)
#38 /opt/buildkit/extern/phpunit5/phpunit5.phar(598): PHPUnit_TextUI_Command::main()
#39 {main}
```5.11https://lab.civicrm.org/dev/core/-/issues/3599Test mailings create new contacts even when "Add Contacts" permission is not ...2022-06-11T14:55:17ZJonGoldTest mailings create new contacts even when "Add Contacts" permission is not present.### Overview
When sending a test mail, CiviCRM will check if the email matches an existing contact in the database. If it does, it uses that contact ID; if not, it creates a contact. However, it creates a contact without regard for wh...### Overview
When sending a test mail, CiviCRM will check if the email matches an existing contact in the database. If it does, it uses that contact ID; if not, it creates a contact. However, it creates a contact without regard for whether a user has permission to add contacts.
### Steps to replicate
* Create a user that does not have the "Add Contacts" permission.
* With that user, create a new mailing (traditional or Mosaico, doesn't matter).
* Send a test ("draft") mail to an email address that doesn't exist in the database.
### Expected behavior
No new contact is created.
### Actual behavior
A new contact is created.5.29.0JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3598Proposal: Scheduled Reminders should not (by default) use Smarty on new installs2023-01-27T15:27:33ZJonGoldProposal: Scheduled Reminders should not (by default) use Smarty on new installsWhen token rendering is enabled, and the `TokenProcessor` context says to use Smarty, the TokenRow rendering barfs on any curly brace that isn't a token or Smarty. Which includes, for instance, [inline CSS](https://civicrm.stackexchange...When token rendering is enabled, and the `TokenProcessor` context says to use Smarty, the TokenRow rendering barfs on any curly brace that isn't a token or Smarty. Which includes, for instance, [inline CSS](https://civicrm.stackexchange.com/q/17643/12).
At the time that SE post was made, Smarty stripped out the CSS. It no longer does, and instead generates large numbers of errors. Rather than fight that battle, I'm curious whether Smarty on scheduled reminders is necessary and serves a purpose in a post-5.43 TokenProcessor world.
If we decided that Scheduled Reminder TokenProcessors had [Smarty set to false](https://github.com/civicrm/civicrm-core/blob/ce0d67d6a686fcfae1d7046e615107f25cec501a/CRM/Core/BAO/ActionSchedule.php#L647) this problem would go away.
Does anyone use Smarty here? Should they?JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3597Contact with non-primary on-hold email addresses prevent delivery2022-06-11T14:55:15ZAlanDixonContact with non-primary on-hold email addresses prevent deliveryHere's the detailed origin of the problem. I have a client who complained of his list being too small. Normally I just explain the usual reasons a group size is smaller than what gets sent, but he was adamant that something else was goin...Here's the detailed origin of the problem. I have a client who complained of his list being too small. Normally I just explain the usual reasons a group size is smaller than what gets sent, but he was adamant that something else was going on.
So, I created a mailing to that group and it calculated and found 9560 recipients, same as what it reported on screen.
Next I generated a list of who I think should get the mailing, using this sql:
`select count(distinct(gc.contact_id)), gc.status, e.on_hold, c.do_not_email, c.is_opt_out from civicrm_group_contact gc inner join civicrm_email e on gc.contact_id = e.contact_id inner join civicrm_contact c on gc.contact_id = c.id where group_id = 19 and e.is_primary = 1 group by status, on_hold, do_not_email, is_opt_out;`
which gave me:
```
+--------------------------------+---------+---------+--------------+------------+
| count(distinct(gc.contact_id)) | status | on_hold | do_not_email | is_opt_out |
+--------------------------------+---------+---------+--------------+------------+
| 10374 | Added | 0 | 0 | 0 |
| 1094 | Added | 0 | 0 | 1 |
| 2 | Added | 0 | 1 | 0 |
| 2 | Added | 0 | 1 | 1 |
| 3792 | Added | 1 | 0 | 0 |
| 4 | Added | 1 | 0 | 1 |
| 1 | Added | 2 | 0 | 0 |
| 1 | Pending | 0 | 0 | 0 |
| 2 | Pending | 1 | 0 | 0 |
| 2547 | Removed | 0 | 0 | 0 |
| 312 | Removed | 0 | 0 | 1 |
| 2 | Removed | 0 | 1 | 0 |
| 2 | Removed | 0 | 1 | 1 |
| 67 | Removed | 1 | 0 | 0 |
| 1 | Removed | 1 | 1 | 1 |
+--------------------------------+---------+---------+--------------+------------+
```
So that left me with about 800 contacts that were not going to be mailed for unaccounted reasons.
I looked up the first one that differed and discovered that it had a non-primary email address on hold (but also a primary one that was not on hold).
My conclusion is that this claim: https://civicrm.stackexchange.com/questions/11517/on-hold-what-happens-with-mailings
is false.https://lab.civicrm.org/dev/core/-/issues/3596Manual Recipients in Scheduled Reminders2022-06-11T14:55:13ZguyiacManual Recipients in Scheduled RemindersI can't test this on the dev, but in our Civi (Drupal 7 / Civi 5.24.5) when you save a scheduled reminder for an event and add manual recipients (per the attached image), the reminder gets sent to the manual recipients right away when th...I can't test this on the dev, but in our Civi (Drupal 7 / Civi 5.24.5) when you save a scheduled reminder for an event and add manual recipients (per the attached image), the reminder gets sent to the manual recipients right away when the reminder is saved (the others don't get it until the scheduled time). Am I misreading the help bullets and this is expected functionality, or is this a bug?
![image](/uploads/05ab358390d318a31a161fb42806464c/image.png)https://lab.civicrm.org/dev/core/-/issues/3595Mailing priority2024-02-16T05:03:23ZmfbMailing priorityOrganizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simult...Organizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simultaneously a press release is sent to press contacts. The latter would have higher priority.
CiviMail would send messages for mailings with a high priority before those with a lower priority.https://lab.civicrm.org/dev/core/-/issues/3594Core should provide bounce handling (and queueing) for transactional email2024-02-15T05:03:25ZmfbCore should provide bounce handling (and queueing) for transactional emailCurrently CiviCRM core does not provide bounce handling for transactional email. This can result in automated processes sending out a large volume of scheduled reminders, etc. (as well as user-triggered forms sending out subscription co...Currently CiviCRM core does not provide bounce handling for transactional email. This can result in automated processes sending out a large volume of scheduled reminders, etc. (as well as user-triggered forms sending out subscription confirmations, etc.) with only manual bounce handling by humans.
An extension exists to provide bounce handling and other features for transactional emails: https://civicrm.org/extensions/transactional-emails
Arguably, however, this feature should be provided by core itself, so all civi instances have bounce handling for any email message they send.
Assuming we continue the architecture where it is the CiviMail component that provides bounce handling, a solution could be that enabling CiviMail overrides how transactional email is sent, adding the extra bounce processing and other features.
Ideally, transactional email could also be queued by CiviMail, so failures are not fatal and delivery attempts can be retried. Currently, if you are over-quota or your mail delivery provider temporarily suspended your account, then you have real problems when doing some critical thing that sends a transactional email message.https://lab.civicrm.org/dev/core/-/issues/3593{contact.hash} not populated in Flexmailer/Mosaico2022-06-11T14:55:07ZBjörn Endres{contact.hash} not populated in Flexmailer/MosaicoI don't know if this is the right forum, but but seems like the `{contact.hash}` token is not populated when using Flexmailer/Mosaico.
Observed on:
* CiviCRM `5.5.1`
* Flexmailer `0.2-6ff530c`
* Mosaico `2.0-beta4`I don't know if this is the right forum, but but seems like the `{contact.hash}` token is not populated when using Flexmailer/Mosaico.
Observed on:
* CiviCRM `5.5.1`
* Flexmailer `0.2-6ff530c`
* Mosaico `2.0-beta4`5.37.0https://lab.civicrm.org/dev/core/-/issues/3592(MOVED) [Feature] Please give us the option to disable subtotals for Soft Cre...2022-06-11T14:55:06Zswebervna(MOVED) [Feature] Please give us the option to disable subtotals for Soft Credit Contribution ReportsIssue moved to: https://lab.civicrm.org/dev/report/-/issues/51Issue moved to: https://lab.civicrm.org/dev/report/-/issues/51https://lab.civicrm.org/dev/core/-/issues/3591Public view link for draft mailing is visible to public before mailing is sen...2023-06-04T14:17:12ZlolcodePublic view link for draft mailing is visible to public before mailing is sent and even before it is completeIt doesn't seem quite right that the mailing is public even when it is in draft state.
Of course a bot or interested party would need to be cycling through mailing ids to access it.
It also shows even if the mailing is clearly not compl...It doesn't seem quite right that the mailing is public even when it is in draft state.
Of course a bot or interested party would need to be cycling through mailing ids to access it.
It also shows even if the mailing is clearly not complete. For example if the user doesn't select a mailing header before saving the draft then viewing the public page can also result in errors such as `Error: Call to a member function headers() on null in civicrm_api3_mailing_preview() (line 597 of /sites/all/modules/civicrm/api/v3/Mailing.php).`https://lab.civicrm.org/dev/core/-/issues/3590Email to activity processing - creates duplicate contacts where verp is present2024-01-29T09:59:12ZeileenEmail to activity processing - creates duplicate contacts where verp is presentWhen contacts reply to a civimail from mail@domain.com they use the verp reply to - eg.
mail+r.2057.84206.03323c082c189038@domain.com
The address mail@domain.com exists in the db but a new contact is created rather than matching themWhen contacts reply to a civimail from mail@domain.com they use the verp reply to - eg.
mail+r.2057.84206.03323c082c189038@domain.com
The address mail@domain.com exists in the db but a new contact is created rather than matching themhttps://lab.civicrm.org/dev/core/-/issues/3588Token unsubscribeUrl seemingly resolves to optOutUrl2022-06-11T14:55:01ZmarcelklehrToken unsubscribeUrl seemingly resolves to optOutUrlWhen using the action.unsubscribeToken in a mailing, the resulting URL prompts the user to confirm unsubscribing from all existing newletters instead of just the one that the link came with.When using the action.unsubscribeToken in a mailing, the resulting URL prompts the user to confirm unsubscribing from all existing newletters instead of just the one that the link came with.