CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2024-03-24T05:03:28Zhttps://lab.civicrm.org/dev/core/-/issues/3775Disable Deadlock retries for Mailing Jobs2024-03-24T05:03:28ZseamusleeDisable Deadlock retries for Mailing JobsSo CiviCRM added a feature which re-tries the same query x number of times by default that is 3 when either a deadlock or a lock wait timeout is hit which is useful, but in the concept of mailing job processing it is not so useful becaus...So CiviCRM added a feature which re-tries the same query x number of times by default that is 3 when either a deadlock or a lock wait timeout is hit which is useful, but in the concept of mailing job processing it is not so useful because it means it can get tied up on get_lock queries re-trying them a silly number of times when it should be moving onto the next thingseamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/3753Proposal: Allow negative rules for ACLs2023-09-08T10:52:34ZTobias Voigttobias.voigt@civiservice.deProposal: Allow negative rules for ACLs**PROBLEM:**
When defining ACL rules, it is only possible to 'allow' certain behaviour but not to 'disallow' it. This makes it tedious to define a set of rules that restricts access to e.g. groups of contacts or sets of custom fields - ...**PROBLEM:**
When defining ACL rules, it is only possible to 'allow' certain behaviour but not to 'disallow' it. This makes it tedious to define a set of rules that restricts access to e.g. groups of contacts or sets of custom fields - especially if you have many (groups or sets of custom fields).
**USE CASE:**
In one of our client's system, we have many groups and many sets of custom fields. What I want to achieve is:
1. define a privileged group / role that has exclusive access to **one** set of custom fields
2. allow access to all other sets of custom fields for all users
3. prevent 'normal' users from giving themselves access to the exclusive information by adding themselves to the privileged group
Because I can only define 'positive' ACL rules, here's what I have to do:
- turn off the WP permission for custom fields for the relevant roles (to be able to define my rules via CiviCRM ACLs)
- allow access to all sets of custom fields for 'everyone' (excluding the one I want to restrict access for) - **by creating a rule for each set of custom fields**
- define a rule to allow access to the restricted set of custom fields to the privileged role
Depending on the number of sets of custom fields, this can be a tedious process. Not only that - if I create a new set of custom fields at a later time, I have to remember to create a new ACL rule for that as well.
Now for the third task (preventing users from adding themselves to the privileged group). To achive this I furthermore have to:
- turn off the WP permissions for viewing and editing all contacts for the relevant roles
- allow access to all groups for 'everyone' (excluding the one I want to restrict access for) - **by creating a rule for each contact group**
- define a rule to allow access to the restricted contact group only for the privileged role
Again: With about 50+ groups, that's a tedious task. And again: I have to remember to create a new ACL rule each time I add a new group.
All in all I could end up with a multitude of rules (in our case 70+) that are really hard to maintain.
**PROPOSAL:**
This task could be so much easier, if it was possible to define 'negative' ACL rules. If this was possible, I could:
- turn off the relevant WP permissions
- allow access to all sets of custom fields for everyone (1 rule)
- allow access to all contact groups for everyone (1 rule)
- disallow access to the restricted set of custom fields for everyone (1 rule)
- disallow access to the restricted contact group for everyone (1 rule)
- allow access to the restricted set of custom fields for the privileged role (1 rule)
- allow access to the restricted contact group for the privileged role (1 rule)
As is implied above, **there would have to be some form of priorisation** of rules to make this work.
This way I would end up with only 6 rules and would not have to remember to create new rules whenever I create a new set of custom fields or a new contact group.5.64.0https://lab.civicrm.org/dev/core/-/issues/3740Set default lock level to READ COMMITTED2023-10-20T07:57:25ZJoeMurraySet default lock level to READ COMMITTEDDrupal recently changed the default lock level slightly down from REPEATABLE READ to READ COMMITTED, which has the effect of avoiding some db locks. See https://www.drupal.org/node/3269885 We have seen over recent years some issues with ...Drupal recently changed the default lock level slightly down from REPEATABLE READ to READ COMMITTED, which has the effect of avoiding some db locks. See https://www.drupal.org/node/3269885 We have seen over recent years some issues with deadlocks, primarily on rebuilding group contact cache when there are hierarchies of smart groups. Should CiviCRM be more explicit about what isolation level we expect in order to get more repeatable results, and should we follow Drupal in its move to READ COMMITTED? I tend to think so.
See responses in MM chat from @demeritcowboy
https://chat.civicrm.org/civicrm/pl/gnmhfdxes7rgb8k7ppoaezo45ahttps://lab.civicrm.org/dev/core/-/issues/3677API thoughts: Global switch to turn off all permission checking, not just to ...2022-06-27T21:54:03ZDaveDAPI thoughts: Global switch to turn off all permission checking, not just to the "top-level" api call that was madeI've been thinking about https://lab.civicrm.org/dev/core/-/issues/3671, and my thinking went something like:
* This worked before, so it should go back to the way it was.
* But wait, isn't this really exposing a bug in webform because ...I've been thinking about https://lab.civicrm.org/dev/core/-/issues/3671, and my thinking went something like:
* This worked before, so it should go back to the way it was.
* But wait, isn't this really exposing a bug in webform because how did the person update a case if they don't have permissions to cases?
* But wait, what's this? Every api call in webform purposely turns off permissions? https://github.com/colemanw/webform_civicrm/blob/0bedf9411bf3f0a682ce7a1b3fad9a682093b240/src/Utils.php#L654
```php
function wf_civicrm_api($entity, $operation, $params) {
...
$params += [
'check_permissions' => FALSE,
];
```
* So webform is saying: "Core, I don't want you to check any permissions. Anyone who has (drupal) permission to use this form should have permission to do whatever the form does."
* I don't remember where but I think I saw a similar type of thing come up with afform, or it seems likely to, where "I've only given certain people permission to reach this form, but if they have it, they should be able to do whatever the form does."
So one way to address this use-case, in a way that doesn't require core to lower permission checking for everyone else, is to provide a way for webform and others to set a global kill-switch that turns off all permission checking. That way secondary permission checks that get called from within the api itself like the one in Utils_Recent would still be on for everyone else, but could be turned off for forms that don't want it.
I'm not confident this is the _best_ way, but am just putting it out there.https://lab.civicrm.org/dev/core/-/issues/3648SMS limit is hardcoded at 460 but should be changeable somewhere2024-01-29T09:44:22ZjasonobrownSMS limit is hardcoded at 460 but should be changeable somewhereTwillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the sys...Twillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the system now accepts (and sends) messages up to 1600 characters. It seems logical that the character limit should able to be adjusted and stored somewhere rather than set as a static value in the code. We are currently using the larger limit, and would really like to see this implemented asap so that we can stop manually updating the code each time we update civi.https://lab.civicrm.org/dev/core/-/issues/3646Support tracking URLs with tokens in query strings2022-06-11T14:58:11ZlarsssandergreenSupport tracking URLs with tokens in query stringsURLs with tokens in mailings do not have click tracking. This is an important metric for fundraising (e.g. to see how many and even which contacts click through to contribution pages). @artfulrobot put together some code that worked well...URLs with tokens in mailings do not have click tracking. This is an important metric for fundraising (e.g. to see how many and even which contacts click through to contribution pages). @artfulrobot put together some code that worked well to track these clicks in Flexmailer, but that code was not merged before Flexmailer was merged into core.
https://github.com/civicrm/org.civicrm.flexmailer/issues/30
We used this without issue for four months and found it worked very well. @artfulrobot was using it since March. Would it be possible to merge this into core. Rich, would you be willing to copy your code to a patch against core?RichRichhttps://lab.civicrm.org/dev/core/-/issues/3645Eventually google will require OAUTH for bounce processing and may require it...2022-12-08T20:55:07ZDaveDEventually google will require OAUTH for bounce processing and may require it for outbound SMTP through gmail serversAs noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue supp...As noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue support for just username/password.
The way I read it, this is mostly applying to POP/IMAP clients for reading mail, and they are *hinting* that someday they might require it for outbound SMTP, i.e. at Administer - System Settings - Outbound Mail you choose SMTP and have it set to use gmail servers. The relevant quote is
> No change is required for scanners or other devices using simple mail transfer protocol (SMTP) or LSAs to send emails. If you replace your device, look for one that sends email using OAuth.
(LSA means less secure apps, which is google-speak for username/password)
They are targeting June 2020 - Feb 2021 for IMAP etc, but as noted above they have not specifically set a date for changes regarding sending outbound emails.
It occurs to me though that bounce processing and the email processor use POP/IMAP. For the email processor you can work around it by just forwarding it to a non-gmail address and polling that, but bounce processing might be harder depending on setup.https://lab.civicrm.org/dev/core/-/issues/3638Provide a fullPage property for message templates2024-02-22T05:03:26ZmfbProvide a fullPage property for message templatesIn various parts of the UI, CiviCRM allows users to send a message template either on its own, or wrapped in a header and footer.
e.g. you can send a message template to a contact.
Or, you can create a mailing based on a message templa...In various parts of the UI, CiviCRM allows users to send a message template either on its own, or wrapped in a header and footer.
e.g. you can send a message template to a contact.
Or, you can create a mailing based on a message template, with additional header and footer.
CKEditor needs different settings at civicrm/admin/ckeditor?preset=civimail re: allowed HTML tags to support these two scenarios. With fullPage = true, the HTML is forcibly wrapped with html/head/body tags, and so a message template that includes its header and footer can be composed. With fullPage = false, html/head/body are silently stripped, so this is useful for message templates that should be used with a header and footer.
However, there is only one CKEditor setting for both of these scenarios.
I would propose a new database column where we can indicate if a message template is "fullPage" - i.e. includes header and footer, or not fullPage, i.e. designed to be used with header and footer.
This would allow us to toggle the fullPage setting appropriately. It would also allow us to prevent a user from sending a message template that needs a header and footer without one, or wrapping a message template that includes its header and footer with an additional header and footer.https://lab.civicrm.org/dev/core/-/issues/3636CiviCRM, Find Mailings. If there are no results, the same message "There are ...2022-06-11T14:57:49Zjustinfreeman (Agileware)CiviCRM, Find Mailings. If there are no results, the same message "There are no Unscheduled Mailings" is shownCiviCRM, Find Mailings. If there are no results, the same message "There are no Unscheduled Mailings" is shown regardless of the search criteria. If a different mailing status was used for search, it does not make sense to refer to "Unsc...CiviCRM, Find Mailings. If there are no results, the same message "There are no Unscheduled Mailings" is shown regardless of the search criteria. If a different mailing status was used for search, it does not make sense to refer to "Unscheduled Mailings".
This text should be changed to something else like: "There are no mailings matching this criteria".
Using CiviCRM 5.7.2
Agileware Ref: CIVICRM-1121
![Find_Mailings-2](/uploads/34e6b1996338721a1a2a333a3a16cd2a/Find_Mailings-2.png)https://lab.civicrm.org/dev/core/-/issues/3635Emails marked as bounce when quota is full2024-02-22T05:03:25ZEdselopezEmails marked as bounce when quota is fullThere are cases (usually when using Gmail SMTP) when the Outbound email fails to deliver a mailing due to the user's daily quota being exhausted. In this case, if the mailing is re-used unknowingly, attempts to re-send the mailing adds t...There are cases (usually when using Gmail SMTP) when the Outbound email fails to deliver a mailing due to the user's daily quota being exhausted. In this case, if the mailing is re-used unknowingly, attempts to re-send the mailing adds to the bounce threshold, finally surpassing the limit of three bounces, marking the email as "on hold". To solve this, I'm thinking bounce type of "Quota" should be excluded from the calculation of marking the email as on hold. Please let me know if this is reasonable and I will provide a patch.https://lab.civicrm.org/dev/core/-/issues/3634When using a smart group as a mailing list, users who unsubscribe from the sm...2022-06-11T14:57:45Ztom.mWhen using a smart group as a mailing list, users who unsubscribe from the smart group are still included in the mailing.It looks like in CRM/MailingBAO/Mailing.php, is not taking into account those who have been removed from the smart group. Civi version 5.3.
```
if (count($includeSmartGroupIDs)) {
$query = CRM_Utils_SQL_Select::from($contact)
...It looks like in CRM/MailingBAO/Mailing.php, is not taking into account those who have been removed from the smart group. Civi version 5.3.
```
if (count($includeSmartGroupIDs)) {
$query = CRM_Utils_SQL_Select::from($contact)
->select("$contact.id as contact_id, $entityTable.id as $entityColumn")
->join($entityTable, " INNER JOIN $entityTable ON $entityTable.contact_id = $contact.id ")
->join('gc', " INNER JOIN civicrm_group_contact_cache gc ON $contact.id = gc.contact_id ")
->join('mg', " INNER JOIN civicrm_mailing_group mg ON gc.group_id = mg.entity_id AND mg.search_id IS NULL ")
->join('temp', " LEFT JOIN $excludeTempTablename temp ON $contact.id = temp.contact_id ")
->where('gc.group_id IN (#groups)')
->merge($criteria)
->replaceInto($includedTempTablename, array('contact_id', $entityColumn))
->param('#groups', $includeSmartGroupIDs)
->param('#mailingID', $mailingID)
->execute();
}
```https://lab.civicrm.org/dev/core/-/issues/3631Unexpected behavior from api.MailingEventSubscribe.create2024-02-21T05:03:31ZginkgofjgUnexpected behavior from api.MailingEventSubscribe.createIn creating a custom e-newsletter sign-up form for a client, I encountered the following unexpected behaviors. I'm happy to submit one or more corrective pull requests but would first like to:
* confirm that this is not an unexpected us...In creating a custom e-newsletter sign-up form for a client, I encountered the following unexpected behaviors. I'm happy to submit one or more corrective pull requests but would first like to:
* confirm that this is not an unexpected use of this API,
* confirm that these behaviors are unexpected,
* get two cents from others on the approach, and
* get acceptance criteria for the PR(s) -- i.e., what tests do I need to write/pass?
The unexpecteds:
1. This is cosmetic, but the API metadata are wrong. The API Explorer describes the params as though they were for unsubscription.
2. Suppose contact X is already a member of a mailing group Y. If their IDs are passed to this API, the contact's status in the group is "downgraded" to "Pending" and the user must reconfirm membership by clicking the link emailed to them.
3. The contact's DO NOT MAIL and NO BULK EMAILS Communication Preferences are not updated. These changes should not be effected by the initial API call, but after the user clicks the confirmation link emailed to her.
The approaches:
1. This should be a trivial update to the API's `_spec()` function.
2. The API delegates to `CRM_Mailing_Event_BAO_Subscribe::subscribe()`. I would fix the problem in the delegate by checking for group membership and bailing out early if the record already exists with the appropriate status.
3. Confirmation is handled by `CRM_Mailing_Page_Confirm` which delegates to `CRM_Mailing_Event_BAO_Confirm::confirm()`. I would unset the communication preferences in question in the delegate.ginkgofjgginkgofjghttps://lab.civicrm.org/dev/core/-/issues/3620"View in Browser" does not feed mailing statistics2024-02-19T05:03:28Ztotten"View in Browser" does not feed mailing statisticsI just want to log the existence of the issue generally... suppose you send a newsletter and it has a "View in Browser" link.
Suppose the recipient uses that link. The view the mailing and then drill-down, clicking through some more lin...I just want to log the existence of the issue generally... suppose you send a newsletter and it has a "View in Browser" link.
Suppose the recipient uses that link. The view the mailing and then drill-down, clicking through some more links.
What happens to the stats?
* The web-based view does not count as an "Open". The MUA may or may not report back the "Open" on the original message.
* You will not have any "Click Through" events logged.
This is because the "View in browser" variant of the newsletter is not instrumented with tracking codes (while the MUA variant of the newsletter does have tracking codes).https://lab.civicrm.org/dev/core/-/issues/3616Should tracking pixel be placed at start of email for when Gmail clips messages?2024-02-18T05:03:26ZlarsssandergreenShould tracking pixel be placed at start of email for when Gmail clips messages?With Mosaico templates, it isn't that difficult to reach Gmail's limit of 96-102kB with an email newsletter, after which your email message is clipped and not displayed in full. This is unfortunate in itself, but it also breaks open trac...With Mosaico templates, it isn't that difficult to reach Gmail's limit of 96-102kB with an email newsletter, after which your email message is clipped and not displayed in full. This is unfortunate in itself, but it also breaks open tracking as the tracking pixel is only embedded at the very end of the message and thus isn't fetched unless the reader clicks "View entire message".
Would it be worthwhile to consider putting the tracking pixel at the start of the message to avoid this issue? Obviously, less than ideal for those who don't display images, but I'm not sure how these two might balance out or if there is a clever way to have the best of both worlds.
I'd be happy to provide a patch, but not sure what the best way forward is here.
(Was a former Flexmailer issue, reposted here as I think this is still important.)https://lab.civicrm.org/dev/core/-/issues/3615<link> URLs are tracked and shouldn't be2022-06-13T09:15:05Zlarsssandergreen<link> URLs are tracked and shouldn't beIf you use <link href=URL> in a mailing template, for example to load a webfont, that URL is converted to a tracked URL. Presumably it shouldn't be as this makes a mess of the clickthrough statistics (every one of these links will be cou...If you use <link href=URL> in a mailing template, for example to load a webfont, that URL is converted to a tracked URL. Presumably it shouldn't be as this makes a mess of the clickthrough statistics (every one of these links will be counted as clicked as long as the user's mail client fetches the resource). In other words, a link being fetched in a not a click and shouldn't be counted as one.
Do we need to track any URLs that aren't inside an <a> tag? It might be easier to limit to <a... rather than trying to exclude <link.... The only valid way to write an <a> is with the tag right after the <, so I think we could just look for <a instead of <.
I can provide a patch if this approach is supported.https://lab.civicrm.org/dev/core/-/issues/3607Feature: Provide an administrative option to make unsubscribe affect all matc...2024-02-17T05:03:32ZlolcodeFeature: Provide an administrative option to make unsubscribe affect all matching email addressesOur organization would like the unsubscribe action to unsubscribe all contacts with the same email from the group. In cases of duplicate contacts they unsubscribe once but one of the duplicates will still get an email and then we get CAS...Our organization would like the unsubscribe action to unsubscribe all contacts with the same email from the group. In cases of duplicate contacts they unsubscribe once but one of the duplicates will still get an email and then we get CASL complaints. Due to CASL we are trying to be cautious by default here.
We suggest an administrative setting for Civimail: "Unsubscribe contact (radio) or unsubscribe email address (radio)"
There was some support for this option expressed in a discussion here: https://chat.civicrm.org/civicrm/pl/sxd4jpwxpbynx8y15994ua7jea
A good point is that this needs to be guarded by token access to stop attackers being able to unsubscribe prominent emails from organisational lists (for example).
A further suggestion was made to make this a user selection option on the unsubscribe page. I feel that this would be a good but separate feature request.https://lab.civicrm.org/dev/core/-/issues/3606send test email may create duplicate contacts2022-06-11T14:55:49Zlcdwebsend test email may create duplicate contactsto reproduce:
1. construct a new email
2. in the "send test email to:" field list two email addresses that match existing contacts. separate them with a comma and space
3. trigger the test email
the first email will match the existi...to reproduce:
1. construct a new email
2. in the "send test email to:" field list two email addresses that match existing contacts. separate them with a comma and space
3. trigger the test email
the first email will match the existing contact. the second email will result in a new duplicate contact getting created. the issue is if you have a space after the comma separating the email addresses. the values are not trimmed when parsed, causing a failed match and the creation of a duplicate.5.6lcdweblcdwebhttps://lab.civicrm.org/dev/core/-/issues/3605"Tracking Click-Throughs" option in mailings generates 404 links when using m...2024-02-17T05:03:31Zjensschuppe"Tracking Click-Throughs" option in mailings generates 404 links when using multi-language with path prefixThis occurred in a Drupal environment with multiple languages and the following configuration:
- Languages in Drupal:
- English (activated)
- German (activated, default)
- Languages in CiviCRM:
- Default language: English
- avai...This occurred in a Drupal environment with multiple languages and the following configuration:
- Languages in Drupal:
- English (activated)
- German (activated, default)
- Languages in CiviCRM:
- Default language: English
- available: German, English
- Inherit CMS language: yes
Conditions under which the erroneous behavior can be reproduced:
- Mailing language: English
- UI language: German
- language prefix in the URL: none or "de"
or:
- Mailing language: German
- UI language: English
- language prefix in the URL: none or "en"
ergo: Using a UI language different from the mailing language and having path prefixes for language detection.
The result is links to the tracking script being prefixed with the language code of the UI language, like so:
https://example.org/de/sites/all/modules/civicrm/extern/url.php?u=123&qid=12345
which is producing a 404 as obviously the path to the script file is not valid due to the language prefix.
Apparently, this code in CRM/Mailing/BAO/TrackableURL.php:93 is where the URL is constructed, but with `userFrameworkResourceURL` being prefixed with the language code:
```
$redirect = $config->userFrameworkResourceURL . "extern/url.php?u=$id";
```https://lab.civicrm.org/dev/core/-/issues/3598Proposal: Scheduled Reminders should not (by default) use Smarty on new installs2023-01-27T15:27:33ZJonGoldProposal: Scheduled Reminders should not (by default) use Smarty on new installsWhen token rendering is enabled, and the `TokenProcessor` context says to use Smarty, the TokenRow rendering barfs on any curly brace that isn't a token or Smarty. Which includes, for instance, [inline CSS](https://civicrm.stackexchange...When token rendering is enabled, and the `TokenProcessor` context says to use Smarty, the TokenRow rendering barfs on any curly brace that isn't a token or Smarty. Which includes, for instance, [inline CSS](https://civicrm.stackexchange.com/q/17643/12).
At the time that SE post was made, Smarty stripped out the CSS. It no longer does, and instead generates large numbers of errors. Rather than fight that battle, I'm curious whether Smarty on scheduled reminders is necessary and serves a purpose in a post-5.43 TokenProcessor world.
If we decided that Scheduled Reminder TokenProcessors had [Smarty set to false](https://github.com/civicrm/civicrm-core/blob/ce0d67d6a686fcfae1d7046e615107f25cec501a/CRM/Core/BAO/ActionSchedule.php#L647) this problem would go away.
Does anyone use Smarty here? Should they?JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3595Mailing priority2024-02-16T05:03:23ZmfbMailing priorityOrganizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simult...Organizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simultaneously a press release is sent to press contacts. The latter would have higher priority.
CiviMail would send messages for mailings with a high priority before those with a lower priority.