CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2024-02-21T05:03:31Zhttps://lab.civicrm.org/dev/core/-/issues/3631Unexpected behavior from api.MailingEventSubscribe.create2024-02-21T05:03:31ZginkgofjgUnexpected behavior from api.MailingEventSubscribe.createIn creating a custom e-newsletter sign-up form for a client, I encountered the following unexpected behaviors. I'm happy to submit one or more corrective pull requests but would first like to:
* confirm that this is not an unexpected us...In creating a custom e-newsletter sign-up form for a client, I encountered the following unexpected behaviors. I'm happy to submit one or more corrective pull requests but would first like to:
* confirm that this is not an unexpected use of this API,
* confirm that these behaviors are unexpected,
* get two cents from others on the approach, and
* get acceptance criteria for the PR(s) -- i.e., what tests do I need to write/pass?
The unexpecteds:
1. This is cosmetic, but the API metadata are wrong. The API Explorer describes the params as though they were for unsubscription.
2. Suppose contact X is already a member of a mailing group Y. If their IDs are passed to this API, the contact's status in the group is "downgraded" to "Pending" and the user must reconfirm membership by clicking the link emailed to them.
3. The contact's DO NOT MAIL and NO BULK EMAILS Communication Preferences are not updated. These changes should not be effected by the initial API call, but after the user clicks the confirmation link emailed to her.
The approaches:
1. This should be a trivial update to the API's `_spec()` function.
2. The API delegates to `CRM_Mailing_Event_BAO_Subscribe::subscribe()`. I would fix the problem in the delegate by checking for group membership and bailing out early if the record already exists with the appropriate status.
3. Confirmation is handled by `CRM_Mailing_Page_Confirm` which delegates to `CRM_Mailing_Event_BAO_Confirm::confirm()`. I would unset the communication preferences in question in the delegate.ginkgofjgginkgofjghttps://lab.civicrm.org/dev/core/-/issues/3620"View in Browser" does not feed mailing statistics2024-02-19T05:03:28Ztotten"View in Browser" does not feed mailing statisticsI just want to log the existence of the issue generally... suppose you send a newsletter and it has a "View in Browser" link.
Suppose the recipient uses that link. The view the mailing and then drill-down, clicking through some more lin...I just want to log the existence of the issue generally... suppose you send a newsletter and it has a "View in Browser" link.
Suppose the recipient uses that link. The view the mailing and then drill-down, clicking through some more links.
What happens to the stats?
* The web-based view does not count as an "Open". The MUA may or may not report back the "Open" on the original message.
* You will not have any "Click Through" events logged.
This is because the "View in browser" variant of the newsletter is not instrumented with tracking codes (while the MUA variant of the newsletter does have tracking codes).https://lab.civicrm.org/dev/core/-/issues/3616Should tracking pixel be placed at start of email for when Gmail clips messages?2024-02-18T05:03:26ZlarsssandergreenShould tracking pixel be placed at start of email for when Gmail clips messages?With Mosaico templates, it isn't that difficult to reach Gmail's limit of 96-102kB with an email newsletter, after which your email message is clipped and not displayed in full. This is unfortunate in itself, but it also breaks open trac...With Mosaico templates, it isn't that difficult to reach Gmail's limit of 96-102kB with an email newsletter, after which your email message is clipped and not displayed in full. This is unfortunate in itself, but it also breaks open tracking as the tracking pixel is only embedded at the very end of the message and thus isn't fetched unless the reader clicks "View entire message".
Would it be worthwhile to consider putting the tracking pixel at the start of the message to avoid this issue? Obviously, less than ideal for those who don't display images, but I'm not sure how these two might balance out or if there is a clever way to have the best of both worlds.
I'd be happy to provide a patch, but not sure what the best way forward is here.
(Was a former Flexmailer issue, reposted here as I think this is still important.)https://lab.civicrm.org/dev/core/-/issues/3607Feature: Provide an administrative option to make unsubscribe affect all matc...2024-02-17T05:03:32ZlolcodeFeature: Provide an administrative option to make unsubscribe affect all matching email addressesOur organization would like the unsubscribe action to unsubscribe all contacts with the same email from the group. In cases of duplicate contacts they unsubscribe once but one of the duplicates will still get an email and then we get CAS...Our organization would like the unsubscribe action to unsubscribe all contacts with the same email from the group. In cases of duplicate contacts they unsubscribe once but one of the duplicates will still get an email and then we get CASL complaints. Due to CASL we are trying to be cautious by default here.
We suggest an administrative setting for Civimail: "Unsubscribe contact (radio) or unsubscribe email address (radio)"
There was some support for this option expressed in a discussion here: https://chat.civicrm.org/civicrm/pl/sxd4jpwxpbynx8y15994ua7jea
A good point is that this needs to be guarded by token access to stop attackers being able to unsubscribe prominent emails from organisational lists (for example).
A further suggestion was made to make this a user selection option on the unsubscribe page. I feel that this would be a good but separate feature request.https://lab.civicrm.org/dev/core/-/issues/3605"Tracking Click-Throughs" option in mailings generates 404 links when using m...2024-02-17T05:03:31Zjensschuppe"Tracking Click-Throughs" option in mailings generates 404 links when using multi-language with path prefixThis occurred in a Drupal environment with multiple languages and the following configuration:
- Languages in Drupal:
- English (activated)
- German (activated, default)
- Languages in CiviCRM:
- Default language: English
- avai...This occurred in a Drupal environment with multiple languages and the following configuration:
- Languages in Drupal:
- English (activated)
- German (activated, default)
- Languages in CiviCRM:
- Default language: English
- available: German, English
- Inherit CMS language: yes
Conditions under which the erroneous behavior can be reproduced:
- Mailing language: English
- UI language: German
- language prefix in the URL: none or "de"
or:
- Mailing language: German
- UI language: English
- language prefix in the URL: none or "en"
ergo: Using a UI language different from the mailing language and having path prefixes for language detection.
The result is links to the tracking script being prefixed with the language code of the UI language, like so:
https://example.org/de/sites/all/modules/civicrm/extern/url.php?u=123&qid=12345
which is producing a 404 as obviously the path to the script file is not valid due to the language prefix.
Apparently, this code in CRM/Mailing/BAO/TrackableURL.php:93 is where the URL is constructed, but with `userFrameworkResourceURL` being prefixed with the language code:
```
$redirect = $config->userFrameworkResourceURL . "extern/url.php?u=$id";
```https://lab.civicrm.org/dev/core/-/issues/4639New approaches to ACL caching (and smart groups?)2024-02-16T16:22:42ZJonGoldNew approaches to ACL caching (and smart groups?)A lot of work has gone into improving the efficiency and frequency of ACL calculation. Let's attack the performance of writing the results to db?
I have a site where contacts change frequently, and ACLed users need to wait for 100,000 ...A lot of work has gone into improving the efficiency and frequency of ACL calculation. Let's attack the performance of writing the results to db?
I have a site where contacts change frequently, and ACLed users need to wait for 100,000 records to be written each time. Disabling opportunistic flushes only partly mitigates the issue.
Here are some ideas:
* Could we move ACL/smart group caching into `\Civi::cache`? Then we could use Redis etc.
* Instead of truncating and writing all contacts - what if we only `INSERT` and `DELETE` changes from the existing cache?
* When a contact's edited, could we just calculate cache for that contact? For any ACL based on a static group, this should be trivial. Even with smart groups/custom code - are there scenarios where we'd ever need to calculate ACLs for anyone but the changed contact and all their related contacts?https://lab.civicrm.org/dev/core/-/issues/3595Mailing priority2024-02-16T05:03:23ZmfbMailing priorityOrganizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simult...Organizations that send mailings to large lists, alongside more important messages sent to smaller lists, may need a mailing priority feature.
For example, a newsletter might go out to subscribers announcing a new campaign, while simultaneously a press release is sent to press contacts. The latter would have higher priority.
CiviMail would send messages for mailings with a high priority before those with a lower priority.https://lab.civicrm.org/dev/core/-/issues/4631CiviGrant labels are...not good2024-02-15T15:50:15ZJonGoldCiviGrant labels are...not goodThe CiviGrant fields are strange.
The "Amount Requested" field has this description: *Requested grant amount, in original currency (optional)*.
"Total Amount" has this description: *Requested grant amount, in default currency*.
If you...The CiviGrant fields are strange.
The "Amount Requested" field has this description: *Requested grant amount, in original currency (optional)*.
"Total Amount" has this description: *Requested grant amount, in default currency*.
If you're creating a FormBuilder form for grant requests, it makes sense that you'd want a "Requested Amount" field - but you actually need to use "Total Amount". This is unintuitive, and there's nothing in the field labels to suggest anything about original vs. default currency.
I don't think "Total Amount" makes any real sense here as a label - if anything, I'd expect that to be the amount granted (except that's the "Amount Granted" field).
**Proposal**
"Total Amount" gets relabeled as "Total Amount Requested".
"Amount Requested" gets relabeled as "Amount Requested in Original Currency".https://lab.civicrm.org/dev/core/-/issues/3594Core should provide bounce handling (and queueing) for transactional email2024-02-15T05:03:25ZmfbCore should provide bounce handling (and queueing) for transactional emailCurrently CiviCRM core does not provide bounce handling for transactional email. This can result in automated processes sending out a large volume of scheduled reminders, etc. (as well as user-triggered forms sending out subscription co...Currently CiviCRM core does not provide bounce handling for transactional email. This can result in automated processes sending out a large volume of scheduled reminders, etc. (as well as user-triggered forms sending out subscription confirmations, etc.) with only manual bounce handling by humans.
An extension exists to provide bounce handling and other features for transactional emails: https://civicrm.org/extensions/transactional-emails
Arguably, however, this feature should be provided by core itself, so all civi instances have bounce handling for any email message they send.
Assuming we continue the architecture where it is the CiviMail component that provides bounce handling, a solution could be that enabling CiviMail overrides how transactional email is sent, adding the extra bounce processing and other features.
Ideally, transactional email could also be queued by CiviMail, so failures are not fatal and delivery attempts can be retried. Currently, if you are over-quota or your mail delivery provider temporarily suspended your account, then you have real problems when doing some critical thing that sends a transactional email message.https://lab.civicrm.org/dev/core/-/issues/3578Send order when A/B testing2024-02-15T05:03:25ZufundoSend order when A/B testingAs far as I can see current behaviour when processing an A/B test mailing is that CiviMail sends out to all A recipients then all B recipients.
If you have rate limiting on outbound email that is significant relative to your list size, ...As far as I can see current behaviour when processing an A/B test mailing is that CiviMail sends out to all A recipients then all B recipients.
If you have rate limiting on outbound email that is significant relative to your list size, this can result in a significant difference in (average) send time between the A group and the B group, which may then be confounding your A/B test results.
I wonder how hard it would be to implement some kind of interpolation (?) for the sends for an A/B mailing -- something like firing off equal numbers of As and Bs in each mailing batch as they are processed?https://lab.civicrm.org/dev/core/-/issues/5000SearchKit: Allow formatting of the totals in the footer2024-02-14T10:06:39Zsimon.hermannSearchKit: Allow formatting of the totals in the footerSearchKit allows to create a footer row, where the total of a column can be displayed. However, this total does not allow for any formatting or reproduces the formatting of the columns it sums.
It would be great if there would be either ...SearchKit allows to create a footer row, where the total of a column can be displayed. However, this total does not allow for any formatting or reproduces the formatting of the columns it sums.
It would be great if there would be either an option to choose a formatting (decimal separators, thousand separators, currencies if applicable etc) or if the total would reflect the formatting of the column it sums.https://lab.civicrm.org/dev/core/-/issues/4982Form builder - Extension provided entity settings2024-02-14T09:53:44ZseamusleeForm builder - Extension provided entity settingsJMA is in the progress of working on a geographically targeted petition extension
At the moment we have some additional settings that we get a user to set when creating the petition page in Form Builder. These get injected by the alter ...JMA is in the progress of working on a geographically targeted petition extension
At the moment we have some additional settings that we get a user to set when creating the petition page in Form Builder. These get injected by the alter angular hook https://lab.jmaconsulting.biz/extensions/geotargetedpetitions/-/blob/main/geotargetedpetitions.php?ref_type=heads#L76 however this means that even on non petition related form builder forms they will show up.
This is about seeing if there is a better way to achieve this and or see if we can get them put onto the specific settings page for say GeotargetedPetition Entity.
@JoeMurray @Edselopez @colemanwhttps://lab.civicrm.org/dev/core/-/issues/4979Should `die()` be called when a DB Query fails?2024-02-14T09:49:16ZhaystackShould `die()` be called when a DB Query fails?Overview
----------------------------------------
As the title says, when a DB Query fails, `CRM_Utils_File::runSqlQuery()` [calls](https://github.com/civicrm/civicrm-core/blob/fd4913060d8e944a99abe30c22ba670ed0e51a9a/CRM/Utils/File.php#...Overview
----------------------------------------
As the title says, when a DB Query fails, `CRM_Utils_File::runSqlQuery()` [calls](https://github.com/civicrm/civicrm-core/blob/fd4913060d8e944a99abe30c22ba670ed0e51a9a/CRM/Utils/File.php#L327) `die()` and all script execution stop *ahem* dead. Is this intentional?
Reproduction steps
----------------------------------------
1. Install the "Deduper" Extension (v1.7) via the API (sorry @eileen :dove:)
1. See the unrecoverable error "**Cannot execute INSERT INTO civicrm_contact_name_pair_family (id, name_a, name_b, is_most_common_form, is_active) VALUES (65, 'Takahashi', '髙𣘺', 0, 1)**".
Current behaviour
----------------------------------------
There's no way to trap this and continue with a script that installs Extensions via the API.
Edit: apparently there [is a way](https://stackoverflow.com/a/16925225) but this seems a less than sensible way to go about trapping the error.
Expected behaviour
----------------------------------------
A script that installs Extensions via the API should be able to trap the error as an `Exception` with `try/catch` and act accordingly.https://lab.civicrm.org/dev/core/-/issues/3574Password for mail accounts should not be stored in plain text2024-02-14T05:03:29ZscardiniusPassword for mail accounts should not be stored in plain textMySQL field civicrm_mail_settings.password contains password in plain text. Should it? I think not
This issue could by linked with https://lab.civicrm.org/dev/core/issues/236MySQL field civicrm_mail_settings.password contains password in plain text. Should it? I think not
This issue could by linked with https://lab.civicrm.org/dev/core/issues/236https://lab.civicrm.org/dev/core/-/issues/3573Unsubscribe fails but appears to have worked for resending to previous mailin...2024-02-14T05:03:29ZRichUnsubscribe fails but appears to have worked for resending to previous mailing to search resultsRecreate with 5.2.2 (on Dupal 7)
1. create mailing group A
2. create other group B
4. add contact into group: B
5. do mailing to contacts in group B, using group A as unsubscribe. (i.e. Find » In Group B » Action: create CiviMail mailin...Recreate with 5.2.2 (on Dupal 7)
1. create mailing group A
2. create other group B
4. add contact into group: B
5. do mailing to contacts in group B, using group A as unsubscribe. (i.e. Find » In Group B » Action: create CiviMail mailing)
7. Create and send mailing: sent to recipients of mlg1
8. click unsubscribe.
## Expect
to see confirm form saying "you will be unsubscribed from A" because A was the base group used for the original mailing. (When sending to previous mailing recipients there is no option to select an unsubscribe group)
## Actual
User sees "yes you've been unsubscribed from the mailing group/list" [code ref](https://lab.civicrm.org/dev/core/blob/master/CRM/Mailing/Form/Unsubscribe.php#L71) but nothing has actually happened!https://lab.civicrm.org/dev/core/-/issues/3568Improve process for creating plain text version of mailing2024-02-13T05:03:24ZMichael McAndrewImprove process for creating plain text version of mailing## Overview
Convert any html contained in tokens to plain text for plain text emails
## Before
HTML in tokens that contained HTML was being outputted to plain text versions of emails.
## After
HTML in tokens is converted to plain te...## Overview
Convert any html contained in tokens to plain text for plain text emails
## Before
HTML in tokens that contained HTML was being outputted to plain text versions of emails.
## After
HTML in tokens is converted to plain text.
## Technical Details
This issue is NOT about generating plain text and HTML versions of tokens. It is about converting an entire email from HTML to plain text when no plain text version is given. This conversion currently proceeds as follows:
1. Check to see if a plain text version has been submitted. If not
2. Convert the HTML to plain text
3. Do token substitution
The problem is that if the token contains html, that HTML persists in the email. The proposed solution is to do the conversion from HTML to plain text after token substitution, i.e.
1. Check to see if a plain text version has been submitted. If not
2. Convert the HTML to plain text
3. Do token substitution
This patch is split between [PR 12061 on core](https://github.com/civicrm/civicrm-core/pull/12061) and [PR 20 on flexmailer](https://github.com/civicrm/org.civicrm.flexmailer/pull/20).
The flexmailer patch will continue to report a single failure until the core patch is merged, at which point we can hopefully run the tests against flexmailer again and merge this patch.
@totten suggested defining tokens with TokenProcessor, whose tokens auto-convert from HTML to plain text and visa versa. However, I am not sure this is going to work for the following reason. When you convert from HTML to plain text (at least with the HTML to plain text converter that we are using) the structure of the text is altered and links appear at the bottom of the converted text. Therefore, if you run multiple conversions, you will get multiple lists of links scattered throughout the email. Best to do html to plain text conversion at the end of the process.
All this is not to say that one shouldn't define plain text and HTML versions of tokens. I think it is fine to do that if you want to. I just think that automatic conversion of HTML to plain text should happen one time only, after the entire email has been assembled.
Note that [PR 12061 on core](https://github.com/civicrm/civicrm-core/pull/12061) removes the check for nofollow in the email html output of the test CRM_Mailing_BaseMailingSystemTest::testHtmlWithOpenAndUrlTracking. The intent of https://issues.civicrm.org/jira/browse/CRM-21768 is not affected. nofollow still appears in the public view (e.g. http://example.org/civicrm/mailing/view) but it is not tested.
I would have added it to a test of civicrm/mailing/view but I could not find any tests there.
This issue was originally discussed here: https://issues.civicrm.org/jira/browse/CRM-21197 and a PR submitted here: https://github.com/civicrm/civicrm-core/pull/10998. But the PR was rejected because it caused issues with Flexmailer.
Also, on the seperate issue of click tracking plain text emails. @totten - you said [here](https://github.com/civicrm/civicrm-core/pull/10998#discussion_r158175327) that you thought it was a significant policy change. It isn't as significant as you think. *Links that are tracked in HTML continue to be tracked in plain text*. Links that are not tracked in HTML (because they were not in an `<a>` tag) are not tracked in plain text either. i.e. there is no significant loss of functionality.https://lab.civicrm.org/dev/core/-/issues/3563Include optional unsubscribe group field on regular mailings2024-02-13T05:03:23ZlarsssandergreenInclude optional unsubscribe group field on regular mailingsWhen you create a mailing from search results, you are required to add an unsubscribe group. When you create a regular mailing and add groups to the recipients, you can't specify the unsubscribe group. Sometimes, it would be helpful to b...When you create a mailing from search results, you are required to add an unsubscribe group. When you create a regular mailing and add groups to the recipients, you can't specify the unsubscribe group. Sometimes, it would be helpful to be able to specify an unsubscribe group that is different than the group the contact is a member of. For instance, if you send a mailing to a few groups based on your main mailing list, perhaps a set of smart group that are subsets of the main list, you may want contacts to unsubscribe from your main list instead of the smart group.
Proposal: Add an optional unsubscribe group to all mailings. If it is blank, unsubscribes work as they do now. If it is populated, all recipients are unsubscribed from the unsubscribe group (exactly as this works for a mailing based on a search now).
EDIT: On @JoeMurray's suggestion, contacts would be unsubscribed from both the original group that are recipients of the mailing and the specified unsubscribe group.
I can give this a go if there is support for the concept.https://lab.civicrm.org/dev/core/-/issues/3539When communcating with external network services, detect transient failures a...2024-02-12T13:57:20ZmfbWhen communcating with external network services, detect transient failures and queue for retryFor improved resiliency, CiviCRM should detect transient failures when communicating with external network services and queue for retry. e.g.
* Transactional email: Queue for sending rather than sending during the request process?
* Bul...For improved resiliency, CiviCRM should detect transient failures when communicating with external network services and queue for retry. e.g.
* Transactional email: Queue for sending rather than sending during the request process?
* Bulk mail: Catch transient failures and ensure that they are retried. PRs: https://github.com/civicrm/civicrm-core/pull/11838 https://github.com/civicrm/civicrm-core/pull/11840 (and there are additional failure scenarios possible)
* ...https://lab.civicrm.org/dev/core/-/issues/3541[Meta] Does CiviCRM have a single defined application HTTP entry point which ...2024-02-11T05:03:23Zxurizaemon[Meta] Does CiviCRM have a single defined application HTTP entry point which routes all requests?Various issues arise when we have multiple entry points to the application (eg scripts in `extern/`, direct call scripts like KCFinder `integration/civicrm.php`, IPN handlers).
These issues stem from duplicated / non-identical approache...Various issues arise when we have multiple entry points to the application (eg scripts in `extern/`, direct call scripts like KCFinder `integration/civicrm.php`, IPN handlers).
These issues stem from duplicated / non-identical approaches to bootstrapping / path discovery / session management.
The end goal here is to eliminate alternate entry points and ensure any callback uses CiviCRM's core routing.
As much as possible auth/bootstrap/url/path resolution should be handled in one place only (each) for maintainability.
## Background issues / examples:
* [CRM-17212](https://issues.civicrm.org/jira/browse/CRM-17212)
* [CRM-13249](https://issues.civicrm.org/jira/browse/CRM-13249)
* [CRM-12246](https://issues.civicrm.org/jira/browse/CRM-12246)
* cloud-native#6
* cloud-native#7
## Targets
* [`extern/authorizeIPN.php`](https://github.com/civicrm/civicrm-core/blob/master/extern/authorizeIPN.php), [`ipn.php`](https://github.com/civicrm/civicrm-core/blob/master/extern/ipn.php), [`pxIPN.php`](https://github.com/civicrm/civicrm-core/blob/master/extern/pxIPN.php) - IPN handlers → use `civicrm/ipn/%` entry point
* [`extern/cxn.php`](https://github.com/civicrm/civicrm-core/blob/master/extern/cxn.php)
* [`extern/open.php`](https://github.com/civicrm/civicrm-core/blob/master/extern/open.php) - endpoint to track mail opens via image URL
* [`extern/rest.php`](https://github.com/civicrm/civicrm-core/blob/master/extern/rest.php) - API endpoint → `civicrm/api/v3`, `civicrm/api/v4` etc
* [`extern/soap.php`](https://github.com/civicrm/civicrm-core/blob/master/extern/soap.php) - SOAP interface, still used by CiviSMTP? → `civicrm/soap`
* [`extern/url.php`](https://github.com/civicrm/civicrm-core/blob/master/extern/url.php) - Mailing redirect → `civicrm/redirect` ?
* [`packages/kcfinder/integration/civicrm.php`](https://github.com/civicrm/civicrm-packages/blob/master/kcfinder/integration/civicrm.php) - KCFinder integration for CiviCRM - IDK
IPN handlers should be tackled as those core payment processors are moved to extensions.https://lab.civicrm.org/dev/core/-/issues/3537Does CiviCRM handle UNIX filesystem functionality like symlinks?2024-02-10T05:03:27ZxurizaemonDoes CiviCRM handle UNIX filesystem functionality like symlinks?Some cloud hosting providers dynamically assign storage (disk) and may use symlinks to associate dynamic storage with predictable filesystem locations. CiviCRM seems to be confused by the presence of symlinks in the CiviCRM install path ...Some cloud hosting providers dynamically assign storage (disk) and may use symlinks to associate dynamic storage with predictable filesystem locations. CiviCRM seems to be confused by the presence of symlinks in the CiviCRM install path (and possibly resource paths).
This manifests as "weird" resource URLs or inability to resolve filesystem paths.
> This doesn't work in all configurations. For example, if your directory structure uses symlinks, it may not resolve correctly.
For examples, https://civicrm.stackexchange.com/search?q=symlink eg
* https://civicrm.stackexchange.com/questions/19495/set-civicrm-files/19498#19498
* https://civicrm.stackexchange.com/questions/20064/help-needed-in-setting-up-rest-api-in-civi-crm/20076#20076
* https://civicrm.stackexchange.com/questions/20265/mailchimp-plugin-creating-webhook-with-invalid-url
* https://civicrm.stackexchange.com/questions/21925/incorrect-resource-url-warning-but-menu-arrows-are-present
For WP specifically, https://github.com/civicrm/civicrm-core/pull/11086