CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-09-25T19:36:31Zhttps://lab.civicrm.org/dev/core/-/issues/3682Feature request - `{domain.logo}`2023-09-25T19:36:31ZeileenFeature request - `{domain.logo}`Tim & I discussed the other day adding the `{domain.logo}` token. As always when this comes up we started with 'it would be very easy to do this minimal version' and by the end of the discussion the scope was un-manageable as unsponsored...Tim & I discussed the other day adding the `{domain.logo}` token. As always when this comes up we started with 'it would be very easy to do this minimal version' and by the end of the discussion the scope was un-manageable as unsponsored work.
However, I think the thing we CAN do is document it....
**The easy way**
`{domain.logo}`
refers to the image (if any) in the image_url field for the domain contact. That's it.
Probably we would need a version with `<img>` tags & one without.
**But then ... sizing & ^^ is too basic/ limited**
So next most complex is to use a setting. Once we start down this path we wind up having to develop a UI, handling sizing and it seems we might as well skip to....
**Mailing Component tokens**
In this version we would add a new component type to `civicrm_mailing_component` and rows of this type would be available as tokens using `{domain.{civicrm_mailing_component.name}}`. Placeholders ones for '{domain.logo}' and '{domain.header}' would be added. The existing UI could be used.
**And still spiralling**
- from there we got into trying to make it possible to nest the above, to use regions, to do the sort of thing the `entity_messages` token attempted to whereby we could store entity-related snippets (the idea was both to be able to insert a contribution specific block and to store entity-specific text to re-use - e.g to re-send an invoice with the original form-base message text)https://lab.civicrm.org/dev/core/-/issues/3648SMS limit is hardcoded at 460 but should be changeable somewhere2024-01-29T09:44:22ZjasonobrownSMS limit is hardcoded at 460 but should be changeable somewhereTwillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the sys...Twillio (and probably other sms providers) allows SMS messages up to 1600 characters, but Civi has the limit hardcoded at 460. I've verified that by simply going to CRM/SMS/Provider.php and changing MAX_SMS_CHAR from 460 to 1600 the system now accepts (and sends) messages up to 1600 characters. It seems logical that the character limit should able to be adjusted and stored somewhere rather than set as a static value in the code. We are currently using the larger limit, and would really like to see this implemented asap so that we can stop manually updating the code each time we update civi.https://lab.civicrm.org/dev/core/-/issues/3645Eventually google will require OAUTH for bounce processing and may require it...2022-12-08T20:55:07ZDaveDEventually google will require OAUTH for bounce processing and may require it for outbound SMTP through gmail serversAs noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue supp...As noted on [stackexchange](https://civicrm.stackexchange.com/questions/34142/sending-mail-via-gmail-google-announce-username-password-authentication-will-b), Google sent out an email outlining plans to require OAUTH and discontinue support for just username/password.
The way I read it, this is mostly applying to POP/IMAP clients for reading mail, and they are *hinting* that someday they might require it for outbound SMTP, i.e. at Administer - System Settings - Outbound Mail you choose SMTP and have it set to use gmail servers. The relevant quote is
> No change is required for scanners or other devices using simple mail transfer protocol (SMTP) or LSAs to send emails. If you replace your device, look for one that sends email using OAuth.
(LSA means less secure apps, which is google-speak for username/password)
They are targeting June 2020 - Feb 2021 for IMAP etc, but as noted above they have not specifically set a date for changes regarding sending outbound emails.
It occurs to me though that bounce processing and the email processor use POP/IMAP. For the email processor you can work around it by just forwarding it to a non-gmail address and polling that, but bounce processing might be harder depending on setup.https://lab.civicrm.org/dev/core/-/issues/3613System Workflow Messages - Improve localization experience2024-01-14T14:55:57ZtottenSystem Workflow Messages - Improve localization experience## Background
Several CiviCRM components - such as CiviContribute, CiviCase, and CiviEvent - send automated email notifications. Many organizations find it useful to customize/tune these emails, and the emails may be administered in the...## Background
Several CiviCRM components - such as CiviContribute, CiviCase, and CiviEvent - send automated email notifications. Many organizations find it useful to customize/tune these emails, and the emails may be administered in the web UI ("Administer: Message Templates: System Workflow Messages").
In principle, this is a powerful tool in which an administrator may use Smarty templating to produce highly tuned messages -- and it requires only the ability to understand template notations (e.g. `{$contact.first_name}` or `{$event.start_date}`). However, in practice, this is often difficult -- real-world templates are long and complex, and they may be used with data of varying quality/character, and the workflow to test a template can be tedious.
## Goals
This epic (*overarching issue*) aims to improve the experience of maintaining system workflow templates. We are specifically funded to work on it from the following perspective: You have an organization which maintains multiple templates for use in multiple countries/locales -- periodically, as campaigns/operations are revised, you may need to engage with a few people to update+retest a few templates. We have three main themes to improve upon:
* __Drafting/Workflow__: Currently, Civi has exactly one variant/revision of a template, and it is always live. There is no way to develop a draft (new revisions) or to keep a record of past revisions.
* __~~Translation~~ Localization__: For any given message, you may need to compose different versions for different locales.
* __Previews/Samples__: Many templates are sensitive to fine-grained details. This is true in multiple ways, e.g. (1) the template adjusts based on *available data* ("Do we have a mailing address for this person? Do we have a full-name, or just an email? Is this one-off contribution or a recurring contribution?"), and (2) the template includes *precise codes* that must be well-formed (e.g. matching tags, matching quotes, using an email-friendly dialect of HTML/CSS).
## Related work
* https://lab.civicrm.org/extensions/msgtpltools
* https://lab.civicrm.org/community/feature-request/-/issues/26
* https://github.com/eileenmcnaughton/civi-data-translate
* https://lab.civicrm.org/extensions/l10nx
## Subtopics / Brainstorming
(Some of the descriptions here are terse - but they may get more explanation in the wireframe.)
* __Sample data presentation__: While composing a message, how do you show a preview based on sample data? Some ideas:
* __Edit-Preview Split-Panes__
* __Pro__: very amenable to live-update
* __Con__: needs wide screen (prob dialog). agreeable with O(10) sample records - but disagreeable with O(1000) records
* __Edit-Preview Accordion__
* __Pro__: fairly amenable to live-update. full width.
* __Con__: agreeable with O(10) sample records - but disagreeable with O(1000) records
* __Explore-Preview Panes in dialog__
* __Pro__: agreeable with O(1000) records
* __Con__: requires more navigating.
* __Comment__: live-update might be possible with separate popup - but more difficult
* __Comment__: might mitigate navigation work with a hotkey
* __Sample data source__: When viewing a preview based on some example record(s), where does the example come from?
* __Basic/prepackaged__: Define some JSON files, which are linked to the workflows.
* __Comment__: Requires metadata to say, "workflows $a+$b can use sample data $c+$d"
* __Pro__: Some templates have very nuanced purposes and may have special vars. This allows you to mock data that's attuned to edge-cases, and you can use these samples in any environment (dev-site/public-demo-site/live-site). No mixing-up real records and sample records. Can simulate complex data (e.g. "tpl for use-case with 2 contribution records")
* __Con__: More upfront work in making sample data. User cannot test against real data.
* __Comment__: Should have unit-test to ensure sample-data-structure stays in sync with real-data-structure
* __Data search__: Search for a live record (eg `Contribution` or `Membership`) to use as a sample.
* __Comment__: Requires metadata to say, "workflows $a+$b need data from entity $c"
* __Pro__: You can use real data, which can be subjectively attuned.
* __Con__: You have to find real data that's *useful*. May be awkward if you're trying to test edge-cases, unless you're willing to add mock/synthetic records to the DB. All mail-merge data must be selectable from one record (eg "Contribution #123"; eg no other inputs besides DB content).
* __Variants__: The UI for this can take on a few variations, eg
* In "Edit Msg Tpl", show a searchable tree. Filter by contact name and select the target record.
* In "Edit Msg Tpl", ask the user to enter a numeric ID.
* In "View Contact", add links/actions for generating previews.
* __Flagged records__: As an adminsitrator, find real records and save them in a list (e.g. "Record #$x is a sample for use in workflows $a, $b")
* __Comment__: Requires somewhere to store the list (e.g. custom-field or new-field or new-table)
* __Pro__: You can use real data, which can be subjectively attuned. The list of samples is curated/abbreviated.
* __Con__: The individual composing the template may not know how to flag/unflag sample records. Requires configuration (and extant samples) before you can use email preview.
* __Grain of ~~translation~~ localization__
* __Record level__: For each `(locale,workflow)`, create another `civicrm_msg_template` record.
* __Field level__: Each `(workflow)` only has one `civicrm_msg_template` record. Attached to that record are multiple translations (ex: `field=subject,locale=fr_FR,text="Bonjour"`).
* __String level__: Each `(workflow)` only has one `civicrm_msg_template` record, and that record has one `body_text`. However, the `body_text` uses translatable strings (ex: 'body_text={cts id=greeting}Greetings,{/cts}`)
* Trade-off
| Record-level | Field-level | String-level |
| -- | -- | -- |
| Each translation goes through a separate, well-delineated drafting workflow. | Translation workflow is batched (per template). All translations must be ready to activate together. | Orthogonal workflows for templates vs strings. |
| Message structure (paras/sections) is localized. | Message structure (paras/sections) is localized. | Message structure is uniform across locales. (Realistic? What abouts `<table>` LTR/RTL? |
| Cannot re-use translated snippets. | Cannot re-use translated snippets. | Re-use translated snippets. |
## Wireframes
There are a few different ways to approach this. To get a sense for the usability and work, we can post a few alternative wireframes. (*This section may be updated as more are added.*)
* __(0) Status quo__: [_0__Sys_Wf_Msgs](/uploads/ad96003d71133d298f95ce79902a7583/_0__Sys_Wf_Msgs.png), [_0__Edit_Msg_Tpl](/uploads/81bc234651182b65993bc28c8052b68b/_0__Edit_Msg_Tpl.png)
* __(1) Record-level translations w/previews and drafting workflow__: [_1Rec__Sys_Wf_Msgs](/uploads/532b842406c17c31be5ae9195ceb6aa8/_1Rec__Sys_Wf_Msgs.png), [_1Rec__Edit_Msg_Tpl](/uploads/13a1f717e829e3de480750c4893969a7/_1Rec__Edit_Msg_Tpl.png) (*Note: The blue-arcs indicate a couple variations on how to include the preview/sample functionality. This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(2) Field-level translations__: [_2Fld__Sys_Wf_Msgs](/uploads/7e1f46e722d8afc706a92c56e295c55e/_2Fld__Sys_Wf_Msgs.png), [_2Fld__Edit_Msg_Tpl](/uploads/4bc932b7bb3f304853b0c107b98d41b5/_2Fld__Edit_Msg_Tpl.png) (*Note: This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(3) String-level translations__: [_3Subfld__Sys_Wf_Msgs](/uploads/f52ffbb764917954790fbbd3ed807acf/_3Subfld__Sys_Wf_Msgs.png), [_3Subfld__Edit_Msg_Tpl](/uploads/b20bc83d302d55a485d975aff6ca70e1/_3Subfld__Edit_Msg_Tpl.png) (*Note: This tracks close to status-quo, and incremental changes are highlighted with yellow-dots.*)
* __(4) Record-level translations w/browse-edit tree + accordion-preview__: [_4Tree__Browse-Edit](/uploads/8cc47238a9d597f80950c1d98c0ea43a/_4Tree__Browse-Edit.png)
* __(5) Field-level translations w/browse-edit tree + accordion-preview__ [_5Tree__Browse-Edit](/uploads/aab231199d0b4217a2f44f9ad45aecb1/_5Tree__Browse-Edit.png)
* __(6) Field-level translations w/matrix, preview-dialog, and drafting workflow__: [_6Mat__Sys_Wf_Msgs](/uploads/3a32390f36f50d8e158e1155dc586969/_6Mat__Sys_Wf_Msgs.png), [_6Mat__Edit_Msg_Tpl](/uploads/353f97846fd9d2b1e037ec92e60101fc/_6Mat__Edit_Msg_Tpl.png) (*Note: This assumes that there is a separate string-storage service like [civi-data-translate](https://github.com/eileenmcnaughton/civi-data-translate), and it assumes that the `String` record has some workflow property like like `is_draft` or `revision`. So each string might have the properties `entity`,`entity_id`,`field`,`locale`,`text`,`is_draft`.*)
* __(7) As with 6, but with various refinements__: [_7Mat__Sys_Wf_Msgs](/uploads/73d37acab4307054b4fe4f3291ae7dcd/_7Mat__Sys_Wf_Msgs.png),
[_7Mat__Edit_Msg_Tpl](/uploads/4807652bd20bfb5bf0912bf24f05d444/_7Mat__Edit_Msg_Tpl.png)https://lab.civicrm.org/dev/core/-/issues/3587Unable to load the recipients group when sending mailing with Civimail2023-10-05T15:41:24ZmarzastUnable to load the recipients group when sending mailing with CivimailI am running CiviCRM 5.17.4 on Wordpress 5.2.3. The problem I am facing is that I am unable to send mailing to mailing lists (defined as groups) via CiviMail.
When I try to enter the group in the Recipients box I get Loading Failed mess...I am running CiviCRM 5.17.4 on Wordpress 5.2.3. The problem I am facing is that I am unable to send mailing to mailing lists (defined as groups) via CiviMail.
When I try to enter the group in the Recipients box I get Loading Failed message and none of the available groups is loaded (see image below). The problem appears for existing and new mailings.
Interestingly when I re-use the existing mailing I can still send it to a previously used group (though I cannot change the recipient lists). The groups load normally in manage group page.
![aE8nz](/uploads/dca6a15c237cbb7612df433a31b769d5/aE8nz.png)
*Issue already reported on CiviCRM Stack Exchange: https://civicrm.stackexchange.com/questions/33070/unable-to-load-the-recipients-group-when-sending-mailing-with-civimail*https://lab.civicrm.org/dev/core/-/issues/3586Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process ...2023-09-12T02:13:33Zjustinfreeman (Agileware)Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process any emails if a single email is sent to the bounce mailbox with an invalid returnPathCiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service ema...CiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service email" causing all other emails to remain unprocessed. CiviCRM mail reports are therefore incorrect and ultimately the CiviCRM Fetch Bounces scheduled job fails to perform the job it was designed to do.
Example invalid returnPaths as observed in production:
```
x2108-arfsfufwyfgkvni2x7ww5wvac3eifc2wx3vr1fcwbbysieqppis4ytnyip09xcohck9s1vrm77nbg4xh431tthdsmtmfvfpa@rspf.mxthunderstruck.net
```
```
SRS0=3Ypr33=QM=crm.nothappyjohn.org.au=bounce+b.25211.6039846.eb7e7e8ba282f9d0@airquotes.com.au
```
This problem has been highlighted by the introduction of "Scheduled Job Failures" feature which raises the visibility of this type of problem.
Bounce mailboxes have been observed with 10k to 50k unprocessed emails, depending on the time when the "Denial of service email" was received.
Agileware Ref: CIVICRM-1970https://lab.civicrm.org/dev/core/-/issues/3584Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially w...2023-11-09T13:25:31Zjustinfreeman (Agileware)Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and mailing will not sendCiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to ...CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to complete command, DNS not available or timed out 451 Domain of sender address does not resolve_
When the CiviCRM Mailing Job executes again, the process repeats, the same invalid email addresses are used, the same errors are returned and the mailing again will not send.
This process will repeat until someone intervenes, cancelling the mailing, locating and removing the contacts which have the invalid email addresses.
The CiviCRM Mailing Job does not currently check the actual SMTP error code and just treats all SMTP errors as "SMTP Connection Errors". Counting up to 5 and then aborting the job when the threshold is reached.
I think it's fair to call this a "denial of service" because a bad actor could sign up a bunch of fake Contacts to a CiviCRM mailing list with either invalid email domains or valid email domains which are then rendered invalid - and thus cause that mailing list to cease processing.
**Possible solutions**
Change the CiviCRM Mailing Job to check the return SMTP error code and only increment the count for valid "SMTP Connection Errors".
Change the CiviCRM Mailing Job to continue with the mailing, regardless of the 5 errors encountered, which will also skip sending the emails to those Contacts with an invalid email domain. As shown in the patch below.
```
Index: CRM/Mailing/BAO/MailingJob.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/CRM/Mailing/BAO/MailingJob.php b/CRM/Mailing/BAO/MailingJob.php
--- a/CRM/Mailing/BAO/MailingJob.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/CRM/Mailing/BAO/MailingJob.php (date 1620794367592)
@@ -672,20 +672,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryGroup = TRUE;
+ CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $this->writeToDB(
- $deliveredParams,
- $targetParams,
- $mailing,
- $job_date
- );
-
- CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- CRM_Utils_System::civiExit();
}
// Register the bounce event.
Index: ext/flexmailer/src/Listener/DefaultSender.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/ext/flexmailer/src/Listener/DefaultSender.php b/ext/flexmailer/src/Listener/DefaultSender.php
--- a/ext/flexmailer/src/Listener/DefaultSender.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/ext/flexmailer/src/Listener/DefaultSender.php (date 1620794390628)
@@ -76,15 +76,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryBatch = TRUE;
+ \CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $job->writeToDB($deliveredParams, $targetParams, $mailing, $job_date);
-
- \CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- \CRM_Utils_System::civiExit();
}
else {
$this->recordBounce($job, $task, $result->getMessage());
```
Agileware Ref: CIVICRM-1728https://lab.civicrm.org/dev/core/-/issues/3567CiviMail: Provide transparency about (non)recipient details2023-09-05T10:28:43ZtottenCiviMail: Provide transparency about (non)recipient detailsTwo stories which are not-uncommon:
## Story 1
Someone starts using CiviMail. They spend a bunch of time arranging for the list of contacts (e.g. migrating from a previous system) and then go to send a mailing. They start composing a m...Two stories which are not-uncommon:
## Story 1
Someone starts using CiviMail. They spend a bunch of time arranging for the list of contacts (e.g. migrating from a previous system) and then go to send a mailing. They start composing a message and realize that the number of recipients is way off. What's going on? They start searching Google, Stackexchange, etc trying to understand why. Eventually, they learn that there are a half-dozen rules about who gets included or excluded from a mailing. (*Does the contact have an email address? Are they deceased? Have they opted-out of the specific mailing list? Have they opted-out of email generally? Have they had bounces? Does an extension - like GDPR - add more constraints?*) But they haven't solved the issue yet: now they need to figure out which criteria are affecting them, to what extent, and how to adapt.
And of course people have this problem - the "New Mailing" UX only presents the affirmative list of active recipients. It doesn't give any hint or detail about the omitted recipients.
![Screen_Shot_2019-09-03_at_12.58.40_PM](/uploads/73d40846bf379562ed119a317163b4e9/Screen_Shot_2019-09-03_at_12.58.40_PM.png)
## Story 2
A new user composes an email blast and discovers this incredible drop-down button with a list of tokens. They start throwing in tokens like there's no tomorrow. The blast goes out, and recipients complain: "Why are you sending me these weird messages with skipped words?"
And, again, of course people have this problem - the "New Mailing" UX lets you choose tokens without knowing if your recipient-data is up to the job.
## Discussion
These two stories are thematically similar: in both cases, we have a long list of potential recipients, and we have obscure/hidden criteria which determines whether the mailing goes out successfully to the target audience.
In both cases, there needs to be more transparency - adding UI-elements/UX-mechanisms to communicate what's going on. For example, the data-table in the pop-up might be updated with more columns/icons/color-coding.
I've logged the issue because I think, at some point, there should be discussion around design/resourcing. I don't want to prejudge that by saying the UX mechanisms to resolve these two issues must or must-not be the same - they might be totally different, or they might be one data-table. Either way, I think it's good to have a record of the issue.
## Related Links
* https://chat.civicrm.org/civicrm/pl/nn5rjzwiz3fc78pgojmhubx89a
* https://civicrm.org/blog/simonparkervitiligosocietyorguk/solution-all-recipients-not-showing-up-for-sending-mass-email
* https://civicrm.stackexchange.com/questions/5086/civimail-does-not-send-to-a-whole-group
* https://civicrm.stackexchange.com/questions/6346/after-import-to-group-individuals-in-that-group-not-showing-up-as-recipients-to
* https://civicrm.stackexchange.com/questions/26320/contacts-visible-in-mailing-list-group-not-on-hold-or-otherwise-dnd-but-skippe
* https://civicrm.stackexchange.com/questions/18199/in-mailing-all-recipients-are-not-showing/18201#18201
* https://civicrm.org/extensions/zombie-checkhttps://lab.civicrm.org/dev/core/-/issues/3559CiviCRM bounce processing, processed emails are never deleted from the mailbo...2023-09-08T22:34:57Zjustinfreeman (Agileware)CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little valueCiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox,...CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox, or
1. A new Scheduled Job process should be implemented which deletes all processed email which is older than 30 days
Relates to https://lab.civicrm.org/dev/mail/-/issues/108
Agileware Ref: CIVICRM-1977https://lab.civicrm.org/dev/core/-/issues/3556Deleting a sent mailing leaves activities with invalid links to a mailing rep...2023-09-05T10:00:34ZlolcodeDeleting a sent mailing leaves activities with invalid links to a mailing report that doesn't workAssuming: Civicrm uses the default setting "Enable CiviMail to create activities on delivery"
If a mailing is created, sent and then deleted then the recipients of that mailing will have activities that link to a mailing report using an...Assuming: Civicrm uses the default setting "Enable CiviMail to create activities on delivery"
If a mailing is created, sent and then deleted then the recipients of that mailing will have activities that link to a mailing report using an invalid id. If the link is followed the error will be:
> Sorry, due to an error, we are unable to fulfill your request at the moment. You may want to contact your administrator or service provider with more details about what action you were performing when this occurred.
> Expected one Mailing but found 0
If clicked from the popup activity view it will just be a network error.
The body of the activity will also be blank after the mailing is deleted.
I am not convinced that allowing sent mailings to be deleted at all is a good design.
Some kind of an archive function might be better.
In the short term however the link should probably be removed from these activities.https://lab.civicrm.org/dev/core/-/issues/3552Set time limit for bounce types with hold_threshold > 12023-09-05T10:00:02ZMichael McAndrewSet time limit for bounce types with hold_threshold > 1Bounce types can be split into those with a hold_threshold = 1 where a single bounce should cause this contact to be put on hold (e.g. mailbox non existent), and those where more than one bounce is required (mailbox full).
We don't curr...Bounce types can be split into those with a hold_threshold = 1 where a single bounce should cause this contact to be put on hold (e.g. mailbox non existent), and those where more than one bounce is required (mailbox full).
We don't currently set a time limit on those where more than one bounce is required, which is an oversight. For example, the bounce threshold for Mailbox full is 3. This means that as soon as we have received more than 3 mailbox full messages for a contact, their email will be put on hold. This might make sense if it was 3 in the last year, but doesn't really make sense if it was three in the last 10 years.
Similar arguments can be made for other bounce types, like Away, which is set to 30. If I send a fair amount of email to people that are out of the office quite a lot, over time a significant chunk of these will send me > 30 out of the office messages.
Hence we should only consider messages received in a certain time period. I am going to suggest one year. I would be up for making it user configurable (number of days) if people feel strongly about it.
Let me know what you think.https://lab.civicrm.org/dev/core/-/issues/3539When communcating with external network services, detect transient failures a...2024-02-12T13:57:20ZmfbWhen communcating with external network services, detect transient failures and queue for retryFor improved resiliency, CiviCRM should detect transient failures when communicating with external network services and queue for retry. e.g.
* Transactional email: Queue for sending rather than sending during the request process?
* Bul...For improved resiliency, CiviCRM should detect transient failures when communicating with external network services and queue for retry. e.g.
* Transactional email: Queue for sending rather than sending during the request process?
* Bulk mail: Catch transient failures and ensure that they are retried. PRs: https://github.com/civicrm/civicrm-core/pull/11838 https://github.com/civicrm/civicrm-core/pull/11840 (and there are additional failure scenarios possible)
* ...https://lab.civicrm.org/dev/core/-/issues/3514Define interfaces for interacting with newly cleaned up import code2023-04-03T02:29:57ZeileenDefine interfaces for interacting with newly cleaned up import codeThis is a bit of a placeholder but I think it will be necessary for integrations to
- ~~add new user_job_types - currently this is a hard-coded list and the Parser class is hard-coded into it - we probably need to solve this to get the ...This is a bit of a placeholder but I think it will be necessary for integrations to
- ~~add new user_job_types - currently this is a hard-coded list and the Parser class is hard-coded into it - we probably need to solve this to get the csvimporter working~~ these are declared by any class that `implements UserJobInterface`
- alter the metadate - eg inject 'full_name' into the available fields to map to. UPDATE - am wondering if being able to add pseudofields for the api is enough - would work for full_name
- intervene once `getMappedRow` has run
- ideally we would extract out the `lookup` code where matching contacts are found or matching entities and add some interaction there - assuming the hook in `findDuplicates` is not enough. UPDATE - I think intervening at the end of `getMappedRow` will work
- be able to alter the validation of the mapping. I know a form rule could be added by hook but I think a specific hook is cleaner as the hook would also need to alter existing ruleshttps://lab.civicrm.org/dev/core/-/issues/3501Featrure Request: tracking engagement on contact records2022-09-27T23:23:59ZhescoFeatrure Request: tracking engagement on contact recordsOverview
----------------------------------------
_Please describe your improvement in detail._
I have been watching a seminar program entitled "The Web 3.0 Future", offered by the ASK Method.
They have been reviewing the implication...Overview
----------------------------------------
_Please describe your improvement in detail._
I have been watching a seminar program entitled "The Web 3.0 Future", offered by the ASK Method.
They have been reviewing the implications of changes to iOS 14 and 15. iPhones account for better than 50% of ne cell phone purchases. Their recent upgrades position privacy as a unique selling proposition for Apple. This has the effect of disrupting the business model of meta/facebook/instagram/etc and google/youtube/etc (responsible for roughly 85% of available ad inventory between them). The ripple effects impact all the businesses and organizations whose strategies depend on those platforms for lead generation.
In addition, the iOS 15 upgrade, along with corporate strategies within Apple, have rendered email open-rates as bogus and undependable. The only reliable data we have on email list subscription engagement levels it seems are click-throughs.
For a number of reasons discussed in today's segment if this seminar, it is becoming extremely important, in terms of email deliverability to be able to track by contact and to segment our lists by engagement history.
CiviMail does an excellent job of reporting on performance metrics for a mailing. What it fails to do it seems is to expose that data on individual contact records so we can meaningfully measure engagement by contact and use those measurements to segment our lists in appropriate ways for purposes both of list hygiene, and also for protecting sender reputation with the large email providers by managing email frequency by engagement history.
Example use-case
----------------------------------------
1. Click on **Contacts -> New Individual**.
1. Enter **First Name** and **Last Name** and click **Save**.
Send a CiviMail.
Current behaviour
----------------------------------------
_What is currently possible? What limit ?_
CiviMail does an excellent job of reporting on performance metrics for a mailing. What it fails to do it seems is to expose that data on individual contact records so we can meaningfully measure engagement by contact and use those measurements to segment our lists in appropriate ways for purposes of relationship building, list hygiene and protecting sender reputation with the large email providers by managing email frequency based on engagement history.
Proposed behaviour
----------------------------------------
_What should happen? How is this better? If appropriate/available, include any wireframes or mockups._
CiviMail needs a means for exposing its engagement data on a contact record. In the activity view, I want to see not just that an email was sent, but also whether it was delivered, opened and clicked on. If opened, how often, when and from what IPs? If clicked on, the activity view should allow me to drill down to examine which links were clicked and how often, and perhaps also provide a link back to the mailing itself and its performance report.
CiviMail would benefit from shipping with a default rule, but also having hooks allowing CiviRules to be used to define an installation-specific two-deminsional engagement score for a contact, based on frequency and recency of engagement. The contact search tools need to a means for searching based on this engagement score. Smart groups would benefit being dynamically built based on this engagement score.
Comments
----------------------------------------
_Anything else you would like the reviewer to note._https://lab.civicrm.org/dev/core/-/issues/3493Afform: Failed to find key by ID or tag (SIGN)2022-12-19T21:18:09ZandyburnsAfform: Failed to find key by ID or tag (SIGN)I created a simple afform submitting a name and it just spins and shows this console error. On Civi 5.50.0, WP 5.8.4 and multisite. I had this issue on 5.46.3 as well. It does submit the contact even though it appears not to.
![image](/...I created a simple afform submitting a name and it just spins and shows this console error. On Civi 5.50.0, WP 5.8.4 and multisite. I had this issue on 5.46.3 as well. It does submit the contact even though it appears not to.
![image](/uploads/98a9537c12d29cda91345959e268d8a0/image.png)
Sometimes only the angular error:
![image](/uploads/4e77c2cbad1b88909dc62e7b4e3d4500/image.png)
No error reported to civi logs.https://lab.civicrm.org/dev/core/-/issues/3490Upgrader - Apply extension updates after core updates2023-02-08T19:08:30ZtottenUpgrader - Apply extension updates after core updatesOverview
----------------------------------------
Functionality in `civicrm-core` is increasingly organized with internal extensions (`civicrm-core:ext/*`). The main upgrade workflow should incorporate the upgrade-steps for extensions.
...Overview
----------------------------------------
Functionality in `civicrm-core` is increasingly organized with internal extensions (`civicrm-core:ext/*`). The main upgrade workflow should incorporate the upgrade-steps for extensions.
Example use-case
----------------------------------------
1. Extract tarball with a new version of `civicrm-core`, including updated `ext/*`
1. Run the main upgrader (`civicrm/upgrade?reset=1`, `cv core:upgrade`, `drush civicrm-upgrade-db`, or `wp civicrm upgrade-db`)
Current behaviour
----------------------------------------
It updates the schema for things like `civicrm_contact` (eg `CRM/Upgrade/*`) but not things like `civicrm_search_segment` (eg `ext/search_kit/CRM/Search/Upgrader.php`).
As a sysadmin, you must run the upgrades separately.
Most are not habituated to running these together. This leads to confusing user-stories like https://civicrm.stackexchange.com/questions/42094/civimail-issue-db-error-no-such-table-when-sending-mailing (where the first upgrade seemed to work - but you get random errors because the second upgrade hasn't run).
Proposed behaviour
----------------------------------------
Run both.
The core-upgrade should delegate/incorporate the ext-upgrade, and it should preserve essential elements of the ext-upgrade contract. Specifically: ext-upgrades can use features/services/APIs from core. This implies that (1) ext-upgrades run after core-upgrades and (2) when it gets to ext-upgrades, it should follow a normal/open dispatch-policy.
Comments
----------------------------------------
Filing issue after some Mattermost discussion with rudanpal, @colemanw, @bgm, @totten: https://chat.civicrm.org/civicrm/pl/b76y4gy517yxdet331ueqrxgrw
I posted a draft at https://github.com/civicrm/civicrm-core/compare/5.50...totten:run-ext-upgrades?expand=1
As mentioned in PHP comments+Mattermost, there's an issue with the current draft -- when it gets to ext-upgrade tasks, it would (unexpectedly) continue to apply the limited dispatch-policy used for core-upgrades (`upgrade.main` vs `upgrade.finish`). Some ways to address that:
* __Runner state__: While the upgrader goes through phases (*ex: 1-core upgrade, 2-ext upgrade, 3-new exts*), keep a stored value naming the phase. this stored value determines the active dispatch-policy
* ex: `Civi::settings()->set('upgrade_phase', '...')`
* __Task wrapping/task tagging__: When taking ext-upgrade tasks and putting them into the core-upgrade queue, put some wrapper or tag on each one to indicate the phase/policy that it relies on
* wrapper example: `CRM_Upgrade_Form::wrappedTask(string $dispatchPolicy, array $realCallback)
* tag example: `$queue->createItem($task, ['dispatchPolicy' => string])`https://lab.civicrm.org/dev/core/-/issues/3478Afform - email delivery2023-06-29T15:47:01Zaydunsaidan.saunders@squiffle.ukAfform - email deliveryOverview
----------------------------------------
Add an action to enable scheduled Email Delivery of SearchKit results.
Reports provides settings to enable scheduled Email Delivery of reports. The lack of an option to do the same dete...Overview
----------------------------------------
Add an action to enable scheduled Email Delivery of SearchKit results.
Reports provides settings to enable scheduled Email Delivery of reports. The lack of an option to do the same deters some people from using SearchKit - see https://civicrm.stackexchange.com/q/41966/225
Comments
----------------------------------------
Not a high priority item but noting for parity with Reports functionality.https://lab.civicrm.org/dev/core/-/issues/3470Search Kit: Mailing labels don't work2022-12-19T15:47:10ZJonGoldSearch Kit: Mailing labels don't workMailing labels don't work with Search Kit.
**Steps to Replicate**
* Do a search for contacts in search kit (with a small number of contacts to avoid hitting #3222).
* Select 1 or more contacts, select **Mailing Labels** from **Actions**...Mailing labels don't work with Search Kit.
**Steps to Replicate**
* Do a search for contacts in search kit (with a small number of contacts to avoid hitting #3222).
* Select 1 or more contacts, select **Mailing Labels** from **Actions**.
* Select a label and press **Make Mailing Labels**.
**Expected Result**
Mailing labels.
**Actual Result**
Spins indefinitely.
This doesn't happen when making PDFs, or if you grab the last HTTP request from the devtools Network tab and open it in a new window.
It happens because TCPDF isn't expecting to be in a "snippet" context at this point - it's just spitting out raw HTML response headers direct to the browser (`CRM_Contact_Form_Task_Label::createLabel()` calls `$pdf->Output()`). "Make documents" uses the more modern `CRM_Utils_PDF_Utils::html2pdf(), which uses DOMPDF or wkHTMLtoPDF, and knows how to handle this. Unfortunately, TCPDF has the mailing label functionality, so we're stuck with it.
**How to proceed**
I could probably make a fix that involves saving the file to disk first - but I think a better fix is to modify the Search Kit "Actions" JS to open the labels in a new window, but I can't find where that happens.https://lab.civicrm.org/dev/core/-/issues/3462Event location search2024-01-31T10:18:23Zaydunsaidan.saunders@squiffle.ukEvent location searchOverview
----------------------------------------
Following on from #2103 - when configuring an event and reusing a location, Civi shows a message like 'This location is used by 2 other events' but does not indicate which events.
It w...Overview
----------------------------------------
Following on from #2103 - when configuring an event and reusing a location, Civi shows a message like 'This location is used by 2 other events' but does not indicate which events.
It would be useful to show a list of those locations, or include a link to search for them.
Note that SearchKit displays now handle LocBlocks (see #2676)https://lab.civicrm.org/dev/core/-/issues/3453Afform - relationships fill from other entity2022-12-07T01:43:54ZsamuelsovAfform - relationships fill from other entityFollow-up from https://github.com/civicrm/civicrm-core/pull/23296#issuecomment-1119014194
Let say we want to do a form that allow an employer to update :
- the main contact of the organization
- all the employees
Thanks to https://lab....Follow-up from https://github.com/civicrm/civicrm-core/pull/23296#issuecomment-1119014194
Let say we want to do a form that allow an employer to update :
- the main contact of the organization
- all the employees
Thanks to https://lab.civicrm.org/dev/core/-/issues/3117 it's now possible to do a form that will allow to create in one step :
- the organization
- the main contact as Individual 1 with a relationship between the organization and Individual 1
- the employees as Individual 2 with a relationship between the organization and Invividual 2 (which is multiple)
However, there is no way to have an edit mode for such a form. It's possible to add the organization id as an argument but we also need a way to pre-populate the main contacts / list of employees as an option based on the relationships definition.colemanwcolemanw