CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2024-03-18T15:33:44Zhttps://lab.civicrm.org/dev/core/-/issues/5082CiviCRM 5.70.0, 5.71.0 - With URL tracking enabled, a personalised "View in y...2024-03-18T15:33:44Zjustinfreeman (Agileware)CiviCRM 5.70.0, 5.71.0 - With URL tracking enabled, a personalised "View in your browser" link incorrectly replaces ? with & which causes CiviCRM to respond with error: "You do not have permission to access this page"CiviCRM 5.70.0, 5.71.0 - With URL tracking enabled, a personalised "View in your browser" link incorrectly replaces ? with & which causes CiviCRM to respond with error: "You do not have permission to access this page".
This happens when...CiviCRM 5.70.0, 5.71.0 - With URL tracking enabled, a personalised "View in your browser" link incorrectly replaces ? with & which causes CiviCRM to respond with error: "You do not have permission to access this page".
This happens when using a personalised "View in your browser" URL like this in the mailing, note the use of tokens:
https://goodcause.org.au/civicrm/mailing/view?id={mailing.key}&{contact.checksum}&cid={contact.contact_id}
Which is then incorrectly converted to - this only happens with URL tracking enabled. When URL tracking is disabled, no problems at all.
https://goodcause.org.au/civicrm/mailing/view&id=38&cs=838eae033aa8c2edb56f25b54a1edde5_1709775006_2880&cid=389
And then CiviCRM to respond with error: "You do not have permission to access this page"
The fix and workaround for this issue is to instead just use this token as the URL, which will render correctly.
{mailing.viewUrl}
It's not unreasonable to expect the personalised "View in your browser" URL will work, the first URL parameter should not be converted in this way. This may have implications for other types of URLs too.
Agileware Ref: CIVICRM-2230https://lab.civicrm.org/dev/core/-/issues/5060Automated reminder to people that didn't subcribe to groups they applied subs...2024-03-14T22:45:20Znikola.mladenovicAutomated reminder to people that didn't subcribe to groups they applied subscription forOverview
----------------------------------------
Subscribing for newsletter usually ends up as a one way street, if they want to have subscribers and be under wing of GDPR. If the user doesn't confirm their subscription, they would eith...Overview
----------------------------------------
Subscribing for newsletter usually ends up as a one way street, if they want to have subscribers and be under wing of GDPR. If the user doesn't confirm their subscription, they would either have to find original email or resubscribe if they cant find it, but usually users just forget they did subscribe.
Recommended improvement was discussed on Mattermost to check if such feature existed in civicrm and few users did raise concern and that they would like to have this feature request:
[Mattermost chat](https://chat.civicrm.org/civicrm/pl/bg6x9u7qmf8fdkwn8tz4o9ft1r )
Current behaviour
----------------------------------------
Subscribing to newsletter only sends email as user applies for newsletter
Proposed behaviour
----------------------------------------
Scheduled job for example, which would check if there are any users which are pending for some groups, check the date they applied and then send another nudge for user to join in. This feature should be limited to only once per fresh request for all pending users. Proposal for after how much this automated schedueld job would contact users should be a week, 2 weeks, month, 2 months, 3 months.
Comments
----------------------------------------
_Anything else you would like the reviewer to note._https://lab.civicrm.org/dev/core/-/issues/5055Does anyone use CRM_Mailing_Form_Approve?2024-03-07T13:37:04ZherbdoolDoes anyone use CRM_Mailing_Form_Approve?As part of clean up of accordions, we noticed that it's hard to test template for `civicrm/mailing/approve` which calls `CRM_Mailing_Form_Approve`. Is this being used? There's a permission as well which seems like it can't even be set. U...As part of clean up of accordions, we noticed that it's hard to test template for `civicrm/mailing/approve` which calls `CRM_Mailing_Form_Approve`. Is this being used? There's a permission as well which seems like it can't even be set. Unless I'm missing something.https://lab.civicrm.org/dev/core/-/issues/5027Event Confirmation Email Text Token Should be Set as text/html2024-02-28T12:46:59ZpbarmakEvent Confirmation Email Text Token Should be Set as text/htmlPer a chat with Eileen and Seamus, adding a ticket here. The Event confirmation email text is currently not rendering any input HTML tags. It seems this is because the token is not set as "text/html".
Seamus recommended changing https:/...Per a chat with Eileen and Seamus, adding a ticket here. The Event confirmation email text is currently not rendering any input HTML tags. It seems this is because the token is not set as "text/html".
Seamus recommended changing https://github.com/civicrm/civicrm-core/blob/master/CRM/Event/Tokens.php#L239 from this:
```
else {
$tokens[$fieldName]['text/plain'] = $event[$fieldName];
}
```
to this:
```
else {
$type = ($fieldName === 'confirm_email_text' ? 'text/html' : 'text/plain');
$tokens[$fieldName][$type] = $event[$fieldName];
}
```
For my testing, I just added this line around line 227:
`$tokens['confirm_email_text']['text/html'] = $event['confirm_email_text'];`
Either of those seem to solve the issue, converting the value of that field to actual html code.https://lab.civicrm.org/dev/core/-/issues/5022MailingEventSubscribe Error2024-02-28T12:46:24ZjmargrafMailingEventSubscribe ErrorOverview
----------------------------------------
MailingEventSubscribe does result in an error message
Reproduction steps
----------------------------------------
1. Check the ID of an existing mailing list group (eg. 60)
2. Open AP...Overview
----------------------------------------
MailingEventSubscribe does result in an error message
Reproduction steps
----------------------------------------
1. Check the ID of an existing mailing list group (eg. 60)
2. Open API-Explorer
3. Send API-Request:
```
$result = civicrm_api3('MailingEventSubscribe', 'create', [
'group_id' => 60,
'email' => "test@example.com",
]);
```
4. Got Error Message:
```
{
"error_code": 0,
"entity": "MailingEventSubscribe",
"action": "create",
"is_error": 1,
"error_message": "Failed to find key by ID or tag (I2s19RqQP0fDuuRcyoyhUhRQiAY)"
}
```
Current behaviour
----------------------------------------
Error Message: "error_message": "Failed to find key by ID or tag (I2s19RqQP0fDuuRcyoyhUhRQiAY)"
Expected behaviour
----------------------------------------
Subscribe to Mailing-List
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _Firefox 59.0.1/Chrome 78.0.3904/Safari 13_
* __CiviCRM:__ 5.69.5
* __PHP:__ 8.1
* __CMS:__ Drupal 10.2.3
* __Database:__ MariaDB
* __Web Server:__ Apachehttps://lab.civicrm.org/dev/core/-/issues/4875Better tracking of Scheduled Reminders including Reporting2024-02-21T19:56:17ZshaneonabikeBetter tracking of Scheduled Reminders including Reporting## Overview
_It has come up quite a few times with several clients of ours, whereby they want to see what Scheduled messages went out, if they bounced, and other details. In a few cases, the Scheduled Reminders were not configured prope...## Overview
_It has come up quite a few times with several clients of ours, whereby they want to see what Scheduled messages went out, if they bounced, and other details. In a few cases, the Scheduled Reminders were not configured properly by the client, which is understandable but that's another UI issue._
_So I think it could be really helpful to provide this type of tracking, including click-through, FFWD, etc for Scheduled Reminders like regular mailings._
## Example use-case
1. Set up a Scheduled Reminder for an Event or Membership Renewal
2. Go to Reports to view the progress of Scheduled Reminders
3. Filter by click-throughs, bounce, etc.
## Current behaviour
_Same as above. Presently, the only workaround in some cases like Scheduled Reminders is to create a "fake" mailing that is sent out to one person. Then copying the code generated into the Scheduled Reminder allows a person to actually track the Scheduled Reminder information like you would with a regular mailing because it is associated to our "fake" mailing. But this is super not sustainable or even doable for Event Reminders._
## Proposed behaviour
_Basically, I think it could be extremely useful to integrate the same functionality for Mailing tracking (including bounce) to Scheduled Reminders._https://lab.civicrm.org/dev/core/-/issues/4846API Deprecation2023-12-20T19:19:58ZtreseroAPI DeprecationLots of these in the log for mailings
[warning] Deprecated function CRM_Mailing_BAO_MailingComponent::retrieve, use API.
CRM_Core_Error::deprecatedFunctionWarning
CRM_Mailing_BAO_MailingComponent::retrieve
CRM_Mailing_Form_Component::se...Lots of these in the log for mailings
[warning] Deprecated function CRM_Mailing_BAO_MailingComponent::retrieve, use API.
CRM_Core_Error::deprecatedFunctionWarning
CRM_Mailing_BAO_MailingComponent::retrieve
CRM_Mailing_Form_Component::setDefaultValues
CRM_Core_Form::buildForm
Array
(
[civi.tag] => deprecated
)
Trying to make an email template, and I get this.
Wish I could help more.https://lab.civicrm.org/dev/core/-/issues/4782Mailing system doesn't work with DigitalOcean managed databases2023-11-20T12:13:45ZrobertgarrigosMailing system doesn't work with DigitalOcean managed databasesOverview
----------------------------------------
When creating a new mailing adding a group in the recipients field triggers an error:
Possibly unhandled rejection: {"error_code":-1,"sql":"CREATE TEMPORARY TABLE civicrm_tmp_e_exrecipie...Overview
----------------------------------------
When creating a new mailing adding a group in the recipients field triggers an error:
Possibly unhandled rejection: {"error_code":-1,"sql":"CREATE TEMPORARY TABLE civicrm_tmp_e_exrecipient_dc8f2500b15714f5703ad66bc5a8bb37 (contact_id int primary key) ENGINE=MEMORY COLLATE utf8mb4_unicode_ci","debug_info":"CREATE TEMPORARY TABLE civicrm_tmp_e_exrecipient_dc8f2500b15714f5703ad66bc5a8bb37 (contact_id int primary key) ENGINE=MEMORY COLLATE utf8mb4_unicode_ci [nativecode=3161 ** Storage engine MEMORY is disabled (Table creation is disallowed).]","entity":"Mailing","action":"create","is_error":1,"error_message":"DB Error: unknown error","debug_information":"CREATE TEMPORARY TABLE civicrm_tmp_e_exrecipient_dc8f2500b15714f5703ad66bc5a8bb37 (contact_id int primary key) ENGINE=MEMORY COLLATE utf8mb4_unicode_ci”}
https://civicrm.stackexchange.com/questions/45905/storage-engine-memory-is-disabled-with-digitalocean-managed-database
Reproduction steps
----------------------------------------
1. Create a new mailing
1. add a group to the recipients field
1. Got error on console as well as "estimating" label get locked
1. You need to have the civicrm database hosted in a DigitalOcean manage database cluster
Expected behaviour
----------------------------------------
Groups should get added and count of contacts should be shown next to the recipients field
Comments
----------------------------------------
Fixed by changing line 67 of CRM/Utils/SQL/TempTable.php from `const MEMORY = 'ENGINE=MEMORY';` to `const MEMORY = 'ENGINE=InnoDB';`
I'm aware that you could think that this is a DigitalOcean Problem (they have unsetted the ability to set internal_tmp_mem_storage_engine to MEMORY), but I wonder if the Memory Engine is necessary at all.https://lab.civicrm.org/dev/core/-/issues/4733Where art thou APIv4: MailingEventSubscribe Create? Missing parity with civic...2023-11-02T15:16:06Zjustinfreeman (Agileware)Where art thou APIv4: MailingEventSubscribe Create? Missing parity with civicrm_api3('MailingEventSubscribe', 'Create')Where art thou APIv4: MailingEventSubscribe Create? Missing parity with civicrm_api3('MailingEventSubscribe', 'Create')
Agileware Ref: CVAP-50Where art thou APIv4: MailingEventSubscribe Create? Missing parity with civicrm_api3('MailingEventSubscribe', 'Create')
Agileware Ref: CVAP-50https://lab.civicrm.org/dev/core/-/issues/4626Reduce calls to CRM_Utils_System::url() in mailings2023-10-05T11:45:56ZJonGoldReduce calls to CRM_Utils_System::url() in mailingsOn Drupal 8+ and possibly on other CMSes, `CRM_Utils_System::url()` takes an absurdly long time to run. Seems like `\Drupal\Core\Utility\UnroutedUrlAssembler::assemble()` is the main culprit, but even that's only part of the problem.
W...On Drupal 8+ and possibly on other CMSes, `CRM_Utils_System::url()` takes an absurdly long time to run. Seems like `\Drupal\Core\Utility\UnroutedUrlAssembler::assemble()` is the main culprit, but even that's only part of the problem.
When generating VERP URLs, we call `CRM_Utils_System::url()` 5 times per email, to generate almost-identical URLs. Simply reducing that to one call per email cut my test mail sending by just over 25%!
There are further improvements to be made, I'd love to hear folks' feedback.
* Do we even need to call `\Drupal\Core\Url::fromUri()` every time we call `CRM_Utils_System::url()`? Is there a faster way to generate URLs?
* Many email service providers (Mailjet, Sparkpost etc.) emit a webhook on bounce. What if we created a "Disable VERP" setting?JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/4608View mailing access is not allowed for users authenticated via checksum2023-09-26T12:15:20ZlarsssandergreenView mailing access is not allowed for users authenticated via checksumIf you send a mailing with a link to view the mailing in your browser that includes contact id and checksum (which is what the default "View in browser" link Mosaico's Versafix template does), you would expect that you'd be able to view ...If you send a mailing with a link to view the mailing in your browser that includes contact id and checksum (which is what the default "View in browser" link Mosaico's Versafix template does), you would expect that you'd be able to view the mailing even if the Publication is set to User and User Admin Only. However, if you open the link in a private browsing window, you'll get access denied.
I think the expected behaviour would be that you can access the mailing when authenticated via checksum. Otherwise, that View in browser link is broken for any mailing that isn't public.
If the mailing isn't public, the [page just checks](https://github.com/civicrm/civicrm-core/blob/0c5f17226c73474e40285a87d8509b272f4f4a67/CRM/Mailing/Page/View.php#L27) for administer CiviCRM, approve mailings or access CiviMail. Maybe we need a new permission like `CiviMail: access non-public mailings` or maybe we can simply allow any logged in user to view mailings (since it says "User and User Admin Only" and a logged in contact is a user).https://lab.civicrm.org/dev/core/-/issues/4567Address CiviCRM Mailing table complexity - make queries easier & data more pr...2023-09-27T01:01:44ZeileenAddress CiviCRM Mailing table complexity - make queries easier & data more prunableIssue
----------------------------------------
1) on large databases the civicrm_mailing screen is slow-to-impossible to load as the query cannot get the status of the mailing from the civicrm_mailing table and must do a many-to-one joi...Issue
----------------------------------------
1) on large databases the civicrm_mailing screen is slow-to-impossible to load as the query cannot get the status of the mailing from the civicrm_mailing table and must do a many-to-one join with DISTINCT to get the status of the mailing. This is not well suited to paging because of the many to 1 so many more records are retrieved than are needed
2) the mailing job records are really just a form of queue management. ie their logic is temporary and related to active sending. Once the send has happened it should be possible to delete them - but the mailing_job records are the only way the civicrm_mailing_event_queue table links to Bounce entries
3) the verp hash needs to be of limited length. While efforts were made in the past to reduce the length the ids of the values in the hash can get large enough to start to blow it up. It really doesn't need to have a queue_id AND a job_id along with the hash
4) complexity complexity complexity. The mailing_job has fields like 'parent_id' and job_type' = 'child'. It's just confusing as well as bloated and unnecessary to refer to it outside it's 'real' function - tracking the actual sending
5) I have some historical nervousness about the civimail system sending out stuff it shouldn't - I feel like great transparency about loading from civicrm_mailing & then processing would be possible with
Proposal
----------------------------------------
**Stage 1**
1) adding the field mailing_id to civicrm_mailing_event_queue
2) altering the verp calculation to stop including the job_id on new mailings(we would need to handle it's presence for a while in bounce processing)
3) altering the job_id index in civicrm_event_queue to set to NULL on delete
4) add columns for status_id, start_date, end_date to civicrm_mailing, update as appropriate.
5) add a batch script to allow people to populate the columns above in their own time.
6) use more efficient queries on sites that have fully migrated (eg. conditional on status_id being populated for all mailings then the civicrm_mailing dashboard could use a more performant query)
<details>
<summary>Handy query for getting row counts in these tables</summary>
```
SELECT 'civicrm_mailing' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing
UNION
SELECT 'civicrm_mailing_job' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_job
UNION
SELECT 'civicrm_mailing_event_queue' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_queue
UNION
SELECT 'civicrm_mailing_event_bounce' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_bounce
UNION
SELECT 'civicrm_mailing_event_confirm' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_confirm
UNION
SELECT 'civicrm_mailing_event_delivered' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_delivered
UNION
SELECT 'civicrm_mailing_event_forward' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_forward
UNION
SELECT 'civicrm_mailing_event_opened' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_opened
UNION
SELECT 'civicrm_mailing_event_reply' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_reply
UNION
SELECT 'civicrm_mailing_event_subscribe' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_subscribe
UNION
SELECT 'civicrm_mailing_event_trackable_url_open' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_trackable_url_open
UNION
SELECT 'civicrm_mailing_event_unsubscribe' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_unsubscribe
```
</details>
**Stage 2**
1) deprecate the field civicrm_mailing.is_completed
6) set up some processes around deleting mailing_job records once no longer needed.
8)remove the interim queries for the partially migrated mailing records
*Stage 3*
7) eventually the civicrm_mailing.is_completed column could go
8) stop handling the job id in the verp (probably after a few years)
9) add self-cleanup on civicrm_mailing_job records (how would we ideally clean them up once completed?)
10) migrate mailing_job to our queue system
**Technical notes**
The way in which mailing events link back to the mailing looks like (eg. for bounce)
- civicrm_mailing_event_bounce.event_queue_id => civicrm_mailing_event_queue.id
- civicrm_mailing_event_queue.job_id => civicrm_mailing_job.id
- civicrm_mailing_job.mailing_id => civicrm_mailing.id
A similar pattern is followed for
civicrm_mailing_event_delivered
civicrm_mailing_event_forward
civicrm_mailing_event_opened
civicrm_mailing_event_reply
civicrm_mailing_event_subscribe
civicrm_mailing_event_confirm links back through civicrm_mailing_event_subscribe
**Still confusing**
How do civicrm_mailing_recipients & civicrm_mailing_event_queue relate to each other?**** UPDATE - it seems like mailing_event_recipients doesn't add much extra over civicrm_mailing_event_queue but it is used to serve 2 purposes
1) it fulfills the role of a temp table when generating the entries for civicrm_mailing_queue
2) it is used for include / exclude queries - this could be done by event_queue too but perhaps would be more blocking
It does feel like it doesn't add anything that just using queue wouldn't, if we pointed the various queries at queue
What is the status with this historic evilness https://lab.civicrm.org/dev/core/-/issues/4385https://lab.civicrm.org/dev/core/-/issues/4561On new mailing screen the widget for include/exclude double-escapes characters2023-09-13T12:22:17ZDaveDOn new mailing screen the widget for include/exclude double-escapes characters1. Create a group like "2023 Donated >$100". You don't have to actually create a search, just manually create a group with that title.
2. On the new mailing screen, try filtering by typing ">$100". It won't find it because the title disp...1. Create a group like "2023 Donated >$100". You don't have to actually create a search, just manually create a group with that title.
2. On the new mailing screen, try filtering by typing ">$100". It won't find it because the title displays as `>$100`. Searching for `gt;` will find it but that's silly.https://lab.civicrm.org/dev/core/-/issues/4539Very slow query when sending mailings via CiviMail2023-09-12T05:08:13ZwmortadaVery slow query when sending mailings via CiviMailOverview
----------------------------------------
We have come across an issue where the database connection was timing out due to slow, long-running queries when the Send Scheduled Mailings job was running. We became aware of this when...Overview
----------------------------------------
We have come across an issue where the database connection was timing out due to slow, long-running queries when the Send Scheduled Mailings job was running. We became aware of this when we set a maximum execution time limit of 3 minutes for MySQL queries. Mailings for one of our clients were not being sent because the database connection was timing out.
As a workaround, we have increased the maximum execution time to 10 minutes and reduced the Mailer Batch Limit and Mailer Job Size to 1,000. However, we'd like to fix the underlying issue that the queries are running so slowly.
When sending a mailing CiviCRM runs a query of the form:
```sql
SELECT r.contact_id, r.email_id, r.phone_id
FROM civicrm_mailing_recipients r
INNER JOIN civicrm_contact c on
(c.id = r.contact_id
AND c.is_deleted = 0
AND c.is_deceased = 0
AND c.do_not_email = 0
AND c.is_opt_out = 0
)
INNER JOIN civicrm_email e ON (r.email_id = e.id AND e.on_hold = 0)
WHERE r.mailing_id = 3271
LIMIT 18000, 1000
```
There are no indices for `is_deceased`, `do_not_email` or `is_opt_out` so this query is very slow. (There is an index for `is_deleted`. There should be an index for `is_deceased` but this appears to be missing from this site.)
We have tested adding these indices in a development environment to see what difference it would make and this increase the speed of the query by a factor of one thousand:
- 20,000 rows with no indices = 80 seconds
- 20,000 rows with indices = 0.085 seconds
Reproduction steps
----------------------------------------
This is likely to only affect sites with a large number of contacts and large number of rows in the `civicrm_mailing_recipients` table (in this case 46,000 contacts and 9 million rows respectively).
1. Set a maximum execution time for MySQL of 3 minutes (say)
2. Create a mailing to a large group of contacts (>20,000)
3. Schedule the mailing to send
4. Send Scheduled Mailings job fails with error: "Finished execution of Send Scheduled Mailings with result: Failure, Error message: DB Error: unknown error"
Current behaviour
----------------------------------------
No indices ~~`is_deceased`~~, `do_not_email` or `is_opt_out` in `civicrm_contact` so queries run very slowly.
Expected behaviour
----------------------------------------
Indices for the above fields so that the query runs quickly.
Environment information
----------------------------------------
* __CiviCRM:__ _5.62_
* __PHP:__ _8.1__
* __CMS:__ _N/A_
* __Database:__ _MySQL 5.7.43_
Comments
----------------------------------------
There are other speed improvements in #4045 but these relate to the speed of the user interface rather than the speed of sending emails.
Perhaps this can also help to [reduce our collective CO2 emissions](https://civicrm.org/blog/systopia/beginning-green-how-can-we-make-civicrm-more-sustainable)?https://lab.civicrm.org/dev/core/-/issues/4508CiviMail Auto complete includes unsent mailings2023-08-24T13:43:34ZseamusleeCiviMail Auto complete includes unsent mailingsAs reported by Jon Goldberg it seems the current CiviMail Recipients Auto complete includes unsent mailings https://github.com/civicrm/civicrm-core/pull/27071#issuecomment-1682481080As reported by Jon Goldberg it seems the current CiviMail Recipients Auto complete includes unsent mailings https://github.com/civicrm/civicrm-core/pull/27071#issuecomment-1682481080https://lab.civicrm.org/dev/core/-/issues/4434Don't allow uploading attachments to bulk mailings if they exceed the size limit2023-07-21T09:57:50ZDaveDDon't allow uploading attachments to bulk mailings if they exceed the size limitThere's been a couple times on multiple sites where the mailing bounces and it's not clear to the average office staffer why, but it's because they've uploaded a 30MB pdf to the mailing.
It should just not allow doing that in the first ...There's been a couple times on multiple sites where the mailing bounces and it's not clear to the average office staffer why, but it's because they've uploaded a 30MB pdf to the mailing.
It should just not allow doing that in the first place if the config settings have a smaller limit.https://lab.civicrm.org/dev/core/-/issues/4421Scheduled jobs stopped working after an update last week - error in MailingEv...2023-11-23T06:34:09ZTobias KrauseScheduled jobs stopped working after an update last week - error in MailingEventUnsubscribe.phpI realized this morning that the CiviCRM cron job for the scheduled jobs stopped working after we updated last week from CiviCRM 5.61.2 to 5.61.4. Today I updated to the most current version 5.63.1 but the error still exists.
The proble...I realized this morning that the CiviCRM cron job for the scheduled jobs stopped working after we updated last week from CiviCRM 5.61.2 to 5.61.4. Today I updated to the most current version 5.63.1 but the error still exists.
The problem became obvious as the message "Cron job not running" appeared when I logged in this morning (I did not so for the whole last week). When I checked the list of scheduled jobs I found out that the "Bounces fetcher" job run successfully but all the other jobs did not. So I run our cron job task
`/var/www/civi_live/vendor/civicrm/cv/bin/cv api job.execute --user=admin --cwd=/var/www/civi_live/httpdocs`
manually in CLI and got the following error:
```
In MailingEventUnsubscribe.php line 47:
count(): Argument #1 ($value) must be of type Countable|array, null given
```
The following file is the one: vendor/civicrm/civicrm-core/api/v3/MailingEventUnsubscribe.php
Here it is the following code part in the function civicrm_api3_mailing_event_unsubscribe_create:
```
$groups = CRM_Mailing_Event_BAO_MailingEventUnsubscribe::unsub_from_mailing($job, $queue, $hash);
if (count($groups)) {
CRM_Mailing_Event_BAO_MailingEventUnsubscribe::send_unsub_response($queue, $groups, FALSE, $job);
return civicrm_api3_create_success($params);
}
```
CRM_Mailing_Event_BAO_MailingEventUnsubscribe::unsub_from_mailing can return an array or NULL. I am not sure why this error appeared after the last update as neither CRM_Mailing_Event_BAO_MailingEventUnsubscribe::unsub_from_mailing nor civicrm_api3_mailing_event_unsubscribe_create() has changed but it may be related to some other changes somewhere.
When I change this code part to the following the scheduled jobs are finished:
```
$groups = CRM_Mailing_Event_BAO_MailingEventUnsubscribe::unsub_from_mailing($job, $queue, $hash);
if ($groups && count($groups)) {
CRM_Mailing_Event_BAO_MailingEventUnsubscribe::send_unsub_response($queue, $groups, FALSE, $job);
return civicrm_api3_create_success($params);
}
```https://lab.civicrm.org/dev/core/-/issues/4385version=4 is not passed to Mailing.create on APIv4 so MailingJob gets scheduled2023-09-13T01:30:02ZSandor Semseyversion=4 is not passed to Mailing.create on APIv4 so MailingJob gets scheduledOverview
----------------------------------------
I'd like to create a single Mailing entity through APIv4 without scheduling a MailingJob and calculating recipients.
Reproduction steps
----------------------------------------
1. Call M...Overview
----------------------------------------
I'd like to create a single Mailing entity through APIv4 without scheduling a MailingJob and calculating recipients.
Reproduction steps
----------------------------------------
1. Call Mailing.create through API Explorer or `cv`
1. Notice that a MailingJob is created
Current behaviour
----------------------------------------
According to https://github.com/civicrm/civicrm-core/blob/94b9b95547d3f8292740c059f77f83cc45e57b57/CRM/Mailing/BAO/Mailing.php#L1622-L1629 this shouldn't happen by default, but it does: `$params['version']` is empty and `self::doSubmitActions()` is called.
Expected behaviour
----------------------------------------
`$params['version']` is populated so `CRM_Mailing_BAO_Mailing::doSubmitActions()` doesn't fire.
Comments
----------------------------------------
This is similar to https://github.com/civicrm/civicrm-core/blob/94b9b95547d3f8292740c059f77f83cc45e57b57/CRM/Member/BAO/Membership.php#L334-L343. In that case `Civi\Api4\Service\Spec\Provider\MembershipCreationSpecProvider` injects `version` parameter into the API call, but for Mailing such spec provider doesn't exists.
I can do some work with this, I'm just not sure if that spec provider is missing by design or not...
Also I can pass in `version=4` explicitly or use `_skip_evil_bao_auto_schedule_` and `_skip_evil_bao_auto_recipients_` to prevent submit actions to happen.https://lab.civicrm.org/dev/core/-/issues/4342It should be possible to remove a hidden smart group from recipients of a mai...2023-06-28T06:56:18ZlarsssandergreenIt should be possible to remove a hidden smart group from recipients of a mailing that has been copiedIf you create a mailing by copying another mailing that was sent to search results, it is impossible to remove the hidden smart group from the recipients of your new mailing. This can be frustrating, because your intent in copying a mail...If you create a mailing by copying another mailing that was sent to search results, it is impossible to remove the hidden smart group from the recipients of your new mailing. This can be frustrating, because your intent in copying a mailing may not have been to send it to the same contacts, rather to re-use the content. With Mosaico, you can't easily duplicate the content, so this is a pain.
Ideally, hidden smart groups would be mandatory for the original mailing and could be removed for copies. However, that isn't possible because mandatory groups are simply [included groups that are hidden](https://github.com/civicrm/civicrm-core/blob/f0648c892ab4795b9bec873c840e06009ea4800a/ang/crmMailing/BlockRecipients.html#L6).
Options I see:
1. Don't include hidden groups when copying a mailing. Would solve this issue, but then some users are going to expect identical recipients when they copy a mailing because they are copying the mailing in order to resend to the same recipients, changing the content.
1. Add a Copy without recipients option. This seems like it would add UI complexity for something pretty specific.
2. Allow hidden smart groups to be removed from mailings in general, with a confirmation. This feels like the best option to me. Maybe I've started a mailing from search results and later decide I don't want to send it to those search results. Clearly we want to prevent users from removing a hidden group by accident, but as an intentional decision the user confirms, I think this makes sense. Would have to look at how to handle the unsubscribe group in this case.https://lab.civicrm.org/dev/core/-/issues/4328Double opt-in requires traditional bounce handling to be enabled2023-06-28T06:55:29ZJonGoldDouble opt-in requires traditional bounce handling to be enabledOverview
----------------------------------------
The default double opt-in email says:
> You have a pending subscription to the {subscribe.group} mailing list. To confirm this subscription, reply to this email or click <a href="{subscri...Overview
----------------------------------------
The default double opt-in email says:
> You have a pending subscription to the {subscribe.group} mailing list. To confirm this subscription, reply to this email or click <a href="{subscribe.url}">here</a>.
However, "reply to this email" requires traditional bounce handling (with a bounce mailbox) to be set up. In this day and age that's less common compared to using a third party mailer that sends bounce notifications via API (to [Airmail](https://civicrm.org/extensions/airmail) or one of its many cousins).
Moreover, it's a usability issue to put "here" as the clickable text. It should be more like: "Please click here to <a href="{subscribe.url}">confirm your subscription.</a>.
I want to get concept approval - and also guidance on whether we should update existing templates or just new installs. Personally I think we should update any non-customized template.