CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-09-18T12:24:16Zhttps://lab.civicrm.org/dev/core/-/issues/4572Proposal: Retrieval of Defaults w/ Form Builder and Form Processor2023-09-18T12:24:16ZMariaVProposal: Retrieval of Defaults w/ Form Builder and Form ProcessorOverview
----------------------------------------
At the CiviSprint in Germany we found out that Submission forms (Form Builder) currently do not support use cases such as self-service forms that allow users to check and change their exi...Overview
----------------------------------------
At the CiviSprint in Germany we found out that Submission forms (Form Builder) currently do not support use cases such as self-service forms that allow users to check and change their existing contact data.
A lot of Wordpress projects use CalderaForms together with Form Processor to transfer the data to CiviCRM. In CalderaForms there is an option to activate Retrieval of Defaults:
![grafik](/uploads/912921154b32fbf34052c77dcf79b597/grafik.png)
As well as in Form Processor:
![grafik](/uploads/8802a13e8aa75798be4a25bfa9126fcf/grafik.png)
Caldera Forms allows passing URL parameters to the form processor – e.g. a contact CiviCRM ID along with a checksum to prove that the user is allowed to see and edit data for this particular contact, as a way of authentication without requiring a login. These URLs look as follows:
_https//link.org/example-page/cid={contact.contact_id}&{contact.checksum}_
With this information the form processor can retrieve data from within CiviCRM when the form is loaded and prefill fields in the form.
What works in the Form Builder already?
----------------------------------------
- Adding Form Processor as an entity
- Using Form Processor fields for a form
- Submitting/Creating data entered in submission form
- Using the entity id as an URL parameter (requires login)
What is missing?
----------------------------------------
- Option to enable retrieval of defaults (like in CalderaForms) to fill forms with existing data.
- Forms that work for anonymous users, using a checksum for authentication
Comments
----------------------------------------
If you need any further information, please let me know.
I could assist in setting up an example in a test environment since I am not able to implement this function myself.https://lab.civicrm.org/dev/core/-/issues/4571Proposal: Add possibilty to assign custom CSS classes to containers in Form B...2024-03-12T13:14:12ZTobias Voigttobias.voigt@civiservice.deProposal: Add possibilty to assign custom CSS classes to containers in Form BuilderOverview
----------------------------------------
This issue intends to achieve more control and flexibility in styling the frontend appearance of form builder forms. To achieve this we suggest to add the possibility of assigning custom ...Overview
----------------------------------------
This issue intends to achieve more control and flexibility in styling the frontend appearance of form builder forms. To achieve this we suggest to add the possibility of assigning custom CSS classes to containers in the configuration backend / GUI of Form Builder.
This is meant to be an addition to the [Form Builder Roadmap](https://lab.civicrm.org/dev/core/-/issues/3761).
Current behaviour
----------------------------------------
Currently it is possible to select a predefined CSS class from the 'style' dropdown menu ("Panel Pane"). When this style is selected, an additional class is added to the div container ("af-container-style-pane").
![Screenshot](/uploads/306fe60b466b20d838972ff125ebb988/Bildschirmfoto_vom_2023-09-13_16-48-18.png)
Proposed behaviour
----------------------------------------
We propose to give users the possibilty to add custom CSS classes to form builder containers in a new input (text) field to allow more control and flexibility in styling the frontend appearance of form builder forms.
![Mockup](/uploads/047b3bc36039cf5f138dc9342834e25c/mockup-custom-css-classes.png)
Comments
----------------------------------------
Some major parts of the relevant code seems to be in these files:
https://github.com/civicrm/civicrm-core/blob/master/ext/afform/admin/ang/afGuiEditor/afGuiMenuItemStyle.component.js
https://github.com/civicrm/civicrm-core/blob/master/ext/afform/admin/ang/afGuiEditor/afGuiMenuItemStyle.htmlhttps://lab.civicrm.org/dev/core/-/issues/4570Proposal: More logging in Form Builder2023-09-14T12:58:32ZAndreasandreas.howiller@civiservice.deProposal: More logging in Form BuilderOverview
----------------------------------------
This issue explains a new proposal for the "Other improvements" section in the [Form Builder roadmap](https://lab.civicrm.org/dev/core/-/issues/3761): Extending the existing submission lo...Overview
----------------------------------------
This issue explains a new proposal for the "Other improvements" section in the [Form Builder roadmap](https://lab.civicrm.org/dev/core/-/issues/3761): Extending the existing submission log and adding a debug log.
Current behaviour
----------------------------------------
Currently, there is a simple log for submissions that does not contain submission data.
Proposed behaviour
----------------------------------------
There is already a submission log per form, but it only records the submitter and the time.
With some additional capabilities of FormBuilder, it becomes more important for both reporting and debugging to capture more information about the submissions themselves on the one hand, but also about their processing on submission on the other hand (examples: deduplication, processing by Form Processor etc.).
There should therefore be two logs:
1. the submission log should contain the actual submission data in addition to the metadata (cf. #3744).
2. an additional debug log should contain information about the processing of the data (like the CMRF log in WordPress or Drupal for [CiviMcRestFace](https://github.com/CiviMRF) submissions).
Comments
----------------------------------------
For reasons of data protection, it should be configurable by form whether no logging, logging of metadata or extended logging is desired.https://lab.civicrm.org/dev/core/-/issues/4569Proposal: Adding confirmation messages flexibly to Form Builder2023-09-14T15:03:30ZAndreasandreas.howiller@civiservice.deProposal: Adding confirmation messages flexibly to Form BuilderOverview
----------------------------------------
This issue explains a new proposal for the "Support more workflows" section in the [Form Builder roadmap](https://lab.civicrm.org/dev/core/-/issues/3761): Add confirmation messages flexib...Overview
----------------------------------------
This issue explains a new proposal for the "Support more workflows" section in the [Form Builder roadmap](https://lab.civicrm.org/dev/core/-/issues/3761): Add confirmation messages flexibly as we know it from Drupal webforms or Caldera Forms.
Current behaviour
----------------------------------------
Currently, there is no success message at all when I submit a form - as a user, I do not receive any feedback as to whether my data has really been processed.
Proposed behaviour
----------------------------------------
There should be a message for a successful submission resp. a message for errors.
For simple use cases such as editing data in the backend, the definition of messages per form (cf. #2957) is certainly sufficient. For more complex applications, more flexibility would also make sense in view of other features on the Form Builder roadmap, e.g. displaying messages after forwarding to a multi-step form (cf. #4574) or modal forwarding. To put it short: We should do it the way webforms does:
![confirmation_webforms](/uploads/74fa65d8d384b1f6b5de1da76be0e56c/confirmation_webforms.png)
Comments
----------------------------------------
1. There should probably be a translated fallback / standard message as well.
2. In order to be able to design success messages more individually, it should be possible to include form entries in the success message. Example: Dear Frank, thank you for signing our petition. In Caldera Forms, for example, this is comfortably solved via slugs / "magic tags":
![grafik](/uploads/097c919b39a4b575a524cf4b96b4f50a/grafik.png)
Cf. "post-submit tokens" on Form Builder roadmap and #4576https://lab.civicrm.org/dev/core/-/issues/4567Address CiviCRM Mailing table complexity - make queries easier & data more pr...2023-09-27T01:01:44ZeileenAddress CiviCRM Mailing table complexity - make queries easier & data more prunableIssue
----------------------------------------
1) on large databases the civicrm_mailing screen is slow-to-impossible to load as the query cannot get the status of the mailing from the civicrm_mailing table and must do a many-to-one joi...Issue
----------------------------------------
1) on large databases the civicrm_mailing screen is slow-to-impossible to load as the query cannot get the status of the mailing from the civicrm_mailing table and must do a many-to-one join with DISTINCT to get the status of the mailing. This is not well suited to paging because of the many to 1 so many more records are retrieved than are needed
2) the mailing job records are really just a form of queue management. ie their logic is temporary and related to active sending. Once the send has happened it should be possible to delete them - but the mailing_job records are the only way the civicrm_mailing_event_queue table links to Bounce entries
3) the verp hash needs to be of limited length. While efforts were made in the past to reduce the length the ids of the values in the hash can get large enough to start to blow it up. It really doesn't need to have a queue_id AND a job_id along with the hash
4) complexity complexity complexity. The mailing_job has fields like 'parent_id' and job_type' = 'child'. It's just confusing as well as bloated and unnecessary to refer to it outside it's 'real' function - tracking the actual sending
5) I have some historical nervousness about the civimail system sending out stuff it shouldn't - I feel like great transparency about loading from civicrm_mailing & then processing would be possible with
Proposal
----------------------------------------
**Stage 1**
1) adding the field mailing_id to civicrm_mailing_event_queue
2) altering the verp calculation to stop including the job_id on new mailings(we would need to handle it's presence for a while in bounce processing)
3) altering the job_id index in civicrm_event_queue to set to NULL on delete
4) add columns for status_id, start_date, end_date to civicrm_mailing, update as appropriate.
5) add a batch script to allow people to populate the columns above in their own time.
6) use more efficient queries on sites that have fully migrated (eg. conditional on status_id being populated for all mailings then the civicrm_mailing dashboard could use a more performant query)
<details>
<summary>Handy query for getting row counts in these tables</summary>
```
SELECT 'civicrm_mailing' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing
UNION
SELECT 'civicrm_mailing_job' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_job
UNION
SELECT 'civicrm_mailing_event_queue' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_queue
UNION
SELECT 'civicrm_mailing_event_bounce' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_bounce
UNION
SELECT 'civicrm_mailing_event_confirm' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_confirm
UNION
SELECT 'civicrm_mailing_event_delivered' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_delivered
UNION
SELECT 'civicrm_mailing_event_forward' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_forward
UNION
SELECT 'civicrm_mailing_event_opened' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_opened
UNION
SELECT 'civicrm_mailing_event_reply' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_reply
UNION
SELECT 'civicrm_mailing_event_subscribe' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_subscribe
UNION
SELECT 'civicrm_mailing_event_trackable_url_open' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_trackable_url_open
UNION
SELECT 'civicrm_mailing_event_unsubscribe' as mailing_table, COUNT(*) as number_rows
FROM civicrm_mailing_event_unsubscribe
```
</details>
**Stage 2**
1) deprecate the field civicrm_mailing.is_completed
6) set up some processes around deleting mailing_job records once no longer needed.
8)remove the interim queries for the partially migrated mailing records
*Stage 3*
7) eventually the civicrm_mailing.is_completed column could go
8) stop handling the job id in the verp (probably after a few years)
9) add self-cleanup on civicrm_mailing_job records (how would we ideally clean them up once completed?)
10) migrate mailing_job to our queue system
**Technical notes**
The way in which mailing events link back to the mailing looks like (eg. for bounce)
- civicrm_mailing_event_bounce.event_queue_id => civicrm_mailing_event_queue.id
- civicrm_mailing_event_queue.job_id => civicrm_mailing_job.id
- civicrm_mailing_job.mailing_id => civicrm_mailing.id
A similar pattern is followed for
civicrm_mailing_event_delivered
civicrm_mailing_event_forward
civicrm_mailing_event_opened
civicrm_mailing_event_reply
civicrm_mailing_event_subscribe
civicrm_mailing_event_confirm links back through civicrm_mailing_event_subscribe
**Still confusing**
How do civicrm_mailing_recipients & civicrm_mailing_event_queue relate to each other?**** UPDATE - it seems like mailing_event_recipients doesn't add much extra over civicrm_mailing_event_queue but it is used to serve 2 purposes
1) it fulfills the role of a temp table when generating the entries for civicrm_mailing_queue
2) it is used for include / exclude queries - this could be done by event_queue too but perhaps would be more blocking
It does feel like it doesn't add anything that just using queue wouldn't, if we pointed the various queries at queue
What is the status with this historic evilness https://lab.civicrm.org/dev/core/-/issues/4385https://lab.civicrm.org/dev/core/-/issues/4565Form Builder: Dropdown field options do not filter despite UI configuration2023-09-26T12:13:02ZScottMasonForm Builder: Dropdown field options do not filter despite UI configurationFields on FB forms appear to be ignoring configuration.
To replicate, create a FB form. Add dropdown field. Select 'customize options'. Remove options from the list
![image](/uploads/55fb079c2cf1c243fd5280edbf218fa1/image.png)
Despite...Fields on FB forms appear to be ignoring configuration.
To replicate, create a FB form. Add dropdown field. Select 'customize options'. Remove options from the list
![image](/uploads/55fb079c2cf1c243fd5280edbf218fa1/image.png)
Despite cofiguration, options still appear in the dropdown on the form.
![image](/uploads/75d2d152d9aaf8199f19cea9c488e309/image.png)https://lab.civicrm.org/dev/core/-/issues/4564Idea: Smart groups without the cache2023-09-13T12:20:47ZAndrew WestIdea: Smart groups without the cacheJust throwing this out there as something we could maybe fund:
Current behaviour
----------------------------------------
Smart groups always write to the civicrm_group_contact_cache table
Proposed behaviour
---------------------------...Just throwing this out there as something we could maybe fund:
Current behaviour
----------------------------------------
Smart groups always write to the civicrm_group_contact_cache table
Proposed behaviour
----------------------------------------
Smart groups optionally write to the civicrm_group_contact_cache table (default: yes)
Overview
----------------------------------------
As I understand it, whenever you request a smart group it stores its contacts in civicrm_group_contact_cache. This is for good reason: it's a cache that means the smart groups aren't regenerated every time they're requested, which can be slow.
It also simplifies the code: you can effectively treat smart groups the same as non-smart groups - as it's just grabbing a list of contacts from a table.
But it's possible to set the cache to expire after 0 seconds, if you need your smart groups to be up to date: ie. they are regenerated each time they're requested. I think a fair few orgs do this.
If you have a 0s cache, you sometimes don't need the data written to a table at all. You just need a transient list of contacts that doesn't need to persist past the end of the request. And it's pretty slow to use the cache table: the 'INSERT INTO' command is slow relative to the original SELECT query. This time can add up if you're querying a bunch of groups at once, or have nested smart groups, or have a lot of contacts in the groups.
So I was tentatively wondering if there could be an API4 'no-smart-group-cache' parameter, that just runs the underlying query and stores it all in memory. I imagine this would only work for API4 smart groups, as it's super-easy to get their underlying search criteria.
The default would remain to use the cache, for full backwards-compatibility. But the parameter could be used in custom code.
Two questions:
1. Anyone else interested in this?
2. How hard would it be / how much of a can of worms is it?https://lab.civicrm.org/dev/core/-/issues/4561On new mailing screen the widget for include/exclude double-escapes characters2023-09-13T12:22:17ZDaveDOn new mailing screen the widget for include/exclude double-escapes characters1. Create a group like "2023 Donated >$100". You don't have to actually create a search, just manually create a group with that title.
2. On the new mailing screen, try filtering by typing ">$100". It won't find it because the title disp...1. Create a group like "2023 Donated >$100". You don't have to actually create a search, just manually create a group with that title.
2. On the new mailing screen, try filtering by typing ">$100". It won't find it because the title displays as `>$100`. Searching for `gt;` will find it but that's silly.https://lab.civicrm.org/dev/core/-/issues/4558Proposal: setting to define alternative frontend URLs for when using a proxy2023-09-13T12:12:51ZbgmProposal: setting to define alternative frontend URLs for when using a proxyWhen CiviCRM runs behind another system, such as a reverse-proxy running on a separate domain that only exposes specific features, we do not have a way to define that URL.
For example:
- Staff access crm.example.org - the service is ru...When CiviCRM runs behind another system, such as a reverse-proxy running on a separate domain that only exposes specific features, we do not have a way to define that URL.
For example:
- Staff access crm.example.org - the service is running behind a VPN and not available to the general public
- However, they send mailings, which include unsubscribe and opt-out links. This feature is provided by a reverse-proxy that forwards those requests specifically.
[CRM/Mailing/BAO/Mailing.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Mailing/BAO/Mailing.php#L1020) defines a few URLs and uses `CRM_Utils_System::url` to generate the links. In the example I found, they were overriding the core PHP file, in order to hardcode the domain.
~~I'm thinking mailing [hook_civicrm_links](https://docs.civicrm.org/dev/en/latest/hooks/hook_civicrm_links/) might be a good semantic fit, even if it's a bit of an odd case, but it is used is weird places, such as in the Extension Manager.~~
- `op` = `mailing.content.actions` ? (mailing.actions could be confusing with actions in the UI, such as edit/delete)
- `objectID` = `job_id` ?
~~Where it gets weirder, is that the array of `$urls` would have to be packaged into the format that the hook expects, then unpackaged back into the expected format from the parent function (unless we just send it as-is, because the fluff is not necessary, and it's up to the hook to check for the `$op`).~~
An alternative could be to add a special `CIVICRM_UF_BASEURL`, but then it would mean more params for `CRM_Utils_System::url` and it seems already messy enough.
Edit: based on Jaap's comment, it probably makes more sense to find a solution for a "reverse-proxy URL".https://lab.civicrm.org/dev/core/-/issues/4557Proposal - Allow empty profile field labels instead of null2023-09-08T15:11:53ZshaneonabikeProposal - Allow empty profile field labels instead of nullPresently, when you create a Custom Profile and add fields you need add a label. In some cases, having a large question makes the label kind of awkward and crammed. It makes for a bad UX experience I believe.
So the workaround by most o...Presently, when you create a Custom Profile and add fields you need add a label. In some cases, having a large question makes the label kind of awkward and crammed. It makes for a bad UX experience I believe.
So the workaround by most of the designers I work with is to simply remove the ```label``` and then drop the question in the Pre text. This works just dandy, except that CiviCRM is adding ```null``` for empty texts. The only way to prevent this is to modify the label in the DB.
I would like to propose that we remove the code that is setting this text to null. I could make a quick patch. I realize the label is being used to create the actual field name, but in the case of the Profile the field name is already created.
![Selection_007](/uploads/48ab9decd2a6b1c3bb5a380a8278f12d/Selection_007.png)https://lab.civicrm.org/dev/core/-/issues/4552Add scheduled job form doesn't go to the right place after save or delete2023-09-06T09:36:22ZDaveDAdd scheduled job form doesn't go to the right place after save or deleteI'm not sure how often people use the UI to add scheduled jobs - I was just testing something and came across this. But what happens is it will create the job but then just go to a blank form. After deleting a scheduled job, it does dele...I'm not sure how often people use the UI to add scheduled jobs - I was just testing something and came across this. But what happens is it will create the job but then just go to a blank form. After deleting a scheduled job, it does delete it but just reloads the confirmation screen.https://lab.civicrm.org/dev/core/-/issues/4547SearchKit: Edit in place for display name should be possible2023-09-03T00:16:46ZlarsssandergreenSearchKit: Edit in place for display name should be possibleIf you have a contact id in a SK display, enabling edit in place will show an autocomplete with contact names when clicked, but the field will show the contact id before clicking, which is confusing UI. On the other hand, a display name ...If you have a contact id in a SK display, enabling edit in place will show an autocomplete with contact names when clicked, but the field will show the contact id before clicking, which is confusing UI. On the other hand, a display name field shows the contact name as desired, but doesn't allow editing. So if you want to have an edit in place and show the contact name, you need both a contact id field and a display name field, which is confusing for the end user.
![image](/uploads/76c8542e1f32afac6074e817252e87ba/image.png)
Why should I click on the random integer on the right to edit the contact, instead of the name on the left?
Could we either make it possible to edit in place on the display name or make it possible to show the display name for the contact id field? Or maybe change the contact id field so it shows the display name by default, rename it to just contact, and add a separate field just for showing the contact id (which is probably a fairly rare need)?https://lab.civicrm.org/dev/core/-/issues/4546SearchKit: Activity Contact Display and Sort Names don't show up2023-09-02T15:31:27ZlarsssandergreenSearchKit: Activity Contact Display and Sort Names don't show upIf you create a new SearchKit search for Activities and add fields `Added by Display Name`, `With Contacts Display Name` and ` Assigned to Display Name`, you'll see that these fields are always blank. Same for `Added by Sort Name` and so...If you create a new SearchKit search for Activities and add fields `Added by Display Name`, `With Contacts Display Name` and ` Assigned to Display Name`, you'll see that these fields are always blank. Same for `Added by Sort Name` and so on.
I think these were added fairly recently, but I suspect that was part of something more general and they never worked.
Adding the display name by adding a with Activity Contact works.https://lab.civicrm.org/dev/core/-/issues/4544Older sites missing index for `is_deceased` on `civicrm_contact`2023-09-06T13:56:12ZwmortadaOlder sites missing index for `is_deceased` on `civicrm_contact`This is a follow up from Eileen's comment about a missing index in https://lab.civicrm.org/dev/core/-/issues/4539#note_150479.
The index for `is_deceased` was added in CiviCRM 4.7 and is defined in [xml/schema/Contact/Contact.xml](https...This is a follow up from Eileen's comment about a missing index in https://lab.civicrm.org/dev/core/-/issues/4539#note_150479.
The index for `is_deceased` was added in CiviCRM 4.7 and is defined in [xml/schema/Contact/Contact.xml](https://github.com/civicrm/civicrm-core/blob/f068222ac25a5a393e9238b617ae91a4d9d29797/xml/schema/Contact/Contact.xml#L722-L726). However, there doesn't appear to be an upgrade task to add this index to existing sites. As a result any site installed before CiviCRM 4.7 is likely to be missing this index. Checking our client sites, I can see a mix of sites that do and don't have it.
The MR where this index was added is: https://github.com/civicrm/civicrm-core/pull/10489
There was a question from Tim about whether there should be an upgrader. There was a later comment from Eileen about whether this index should be removed.
So I guess the question is whether we want this index or not:
- if we **do** want this index we should add an upgrade task so that older sites have this index
- if we **don't** want this index we should add an upgrade task to remove it from sites that have ithttps://lab.civicrm.org/dev/core/-/issues/4540Email issues can cause new membership contributions to fail with pending/inco...2023-11-23T06:38:55ZTOCM_MMatthewsEmail issues can cause new membership contributions to fail with pending/incomplete.Overview
----------------------------------------
On a self hosted WordPress (currently 6.3) / CiviCRM (currently 5.64.0) site, when we had an SMTP server outage (not sending email from our virtual server directly, as that causes other s...Overview
----------------------------------------
On a self hosted WordPress (currently 6.3) / CiviCRM (currently 5.64.0) site, when we had an SMTP server outage (not sending email from our virtual server directly, as that causes other spam-related problems), any attempt to sign up as a member to our site would end in a failure state (contribution pending/incomplete), after processing payment but failing to send the receipt email. Could be related to the request timing out from php-fpm, but whatever that email timeout was, it was longer than 120s when I gave up trying to adjust php-fpm's timeout value. Settings within CiviCRM (outbound mail) have always just used mail(). But we use the WP Mail SMTP plugin for all WordPress related email to go through the trusted mailers directly. And when that was pointing to mail servers that timed out our connections, we'd see the pending/incomplete failures. Flipping that to default/none so it would use the host mailer as well, things started to work again (with the caveat of some emails not getting through because our server's IP will never be fully trusted by all the major mailers, given that it's within a virtual IP range that apparently triggered some alarms before).
https://civicrm.stackexchange.com/questions/45428/email-timeouts-breaking-contributions?noredirect=1#comment53603_45428
Reproduction steps
----------------------------------------
Set your WordPress mailer up to a host that times out the SMTP connection with WP Mail SMTP (we use the free version, non-profit, can't justify Pro upgrade). Key point probably being SMTP connection timeout being longer than php-fpm timeout, as the process terminates ungracefully.
Set up membership contribution page that sends credit card processing receipt. Maybe it was the password reset link from WordPress, but, can't see how?
Current behaviour
----------------------------------------
Users would not get any receipt or password reset email, transaction -- while processed via our provider (Stripe) would show as pending/incomplete transaction.
Expected behaviour
----------------------------------------
The transaction should continue. Maybe emails will get delayed/queued/lost depending on the mailer situation, but this should not break recording of a successful payment, so we'd just have to resend emails instead of go in and mark the transaction as complete after verifying payment has been received and then resend emails.
Environment information
----------------------------------------
* __Browser:__ Every. Testing confirmed with Safari 16.5.2 / Chrome 116.0.x on macOS Ventura 13.4.1(c).
* __CiviCRM:__ 5.64.0 at the time of SMTP failure
* __PHP:__ 8.0.30
* __CMS:__ WordPress 6.3
* __Database:__ MySQL 8.0.32
* __Web Server:__ Apache 2.4.57
Comments
----------------------------------------
_Anything else you would like the reviewer to note._https://lab.civicrm.org/dev/core/-/issues/4539Very slow query when sending mailings via CiviMail2023-09-12T05:08:13ZwmortadaVery slow query when sending mailings via CiviMailOverview
----------------------------------------
We have come across an issue where the database connection was timing out due to slow, long-running queries when the Send Scheduled Mailings job was running. We became aware of this when...Overview
----------------------------------------
We have come across an issue where the database connection was timing out due to slow, long-running queries when the Send Scheduled Mailings job was running. We became aware of this when we set a maximum execution time limit of 3 minutes for MySQL queries. Mailings for one of our clients were not being sent because the database connection was timing out.
As a workaround, we have increased the maximum execution time to 10 minutes and reduced the Mailer Batch Limit and Mailer Job Size to 1,000. However, we'd like to fix the underlying issue that the queries are running so slowly.
When sending a mailing CiviCRM runs a query of the form:
```sql
SELECT r.contact_id, r.email_id, r.phone_id
FROM civicrm_mailing_recipients r
INNER JOIN civicrm_contact c on
(c.id = r.contact_id
AND c.is_deleted = 0
AND c.is_deceased = 0
AND c.do_not_email = 0
AND c.is_opt_out = 0
)
INNER JOIN civicrm_email e ON (r.email_id = e.id AND e.on_hold = 0)
WHERE r.mailing_id = 3271
LIMIT 18000, 1000
```
There are no indices for `is_deceased`, `do_not_email` or `is_opt_out` so this query is very slow. (There is an index for `is_deleted`. There should be an index for `is_deceased` but this appears to be missing from this site.)
We have tested adding these indices in a development environment to see what difference it would make and this increase the speed of the query by a factor of one thousand:
- 20,000 rows with no indices = 80 seconds
- 20,000 rows with indices = 0.085 seconds
Reproduction steps
----------------------------------------
This is likely to only affect sites with a large number of contacts and large number of rows in the `civicrm_mailing_recipients` table (in this case 46,000 contacts and 9 million rows respectively).
1. Set a maximum execution time for MySQL of 3 minutes (say)
2. Create a mailing to a large group of contacts (>20,000)
3. Schedule the mailing to send
4. Send Scheduled Mailings job fails with error: "Finished execution of Send Scheduled Mailings with result: Failure, Error message: DB Error: unknown error"
Current behaviour
----------------------------------------
No indices ~~`is_deceased`~~, `do_not_email` or `is_opt_out` in `civicrm_contact` so queries run very slowly.
Expected behaviour
----------------------------------------
Indices for the above fields so that the query runs quickly.
Environment information
----------------------------------------
* __CiviCRM:__ _5.62_
* __PHP:__ _8.1__
* __CMS:__ _N/A_
* __Database:__ _MySQL 5.7.43_
Comments
----------------------------------------
There are other speed improvements in #4045 but these relate to the speed of the user interface rather than the speed of sending emails.
Perhaps this can also help to [reduce our collective CO2 emissions](https://civicrm.org/blog/systopia/beginning-green-how-can-we-make-civicrm-more-sustainable)?https://lab.civicrm.org/dev/core/-/issues/4536Report improvements2023-09-23T05:02:21ZyashodhaReport improvements_Primary Membership_ filter option in _Membership Details_ report has options that don't make sense.
![dddsdsd](/uploads/0e74060c879cbaf416c85e8eb1067134/dddsdsd.png)
Remove options that don't make sense and add option to choose _All_..._Primary Membership_ filter option in _Membership Details_ report has options that don't make sense.
![dddsdsd](/uploads/0e74060c879cbaf416c85e8eb1067134/dddsdsd.png)
Remove options that don't make sense and add option to choose _All_ as well.yashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/4535View Membership lists template contributions along with actual contribution u...2023-08-29T10:36:47ZRichView Membership lists template contributions along with actual contribution under "Related Contributions and Recurring Contributions"This is confusing. The status column is blank, too, so it looks at a glance as though this is a real payment:
![image](/uploads/61a1120ef3c1f389c2dda90bb0bd38f5/image.png)
I propose that template contributions are NOT shown on that pag...This is confusing. The status column is blank, too, so it looks at a glance as though this is a real payment:
![image](/uploads/61a1120ef3c1f389c2dda90bb0bd38f5/image.png)
I propose that template contributions are NOT shown on that page. Agree?https://lab.civicrm.org/dev/core/-/issues/4534Price set discount by date back office issues2023-08-24T18:33:09ZlarsssandergreenPrice set discount by date back office issuesThese are probably not a priority for anyone, but I ran into these while testing @eileen's recent PR, so thought I might as well note them:
1. Discount set shows as a select on edit participant, but only in certain circumstances (haven'...These are probably not a priority for anyone, but I ran into these while testing @eileen's recent PR, so thought I might as well note them:
1. Discount set shows as a select on edit participant, but only in certain circumstances (haven't quite worked this out, but something to do with number of discount sets and/or whether the dates are current or in the past). It also seems to be wrong, showing the current discount set based on the current date, not the one for the participant. Regardless, it shouldn't be changeable here as that does nothing. It should either always show as frozen or never show at all.
2. On the other hand, discount set should always be selectable on Change Selections, with the default set to the discount set that the participant currently has (then if you change from adult to student ticket or something, they still get the discount based on date, but you also can change the discount if needed).
3. Discount set is selectable when registering a single participant from back office, but not on the Register search task for multiple participants. It should be.https://lab.civicrm.org/dev/core/-/issues/4533Searchkit: "Add new" button text2023-08-24T17:22:23Zmattwiremjw@mjwconsult.co.ukSearchkit: "Add new" button textWhen you enable the "Add New" button you cannot customise the text and it uses the entity name eg. "Add Activity". But if you want to link to a form that does something more specific eg. to create a signature on a petition it would be mu...When you enable the "Add New" button you cannot customise the text and it uses the entity name eg. "Add Activity". But if you want to link to a form that does something more specific eg. to create a signature on a petition it would be much clearer if this button could be customised to say eg. "Add Signature".