CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-09-13T12:22:17Zhttps://lab.civicrm.org/dev/core/-/issues/4561On new mailing screen the widget for include/exclude double-escapes characters2023-09-13T12:22:17ZDaveDOn new mailing screen the widget for include/exclude double-escapes characters1. Create a group like "2023 Donated >$100". You don't have to actually create a search, just manually create a group with that title.
2. On the new mailing screen, try filtering by typing ">$100". It won't find it because the title disp...1. Create a group like "2023 Donated >$100". You don't have to actually create a search, just manually create a group with that title.
2. On the new mailing screen, try filtering by typing ">$100". It won't find it because the title displays as `>$100`. Searching for `gt;` will find it but that's silly.https://lab.civicrm.org/dev/core/-/issues/4558Proposal: setting to define alternative frontend URLs for when using a proxy2023-09-13T12:12:51ZbgmProposal: setting to define alternative frontend URLs for when using a proxyWhen CiviCRM runs behind another system, such as a reverse-proxy running on a separate domain that only exposes specific features, we do not have a way to define that URL.
For example:
- Staff access crm.example.org - the service is ru...When CiviCRM runs behind another system, such as a reverse-proxy running on a separate domain that only exposes specific features, we do not have a way to define that URL.
For example:
- Staff access crm.example.org - the service is running behind a VPN and not available to the general public
- However, they send mailings, which include unsubscribe and opt-out links. This feature is provided by a reverse-proxy that forwards those requests specifically.
[CRM/Mailing/BAO/Mailing.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Mailing/BAO/Mailing.php#L1020) defines a few URLs and uses `CRM_Utils_System::url` to generate the links. In the example I found, they were overriding the core PHP file, in order to hardcode the domain.
~~I'm thinking mailing [hook_civicrm_links](https://docs.civicrm.org/dev/en/latest/hooks/hook_civicrm_links/) might be a good semantic fit, even if it's a bit of an odd case, but it is used is weird places, such as in the Extension Manager.~~
- `op` = `mailing.content.actions` ? (mailing.actions could be confusing with actions in the UI, such as edit/delete)
- `objectID` = `job_id` ?
~~Where it gets weirder, is that the array of `$urls` would have to be packaged into the format that the hook expects, then unpackaged back into the expected format from the parent function (unless we just send it as-is, because the fluff is not necessary, and it's up to the hook to check for the `$op`).~~
An alternative could be to add a special `CIVICRM_UF_BASEURL`, but then it would mean more params for `CRM_Utils_System::url` and it seems already messy enough.
Edit: based on Jaap's comment, it probably makes more sense to find a solution for a "reverse-proxy URL".https://lab.civicrm.org/dev/core/-/issues/4557Proposal - Allow empty profile field labels instead of null2023-09-08T15:11:53ZshaneonabikeProposal - Allow empty profile field labels instead of nullPresently, when you create a Custom Profile and add fields you need add a label. In some cases, having a large question makes the label kind of awkward and crammed. It makes for a bad UX experience I believe.
So the workaround by most o...Presently, when you create a Custom Profile and add fields you need add a label. In some cases, having a large question makes the label kind of awkward and crammed. It makes for a bad UX experience I believe.
So the workaround by most of the designers I work with is to simply remove the ```label``` and then drop the question in the Pre text. This works just dandy, except that CiviCRM is adding ```null``` for empty texts. The only way to prevent this is to modify the label in the DB.
I would like to propose that we remove the code that is setting this text to null. I could make a quick patch. I realize the label is being used to create the actual field name, but in the case of the Profile the field name is already created.
![Selection_007](/uploads/48ab9decd2a6b1c3bb5a380a8278f12d/Selection_007.png)https://lab.civicrm.org/dev/core/-/issues/4552Add scheduled job form doesn't go to the right place after save or delete2023-09-06T09:36:22ZDaveDAdd scheduled job form doesn't go to the right place after save or deleteI'm not sure how often people use the UI to add scheduled jobs - I was just testing something and came across this. But what happens is it will create the job but then just go to a blank form. After deleting a scheduled job, it does dele...I'm not sure how often people use the UI to add scheduled jobs - I was just testing something and came across this. But what happens is it will create the job but then just go to a blank form. After deleting a scheduled job, it does delete it but just reloads the confirmation screen.https://lab.civicrm.org/dev/core/-/issues/4547SearchKit: Edit in place for display name should be possible2023-09-03T00:16:46ZlarsssandergreenSearchKit: Edit in place for display name should be possibleIf you have a contact id in a SK display, enabling edit in place will show an autocomplete with contact names when clicked, but the field will show the contact id before clicking, which is confusing UI. On the other hand, a display name ...If you have a contact id in a SK display, enabling edit in place will show an autocomplete with contact names when clicked, but the field will show the contact id before clicking, which is confusing UI. On the other hand, a display name field shows the contact name as desired, but doesn't allow editing. So if you want to have an edit in place and show the contact name, you need both a contact id field and a display name field, which is confusing for the end user.
![image](/uploads/76c8542e1f32afac6074e817252e87ba/image.png)
Why should I click on the random integer on the right to edit the contact, instead of the name on the left?
Could we either make it possible to edit in place on the display name or make it possible to show the display name for the contact id field? Or maybe change the contact id field so it shows the display name by default, rename it to just contact, and add a separate field just for showing the contact id (which is probably a fairly rare need)?https://lab.civicrm.org/dev/core/-/issues/4546SearchKit: Activity Contact Display and Sort Names don't show up2023-09-02T15:31:27ZlarsssandergreenSearchKit: Activity Contact Display and Sort Names don't show upIf you create a new SearchKit search for Activities and add fields `Added by Display Name`, `With Contacts Display Name` and ` Assigned to Display Name`, you'll see that these fields are always blank. Same for `Added by Sort Name` and so...If you create a new SearchKit search for Activities and add fields `Added by Display Name`, `With Contacts Display Name` and ` Assigned to Display Name`, you'll see that these fields are always blank. Same for `Added by Sort Name` and so on.
I think these were added fairly recently, but I suspect that was part of something more general and they never worked.
Adding the display name by adding a with Activity Contact works.https://lab.civicrm.org/dev/core/-/issues/4544Older sites missing index for `is_deceased` on `civicrm_contact`2023-09-06T13:56:12ZwmortadaOlder sites missing index for `is_deceased` on `civicrm_contact`This is a follow up from Eileen's comment about a missing index in https://lab.civicrm.org/dev/core/-/issues/4539#note_150479.
The index for `is_deceased` was added in CiviCRM 4.7 and is defined in [xml/schema/Contact/Contact.xml](https...This is a follow up from Eileen's comment about a missing index in https://lab.civicrm.org/dev/core/-/issues/4539#note_150479.
The index for `is_deceased` was added in CiviCRM 4.7 and is defined in [xml/schema/Contact/Contact.xml](https://github.com/civicrm/civicrm-core/blob/f068222ac25a5a393e9238b617ae91a4d9d29797/xml/schema/Contact/Contact.xml#L722-L726). However, there doesn't appear to be an upgrade task to add this index to existing sites. As a result any site installed before CiviCRM 4.7 is likely to be missing this index. Checking our client sites, I can see a mix of sites that do and don't have it.
The MR where this index was added is: https://github.com/civicrm/civicrm-core/pull/10489
There was a question from Tim about whether there should be an upgrader. There was a later comment from Eileen about whether this index should be removed.
So I guess the question is whether we want this index or not:
- if we **do** want this index we should add an upgrade task so that older sites have this index
- if we **don't** want this index we should add an upgrade task to remove it from sites that have ithttps://lab.civicrm.org/dev/core/-/issues/4540Email issues can cause new membership contributions to fail with pending/inco...2023-11-23T06:38:55ZTOCM_MMatthewsEmail issues can cause new membership contributions to fail with pending/incomplete.Overview
----------------------------------------
On a self hosted WordPress (currently 6.3) / CiviCRM (currently 5.64.0) site, when we had an SMTP server outage (not sending email from our virtual server directly, as that causes other s...Overview
----------------------------------------
On a self hosted WordPress (currently 6.3) / CiviCRM (currently 5.64.0) site, when we had an SMTP server outage (not sending email from our virtual server directly, as that causes other spam-related problems), any attempt to sign up as a member to our site would end in a failure state (contribution pending/incomplete), after processing payment but failing to send the receipt email. Could be related to the request timing out from php-fpm, but whatever that email timeout was, it was longer than 120s when I gave up trying to adjust php-fpm's timeout value. Settings within CiviCRM (outbound mail) have always just used mail(). But we use the WP Mail SMTP plugin for all WordPress related email to go through the trusted mailers directly. And when that was pointing to mail servers that timed out our connections, we'd see the pending/incomplete failures. Flipping that to default/none so it would use the host mailer as well, things started to work again (with the caveat of some emails not getting through because our server's IP will never be fully trusted by all the major mailers, given that it's within a virtual IP range that apparently triggered some alarms before).
https://civicrm.stackexchange.com/questions/45428/email-timeouts-breaking-contributions?noredirect=1#comment53603_45428
Reproduction steps
----------------------------------------
Set your WordPress mailer up to a host that times out the SMTP connection with WP Mail SMTP (we use the free version, non-profit, can't justify Pro upgrade). Key point probably being SMTP connection timeout being longer than php-fpm timeout, as the process terminates ungracefully.
Set up membership contribution page that sends credit card processing receipt. Maybe it was the password reset link from WordPress, but, can't see how?
Current behaviour
----------------------------------------
Users would not get any receipt or password reset email, transaction -- while processed via our provider (Stripe) would show as pending/incomplete transaction.
Expected behaviour
----------------------------------------
The transaction should continue. Maybe emails will get delayed/queued/lost depending on the mailer situation, but this should not break recording of a successful payment, so we'd just have to resend emails instead of go in and mark the transaction as complete after verifying payment has been received and then resend emails.
Environment information
----------------------------------------
* __Browser:__ Every. Testing confirmed with Safari 16.5.2 / Chrome 116.0.x on macOS Ventura 13.4.1(c).
* __CiviCRM:__ 5.64.0 at the time of SMTP failure
* __PHP:__ 8.0.30
* __CMS:__ WordPress 6.3
* __Database:__ MySQL 8.0.32
* __Web Server:__ Apache 2.4.57
Comments
----------------------------------------
_Anything else you would like the reviewer to note._https://lab.civicrm.org/dev/core/-/issues/4539Very slow query when sending mailings via CiviMail2023-09-12T05:08:13ZwmortadaVery slow query when sending mailings via CiviMailOverview
----------------------------------------
We have come across an issue where the database connection was timing out due to slow, long-running queries when the Send Scheduled Mailings job was running. We became aware of this when...Overview
----------------------------------------
We have come across an issue where the database connection was timing out due to slow, long-running queries when the Send Scheduled Mailings job was running. We became aware of this when we set a maximum execution time limit of 3 minutes for MySQL queries. Mailings for one of our clients were not being sent because the database connection was timing out.
As a workaround, we have increased the maximum execution time to 10 minutes and reduced the Mailer Batch Limit and Mailer Job Size to 1,000. However, we'd like to fix the underlying issue that the queries are running so slowly.
When sending a mailing CiviCRM runs a query of the form:
```sql
SELECT r.contact_id, r.email_id, r.phone_id
FROM civicrm_mailing_recipients r
INNER JOIN civicrm_contact c on
(c.id = r.contact_id
AND c.is_deleted = 0
AND c.is_deceased = 0
AND c.do_not_email = 0
AND c.is_opt_out = 0
)
INNER JOIN civicrm_email e ON (r.email_id = e.id AND e.on_hold = 0)
WHERE r.mailing_id = 3271
LIMIT 18000, 1000
```
There are no indices for `is_deceased`, `do_not_email` or `is_opt_out` so this query is very slow. (There is an index for `is_deleted`. There should be an index for `is_deceased` but this appears to be missing from this site.)
We have tested adding these indices in a development environment to see what difference it would make and this increase the speed of the query by a factor of one thousand:
- 20,000 rows with no indices = 80 seconds
- 20,000 rows with indices = 0.085 seconds
Reproduction steps
----------------------------------------
This is likely to only affect sites with a large number of contacts and large number of rows in the `civicrm_mailing_recipients` table (in this case 46,000 contacts and 9 million rows respectively).
1. Set a maximum execution time for MySQL of 3 minutes (say)
2. Create a mailing to a large group of contacts (>20,000)
3. Schedule the mailing to send
4. Send Scheduled Mailings job fails with error: "Finished execution of Send Scheduled Mailings with result: Failure, Error message: DB Error: unknown error"
Current behaviour
----------------------------------------
No indices ~~`is_deceased`~~, `do_not_email` or `is_opt_out` in `civicrm_contact` so queries run very slowly.
Expected behaviour
----------------------------------------
Indices for the above fields so that the query runs quickly.
Environment information
----------------------------------------
* __CiviCRM:__ _5.62_
* __PHP:__ _8.1__
* __CMS:__ _N/A_
* __Database:__ _MySQL 5.7.43_
Comments
----------------------------------------
There are other speed improvements in #4045 but these relate to the speed of the user interface rather than the speed of sending emails.
Perhaps this can also help to [reduce our collective CO2 emissions](https://civicrm.org/blog/systopia/beginning-green-how-can-we-make-civicrm-more-sustainable)?https://lab.civicrm.org/dev/core/-/issues/4536Report improvements2023-09-23T05:02:21ZyashodhaReport improvements_Primary Membership_ filter option in _Membership Details_ report has options that don't make sense.
![dddsdsd](/uploads/0e74060c879cbaf416c85e8eb1067134/dddsdsd.png)
Remove options that don't make sense and add option to choose _All_..._Primary Membership_ filter option in _Membership Details_ report has options that don't make sense.
![dddsdsd](/uploads/0e74060c879cbaf416c85e8eb1067134/dddsdsd.png)
Remove options that don't make sense and add option to choose _All_ as well.yashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/4535View Membership lists template contributions along with actual contribution u...2023-08-29T10:36:47ZRichView Membership lists template contributions along with actual contribution under "Related Contributions and Recurring Contributions"This is confusing. The status column is blank, too, so it looks at a glance as though this is a real payment:
![image](/uploads/61a1120ef3c1f389c2dda90bb0bd38f5/image.png)
I propose that template contributions are NOT shown on that pag...This is confusing. The status column is blank, too, so it looks at a glance as though this is a real payment:
![image](/uploads/61a1120ef3c1f389c2dda90bb0bd38f5/image.png)
I propose that template contributions are NOT shown on that page. Agree?https://lab.civicrm.org/dev/core/-/issues/4534Price set discount by date back office issues2023-08-24T18:33:09ZlarsssandergreenPrice set discount by date back office issuesThese are probably not a priority for anyone, but I ran into these while testing @eileen's recent PR, so thought I might as well note them:
1. Discount set shows as a select on edit participant, but only in certain circumstances (haven'...These are probably not a priority for anyone, but I ran into these while testing @eileen's recent PR, so thought I might as well note them:
1. Discount set shows as a select on edit participant, but only in certain circumstances (haven't quite worked this out, but something to do with number of discount sets and/or whether the dates are current or in the past). It also seems to be wrong, showing the current discount set based on the current date, not the one for the participant. Regardless, it shouldn't be changeable here as that does nothing. It should either always show as frozen or never show at all.
2. On the other hand, discount set should always be selectable on Change Selections, with the default set to the discount set that the participant currently has (then if you change from adult to student ticket or something, they still get the discount based on date, but you also can change the discount if needed).
3. Discount set is selectable when registering a single participant from back office, but not on the Register search task for multiple participants. It should be.https://lab.civicrm.org/dev/core/-/issues/4533Searchkit: "Add new" button text2023-08-24T17:22:23Zmattwiremjw@mjwconsult.co.ukSearchkit: "Add new" button textWhen you enable the "Add New" button you cannot customise the text and it uses the entity name eg. "Add Activity". But if you want to link to a form that does something more specific eg. to create a signature on a petition it would be mu...When you enable the "Add New" button you cannot customise the text and it uses the entity name eg. "Add Activity". But if you want to link to a form that does something more specific eg. to create a signature on a petition it would be much clearer if this button could be customised to say eg. "Add Signature".https://lab.civicrm.org/dev/core/-/issues/4528Entity Reference custom fields: Configurable delete behavior2024-01-17T22:28:23ZjensschuppeEntity Reference custom fields: Configurable delete behavior## Overview
Currently, when entities referenced in a custom field of the type _Entity Reference_ are deleted, the entity reference field gets cleared (set to `NULL`). This is easy and prevents us from database inconsistencies. But this ...## Overview
Currently, when entities referenced in a custom field of the type _Entity Reference_ are deleted, the entity reference field gets cleared (set to `NULL`). This is easy and prevents us from database inconsistencies. But this can lead to problems:
* inconsistencies when the entity reference field is set to be required
* stale/orphaned entities the entity reference field is attached to
* `civicrm_value_*` records with all-empty columns (or are they being taken care of already)
## Example use-case
Imagine a custom entity named _Qualification_ with (among others) an entity reference custom field referencing a _Contact_ entity, which is set to be required; read: A _Contact_ has multiple _Qualifications_.
## Current behaviour
When deleting the contact, the _Qualification_ entity will not be deleted until done so manually. The reference is just being set to `NULL` instead. Those entities will be meaningless without a referenced contact, the required entity reference custom field will be empty, which breaks the semantic contract.
Question: Is the `SET NULL` behavior a FK constraint on the DB table or part of some logic in Core code?
## Proposed behaviour
When deleting the contact, the _Qualification_ entity should be deleted. This is what we'd have configured the entity reference field to behave, as in other scenarios, the current behavior will be totally fine.
So, we'll need a configuration option for _Entity Reference_ custom fields for selecting what should happen upon deletion of the referenced entity. Available options could be (basically MySQL foreign key constraints):
* Remove Reference (`SET NULL`) - current behavior, maybe also the default option
* Cascade Delete (`CASCADE`) - removes the _`civicrm_value_*`_ record and the entity the custom group is attached to
* Restrict Delete (`RESTRICT`) - just as you can't e. g. delete contacts with financial transactions
* Set Default (`SET DEFAULT`) - to be configured on the custom field
* ~~Keep Reference (`NO ACTION`)~~ - might not be a good idea
## Comments
@colemanw and @michaelmcandrew might be interested.
Currently, this can be partly achieved by implementing a `preDelete` event for retrieving and deleting referencing entities.https://lab.civicrm.org/dev/core/-/issues/4527End Date field in Relationship cannot filter on date in join clause APIv42023-08-24T21:58:32ZseamusleeEnd Date field in Relationship cannot filter on date in join clause APIv4In APIv4 when joining onto relationships for example from contact.get you cannot use a date value e.g. 2023-08-23 you get unknown field error. However if you put the string into quotes it works
https://dmaster.demo.civicrm.org/civicrm/a...In APIv4 when joining onto relationships for example from contact.get you cannot use a date value e.g. 2023-08-23 you get unknown field error. However if you put the string into quotes it works
https://dmaster.demo.civicrm.org/civicrm/api4#/explorer/Contact/get?join=%5B%5B%22Relationship%20AS%20relationship%22,%22LEFT%22,null,%5B%22id%22,%22%3D%22,%22relationship.contact_id_a%22%5D,%5B%22relationship.end_date%22,%22%3E%22,%222023-08-23%22%5D%5D%5Dhttps://lab.civicrm.org/dev/core/-/issues/4526Unselecting price set discount set for back office event registration disappe...2023-08-23T02:51:57ZlarsssandergreenUnselecting price set discount set for back office event registration disappears fee block1. Add a discount set to a quick config price set for an event
2. Add new event registration from back office
3. Unselect the default discount set (which you would do if you wanted to charge the person full price)
The fee block disappea...1. Add a discount set to a quick config price set for an event
2. Add new event registration from back office
3. Unselect the default discount set (which you would do if you wanted to charge the person full price)
The fee block disappears, you cannot select any price options. The discount set is automatically re-selected, but the price options do not re-appear.https://lab.civicrm.org/dev/core/-/issues/4525SearchKit: If using Group by Display Name, selecting one contact selects them...2023-08-22T23:47:57ZlarsssandergreenSearchKit: If using Group by Display Name, selecting one contact selects them all (sort of)See https://civicrm.stackexchange.com/questions/45404/searchkit-deselect-with-group-by-option-and-how-to-save-the-sorting
To reproduce:
1. Search Contacts in SK
2. Group by Display Name
3. Select one contact from the table
Observe tha...See https://civicrm.stackexchange.com/questions/45404/searchkit-deselect-with-group-by-option-and-how-to-save-the-sorting
To reproduce:
1. Search Contacts in SK
2. Group by Display Name
3. Select one contact from the table
Observe that all the contacts are selected, though it says only 1 of N.https://lab.civicrm.org/dev/core/-/issues/4522Membership Detail improvements2023-08-24T14:02:52ZyashodhaMembership Detail improvementsIn the _Membership Details_ report,
add in _Columns_ tab:
- Auto-renew (Yes/no)
- Auto-renew Status
add in _Sorting_ tab:
- Start Date (of membership)
- End Date (of membership)
- State
- Auto-renew
- Auto-renew StatusIn the _Membership Details_ report,
add in _Columns_ tab:
- Auto-renew (Yes/no)
- Auto-renew Status
add in _Sorting_ tab:
- Start Date (of membership)
- End Date (of membership)
- State
- Auto-renew
- Auto-renew Statusyashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/4520Simplify and improve offline event receipt template contribution section details2023-09-23T22:55:28ZlarsssandergreenSimplify and improve offline event receipt template contribution section detailsAn offline event receipt includes the following (in this case, for a pending (pay later) registration):
![image](/uploads/67eefa5dc465c6ee31b71d7523f2c20c/image.png)
Here's a non-pending one:
![image](/uploads/954f9780ed01de7e46519f6b...An offline event receipt includes the following (in this case, for a pending (pay later) registration):
![image](/uploads/67eefa5dc465c6ee31b71d7523f2c20c/image.png)
Here's a non-pending one:
![image](/uploads/954f9780ed01de7e46519f6b36f2dc2b/image.png)
1. I don't think we need to include Registration Date. I don't know when this would be useful information for the recipient. It is almost always going to be the date of the email and even if it isn't (which I think is only possible if you manually change the date while inputting the registration), what purpose does it serve for the recipient? Note that we have Transaction Date just below this and they are going to be the same almost all the time. If it is important to someone that we include both we could do so if they are different, but I don't really see why we need to include Registration Date in an email receipt.
1. Paid By: This should not be included if no payment was recorded, as it is misleading in that case.
1. Total Amount: This should be included and we should show a balance if no payment has been made and the status is pending (pay later) - but not if the status is otherwise, because then we can assume that we aren't charging the person. In this case, we shouldn't include a total at all.
1. Pay later text. This should be included if we register the person as pending (pay later) and this text exists. It looks like there is some attempt to include this in the template, but it doesn't seem to work - I think is_pay_later may not be set for offline registrations.
2. Financial Type: Do we need to include this? In general, I think it doesn't add anything, but I suppose it could be useful to differentiate donations from non-donations. Seems like that could also be misleading though, as only part of the contribution may be tax deductible. I'd lean on the side of removing this.https://lab.civicrm.org/dev/core/-/issues/4519Search kit - if id is not present then the 'select' is misleading2023-09-02T05:12:00ZeileenSearch kit - if id is not present then the 'select' is misleadingIf you create an email search with no `Email ID` field (eg this url civicrm/admin/search#/create/Email?label=emails&params=%7B"version":4,"select":%5B"email","contact_id"%5D,"orderBy":%7B%7D,"where":%5B%5D,"groupBy":%5B%5D,"join":%5B%5D,...If you create an email search with no `Email ID` field (eg this url civicrm/admin/search#/create/Email?label=emails¶ms=%7B"version":4,"select":%5B"email","contact_id"%5D,"orderBy":%7B%7D,"where":%5B%5D,"groupBy":%5B%5D,"join":%5B%5D,"having":%5B%5D%7D )
and then search & select a row they all display as selected - but note that the text says "1 selected of 202 results"
![image](/uploads/a288959b13e588c1cfa9cbfcb29367d9/image.png)
If you take an action the 1 appears
![image](/uploads/fe31fe6460047573d1c832fb89b0e6b7/image.png)
But, it actually acts on all of them for download & seemingly none for the other actions
@colemanw @larssg this is per chat - @larssg thought https://civicrm.stackexchange.com/questions/45404/searchkit-deselect-with-group-by-option-and-how-to-save-the-sorting might relate