Development issueshttps://lab.civicrm.org/groups/dev/-/issues2023-07-21T09:57:50Zhttps://lab.civicrm.org/dev/core/-/issues/4434Don't allow uploading attachments to bulk mailings if they exceed the size limit2023-07-21T09:57:50ZDaveDDon't allow uploading attachments to bulk mailings if they exceed the size limitThere's been a couple times on multiple sites where the mailing bounces and it's not clear to the average office staffer why, but it's because they've uploaded a 30MB pdf to the mailing.
It should just not allow doing that in the first ...There's been a couple times on multiple sites where the mailing bounces and it's not clear to the average office staffer why, but it's because they've uploaded a 30MB pdf to the mailing.
It should just not allow doing that in the first place if the config settings have a smaller limit.https://lab.civicrm.org/dev/core/-/issues/4433E2E_Core_PathUrlTest::testGetUrl_WpAdmin() fails because CiviCRM routing is c...2023-07-24T20:52:34ZtottenE2E_Core_PathUrlTest::testGetUrl_WpAdmin() fails because CiviCRM routing is confusingThe gist of the test: it calls `cv url civicrm/contribute?reset=1` and asserts that the URL will open in the WordPress backend UI (aka `/wp-admin/`).
([Full source](https://github.com/civicrm/civicrm-core/blob/8e0dabd20bebe55d5dd725f301...The gist of the test: it calls `cv url civicrm/contribute?reset=1` and asserts that the URL will open in the WordPress backend UI (aka `/wp-admin/`).
([Full source](https://github.com/civicrm/civicrm-core/blob/8e0dabd20bebe55d5dd725f3015c41019cb8dbed/tests/phpunit/E2E/Core/PathUrlTest.php#L94-L117))
The test is failing. There are currently two patches:
* [25476](https://github.com/civicrm/civicrm-core/pull/25476): Automatically choose frontend/backend based on the route metadata
* [26772](https://github.com/civicrm/civicrm-core/pull/26772): Change the test to specifically request backend
Both have issues. So here's a deep-dive into the context.
Background
----------
* CiviCRM runs on Drupal/Backdrop and WordPress/Joomla.
* On WordPress/Joomla, the "frontend" and "backend" are different sub-applications (e.g. `/` vs `/wp-admin/`). The applications have very different URL structures. To make a hyperlink, you first decide which sub-application to target -- and then pick a page within it.
* On Drupal/Backdrop, it's one application, and all URLs have similar structure. To make a hyperlink, you simply identify the page.
* (*Drupal/Backdrop UX can sometimes distinguish frontend/backend -- but it's a visual choice based on local configuration. It's not a structural part of the URL.*)
* CiviCRM's routing system integrates into every UF/CMS, but (internally) it is closer to Drupal's. There is one `civicrm_menu` with all frontend+backend pages.
* On Drupal/Backdrop, CiviCRM's routes pass directly to CMS routes.
* On WordPress/Joomla, CiviCRM's routing integrates with both subapplications (ie "frontend" and "backend").
* CiviCRM stores a flag `bool is_public` for each route.
* CiviCRM includes routes which can be classified as:
* Purely backend (ex: `civicrm/dashboard`)
* Purely frontend (ex: `civicrm/event/register`)
* Purely web-service (ex: `civicrm/payment/ipn`)
* Multi-homed
* Ex: `civicrm/profile/view` has use-cases for frontend and backend
* Ex: `civicrm/ajax/api4/%` has use-cases for frontend, backend, and web-service
* CiviCRM has a function `CRM_Utils_System::url()`
* At first glance, it resembles Drupal's `\url()`. You typically just give the page (e.g. `url('civicrm/foo/bar')`).
* Over time, several additional parameters were added -- notably, the 6th parameter `bool $frontend` and the 7th parameter `bool $forceBackend`.
Problems
--------
* The status-quo invites bugs between CMS's.
* A developer working on Drupal will have trouble recognizing the importance of the 6th and 7th parameters. (*Those params do nothing on Drupal - and they're buried at the end of method.*)
* The status-quo invites bugs between interactive and automatic processes.
* A developer defines a custom token and tests it interactively; then at runtime, the token is sent by an automatic process. But the interactive/automatic split is orthogonal to frontend/backend split. Sometimes, interactive/automatic agree with each other (*both frontend or both backend*), and sometimes they disagree (*one frontend, one backend*).
* Overall, what tends to happen is:
* Developer writes a call like `CRM_Utils_System::url('civicrm/foo/bar')`, and it looks pretty.
* They test, and it works beautifully on their system.
* They publish, and it fails on other systems.
* Someone writes a patch to add 5 more parameters (`url('civicrm/foo/bar', '', FALSE, NULL, TRUE, TRUE)`). And then it's OK.
* The DX is awkward and invites bugs (in core and contrib) -- but it becomes more annoying for `cv` UX.
* If you need a 5th/6th param in PHP code, then you'll eventually figure it out and commit the update to your codebase. Then you forget about it.
* `cv` has some commands to facilitate manual testing and E2E testing (ie `cv url`, `cv http`, `cv open`). These are things that you improvise. Mismatched URLs can be annoying anytime you use these subcommands.
What to do
----------
There are two PRs to fix the test, and honestly - I don't really like either.
* [26772](https://github.com/civicrm/civicrm-core/pull/26772) makes the red mark go away, but it leaves the underlying issue (poor DX for PHP and poor UX for cv).
* [25476](https://github.com/civicrm/civicrm-core/pull/25476) aims to fix the underlying issue, but it's probably too facile. The current metadata can distinguish "purely frontend" pages from "purely backend" pages, but it cannot recognize "purely web-service" or "multi-homed". It probably messes-up some scenarios for those.
* Fixing this probably requires some more aggressive transition in the contract.
* Ex: Improve the routing metadata so that we can distinguish web-service routes and multi-home routes.
* Ex: Define a different class or function for generating URLs (*with a more usable signature*).
* Another option is to leave `civicrm-core` as-is -- and only update `cv`.
* Ex: Give `cv` the mechanism to resolve a frontend/backend based on metadata.
* Ex: Change `cv` to complain if you don't a specify frontend/backend flag.
* Either way, it feels like a bit of a wasted opportunity to only patch `cv` when we know that other users of `CRM_Utils_System::url()` get confused about the frontend/backend flags.https://lab.civicrm.org/dev/core/-/issues/4432Permission system can be bypassed from the search results action menu2023-07-21T09:59:05ZschorschiiPermission system can be bypassed from the search results action menuOverview
----------------------------------------
I created a simple permission structure where a group of CiviCRM users ("Group A") has write access to a group of contacts ("Group B"). (Every logged in user can read all contacts in our ...Overview
----------------------------------------
I created a simple permission structure where a group of CiviCRM users ("Group A") has write access to a group of contacts ("Group B"). (Every logged in user can read all contacts in our system.)
When a contact is now added to "Group B", users of "Group A" see the edit button on the contact and can add/remove the contact to/from groups on the contact detail page. When removing the contact from "Group B", the edit button disappears. So far, everything as expected.
But when using the actions menu from the search results, users can add/remove group assignments of a contact which is not in "Group B".
Isn't this an inconsistency in the permission system? Or am I missing something? How to avoid group membership changes of contacts which are not in "Group B" by users which are in "Group A"?
Reproduction steps
----------------------------------------
1. Create a role, assign it to "Group A" and create ACL "edit" for "Group B".
2. Log in with a non-admin user which is member of "Group A".
3. Search a contact which is not member of "Group B". Select it in the search results and choose "Group - add contact" or "Group - remove contact".
Current behaviour
----------------------------------------
I can change the group membership of this contact from within the action menu of the search result list.
Expected behaviour
----------------------------------------
Group membership changes should be refused since the contact is not member of "Group B".
When opening the contact's detail view, it works as expected, which means I'm not able to change the group memberships there.
Environment information
----------------------------------------
CiviCRM version 5.63.1 under WordPress
[Corresponding question on StackExchange](https://civicrm.stackexchange.com/questions/45266/permission-system-not-working-as-expected)https://lab.civicrm.org/dev/core/-/issues/4431API4: "!=" has unexpected results on NULL fields2023-07-21T09:59:23ZJonGoldAPI4: "!=" has unexpected results on NULL fieldsOverview
----------------------------------------
Using the `!=` operator when some of the values are `NULL` does not return the `NULL` records.
Reproduction steps
----------------------------------------
1. On a demo site (which by def...Overview
----------------------------------------
Using the `!=` operator when some of the values are `NULL` does not return the `NULL` records.
Reproduction steps
----------------------------------------
1. On a demo site (which by default has no contacts with a subtype), search for contacts that are not of subtype "Parent".
E.g.:
```php
\Civi\Api4\Contact::get(TRUE)
->addWhere('contact_sub_type', '!=', 'Parent')
->execute();
```
Current behaviour
----------------------------------------
No results are returned.
Expected behaviour
----------------------------------------
All results are returned, since no record is a Parent.
Comments
----------------------------------------
I thought at some point, `!=` internally generated `!= and IS NOT NULL` but maybe that was somewhere else.
I just learned about the MySQL null-safe equals operator, which would also solve this problem: https://stackoverflow.com/a/44723097/2832108https://lab.civicrm.org/dev/core/-/issues/4428Membership Fee token error in automatic membership renewal messages2023-09-24T22:49:30ZbwheelerMembership Fee token error in automatic membership renewal messagesOverview
----------------------------------------
This is related to another issue, https://lab.civicrm.org/dev/core/-/issues/3805 which was posted in https://civicrm.stackexchange.com/questions/42438/error-on-membership-fee-token-when-u...Overview
----------------------------------------
This is related to another issue, https://lab.civicrm.org/dev/core/-/issues/3805 which was posted in https://civicrm.stackexchange.com/questions/42438/error-on-membership-fee-token-when-using-print-merge-document/42443#42443
Reproduction steps
----------------------------------------
The original ticket has these reproduction steps:
I did a short test on the demo sandbox.
Selected "Find Membership" and selected one record. I then choose Print/Merge Document and added 2 tokens, {membership.id} and {membership.fee} Then clicked "Preview" to see the pdf outcome.
If I use only the token {membership.id} I do not get any error message and the preview is created.
Adding {membership.fee} I get the following error message below.
You can also reproduce the issue by creating an automatic membership renewal message with {membership.fee} in it and trying to send the renewal message.
Current behaviour
----------------------------------------
In the current version of Civi, the code will run but it will show 0.00 for the membership fee instead of the actual membership fee.
Expected behaviour
----------------------------------------
It should show the correct membership fee.
We propose fixing this with the following code. This won't match exactly the latest version of Civi because we're working off 5.58.1 but if it looks good, we'll submit a review with the latest version of Civi.
```diff
+++ b/sites/all/modules/civicrm/CRM/Member/Tokens.php
@@ -61,8 +61,17 @@ class CRM_Member_Tokens extends CRM_Core_EntityTokens {
*/
public function evaluateToken(\Civi\Token\TokenRow $row, $entity, $field, $prefetch = NULL) {
if ($field === 'fee') {
- $membershipType = CRM_Member_BAO_MembershipType::getMembershipType($this->getFieldValue($row, 'membership_type_id'));
- $row->tokens($entity, $field, \CRM_Utils_Money::formatLocaleNumericRoundedForDefaultCurrency($membershipType['minimum_fee']));
+ $membershipTypeId = $this->getFieldValue($row, 'membership_type_id');
+ if (empty($membershipTypeId) && isset($row->context['membershipId'])) {
+ $membership = CRM_Member_BAO_Membership::findById($row->context['membershipId']);
+ $membershipTypeId = $membership->membership_type_id;
+ }
+ $membershipType = CRM_Member_BAO_MembershipType::getMembershipType($membershipTypeId);
+ $minimumFee = 0;
+ if ($membershipType) {
+ $minimumFee = $membershipType['minimum_fee'];
+ }
+ $row->tokens($entity, $field, \CRM_Utils_Money::formatLocaleNumericRoundedForDefaultCurrency($minimumFee));
}
```https://lab.civicrm.org/dev/core/-/issues/4423Slow contact lookup query in SearchKit2023-07-15T12:58:42ZeileenSlow contact lookup query in SearchKitI think we might be doing something weird when we query for an id in search kit / api4
It takes about 30 seconds to return here - that is what I might expect if I was waiting for it to search without an index (ie a pre-pended wildcard) ...I think we might be doing something weird when we query for an id in search kit / api4
It takes about 30 seconds to return here - that is what I might expect if I was waiting for it to search without an index (ie a pre-pended wildcard) but of course we turned that off using the site setting
![image](/uploads/acb69198d4abda193d10b5e2fd103252/image.png)https://lab.civicrm.org/dev/core/-/issues/4422Is there any reason to save both html and text versions of Inbound Emails?2023-07-13T17:33:27ZlarsssandergreenIs there any reason to save both html and text versions of Inbound Emails?For Email-to-Activity processing, when the email being processed has both html and text versions, both are saved to the details for the activity using the `-ALTERNATIVE ITEM 0-` ... `-ALTERNATIVE ITEM 1-` delimiters (depending on the det...For Email-to-Activity processing, when the email being processed has both html and text versions, both are saved to the details for the activity using the `-ALTERNATIVE ITEM 0-` ... `-ALTERNATIVE ITEM 1-` delimiters (depending on the details of the email, sometimes it seems to fail to recognize that both are present). This is handled while viewing activities (you only get the first version, which is not ideal as it is often the text version), but not in SearchKit.
I don't see any reason to keep two versions of the same email, so my proposal is just to keep the html version if both are present. Simple is better here unless there is some use case for having both.
If there is a need for the text version, perhaps we could add a setting for the mail account that would select text or html if both are present.https://lab.civicrm.org/dev/core/-/issues/4421Scheduled jobs stopped working after an update last week - error in MailingEv...2023-11-23T06:34:09ZTobias KrauseScheduled jobs stopped working after an update last week - error in MailingEventUnsubscribe.phpI realized this morning that the CiviCRM cron job for the scheduled jobs stopped working after we updated last week from CiviCRM 5.61.2 to 5.61.4. Today I updated to the most current version 5.63.1 but the error still exists.
The proble...I realized this morning that the CiviCRM cron job for the scheduled jobs stopped working after we updated last week from CiviCRM 5.61.2 to 5.61.4. Today I updated to the most current version 5.63.1 but the error still exists.
The problem became obvious as the message "Cron job not running" appeared when I logged in this morning (I did not so for the whole last week). When I checked the list of scheduled jobs I found out that the "Bounces fetcher" job run successfully but all the other jobs did not. So I run our cron job task
`/var/www/civi_live/vendor/civicrm/cv/bin/cv api job.execute --user=admin --cwd=/var/www/civi_live/httpdocs`
manually in CLI and got the following error:
```
In MailingEventUnsubscribe.php line 47:
count(): Argument #1 ($value) must be of type Countable|array, null given
```
The following file is the one: vendor/civicrm/civicrm-core/api/v3/MailingEventUnsubscribe.php
Here it is the following code part in the function civicrm_api3_mailing_event_unsubscribe_create:
```
$groups = CRM_Mailing_Event_BAO_MailingEventUnsubscribe::unsub_from_mailing($job, $queue, $hash);
if (count($groups)) {
CRM_Mailing_Event_BAO_MailingEventUnsubscribe::send_unsub_response($queue, $groups, FALSE, $job);
return civicrm_api3_create_success($params);
}
```
CRM_Mailing_Event_BAO_MailingEventUnsubscribe::unsub_from_mailing can return an array or NULL. I am not sure why this error appeared after the last update as neither CRM_Mailing_Event_BAO_MailingEventUnsubscribe::unsub_from_mailing nor civicrm_api3_mailing_event_unsubscribe_create() has changed but it may be related to some other changes somewhere.
When I change this code part to the following the scheduled jobs are finished:
```
$groups = CRM_Mailing_Event_BAO_MailingEventUnsubscribe::unsub_from_mailing($job, $queue, $hash);
if ($groups && count($groups)) {
CRM_Mailing_Event_BAO_MailingEventUnsubscribe::send_unsub_response($queue, $groups, FALSE, $job);
return civicrm_api3_create_success($params);
}
```https://lab.civicrm.org/dev/core/-/issues/4420crmDate support for Date Preferences2023-07-10T06:21:42ZlarsssandergreencrmDate support for Date Preferences[crmDate](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Smarty/plugins/modifier.crmDate.php) only supports Date Formats, not the confusingly separate Date Preferences, which include creditCard format. It would be nice to b...[crmDate](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Smarty/plugins/modifier.crmDate.php) only supports Date Formats, not the confusingly separate Date Preferences, which include creditCard format. It would be nice to be able to use `crmDate:'creditCard'` on Contribution Page Review and Thank You pages, etc.
Or maybe we should use tokens for credit_card_expiration_date instead, @DaveD thinks they support the Preferences.https://lab.civicrm.org/dev/core/-/issues/4415FormBuilder: Add 'is empty' filter2023-07-20T22:00:28Zaydunsaidan.saunders@squiffle.ukFormBuilder: Add 'is empty' filterOverview
----------------------------------------
In FormBuilder, add `Is Empty` / `Is Not Empty` for filter fields.
Booleans can be represented as Yes/No radios. This would let other fields be represented in the same way based on whet...Overview
----------------------------------------
In FormBuilder, add `Is Empty` / `Is Not Empty` for filter fields.
Booleans can be represented as Yes/No radios. This would let other fields be represented in the same way based on whether they are empty or not.
Example use-case
----------------------------------------
1. On the current Contribution Search form there is a 'Contribution is Recurring?' Yes/No which is based on whether the `recur_id` is empty or not.
Current behaviour
----------------------------------------
FormBuilder can add a filter for a specific value of recur_id but not 'is empty'/'is not empty'
Proposed behaviour
----------------------------------------
Allow 'is empty'/'is not empty' to be used with any filter field and be configured as booleans.
Comments
----------------------------------------
General 'is empty'/'is not empty' is available in Apiv4 'where' clauses and SearchKit, so this is about extending to FormBuilder.
See https://chat.civicrm.org/civicrm/pl/js843u3qz3f6pgc8ay183bhuzocolemanwcolemanwhttps://lab.civicrm.org/dev/core/-/issues/4412Add something like API 3 Attachments to API 42023-07-06T06:35:00ZlarsssandergreenAdd something like API 3 Attachments to API 4We can't get the full URL for a file from API 4 File and there is no API 4 Attachments, so API 3 must be used — which seems like a significant gap.
Thought this was worth noting as it has come up [twice](https://civicrm.stackexchange.co...We can't get the full URL for a file from API 4 File and there is no API 4 Attachments, so API 3 must be used — which seems like a significant gap.
Thought this was worth noting as it has come up [twice](https://civicrm.stackexchange.com/questions/45209/get-uploaded-file-full-url-via-api-v4/45210) on [SE](https://civicrm.stackexchange.com/questions/45179/civi-crm-apiv4-image-retrieval) recently.https://lab.civicrm.org/dev/core/-/issues/4406SearchKit: make 'inplace edit' conditional2023-07-09T03:06:31Zaydunsaidan.saunders@squiffle.ukSearchKit: make 'inplace edit' conditionalSuggestion: apply conditional rules (like those on the `Style` option) to `In-place edit`
Use case:
On a listing of Participant Status Types, we want to enable in-place edit if the type is not `Reserved`Suggestion: apply conditional rules (like those on the `Style` option) to `In-place edit`
Use case:
On a listing of Participant Status Types, we want to enable in-place edit if the type is not `Reserved`colemanwcolemanwhttps://lab.civicrm.org/dev/core/-/issues/4404Possibility to reuse the list of languages available for "Preferred language"2023-07-06T06:36:35ZquimgilPossibility to reuse the list of languages available for "Preferred language"Overview
----------------------------------------
Out of the box, CiviCRM comes with a full list of languages used for the field "Preferred language". Right now, there is no way to reuse this list in custom fields. Admins must create the...Overview
----------------------------------------
Out of the box, CiviCRM comes with a full list of languages used for the field "Preferred language". Right now, there is no way to reuse this list in custom fields. Admins must create their own list of languages manually or use different unsupported approaches to reuse the one CiviCRM already has.
Example use-case
----------------------------------------
An international organization working with volunteers wants to know in which languages they have basic, advanced, and native/professional fluency, in separate questions.
Current behaviour
----------------------------------------
The list of languages available in CiviCRM's database cannot be reused for custom fields.
Proposed behaviour
----------------------------------------
Make the "Preferred language" list of languages available to custom fields, just like you can create any list that is then available to other custom fields.
Comments
----------------------------------------
See examples of ad hoc solutions admins are recurring to on https://civicrm.stackexchange.com/questions/6604/how-to-reuse-preferred-language-selecthttps://lab.civicrm.org/dev/core/-/issues/4403SK / API 4 Explorer: event_queue_id, job id, bounce id, etc autocompletes don...2023-07-05T04:30:05ZlarsssandergreenSK / API 4 Explorer: event_queue_id, job id, bounce id, etc autocompletes don't workEdited as this issue is more general than I thought. Fairly minor issue, but:
In SK or API 4 Explorer, for mailing event id fields (event queue id, job id, bounce id, etc), the autocomplete fields don't work. No matter what you input, i...Edited as this issue is more general than I thought. Fairly minor issue, but:
In SK or API 4 Explorer, for mailing event id fields (event queue id, job id, bounce id, etc), the autocomplete fields don't work. No matter what you input, it shows `Loading failed` and you can't make a selection.
To replicate: Choose MailingEventBounce or similar, Where event_queue_id and =.https://lab.civicrm.org/dev/core/-/issues/4402Change unsupervised Household dedupe rule to email only2023-07-06T23:20:28ZlarsssandergreenChange unsupervised Household dedupe rule to email onlyAs @DaveD [pointed out](https://github.com/civicrm/civicrm-core/pull/26673#issuecomment-1613302861), the default unsupervised Household dedupe rule that matches on name or email doesn't make a lot of sense. Presumably, you could easily h...As @DaveD [pointed out](https://github.com/civicrm/civicrm-core/pull/26673#issuecomment-1613302861), the default unsupervised Household dedupe rule that matches on name or email doesn't make a lot of sense. Presumably, you could easily have multiple households with the name "Jones family" or similar in whatever format you choose for household names. You wouldn't want these to be merged automatically.
I think the easiest and clearest solution is only to merge households with the same email, but then that won't help those without emails. We could do something like threshold 20, email 20, name 10, street address 10, phone 10, but I'm not sure we gain much from that. We can keep name or email as the supervised rule.https://lab.civicrm.org/dev/core/-/issues/4401SearchKit - Report replacement - as a standard user without edit searchkit pe...2023-07-06T06:37:04ZsamuelsovSearchKit - Report replacement - as a standard user without edit searchkit permission, add a way to show/hide columns in table displayThere is already an [extension](https://lab.civicrm.org/extensions/search_kit_reports) that recreate the queries of core reports into SearchKit so admin/super user can customize and use them instead of report.
However, as a standard use...There is already an [extension](https://lab.civicrm.org/extensions/search_kit_reports) that recreate the queries of core reports into SearchKit so admin/super user can customize and use them instead of report.
However, as a standard user, edit a Searchkit is too complicated and reports is still the better option if we need to provide a lot of flexibility to users.
Currently, in Reports, standard user have access to customize :
- Columns - no way to do this in SK without editing the SK - hence the proposal
- Sorting - already there by clicking on the columns of the SK table
- Filters - already there by configuring them with FormBuilder
- Title and Format - don't think it is required
- Email delivery - missing, already documented in https://lab.civicrm.org/dev/core/-/issues/3478
So I think the main undocumented missing feature if we want to replace the reports is to have a way for a standard user to choose which columns to show/hide in a table display. It needs to apply to the screen and to the Download action.
As an example, I like the way new reports in Quickbooks works with the ability to switch each column on/off and as a bonus allow to change columns order :
![Rapport](/uploads/550ef9e7d7ffd1f19849b4034faa7765/Rapport.png)
As a first step, I don't think we need to save the user preferences but we will probably want to be able to do that / reset to default eventually.https://lab.civicrm.org/dev/core/-/issues/4400Export of contacts when filtering on custom field exports all contacts in dat...2023-07-06T06:37:41ZtjhellmannExport of contacts when filtering on custom field exports all contacts in databaseOverview
----------------------------------------
When I filter contacts from Advanced Search and select a custom field to filter on, in the next screen, the correct list and number of records comes up. From there if I select All [number...Overview
----------------------------------------
When I filter contacts from Advanced Search and select a custom field to filter on, in the next screen, the correct list and number of records comes up. From there if I select All [number of records] records and select Export contacts, the next screen also shows the correct number of contacts from the previous screen. But when I click 'continue' to export the records, the export contains all the records in the database. This does not happen when the list is filtered for non custom fields.
Reproduction steps
----------------------------------------
1. Visit Advanced Search
1. Select a custom field value as search criteria and click search
1. See expected search results. Now click All Records and Export Contacts.
2. On the next screen see that the expected number of contacts say they will be in the export. Click continue and all the contacts in the database are in the exported file.
Environment information
----------------------------------------
* __CiviCRM:__ 5.62.1
* __PHP:__ 8.0.29
* __CMS:__ Drupal 9.5.9
* __Database:__ MySQL 5.7.39https://lab.civicrm.org/dev/core/-/issues/4399Unable to merge incomplete memberships due to missing end_date2023-07-06T06:38:24Zfreeform.stephUnable to merge incomplete memberships due to missing end_dateWhen a contribution fails when purchasing a membership, a membership remains (as either pending or cancelled) that does not have an end date. When merging contacts any memberships are expected to have an end date.
`"UPDATE civicrm_membe...When a contribution fails when purchasing a membership, a membership remains (as either pending or cancelled) that does not have an end date. When merging contacts any memberships are expected to have an end date.
`"UPDATE civicrm_membership SET end_date = '' WHERE id=ID [nativecode=1292 ** Incorrect date value: '' for column `civicrm`.`civicrm_membership`.`end_date` at row 1]"`https://lab.civicrm.org/dev/core/-/issues/4397What should the Name and Address reserved general Individual dedupe rule do?2023-06-30T00:14:21ZlarsssandergreenWhat should the Name and Address reserved general Individual dedupe rule do?We have a reserved dedupe rule on new installs for Individuals called Name and Address.
What it actually does is matches contacts who have the same first name, last name and street address. What it claims to do is the following:
![image]...We have a reserved dedupe rule on new installs for Individuals called Name and Address.
What it actually does is matches contacts who have the same first name, last name and street address. What it claims to do is the following:
![image](/uploads/d427acb2b1c3ce07c994b68a2371ac62/image.png)
[This rule](https://github.com/civicrm/civicrm-core/blob/master/sql/civicrm_data/civicrm_dedupe_rule/IndividualGeneral.sqldata.php) also includes suffix_id and middle_name with weights of 1, which cannot influence the matches (threshold 15, all other items have weight 5), so these don't do anything.
There is a [hard-coded query](https://github.com/civicrm/civicrm-core/blob/master/CRM/Dedupe/BAO/QueryBuilder/IndividualGeneral.php) for this rule that should match up with the above (right now we are querying middle_name, suffix and birthdate for no reason, which is kind of ironic since this query claims to be optimized).
None of this has changed in at least ten years.
Personally, I don't think matching on first, last and street address is useful (as street addresses are often written slightly differently so are not that likely to match). Maybe we should just eliminate this reserved dedupe rule, since I can't see that it is used for anything. If not, maybe we should make it more useful by making it just first and last name. Or maybe there is some reason this is useful as is and we should simply make this all consistent so it just matches on first, last and address and nothing more.https://lab.civicrm.org/dev/core/-/issues/4395Token support to allow us to migrate sort_name, display_name, address formats...2023-06-28T07:10:08ZeileenToken support to allow us to migrate sort_name, display_name, address formats to tokensWe have a goal to get sort_name, display_name and address formats to resolve using the standard token system, at the same time as the greetings are resolved.
Unfortunately there are some things they support at the moment that our tokens...We have a goal to get sort_name, display_name and address formats to resolve using the standard token system, at the same time as the greetings are resolved.
Unfortunately there are some things they support at the moment that our tokens don't. Our tokens DO support the default filter
`{contact.first_name|default:"donor"}`
and they support the magic space - ie
`{contact.first_name}{ }{contact.last_name}`
which resolves a space in the above, but not if there is no last name.
However, in sort name there is a variant on the magic space we don't currently support in tokens - - ie
`{contact.last_name}{, }{contact.first_name}`
The regex that supports that actually removes the whole `{}` contents if the following tag is empty or renders the contents otherwise.
This is pretty hard to make compatible with smarty :-(, which is a requirement to standardise. So ideally we would
1) make the magic syntax support know usages (within reason) which so far are the magic space, the magic comma and this variant
`{contact.first_name}{ ~ }{contact.nick_name}{ ~ }{contact.last_name}`
2) add a system-check to highlight notify using variations of the magic syntax other than the 3 on our radar that they may not work in future upgrades if we don't add support & direct them to this ticket to comment & explain the use case
3) depending on the variations that come out of 2 add 'micro-conditional support'
Note micro-conditionals was an idea @totten came up with when we were spit-balling this - I've pasted his thoughts below
---------------------------------------------------------------------------------------------------------------------
micro-conditionals: special tokens that conditionally display a handful of symbols
```
---------
{contact.first_name} {contact.last_name} { ~ } {contact.nick_name}
{contact.last_name}{ }{contact.first_name}
{contact.last_name}{, }{contact.first_name}
{contact.first_name}{ (}{contact.nick_name}{) }{contact.last_name}
Dear {contact.first_name}{?friend},
----------
{contact.first_name contact.last_name " ~ " contact.nick_name}
{contact.last_name " " contact.first_name}
{contact.last_name ", " contact.first_name}
{contact.first_name " (" contact.nick_name ") " contact.last_name}
Dear {contact.first_name ?friend},
----------
{{{contact.first_name contact.last_name ~ contact.nick_name}}}
{{{contact.last_name contact.first_name}}}
{{{contact.last_name, contact.first_name}}}
{{{contact.first_name (contact.nick_name) contact.last_name}}}
Dear {{{contact.first_name??"friend"}}},
-------------
```
Also note - first draft of appropriate test https://github.com/civicrm/civicrm-core/pull/26637/files#diff-a5d237ba4e392d0e8ad764a535315f0b1101cc5ca61c39b4e572de42f5c6b4bc