CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-03-08T17:56:47Zhttps://lab.civicrm.org/dev/core/-/issues/3760Add server side validation for afform2023-03-08T17:56:47ZKurund JalmiAdd server side validation for afformCurrently, there is some work done on the frontend to add client side validation and it would be good to add the same at the API end.Currently, there is some work done on the frontend to add client side validation and it would be good to add the same at the API end.Kurund JalmiKurund Jalmihttps://lab.civicrm.org/dev/core/-/issues/4167Transferred to Link in View Event Registration for {participant} Goes to a Pr...2023-03-09T09:20:26ZLKuttnerTransferred to Link in View Event Registration for {participant} Goes to a Previous EventSteps to Reproduce:
- View an event registration that has been transferred to another participant.
- In the Status line, Click the name of the participant that the registration has been transferred to.
- Example: _Status | Transferred (...Steps to Reproduce:
- View an event registration that has been transferred to another participant.
- In the Status line, Click the name of the participant that the registration has been transferred to.
- Example: _Status | Transferred (Transferred to Megan Nedzinski)_
- The link takes you to the registration for the first event that the person was ever registered for.
- The link does not take you to the latest registration that was just transferred to that contact.
- This only happens in the View Event Registration for {participant} page.
- In the Edit Event Registration for {participant} page, clicking the equivalent link does take you to the correct event.
This is using CiviCRM 5.55.1 on Drupal 7. I just noticed this and don't know how long this behavior has been in effect.https://lab.civicrm.org/dev/core/-/issues/3948Oauth authentication against MS Exchange IMAP fails since basic authenticatio...2023-03-11T13:16:37ZspalmstromOauth authentication against MS Exchange IMAP fails since basic authentication blockedOverview
----------------------------------------
----------------------------------------
Bounce processing using Oauth to authenticate against MS Exchange fails since Microsoft blocked basic authentication against IMAP.
Reproduction ...Overview
----------------------------------------
----------------------------------------
Bounce processing using Oauth to authenticate against MS Exchange fails since Microsoft blocked basic authentication against IMAP.
Reproduction steps
----------------------------------------
1. Set up an Azure AD application to allow CiviCRM to check bounce processing against MS Exchange
1. Configure Oauth to use it.
1. Create a Mail account for bounce processing.
1. Save and Test fails.
Current behaviour
----------------------------------------
```
An error occured while sending or receiving mail. The IMAP server did not accept the username and/or password: A0001 NO LOGIN failed.. (See log for more details.)
```
Expected behaviour
----------------------------------------
Success.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _MS Edge_ but probably irrelevant.
* __CiviCRM:__ _5.54.0_ <!-- If this problem relates to an upgrade, then specify both old and new versions -->
* __PHP:__ _7.4__ but probably irrelevant.
* __CMS:__ _Drupal 9.4.8_ but probably not relevant
* __Database:__ _MySQL 8.31_ but probably not relevant
* __Web Server:__ _IIS 10_ but probably not relevant.
Comments
----------------------------------------
Oauth scopes:
```
https://outlook.office.com/IMAP.AccessAsUser.All
https://outlook.office.com/POP.AccessAsUser.All
https://outlook.office.com/SMTP.Send
https://outlook.office.com/User.Read
```
Azure AD App permissions:
```
Microsoft Graph (12)
email Delegated
IMAP.AccessAsUser.All Delegated
Mail.Read Application
Mail.ReadBasi Application
Mail.ReadBasic.All Application
Mail.ReadWrite Application
Mail.Send Application
offline_access Delegated
openid Delegated
POP.AccessAsUser.All Delegated
SMTP.Send Delegated
User.Read Delegated
```
My suspicion is that the extension _never_ worked properly but instead the mail account was using basic authentication so it wasn't an issue. So, when basic authentication was disabled, the application stopped working. Has anyone else seen this issue?https://lab.civicrm.org/dev/core/-/issues/4169Let "Number" and "Money"-type custom fields be nullable2023-03-13T07:39:11ZnoahLet "Number" and "Money"-type custom fields be nullableOverview
----------------------------------------
There are many times when NULL would be a meaningful/useful value in a custom field of type "Number" (float) or "Money" (decimal). However, it is not currently possible, via the form laye...Overview
----------------------------------------
There are many times when NULL would be a meaningful/useful value in a custom field of type "Number" (float) or "Money" (decimal). However, it is not currently possible, via the form layer, to set these fields to NULL. Blank values are currently turned into zero.
Example use-case
----------------------------------------
Let's say we need to track the elevation (meters above or below sea level) of the addresses in our database.
1. Create a custom field "Geographic Elevation" of type "Number", extending the Address entity. Leave the "Default value" blank.
1. Go to a contact and edit one of their addresses, or create a new one.
1. On the create/edit form, leave the "Geographic Elevation" field blank -- because let's say we don't know the elevation at this address.
1. Submit the form and view the saved value.
Current behaviour
----------------------------------------
"Geographic Elevation" is set to zero. That means sea level. But that's not correct -- we actually don't know the elevation.
Proposed behaviour
----------------------------------------
When the field submitted with a blank value (empty string), the database field should be set to NULL, and displayed as blank.
Comments
----------------------------------------
NULL is also a meaningful value for Money fields. Say we have a field on individuals called "Net Worth". Zero (the person is destitute) and NULL (we don't know how much money they have) are quite different.
It _is_ possible to set the field to NULL through the API, but only by passing the string 'null' (due to a PEAR DB limitation).
<details><summary>Show API example</summary>
````php
// NULL will be saved as 0
Civi\Api4\Address::update()
->addValue('Geography.Elevation', NULL)
->addWhere('id', '=', 27)
->execute();
$address = Civi\Api4\Address::get()
->addSelect('Geography.Elevation')
->addWhere('id', '=', 27)
->execute()->single();
// [
// 'id' => 27,
// 'Geography.Elevation' => 0,
// ]
// but 'null' will be saved as NULL
Civi\Api4\Address::update()
->addValue('Geography.Elevation', 'null')
->addWhere('id', '=', 27)
->execute();
$address = Civi\Api4\Address::get()
->addSelect('Geography.Elevation')
->addWhere('id', '=', 27)
->execute()->single();
// [
// 'id' => 27,
// 'Geography.Elevation' => null,
// ]
````
</details>https://lab.civicrm.org/dev/core/-/issues/4170FormBuilder: Allow placing fields outside their entity fieldset2023-03-13T07:41:15ZJonGoldFormBuilder: Allow placing fields outside their entity fieldsetFrom today's FormBuilder meeting:
Currently, the markup for a field doesn't specify its entity, in order to facilitate creating reusable blocks. However, it would be beneficial to allow specifying the entity for a field within the field...From today's FormBuilder meeting:
Currently, the markup for a field doesn't specify its entity, in order to facilitate creating reusable blocks. However, it would be beneficial to allow specifying the entity for a field within the field's markup.
This would allow a UX-friendly way of freely ordering fields on a form without regard to the entity.https://lab.civicrm.org/dev/core/-/issues/4171FormBuilder: Create options to style radio buttons/checkboxes2023-03-13T07:41:35ZJonGoldFormBuilder: Create options to style radio buttons/checkboxesCurrently, there's no way to configure the style of radio buttons/checkboxes within the FormBuilder UI. It also doesn't respect the "Options Per Line" setting of the custom field.
Ideally we could specify vertical/horizontal/columns for...Currently, there's no way to configure the style of radio buttons/checkboxes within the FormBuilder UI. It also doesn't respect the "Options Per Line" setting of the custom field.
Ideally we could specify vertical/horizontal/columns for options.https://lab.civicrm.org/dev/core/-/issues/4172FormBuilder: silent failure if required fields not on form2023-03-13T07:41:58Zaydunsaidan.saunders@squiffle.ukFormBuilder: silent failure if required fields not on formOverview
----------------------------------------
FormBuilder does not show which fields are required to create an entity.
Reproduction steps
----------------------------------------
1. Create a new FormBuilder form
1. Add an activity
1...Overview
----------------------------------------
FormBuilder does not show which fields are required to create an entity.
Reproduction steps
----------------------------------------
1. Create a new FormBuilder form
1. Add an activity
1. Select the type as Meeting
1. Give the form a name, url and Save.
1. Go to the url just created
1. Enter firstname = test, lastname = test, activity subject = test, Submit
Current behaviour
----------------------------------------
The contact is created but the activity is not. Note there is no error message.
Expected behaviour
----------------------------------------
Run-time: Show an error if the activity cannot be created.
Form-save: warn if required fields have not been added to the form, so creation will fail.
Form-edit: indicate which fields are required by the API. Compare FB with API Explorer eg
FB:
![image](/uploads/a4a2cef38dfb59b21a38cb8149974f1f/image.png)
API Explorer:
![image](/uploads/b4b540f37f520c6da5a834f941a03cad/image.png)
Required fields are shown with red asterisk
Environment information
----------------------------------------
* __CiviCRM:__ _Master_ <!-- If this problem relates to an upgrade, then specify both old and new versions -->
Comments
----------------------------------------
Note this is about fields required by the API, not fields marked as required on the form.https://lab.civicrm.org/dev/core/-/issues/1761Should deprecate Financial Type on Contribution records2023-03-13T13:54:21ZRichShould deprecate Financial Type on Contribution recordsOverview
----------------------------------------
Contribution records have a **financial type**. They also have line items, each of which has a financial type. Typically, a contribution has a single line item, and the financial types o...Overview
----------------------------------------
Contribution records have a **financial type**. They also have line items, each of which has a financial type. Typically, a contribution has a single line item, and the financial types of each match. But it's quite possible to have different financial types e.g. contribution and only line item could have different types which does not make sense. Also if a contribution has multiple line items of different types, what should the financial type of the contribution be?
Having financial type on the contribution harks back to simpler times before we allowed line items. But for 90%(?) of cases it doesn't matter much because they're the same. And of course there's a thousand places in core that assumes a single finanical type per contribution, and there's a thousand more in custom code and extensions.
Currently (5.26) you get an api exception if you call Order.create without setting a contribution-level financial type, which is slightly irksome when you're providing line items, not least because it forces new code to do nonsense old stuff, which can lead to developer confusion.
Nb. all the UIs that create contribution records set a financial type to *something*. This may or may not make sense compared with line items (e.g. it's taken from the priceset).
Proposals
---------------
1. I would like to be able to call Order.create with line items and have it pick the financial type from the first line item to use for the Contribution record. This would not affect any existing processes (with the possible exception of processes that currently crash...!)
2. I propose that over time (it will be a long time!) we move all reports/searches etc. to go the extra JOIN and inspect the line items for info on financial types instead of using the contribution's single financial type value.
Comments
----------------------------------------
This is a place to discuss it. Discussion began in [mattermost](https://chat.civicrm.org/civicrm/pl/pjtag6uu1tr7fxaeua51cfhk8w)https://lab.civicrm.org/dev/core/-/issues/2759Intermittent cookies error after contribution2023-03-14T07:16:59ZandyburnsIntermittent cookies error after contributionWe experience a cookies error intermittently after submitting a donation. Currently on Civi 5.35.2 and WP 5.7.2. This is multisite so may be multisite related.
![118401618_379155863073198_2489825481780434605_n](/uploads/1125c66666dd23c3...We experience a cookies error intermittently after submitting a donation. Currently on Civi 5.35.2 and WP 5.7.2. This is multisite so may be multisite related.
![118401618_379155863073198_2489825481780434605_n](/uploads/1125c66666dd23c31102ac79e053f06f/118401618_379155863073198_2489825481780434605_n.png)
A contributor does not get to the thank you page but gets the error above. The donation does succeed. This issue has persisted for a long-time (2+ years) and is a probably the worst civi issue I've encountered seeing it has to do with contribution pages and has no pattern that we can find. Tried all the go to solutions posted on SE with no luck. I would estimate that it occurs ~10-20% of the time.
This is related: https://lab.civicrm.org/dev/core/-/issues/517 and says using PHP 7.2 resolved, but were on 7.2 and now on 7.3
Some steps I've done:
- Set `Clean-up Temporary Data and Files` scheduled job run frequency from Hourly to Daily. It is on every site.
- Changed cookie domain constant in wp-config.php
```
// define('COOKIE_DOMAIN', false);
to
define('COOKIE_DOMAIN', $_SERVER\['HTTP_HOST'\] );
```
- Enabled https://lab.civicrm.org/extensions/qfsessionwarning and set timeout at 120 minutes.
`$civicrm_setting['core']['secure_cache_timeout_minutes'] = 120;`
We have considered changing how garbage collection works as sessions will die if garbage collection is run (server does that every 15 minutes).
[This post](https://civicrm.stackexchange.com/questions/39156/payments-fail-with-paypal-standard-and-could-not-find-a-valid-session-key) seems to have found a solution using the Fatal Error Handler. Is this recommended and how do you implement? Seems like a workaround symptom fix 'solution'.
Also getting this for additional event participants all the time (usually) or not at all depending on when I test: https://lab.civicrm.org/dev/core/-/issues/2708
Help?https://lab.civicrm.org/dev/core/-/issues/4124Past campaigns are not to be assigned via batch update/update contributions2023-03-15T06:27:08ZyashodhaPast campaigns are not to be assigned via batch update/update contributionsSteps to replicate :
- Create a campaign with end date in past
![campaign](/uploads/cefd27e11afe0c929803e28f58f67bbb/campaign.png)
- Create a profile configured for contribution fields (including the campaign field)
- Find Contribution...Steps to replicate :
- Create a campaign with end date in past
![campaign](/uploads/cefd27e11afe0c929803e28f58f67bbb/campaign.png)
- Create a profile configured for contribution fields (including the campaign field)
- Find Contributions, and select a few and use the created profile to update the campaign
- The campaign is NOT available in the drop down
![batch_update](/uploads/cfa7bee25d1cb7baa5bd1664b39e00e5/batch_update.png)
- On new/edit contributions, you are able to choose the past campaigns as well (which is the correct behavior).
![new_contribution](/uploads/82d4ddaaf86af46cab608ef26b333ece/new_contribution.png)yashodhayashodhahttps://lab.civicrm.org/dev/core/-/issues/1770Pagination in Smart Group Alphabetic listings2023-03-15T09:15:59ZmarcusjwilsonPagination in Smart Group Alphabetic listingsOverview
----------------------------------------
When creating a Smart Group (in this case, via the Search Builder) and displaying Contacts within the Smart Group, if we choose to view the Contacts by Alphabet letter, and then paginate ...Overview
----------------------------------------
When creating a Smart Group (in this case, via the Search Builder) and displaying Contacts within the Smart Group, if we choose to view the Contacts by Alphabet letter, and then paginate through that letter (i.e. viewing page 2 of the contacts with surnames starting "H"), then, at the end of the list, the contacts from the start of the "H"s are repeated.
For instance, on page 2 of the "H"s in my smart group, I see that it's displaying "Contact 51 - 67 of 67". But after Contact number 67 it starts to display the "H"s from the start again, in positions 68 to 100 (which shouldn't exist in the view).
Reproduction steps
----------------------------------------
1. Create a Smart Group using Search Builder
2. Navigate to Contacts > Manage Groups. Find the Smart Group and view "Contacts"
3. Navigate to a letter of the alphabet for Contacts where the number of results returned > 50, and view the last page of listings.
Current behaviour
----------------------------------------
Smart Groups Contacts filtered by Letter repeating contacts on last page of listings.
Expected behaviour
----------------------------------------
Smart Groups Contacts filtered by Letter only display each contact once.
Environment information
----------------------------------------
CiviCRM 5.24.3
WordPress 5.4.1
Comments
----------------------------------------
I couldn't test this on the demo sites, as there aren't enough contacts in the demo site databases to display > 50 contacts per letter of the alphabet. Can contact data on demo sites be expanded?https://lab.civicrm.org/dev/core/-/issues/4180SearchKit does not respond to campaigns in mailings2023-03-17T07:47:36ZMariaVSearchKit does not respond to campaigns in mailingsI am trying to build a SearchKit where contacts who have received a mailing with a specific campaign within the last 6 months will no longer be displayed.
**My workaround at the moment:**
- First I enabled the option (CiviMail settings)...I am trying to build a SearchKit where contacts who have received a mailing with a specific campaign within the last 6 months will no longer be displayed.
**My workaround at the moment:**
- First I enabled the option (CiviMail settings) that for each mailing there will be also an activity added.
- The I have a second SearchKit that finds all contacts that have an activity with the campaign within the last 180 days and creates a smart group:
![image](/uploads/7fa530aae317f148486a460c1f63ca26/image.png)
- In my initial SearchKit I choose:
where "in Groups" Not One of "just mentioned smart group"
This works but it has the disadvantage of having 2 SearchKits and another smart group PLUS having an additional activity for each mailing.
**Expected behavior:**
- Having a "without" "Kontakt Mailing" entity where I can just filter those contacts out:
![image](/uploads/753b1ad0c824da4a7794d752ae7c6146/image.png)
I am also confused by the 3 options I have for Mailings:
![image](/uploads/5791966ad972bbc5194e7f796818f0ab/image.png)
What is the difference? Why are 3 options needed? It seems there is no option for the receiver. I have tried all 3 but when I sent out the mailing all the contacts remain in the SearchKit Table.
Is there any other solution that I might have missed?
Thanks in advance for any hint.https://lab.civicrm.org/dev/core/-/issues/4182Create a generic copy or clone api2023-03-17T07:50:37ZeileenCreate a generic copy or clone apiThis is a spin off of https://lab.civicrm.org/dev/core/-/issues/4130
I was looking at supporting import job templates & thought a clone or copy api makes sense.
I'm no longer prioritising this part & I think putting the code in the fo...This is a spin off of https://lab.civicrm.org/dev/core/-/issues/4130
I was looking at supporting import job templates & thought a clone or copy api makes sense.
I'm no longer prioritising this part & I think putting the code in the form layer or BAO layer for the specific entity is enouugh for now.
In order to clone the job the code looks something like
```
$userJob = UserJob::get()->addWhere('id', '=', $templateID)->execute()->first();
unset($userJob['metadata']['DataSource']['table_name'], $userJob['metadata']['submitted_values']['uploadFile'], $userJob['id'], $userJob['created_id'], $userJob['created_date'], $userJob['expires_date']);
$userJobID = UserJob::create(FALSE)->setValues($userJob)->execute()->first()['id'];
```
I feel like there is a case for a generic `Copy` api - in the `DAO` we have [a function with a signature that I think is probably 'a bit much'](https://github.com/civicrm/civicrm-core/blob/8bf6c5853cf76da43765cd8ef79d4323a442f350/CRM/Core/DAO.php#L1845-L1868) - so it might make sense to migrate to a new function... & create a generic API
I checked what Entities implement `Copy` and found a handful -the non static ones are not in the BAO - so we just have one non-std signature which is unused https://github.com/civicrm/civicrm-core/pull/25594
![image](/uploads/c4e1bbdf63fb79efaa06b03bca44fde4/image.png)
One option would be to add a parameter to the schema eg.
'is_not_clonable'
And then we could add that to fields like `created_id` and they would not need to be in the params. We already have the title of the entity so can assume that should be prefixed with `Copy Of`. The `UserJob` would have to override to set some of the deeper valueshttps://lab.civicrm.org/dev/core/-/issues/2966Duplicate contacts when custom field value is 'email', 'phone', etc.2023-03-18T04:08:01Zrita_compucorpDuplicate contacts when custom field value is 'email', 'phone', etc.**Issue**:
when existing contacts are donating (without being logged in) with the same details (email, first name, last name) that are registered in the system, most of the times there is a new contact created instead of merging it to t...**Issue**:
when existing contacts are donating (without being logged in) with the same details (email, first name, last name) that are registered in the system, most of the times there is a new contact created instead of merging it to the existing contact automatically.
**Investigation**:
When a custom field with a checkbox had its title and value set to ‘email’ or 'phone' and the unsupervised dedupe rule was prepared to check for email or phone as well causing the problem. So the problem occurs when:
- the unsupervised dedupe rule has email in it AND
- the custom field’s value is set to ‘email’
However not only the email option can cause duplications, but if you prepare an:
- unsupervised dedupe rule with phone ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24](/uploads/758592a6d5330c1133dcd1206e714daa/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24.png) AND
- the custom field’s value is set to ‘phone’ ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12](/uploads/df40ee69692fda55f51e946b2c841b9d/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12.png)
And even though the clients enter the same email/phone numbers on the donation form, the dedupe rule will be skipped and a new contact with the same information will be created. Probably this can be reproduced with many fields in the system, but only tried these two.
I checked on a few **different civi versions:**
- 4.7.26 - couldn’t reproduce
- 5.35.2 - issue reproducible
- 5.45.alpha1 (core demo civi) - issue reproducible
**Reproduction steps:**
1. create an unsupervised individual rule with the following details: ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24](/uploads/758592a6d5330c1133dcd1206e714daa/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24.png)
2. create a custom field set for Contacts with checkboxes like this: ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12](/uploads/df40ee69692fda55f51e946b2c841b9d/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12.png)
3. add the custom field set that you created in previous step to a profile that can be added to a contribution page
4. create a new (or use an old) contribution page and add the profile that you created in previous step to it in the Profiles tab
5. create a new contact, and make sure it has a First name, Last name and Phone number
6. open the contribution page as anonymous and donate to it, using the exact same First name, Last name and Phone number that you entered in previous step AND select the ‘phone’ option at the custom field you created → _after you submit the donation, a new contact will be created instead of merging it to the existing one, which is wrong behaviour_
7. donate again on the same form, with the same details as before, but instead of selecting ‘phone’ select either the ‘email’ or ‘post’ options → _the contribution will be merged to the original contact, which is the correct behaviour. It is because email and post are not added to the unsupervised dedupe rule in this case._
I think most of the time civi admins create custom fields with values set as numbers, but in some cases they might configure it to be the same as the label name and in that case duplications might occur. I think it might be happening when the value of the custom field is the same as the name of the field in the database (just a thought, can't say for sure).https://lab.civicrm.org/dev/core/-/issues/2357CiviCRM Dedupe Rules, default rules are incorrectly labelled as "and" when ea...2023-03-18T04:18:24Zjustinfreeman (Agileware)CiviCRM Dedupe Rules, default rules are incorrectly labelled as "and" when each rule is actually "or" eg. "Name and Email" should be "Name OR Email". Use of Name is ambiguous. Add "Nickname" to default Organisation dedupe rulesCiviCRM Dedupe Rules, default rules are incorrectly labelled as "and" when each rule is actually "or" eg. "Name and Email" should be "Name OR Email". Use of Name is ambiguous. Add "Nickname" to default Organisation dedupe rules.
Additio...CiviCRM Dedupe Rules, default rules are incorrectly labelled as "and" when each rule is actually "or" eg. "Name and Email" should be "Name OR Email". Use of Name is ambiguous. Add "Nickname" to default Organisation dedupe rules.
Additionally, Name is ambiguous in the context of Household and Organisation Contacts. Should be Household Name and Organisation Name. Organisations for example have Organisation Name and Nickname.
Suggested changes:
1. Change label to use "or" to match the dedupe criteria when or logic applies.
2. Remove ambiguity and be explicit about Name, ie. Household Name, Organisation Name and for Individuals: First Name and Last Name.
3. Add Nickname field to default Organisation dedupe rules.
![Screenshot_20210204_090434](/uploads/0c5b3b687825535aea7e8f0fdc814bf7/Screenshot_20210204_090434.png)
Agileware Ref: CIVICRM-1660https://lab.civicrm.org/dev/core/-/issues/3148Dedupe with multi-select custom fields can trigger IDS2023-03-18T04:20:40ZJonGoldDedupe with multi-select custom fields can trigger IDSWhen deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a che...When deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a checkbox). Note that you need several of these to trigger the "kick" on the IDS (3, I think).
* Create two contacts that are duplicates.
* Find and merge the records.
### Expected result
Contacts merged successfully.
### Actual result
"Your activity is a bit suspicious, hence aborting"
The issue is the POST request, which is passing arguments like `move_custom_12` with the `VALUE_SEPARATOR` control character. This triggers the IDS filter labeled "Detects nullbytes and other dangerous characters".
I'm really not certain what the correct answer is here - I can exempt users from the IDS, and maybe that's the solution to pursue, but it seems like there should be another solution available. Is it possible to exempt certain paths from the IDS, or use an alternate set of rules for a certain path?
Keyword: Intrusion Detection Systemhttps://lab.civicrm.org/dev/core/-/issues/960Proposal: Change supervised dedupe rules to name(s) OR email2023-03-18T04:52:01ZJonGoldProposal: Change supervised dedupe rules to name(s) OR emailI've done countless Civi trainings on deduping, many at CiviCamps and other "official" places. And each time, I need to explain that "yes, unsupervised rules should be more strict than supervised rules. Also yes, the default individual...I've done countless Civi trainings on deduping, many at CiviCamps and other "official" places. And each time, I need to explain that "yes, unsupervised rules should be more strict than supervised rules. Also yes, the default individual supervised rule is far more strict than the default unsupervised rule."
Surely I'm not the only one who feels the same - and yet none of us noticed, because before Civi 5.3/CRM-20565, the supervised rule was basically unused.
Now, there's a code comment that says, [CRM-20565 - Need a good default matching rule before using the dedupe engine for checking on-the-fly.](https://github.com/civicrm/civicrm-core/blob/1eca040f3f6efa0a8a43575cd0a08bae242a95f3/templates/CRM/Contact/Form/Contact.tpl#L334).
I'm proposing we fix this by changing the default supervised rule. The current on-the-fly checking is both a) a default, and b) very poor UX.
I'm also proposing that we do this as the new default for everyone. I'm mindful of the changes to existing systems, and we should probably put in an upgrade note or something - but honestly anyone who's never changed the default supervised rules isn't very likely to be using them.JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/1783Wrong Employer Relationship retained on Merge.2023-03-18T06:41:18ZtommyboboWrong Employer Relationship retained on Merge.Overview
----------------------------------------
If Current Employer record is update to a duplicate that is later merged this results in an inactive relationship being the only employer relationship retained.
Tested on 5.25 but I am ...Overview
----------------------------------------
If Current Employer record is update to a duplicate that is later merged this results in an inactive relationship being the only employer relationship retained.
Tested on 5.25 but I am seeing evidence of this issue going back to 2017.
Reproduction steps
----------------------------------------
1. Contact A has a Current Employer of "Organization 1"
1. They make a contribution and set employer to "Organization One"
1. This sets the relationship to 1 to inactive.
1. CiviCRM Admin notices the error and merges "Organization One" into "Organization 1"
1. CiviCRM Merge saves the inactive "Organization 1" relationship deletes the active "Organization One" to prevent a duplicate, but saves "Organization One" to the Current Employer field.
1. Contact Record is left with inactive employer relationship.
Current behavior
----------------------------------------
CiviCRM is giving preference to Relationships in the retained record during a merge.
It should give preference to Active relationships if there is a conflict.
This may be minor, but a Current Employer fields are a significant source of duplicate records.https://lab.civicrm.org/dev/core/-/issues/2987API call Contribution.repeattransaction fails with recurring contributions th...2023-03-20T01:51:15ZandrewcormickdockeryAPI call Contribution.repeattransaction fails with recurring contributions that have a one-to-one custom group record attachedOverview
----------------------------------------
Re: https://chat.civicrm.org/civicrm/pl/i8fht46isjn5uj4dbdosrqpqmr
The API call Contribution.repeattransaction appears to write a custom group record twice when completing a transaction....Overview
----------------------------------------
Re: https://chat.civicrm.org/civicrm/pl/i8fht46isjn5uj4dbdosrqpqmr
The API call Contribution.repeattransaction appears to write a custom group record twice when completing a transaction. On the second attempt, a database error caused by the clashing unique key is generated. The transaction appears to end up being recorded correctly, however the fatal error means that processes which call this function repeatedly end up not completing. Our use case involves finding all Eway transactions whose next scheduled contribution date is today, and then calling Contribution.repeattransaction for each one.
Reproduction steps
----------------------------------------
1. Ensure a one-to-one custom group exists against contributions.
1. Ensure a recurring contribution exists with a record in that custom group against the transaction.
1. Run `drush cvapi Contribution.repeattransaction contribution_status_id=Completed total_amount=<some value> contribution_recur_id=<contribution_recur_id in previous step> trxn_id=<random unique number>`
Current behaviour
----------------------------------------
Failure message appears: `DB Error: already exists`; process terminates. Record appears to be correctly produced in CiviCRM.
Log details:
```
Dec 07 12:40:11 [error] $Fatal Error Details = Array
(
[callback] => Array
(
[0] => CRM_Core_Error
[1] => exceptionHandler
)
[code] => -5
[message] => DB Error: already exists
[mode] => 16
[debug_info] => INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_field_2`,`custom_field_3`,`custom_field_4`,`entity_id` ) VALUES ( 1,'Some Text','','22943426',123456 ) [nativecode=1062 ** Duplicate entry '123456' for key 'unique_entity_id']
[type] => DB_Error
[user_info] => INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_field_2`,`custom_field_3`,`custom_field_4`,`entity_id` ) VALUES ( 1,'Some Text','','22943426',123456 ) [nativecode=1062 ** Duplicate entry '123456' for key 'unique_entity_id']
[to_string] => [db_error: message="DB Error: already exists" code=-5 mode=callback callback=CRM_Core_Error::exceptionHandler prefix="" info="INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_field_2`,`custom_field_3`,`custom_field_4`,`entity_id` ) VALUES ( 1,'Some Text','','22943426',123456 ) [nativecode=1062 ** Duplicate entry '123456' for key 'unique_entity_id']"]
)
Dec 07 12:40:11 [debug] $backTrace = #0 /path/to/civi/root/civicrm/CRM/Core/Error.php(942): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /path/to/civi/root/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php(922): CRM_Core_Error::exceptionHandler(Object(DB_Error))
#2 /path/to/civi/root/civicrm/vendor/pear/db/DB.php(997): PEAR_Error->__construct("DB Error: already exists", -5, 16, (Array:2), "INSERT INTO civicrm_value_contribution_custom_group_1 ( `hide_from_slider_182`,`custom_...")
#3 /path/to/civi/root/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php(575): DB_Error->__construct(-5, 16, (Array:2), "INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#4 /path/to/civi/root/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php(223): PEAR->_raiseError(Object(DB_mysqli), NULL, -5, 16, (Array:2), "INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", "DB_Error", TRUE)
#5 /path/to/civi/root/civicrm/vendor/pear/db/DB/common.php(1928): PEAR->__call("raiseError", (Array:7))
#6 /path/to/civi/root/civicrm/vendor/pear/db/DB/mysqli.php(936): DB_common->raiseError(-5, NULL, NULL, "INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", "1062 ** Duplicate entry '123456' for key 'unique_entity_id'")
#7 /path/to/civi/root/civicrm/vendor/pear/db/DB/mysqli.php(406): DB_mysqli->mysqliRaiseError()
#8 /path/to/civi/root/civicrm/vendor/pear/db/DB/common.php(1234): DB_mysqli->simpleQuery("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#9 /path/to/civi/root/civicrm/packages/DB/DataObject.php(2696): DB_common->query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#10 /path/to/civi/root/civicrm/packages/DB/DataObject.php(1829): DB_DataObject->_query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#11 /path/to/civi/root/civicrm/CRM/Core/DAO.php(468): DB_DataObject->query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#12 /path/to/civi/root/civicrm/CRM/Core/DAO.php(1621): CRM_Core_DAO->query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", TRUE)
#13 /path/to/civi/root/civicrm/CRM/Core/BAO/CustomValueTable.php(271): CRM_Core_DAO::executeQuery("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", (Array:5))
#14 /path/to/civi/root/civicrm/CRM/Core/BAO/CustomValueTable.php(391): CRM_Core_BAO_CustomValueTable::create((Array:2), "create")
#15 /path/to/civi/root/civicrm/CRM/Core/BAO/CustomValueTable.php(411): CRM_Core_BAO_CustomValueTable::store((Array:5), "civicrm_contribution", 123456, "create")
#16 /path/to/civi/root/civicrm/CRM/Core/DAO.php(2022): CRM_Core_BAO_CustomValueTable::postProcess((Array:5), "civicrm_contribution", 123456, "Contribution", "create")
#17 /path/to/civi/root/civicrm/CRM/Contribute/BAO/Contribution.php(2394): CRM_Core_DAO->copyCustomFields(123455, 123456)
#18 /path/to/civi/root/civicrm/CRM/Contribute/BAO/Contribution.php(4000): CRM_Contribute_BAO_Contribution::repeatTransaction((Array:10), (Array:18))
#19 /path/to/civi/root/civicrm/api/v3/Contribution.php(687): CRM_Contribute_BAO_Contribution::completeOrder((Array:10), "2269", NULL, NULL)
#20 /path/to/civi/root/civicrm/api/v3/Contribution.php(635): _ipn_process_transaction((Array:7), Object(CRM_Contribute_BAO_Contribution), (Array:10), (Array:10))
#21 /path/to/civi/root/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_contribution_repeattransaction((Array:7))
#22 /path/to/civi/root/civicrm/Civi/API/Kernel.php(149): Civi\API\Provider\MagicFunctionProvider->invoke((Array:8))
#23 /path/to/civi/root/civicrm/Civi/API/Kernel.php(81): Civi\API\Kernel->runRequest((Array:8))
#24 /path/to/civi/root/civicrm/api/api.php(132): Civi\API\Kernel->runSafe("contribution", "repeattransaction", (Array:5))
#25 /path/to/civi/root/civicrm/drupal/drush/civicrm.drush.inc(1581): civicrm_api3("Contribution", "repeattransaction", Array:5))
#26 /usr/local/src/drush/includes/command.inc(422): drush_civicrm_api("Contribution.repeattransaction", "contribution_status_id=Completed", "total_amount=11.00", "contribution_recur_id=22663", "trxn_id=aa11bb22cc33dd47")
#27 /usr/local/src/drush/includes/command.inc(231): _drush_invoke_hooks((Array:38), (Array:5))
#28 /usr/local/src/drush/includes/command.inc(199): drush_command("Contribution.repeattransaction", "contribution_status_id=Completed", "total_amount=11.00", "contribution_recur_id=22663", "trxn_id=aa11bb22cc33dd47")
#29 /usr/local/src/drush/lib/Drush/Boot/BaseBoot.php(67): drush_dispatch((Array:38))
#30 /usr/local/src/drush/includes/preflight.inc(67): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#31 /usr/local/src/drush/drush.php(12): drush_main()
#32 {main}
```
Expected behaviour
----------------------------------------
Fatal error not raised; no attempt to write the exact same record twice.
I have mitigated this behaviour with the following patch, but this is clearly a temporary solution and the real root cause needs to be found:
```
diff --git a/CRM/Core/BAO/CustomValueTable.php b/CRM/Core/BAO/CustomValueTable.php
index 8556c4ec21..6773b994c4 100644
--- a/CRM/Core/BAO/CustomValueTable.php
+++ b/CRM/Core/BAO/CustomValueTable.php
@@ -55,7 +55,7 @@ class CRM_Core_BAO_CustomValueTable {
$hookOP = 'edit';
}
else {
- $sqlOP = "INSERT INTO $tableName ";
+ $sqlOP = "INSERT IGNORE INTO $tableName ";
$where = NULL;
$hookOP = 'create';
}
```
Environment information
----------------------------------------
* __Browser:__ _Firefox 94.0.2_
* __CiviCRM:__ _5.43.2_
* __PHP:__ _7.3_
* __CMS:__ _Drupal 7.82_
* __Database:__ _MariaDB 10.4.21_
* __Web Server:__ _Nginx 1.10.3_https://lab.civicrm.org/dev/core/-/issues/1792Wrong status message when deleting a contact if undelete/trash is turned off2023-03-22T16:19:02ZDaveDWrong status message when deleting a contact if undelete/trash is turned off1. Go to Administer - System Settings - Misc
1. Set "Contact Trash and Undelete" to "No".
1. Go to any contact's page.
1. Click the Delete button near the top.
1. Confirm the deletion.
1. The contact is (permanently) deleted correctly, b...1. Go to Administer - System Settings - Misc
1. Set "Contact Trash and Undelete" to "No".
1. Go to any contact's page.
1. Click the Delete button near the top.
1. Confirm the deletion.
1. The contact is (permanently) deleted correctly, but two incorrect messages appear:
1. "[Name] has been moved to the trash."
(It should say permanently deleted, since you turned off trash.)
1. "A Contact with that ID does not exist: [id]"
(Because it's redirecting you to the contact view page for the contact that you just deleted.)
Another path to reproduce the problem is instead of step 3 above do Search - Find Contacts, then select one and from the actions dropdown choose Delete Contacts. It will say moved to trash but it should say permanently deleted. (Choosing Delete Permanently works ok with or without trash turned on.)
I think this might be an ok candidate for a newcomer to tackle, because it's pretty self-contained, not too abstract, and there's an existing unit test to update so provides a jumpstart to working with tests locally.
Relevant code is here:
1. [CRM/Contact/Form/Task/Delete.php in postProcess()](https://github.com/civicrm/civicrm-core/blob/e6d6149bad77c02001efceb0ef487f3ed3f7f8f1/CRM/Contact/Form/Task/Delete.php#L215)
2. [CRM/Contact/Form/Task/Delete.php in preProcess()](https://github.com/civicrm/civicrm-core/blob/e6d6149bad77c02001efceb0ef487f3ed3f7f8f1/CRM/Contact/Form/Task/Delete.php#L52)
3. [Existing test](https://github.com/civicrm/civicrm-core/blob/e6d6149bad77c02001efceb0ef487f3ed3f7f8f1/tests/phpunit/CRM/Contact/Form/Task/DeleteTest.php#L152-L155)
Aside: I'm curious how many people turn off trash because the message seems like something people would have noticed by now. I only noticed it while writing the test.