Development issueshttps://lab.civicrm.org/groups/dev/-/issues2023-11-01T05:03:24Zhttps://lab.civicrm.org/dev/core/-/issues/2997Search Builder (not Search Kit): groups not searched correctly2023-11-01T05:03:24Zaydunsaidan.saunders@squiffle.ukSearch Builder (not Search Kit): groups not searched correctlyOverview
----------------------------------------
In Search Builder, searching for both 'in group' and 'not in group' produces incorrect results.
As [reported in SE](https://civicrm.stackexchange.com/questions/40893/search-builder-fails...Overview
----------------------------------------
In Search Builder, searching for both 'in group' and 'not in group' produces incorrect results.
As [reported in SE](https://civicrm.stackexchange.com/questions/40893/search-builder-fails-in-group-and-not-in-group-search).
![Screenshot_from_2021-12-17_11-18-41](/uploads/10a879d4a29aa0ec727f2ff47c1527e2/Screenshot_from_2021-12-17_11-18-41.png)
NB: this is handled correctly in Search Kit and I doubt anyone will want to put effort into fixing this in Search Builder.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __CiviCRM:__ _Master (5.46.alpha1)_ <!-- If this problem relates to an upgrade, then specify both old and new versions -->https://lab.civicrm.org/dev/core/-/issues/2996Increment recommended php version2021-12-20T20:30:34ZeileenIncrement recommended php versionPHP has just EOLed php 7.3 https://www.php.net/eol.php
We are just about to release the 5.45 ESR and we could take the opportunity to drop support for older php & mysql ie drop support for php 7.2 & mysql 5.6
The gut check in informal ...PHP has just EOLed php 7.3 https://www.php.net/eol.php
We are just about to release the 5.45 ESR and we could take the opportunity to drop support for older php & mysql ie drop support for php 7.2 & mysql 5.6
The gut check in informal discussion was to leave that for the next ESR and instead just increase our recommended versions.
While we support php 8.0 if has pretty low usage at the moment - so 7.4 seemed like the right pick
https://github.com/civicrm/civicrm-core/pull/222655.46.0https://lab.civicrm.org/dev/core/-/issues/2995From Email Address Options Bug2023-11-27T05:03:22ZLKuttnerFrom Email Address Options BugWhen sending some emails, such as using CiviRules and the MailAPI extension, the display name of the specified from email address option is being read from the non-ui editable Name column, while the email address is read from the Label c...When sending some emails, such as using CiviRules and the MailAPI extension, the display name of the specified from email address option is being read from the non-ui editable Name column, while the email address is read from the Label column. This results email being sent with the display name of FIXME when using the from email address that was created during the initial CiviCRM set-up. The Name of the first created email address is "FIXME" <info@example.com"> and is not editable via the user interface. This record can be updated by editing option_group_id 31 value 1 in the database.https://lab.civicrm.org/dev/core/-/issues/2994Support oEmbed for external facing pages2024-03-21T18:50:44ZJoeMurraySupport oEmbed for external facing pagesAn important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different...An important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different site that exposes its content as oEmbed.
There is a good list of oEmbed providers, and CiviCRM should aspire to join it: https://oembed.com/providers.json
We should expose the following 'pages' as oEmbed content, using appropriate standard markup for use in WordPress and Drupal 8.6+ sites, and support the workflow for confirmation, thank you, and search results as appropriate:
1. Phase 1
1. Contribution page
2. Event info page
3. Event registration page
2. Phase 2
1. Something like a profile create/edit page that does not involve a payment but allows submission of a form to create a contact and activity and perhaps more (eg parent-child relationship)
2. Search Kit page
3. Petition page
3. Phase 3
1. Personal Campaign Page
2. Pages from extensions, eg Grant Application Page
3. CiviSurvey pages
Note: I haven't done the research to understand the details of oEmbed and potential challenges. I think this could be done through extension(s).
I'm posting this in order to generate discussion for a possible Roadmap item.https://lab.civicrm.org/dev/core/-/issues/2993Add option to delete old log files during rotation2023-11-02T05:03:25ZJoeMurrayAdd option to delete old log files during rotationOverview
----------------------------------------
_When creating a new log file, optionally remove old ones like logrotate allows (https://linux.die.net/man/8/logrotate)._
Example use-case
----------------------------------------
CiviCR...Overview
----------------------------------------
_When creating a new log file, optionally remove old ones like logrotate allows (https://linux.die.net/man/8/logrotate)._
Example use-case
----------------------------------------
CiviCRM's simplistic log creation does not actually rotate, it just makes more files for the logs and never deletes any (https://github.com/civicrm/civicrm-core/blob/5433c81f5648bf484ef16debeca3c7e6e98d1a11/CRM/Core/Error.php#L684). This can be dangerous for systems where there is little to no active monitoring of logs, eg small orgs without a provider, or on a shoestring budget. It is also dangerous in cases where a change starts to generate high volumes of error messages, eg a client recently misconfigured a civirule and GBs of log were produced every hour. Running out of disk space causes hard crashes of sites.
It is standard practice to have a process in place to ensure that logs do not end up taking too much disk space. We propose allowing a setting to be added to civicrm.settings.php (eg NUM_LOG_FILES_TO_RETAIN) that would allow the number of logs to be retained to be specified. The default setting would be 0, meaning an infinite number, which would retain current behaviour. We would also be happy to make it another number such as 6, which would be 6 months or more than 1.5GB.
My understanding is that settings that can be added to civicrm.settings.php should be included as comments with explanation that can be uncommented, so we will do this.
Current behaviour
----------------------------------------
_CiviCRM generates log files with random hash names, and creates new ones every month or when the file exceeds 256MB. Log file space grows continuously unless log files are manually deleted._
Proposed behaviour
----------------------------------------
_When a new log file is created, if a maximum number of log files has been specified, then delete those over that number, starting with oldest first_
Comments
----------------------------------------
_To reduce the complexity of the settings pages, we propose that this be implemented as a setting that could be optionally specified in civicrm.settings.php._Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/2992cancelling preApproval payment and continue with credit card does not work.2023-11-21T05:03:16Zsunilcancelling preApproval payment and continue with credit card does not work.Overview
----------------------------------------
This issue related to preApproval payment. its present on Contribution and Event registartion form.
E.g Paypal Express.
Reproduction steps
----------------------------------------
1. Se...Overview
----------------------------------------
This issue related to preApproval payment. its present on Contribution and Event registartion form.
E.g Paypal Express.
Reproduction steps
----------------------------------------
1. Set up Paypal Pro account (live and test mode).
1. The configure Processor on Contribution page.
1. Visit online page (live or test mode).
1. Select amount and click on Paypal Button
1. It redirct you to login page, there is link to cancel the process,link look like this
`http://drupal7.test/civicrm/contribute/transact?_qf_Main_display=1&qfKey=CRMContributeControllerContribution67xqxb55jcow0k0gcs8sc8c4cs0c4sc00s4gcwg0skg84ws840_7945&cancel=1&token=EC-1V540554M6477413R` . Click on it. It redirect you to main page.
1. Now this time do the transaction using credit card.
1. Once you reach to CiviCRM confirm page and click to pay amount. You will get the error like this
`Payment Processor Error message :Express Checkout PayerID is missing.`
In Case of event registration, we were not passing qfkye to cancel url. when we cancel the payment on paypal express login page.. we are getting `Could not find valid value for id` Error.
After correcting the cancel url we need to apply same changes to reset `preApproval` data to continue payment with other method without resetting the page.https://lab.civicrm.org/dev/drupal/-/issues/171Drupal 10 requires Guzzle 72022-07-10T15:27:07ZDaveDDrupal 10 requires Guzzle 7It doesn't seem like there is any major change but I'm not super-familiar with everywhere civi uses guzzle: https://github.com/guzzle/guzzle/blob/master/UPGRADING.md#other-backwards-compatibility-breaking-changes
The main thing at the m...It doesn't seem like there is any major change but I'm not super-familiar with everywhere civi uses guzzle: https://github.com/guzzle/guzzle/blob/master/UPGRADING.md#other-backwards-compatibility-breaking-changes
The main thing at the moment is you need to do something like `composer require guzzlehttp/guzzle:"6.3.0 as 7.4.0"` just to get the civi files downloaded, but that's obviously not the right thing to do.5.52.0https://lab.civicrm.org/dev/user-interface/-/issues/44Custom Field Groups Screen - Flip "Preview" and "Settings"2021-12-26T21:28:17ZJonGoldCustom Field Groups Screen - Flip "Preview" and "Settings"This is an "is it everyone or just me" situation - and I think we need to ask folks who are newer to CiviCRM. But personally, I don't use the "Preview" link on the Custom Field Groups, but I *do* use "Settings". It's pretty quick to fl...This is an "is it everyone or just me" situation - and I think we need to ask folks who are newer to CiviCRM. But personally, I don't use the "Preview" link on the Custom Field Groups, but I *do* use "Settings". It's pretty quick to flip them, so mainly looking for UX feedback.
@bgm Do you have the ability to grep the web server logs for Spark over a long enough time period to see which link is used more by Spark users?
More broadly - I think it's worth discussing Spark as a platform for collecting anonymized user behavior data (with consent and transparency obviously).JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/2991Proposal: rework name fields to make name entity2023-12-13T09:20:59ZAndie HuntProposal: rework name fields to make name entity## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handli...## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handling systems, have led me to think that the solution to names isn't creating more or better-labeled fields for more types of names but rather to make name its own entity akin to phone, email, address, website, etc.
Once name is an entity, each contact could have multiple names, one primary, but with a bunch of possible types:
- legal name
- trade name
- nickname
- former name
- name in other language/culture
- name in an external system
- common mistakes
There could also be a bunch of name renderings - ways the name is displayed or inserted:
- sort name
- display name
- name for greetings
- name for listings
The latter could generate the existing greetings and potentially others. In particular, I think there's a real gap for how you insert someone's name in emails where you don't want "Dear", "Hi", or any other greeting as such.
Finally, the duplicate matching process could look for any of the possible names, cutting down on duplicates.
## Example use-case
### Organization has multiple names
An organization has their corporate name, colloquial name, and the name of their main product. These all refer to the same contact, but employees might fill the current employer field as one of the various ways. As an admin, you want to avoid having to merge them all the time. As a routine user, you will want to know all the various ways an organization is termed. As a bookkeeper, you will want to send invoices, checks, receipts, etc. with the correct entity name.
In this case, there are three name entities:
| Organization name | Name type | Is primary |
| ------------------------ | --------- | ---------- |
| Widget Productions | General | true |
| Widget Productions, Inc. | Legal | false |
| WidgetPro | General | false |
| Widgetizer | Product | false |
There would be a couple of name renderings:
| Rendering type | Value |
| -------------- | ------------------------ |
| Display Name | Widget Productions |
| Sort Name | Widget Productions |
| Billing name | Widget Productions, Inc. |
### Minister has variety of prefixes
Someone's name is styled differently when it is listed versus when they are addressed. In this case, a minister is styled "The Reverend Sally Jones" or "Rev. Sally Jones" when listed somewhere but her salutation is "Ms. Jones".
I think she'd have only one entry as a name entity but several renderings:
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------------|------------|-------------|-----------|--------|-----------|------------|
| Ms. | The Reverend | Sally | Lynn | Jones | *null* | General | true |
These values are going to depend upon your sitewide preferences and choices for this contact, but a reasonable default might produce the following:
| Rendering type | Value |
|-----------------|--------------------------|
| Display Name | Sally Lynn Jones |
| Sort Name | Jones, Sally Lynn |
| Addressee | The Reverend Sally Jones |
| Email greeting | Dear Sally |
| Postal greeting | Dear Ms. Jones |
| Short name | Sally |
### An organization has a different name in another language
This should be self-explanatory:
| Organization name | Name type | Is primary |
|--------------------------|-----------|------------|
| Médecins Sans Frontières | General | true |
| Doctors Without Borders | Localized | false |
| Rendering type | Value |
|----------------|--------------------------|
| Display Name | Médecins Sans Frontières |
| Sort Name | Médecins Sans Frontières |
| English Name | Doctors Without Borders |
### Someone's family name comes before their given name
I'm on the fence about how this should be handled, but all I know is that we currently handle it terribly (except maybe in translations that assume it?) One option might be to call the fields first, middle, and last names according to position (*Ai* being the family name, but it comes first):
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Ai | *null* | Weiwei | *null* | General | true |
Alternatively, we could name the fields according to function:
| Prefix | Title | Given name | Additional name | Surname | Suffix | Name type | Is primary |
|--------|--------|------------|-----------------|---------|--------|-----------|------------|
| Mr. | *null* | Weiwei | *null* | Ai | *null* | General | true |
In either case, the renderings should be like
| Rendering type | Value |
|-----------------|---------------|
| Display Name | Ai Weiwei |
| Sort Name | Ai Weiwei |
| Addressee | Mr. Ai Weiwei |
| Email greeting | Dear Weiwei |
| Postal greeting | Dear Mr. Ai |
| Short name | Weiwei |
### Clinic needs to retain legal name for insurance billing
Someone's real name doesn't match their legal name, let alone their nickname. While most organizations wouldn't want or need to record a contact's dead name, a clinic might need it for billing correspondence. In this case, a patient is named Stephen, or Steve for short, but his legal name is Barbara.
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Stephen | Wayne | Smith | *null* | General | true |
| Mr. | *null* | Steve | *null* | Smith | *null* | Nickname | false |
| *null* | *null* | Barbara | Wayne | Smith | *null* | Legal | false |
All of the name renderings would use the primary name or nickname, but the clinic could add an additional name rendering for how they refer to patients when billing to the insurance company.
| Rendering type | Value |
|-----------------|----------------------|
| Display Name | Stephen Wayne Smith |
| Sort Name | Smith, Stephen Wayne |
| Addressee | Mr. Stephen Smith |
| Email greeting | Dear Steve |
| Postal greeting | Dear Mr. Smith |
| Short name | Steve |
| Insurance name | Barbara Smith |
### Household name possibilities
The current household name field befuddles lots of people and, in my opinion, makes households less attractive to use. If a household can have multiple names, it might be more useful.
You might use a Household name field that has everyone all listed out, or you could use the Last name field.
| Household name | Last name | Name type | Is primary |
|-------------------------------------|----------------|-----------|------------|
| Katherine Williams and Kendra Green | *null* | General | true |
| *null* | Williams-Green | General | false |
| Rendering type | Value |
|-----------------|------------------------------------------|
| Display Name | Katherine Williams and Kendra Green |
| Sort Name | Katherine Williams and Kendra Green |
| Addressee | The Williams-Green Household |
| Email greeting | Dear Katherine Williams and Kendra Green |
| Postal greeting | Dear Katherine Williams and Kendra Green |
Or if everyone shares a last name, a single name entity could do it all:
| Given name | Surname | Name type | Is primary |
|---------------|---------|-----------|------------|
| Jack and Jill | Hill | General | true |
(If we don't feel the need to stick with the legacy name field names, there's really no reason that "Given name" couldn't replace Organization name and Household name, too, so this illustrates that.)
| Rendering type | Value |
|-----------------|-------------------------|
| Display Name | Jack and Jill Hill |
| Sort Name | Hill, Jack and Jill |
| Addressee | The Hill Household |
| Email greeting | Dear Jack and Jill |
| Postal greeting | Dear Jack and Jill Hill |
## Current behavior
Right now, each of these situations is pretty annoying. It might require a separate custom field, located away from the name fields. It might make it difficult to automatically and accurately generate greetings for many people.
Currently, there are fields in the `civicrm_contact` table for each name part, plus nickname and legal name.
The only name renderings are display name and sort name (set sitewide), and addressee and email and postal greetings (site-wide presets and defaults, with custom options).
There's no provision for additional renderings or any way to do fallbacks like pick a nickname and fall back to a first name.
There's a limited number of alternative names for a contact.
Alternative names are not compared against primary names when deduping.
## Proposed behavior
### Schema
The name fields would be gone from the contact entity, and a new `civicrm_contact_name` entity is added with one of the following sets of fields:
| Traditional | Radical |
|-------------------|-------------------|
| id | id |
| contact_id | contact_id |
| contact_name_type | contact_name_type |
| is_primary | is_primary |
| first_name | given_name |
| middle_name | additional_name |
| last_name | surname |
| prefix_id | prefix_id |
| suffix_id | suffix_id |
| formal_title | formal_title |
| household_name | |
| organization_name | |
The `contact_name_type` would have reserved options General, Nickname, Legal, and Localized, but more could be added.
A new `civicrm_contact_rendering` entity is added with the following set of fields:
| Fields |
|-------------------------------|
| id |
| contact_id |
| contact_rendering_type |
| contact_rendering_template_id |
| value |
The `contact_rendering_type` would have reserved options Display name, Sort name, Addressee, Email greeting, Postal greeting, and Short name, but more could be added.
The `contact_rendering_template_id` would be akin to the existing `addressee_id`, `email_greeting_id`, and `postal_greeting_id`, while `value` would be the actual generated or custom value.
### Upgrade
On upgrade, the `civicrm_contact_name` would be populated with the main name fields, with nickname and legal name as additional rows. On upgrade, the `civicrm_contact_rendering` would be populated with rows for each contact's display name, sort name, addressee, and greetings.
### Contact rendering templates
The contact rendering templates would have the option to specify a contact name type to provide the default value, with a fallback to the primary contact name. For example, email greeting might default to "Dear {nickname.first_name}", where it picks the value of `first_name` from a contact's name entry with the name type of Nickname, and if that isn't available, it picks the primary name entry's value for `first_name`.
### Dedupe
When matching contacts on any of the name fields, the dedupe process would compare against all rows in the `civicrm_contact_name` field, regardless of `contact_name_type` or if it's the primary name.
## Comments
This is all very early in the process: I think it's important to do something like this, but I don't have my heart on the specific mechanisms here. I think it would need to be evaluated for performance in a variety of ways, but I don't expect it would have too huge of an impact.https://lab.civicrm.org/dev/core/-/issues/2990CiviCampaign - Move data into a single table2024-03-11T20:47:07ZcolemanwCiviCampaign - Move data into a single tableBackground
------------
CiviCampaign is an optional component (disabled by default on new installs). It allows contributions, events, etc. to be associated with a campaign.
Campaigns are basically like tags. The campaign itself contain...Background
------------
CiviCampaign is an optional component (disabled by default on new installs). It allows contributions, events, etc. to be associated with a campaign.
Campaigns are basically like tags. The campaign itself contains a bit more data (type, start & end date, description, goal) than a Tag, but the mechanism for linking an entity with a campaign is a very simple foreign key.
How it works now
-------------
"Tagging" an entity with a campaign works via a column on that entity's table. E.g. `civicrm_contribution_page.campaign_id`. There are currently 10 entities with such a column, allowing them to be "tagged" with a campaign_id.
The problem
-----------
This design involves tight coupling between CiviCampaign and other components, as it involves adding a column to their tables. It also limits the possibilities for which entities can have a campaign_id.
The solution
----------
There is another common pattern in CiviCRM, which is to add a small "bridge" table to join two entities, which is a good alternative to adding a column. `civicrm_entity_tag` is one example. It includes `tag_id`, `entity_id` and `entity_table` columns, and an OptionGroup `tag_used_for` which lists all the possible values for `entity_table`. Importantly, it's easily added-on by extensions; if an extension provides a new entity type which should be taggable, is just adds it to the option list.
The migration
------------
If CiviCampaign were a greenfield project designed today, we wouldn't think twice about structuring the data with a bridge table instead of littering our database with `campaign_id` columns. As a brownfield problem, fixing the structure may be more trouble than it's worth, but I thought I'd at least articulate how it might be possible:
- APIv4 has a new concept of "Extra" calculated fields, which could provide a faux `campaign_id` column for backward compatibility.
- CiviCampaign could implement a post-hook to save campaign_id if it's passed into "create" params. This would keep create and update working as if that column still exists.
- APIv3 would require some hacking to make a pseudo-field to fill in for a missing `campaign_id`.
Random musings
-----------
While researching this, I stumbled across a table called `civicrm_campaign_group`. I have no idea what it does but it looks very similar to how I imagined the new bridge table would be. It includes a `campaign_id`, `entity_id` and `entity_table` column. Why does this table exist??https://lab.civicrm.org/dev/core/-/issues/2989Import of contribution fails when invalid campaign id is provided2022-02-01T23:27:10ZKurund JalmiImport of contribution fails when invalid campaign id is providedCurrently, we do not validate campaign id hence contribution import silently fails to import.Currently, we do not validate campaign id hence contribution import silently fails to import.5.47.0https://lab.civicrm.org/dev/core/-/issues/2988Civi Scss compliler is autoloaded, and its (old) verion conflict with more re...2023-10-30T05:03:24ZjbonlineaCivi Scss compliler is autoloaded, and its (old) verion conflict with more recent ones causing fatail errorsHi there,
It seems there is a issue / conflict between CiviCRM Scss compiler and my wordpress theme (gantry 5) Scss compiler.
We've discussed it [here](https://chat.civicrm.org/civicrm/pl/mfsmpmhewjyd7p7srbe1h9cyfa) where I've linked w...Hi there,
It seems there is a issue / conflict between CiviCRM Scss compiler and my wordpress theme (gantry 5) Scss compiler.
We've discussed it [here](https://chat.civicrm.org/civicrm/pl/mfsmpmhewjyd7p7srbe1h9cyfa) where I've linked what put me on track to think is was due to Civi Scss compiler.
A workaround is to rename the folder `civicrm/civicrm/vendor/scssphp`
Let me know if you want more detail
Cheers
PHP 7.4
WP 5.8.2
Civi 5.40.2
Gantry 5.5.6 (theme I use that also bundle an scss compiler, but there are many others)https://lab.civicrm.org/dev/wordpress/-/issues/116Issue related to Civi and WP Scss compiler causing fatal error2021-12-09T15:17:29ZjbonlineaIssue related to Civi and WP Scss compiler causing fatal errorHi there,
It seems there is a issue / conflict between CiviCRM Scss compiler and my wordpress theme (gantry 5) Scss compiler.
We've discussed it [here](https://chat.civicrm.org/civicrm/pl/mfsmpmhewjyd7p7srbe1h9cyfa) where I've linked w...Hi there,
It seems there is a issue / conflict between CiviCRM Scss compiler and my wordpress theme (gantry 5) Scss compiler.
We've discussed it [here](https://chat.civicrm.org/civicrm/pl/mfsmpmhewjyd7p7srbe1h9cyfa) where I've linked what put me on track to think is was due to Civi Scss compiler.
A workaround is to rename the folder `civicrm/civicrm/vendor/scssphp`
Let me know if you want more detail
Cheers
PHP 7.4
WP 5.8.2
Civi 5.40.2
Gantry 5.5.6 (theme I use that also bundle an scss compiler, but there are many others)https://lab.civicrm.org/dev/core/-/issues/2987API call Contribution.repeattransaction fails with recurring contributions th...2023-03-20T01:51:15ZandrewcormickdockeryAPI call Contribution.repeattransaction fails with recurring contributions that have a one-to-one custom group record attachedOverview
----------------------------------------
Re: https://chat.civicrm.org/civicrm/pl/i8fht46isjn5uj4dbdosrqpqmr
The API call Contribution.repeattransaction appears to write a custom group record twice when completing a transaction....Overview
----------------------------------------
Re: https://chat.civicrm.org/civicrm/pl/i8fht46isjn5uj4dbdosrqpqmr
The API call Contribution.repeattransaction appears to write a custom group record twice when completing a transaction. On the second attempt, a database error caused by the clashing unique key is generated. The transaction appears to end up being recorded correctly, however the fatal error means that processes which call this function repeatedly end up not completing. Our use case involves finding all Eway transactions whose next scheduled contribution date is today, and then calling Contribution.repeattransaction for each one.
Reproduction steps
----------------------------------------
1. Ensure a one-to-one custom group exists against contributions.
1. Ensure a recurring contribution exists with a record in that custom group against the transaction.
1. Run `drush cvapi Contribution.repeattransaction contribution_status_id=Completed total_amount=<some value> contribution_recur_id=<contribution_recur_id in previous step> trxn_id=<random unique number>`
Current behaviour
----------------------------------------
Failure message appears: `DB Error: already exists`; process terminates. Record appears to be correctly produced in CiviCRM.
Log details:
```
Dec 07 12:40:11 [error] $Fatal Error Details = Array
(
[callback] => Array
(
[0] => CRM_Core_Error
[1] => exceptionHandler
)
[code] => -5
[message] => DB Error: already exists
[mode] => 16
[debug_info] => INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_field_2`,`custom_field_3`,`custom_field_4`,`entity_id` ) VALUES ( 1,'Some Text','','22943426',123456 ) [nativecode=1062 ** Duplicate entry '123456' for key 'unique_entity_id']
[type] => DB_Error
[user_info] => INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_field_2`,`custom_field_3`,`custom_field_4`,`entity_id` ) VALUES ( 1,'Some Text','','22943426',123456 ) [nativecode=1062 ** Duplicate entry '123456' for key 'unique_entity_id']
[to_string] => [db_error: message="DB Error: already exists" code=-5 mode=callback callback=CRM_Core_Error::exceptionHandler prefix="" info="INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_field_2`,`custom_field_3`,`custom_field_4`,`entity_id` ) VALUES ( 1,'Some Text','','22943426',123456 ) [nativecode=1062 ** Duplicate entry '123456' for key 'unique_entity_id']"]
)
Dec 07 12:40:11 [debug] $backTrace = #0 /path/to/civi/root/civicrm/CRM/Core/Error.php(942): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /path/to/civi/root/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php(922): CRM_Core_Error::exceptionHandler(Object(DB_Error))
#2 /path/to/civi/root/civicrm/vendor/pear/db/DB.php(997): PEAR_Error->__construct("DB Error: already exists", -5, 16, (Array:2), "INSERT INTO civicrm_value_contribution_custom_group_1 ( `hide_from_slider_182`,`custom_...")
#3 /path/to/civi/root/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php(575): DB_Error->__construct(-5, 16, (Array:2), "INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#4 /path/to/civi/root/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php(223): PEAR->_raiseError(Object(DB_mysqli), NULL, -5, 16, (Array:2), "INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", "DB_Error", TRUE)
#5 /path/to/civi/root/civicrm/vendor/pear/db/DB/common.php(1928): PEAR->__call("raiseError", (Array:7))
#6 /path/to/civi/root/civicrm/vendor/pear/db/DB/mysqli.php(936): DB_common->raiseError(-5, NULL, NULL, "INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", "1062 ** Duplicate entry '123456' for key 'unique_entity_id'")
#7 /path/to/civi/root/civicrm/vendor/pear/db/DB/mysqli.php(406): DB_mysqli->mysqliRaiseError()
#8 /path/to/civi/root/civicrm/vendor/pear/db/DB/common.php(1234): DB_mysqli->simpleQuery("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#9 /path/to/civi/root/civicrm/packages/DB/DataObject.php(2696): DB_common->query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#10 /path/to/civi/root/civicrm/packages/DB/DataObject.php(1829): DB_DataObject->_query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#11 /path/to/civi/root/civicrm/CRM/Core/DAO.php(468): DB_DataObject->query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#12 /path/to/civi/root/civicrm/CRM/Core/DAO.php(1621): CRM_Core_DAO->query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", TRUE)
#13 /path/to/civi/root/civicrm/CRM/Core/BAO/CustomValueTable.php(271): CRM_Core_DAO::executeQuery("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", (Array:5))
#14 /path/to/civi/root/civicrm/CRM/Core/BAO/CustomValueTable.php(391): CRM_Core_BAO_CustomValueTable::create((Array:2), "create")
#15 /path/to/civi/root/civicrm/CRM/Core/BAO/CustomValueTable.php(411): CRM_Core_BAO_CustomValueTable::store((Array:5), "civicrm_contribution", 123456, "create")
#16 /path/to/civi/root/civicrm/CRM/Core/DAO.php(2022): CRM_Core_BAO_CustomValueTable::postProcess((Array:5), "civicrm_contribution", 123456, "Contribution", "create")
#17 /path/to/civi/root/civicrm/CRM/Contribute/BAO/Contribution.php(2394): CRM_Core_DAO->copyCustomFields(123455, 123456)
#18 /path/to/civi/root/civicrm/CRM/Contribute/BAO/Contribution.php(4000): CRM_Contribute_BAO_Contribution::repeatTransaction((Array:10), (Array:18))
#19 /path/to/civi/root/civicrm/api/v3/Contribution.php(687): CRM_Contribute_BAO_Contribution::completeOrder((Array:10), "2269", NULL, NULL)
#20 /path/to/civi/root/civicrm/api/v3/Contribution.php(635): _ipn_process_transaction((Array:7), Object(CRM_Contribute_BAO_Contribution), (Array:10), (Array:10))
#21 /path/to/civi/root/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_contribution_repeattransaction((Array:7))
#22 /path/to/civi/root/civicrm/Civi/API/Kernel.php(149): Civi\API\Provider\MagicFunctionProvider->invoke((Array:8))
#23 /path/to/civi/root/civicrm/Civi/API/Kernel.php(81): Civi\API\Kernel->runRequest((Array:8))
#24 /path/to/civi/root/civicrm/api/api.php(132): Civi\API\Kernel->runSafe("contribution", "repeattransaction", (Array:5))
#25 /path/to/civi/root/civicrm/drupal/drush/civicrm.drush.inc(1581): civicrm_api3("Contribution", "repeattransaction", Array:5))
#26 /usr/local/src/drush/includes/command.inc(422): drush_civicrm_api("Contribution.repeattransaction", "contribution_status_id=Completed", "total_amount=11.00", "contribution_recur_id=22663", "trxn_id=aa11bb22cc33dd47")
#27 /usr/local/src/drush/includes/command.inc(231): _drush_invoke_hooks((Array:38), (Array:5))
#28 /usr/local/src/drush/includes/command.inc(199): drush_command("Contribution.repeattransaction", "contribution_status_id=Completed", "total_amount=11.00", "contribution_recur_id=22663", "trxn_id=aa11bb22cc33dd47")
#29 /usr/local/src/drush/lib/Drush/Boot/BaseBoot.php(67): drush_dispatch((Array:38))
#30 /usr/local/src/drush/includes/preflight.inc(67): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#31 /usr/local/src/drush/drush.php(12): drush_main()
#32 {main}
```
Expected behaviour
----------------------------------------
Fatal error not raised; no attempt to write the exact same record twice.
I have mitigated this behaviour with the following patch, but this is clearly a temporary solution and the real root cause needs to be found:
```
diff --git a/CRM/Core/BAO/CustomValueTable.php b/CRM/Core/BAO/CustomValueTable.php
index 8556c4ec21..6773b994c4 100644
--- a/CRM/Core/BAO/CustomValueTable.php
+++ b/CRM/Core/BAO/CustomValueTable.php
@@ -55,7 +55,7 @@ class CRM_Core_BAO_CustomValueTable {
$hookOP = 'edit';
}
else {
- $sqlOP = "INSERT INTO $tableName ";
+ $sqlOP = "INSERT IGNORE INTO $tableName ";
$where = NULL;
$hookOP = 'create';
}
```
Environment information
----------------------------------------
* __Browser:__ _Firefox 94.0.2_
* __CiviCRM:__ _5.43.2_
* __PHP:__ _7.3_
* __CMS:__ _Drupal 7.82_
* __Database:__ _MariaDB 10.4.21_
* __Web Server:__ _Nginx 1.10.3_https://lab.civicrm.org/dev/drupal/-/issues/170kcfinder error 5002021-12-16T13:14:52Zolivierkcfinder error 500Civicrm 5.43 and 5.44 under Drupal 8, launching kcfinder from ckeditor give an 500 error.
Php script integration/civicrm.php try to call civicrm.config.php located in web/libraries/civicrm/, and this file does not exist.
Copying this fil...Civicrm 5.43 and 5.44 under Drupal 8, launching kcfinder from ckeditor give an 500 error.
Php script integration/civicrm.php try to call civicrm.config.php located in web/libraries/civicrm/, and this file does not exist.
Copying this file from an Drupal 7 installation is not sufficent, because civicrm.settings.php is not searched in correct location.
```
--- ../../mcm65/sites/all/modules/civicrm/civicrm.config.php 2021-11-15 21:42:13.000000000 +0100
+++ libraries/civicrm/civicrm.config.php 2021-12-07 21:14:33.380121826 +0100
@@ -87,6 +87,11 @@
return $confdir;
}
+ if (file_exists($_SERVER['DOCUMENT_ROOT'] . DIRECTORY_SEPARATOR . 'sites' . DIRECTORY_SEPARATOR . 'default' .
+ DIRECTORY_SEPARATOR . 'civicrm.settings.php')) {
+ return $_SERVER['DOCUMENT_ROOT'] . DIRECTORY_SEPARATOR . 'sites' . DIRECTORY_SEPARATOR . 'default';
+ }
+
if (!file_exists($confdir) && !$skipConfigError) {
echo "Could not find valid configuration dir, best guess: $confdir<br/><br/>\n";
exit();
```
Another point, .htaccess in /web directory Deny access to most php files in public directory.
We must allow access to php files under kcfinder directory :
```
# For security reasons, deny access to other PHP files on public sites.
# Note: The following URI conditions are not anchored at the start (^),
# because Drupal may be located in a subdirectory. To further improve
# security, you can replace '!/' with '!^/'.
# Allow access to PHP files in /core (like authorize.php or install.php):
RewriteCond %{REQUEST_URI} !/core/[^/]*\.php$
# Allow access to test-specific PHP files:
RewriteCond %{REQUEST_URI} !/core/modules/system/tests/https?.php
# Allow access to Statistics module's custom front controller.
# Copy and adapt this rule to directly execute PHP files in contributed or
# custom modules or to run another PHP application in the same directory.
RewriteCond %{REQUEST_URI} !/core/modules/statistics/statistics.php$
RewriteCond %{REQUEST_URI} !/libraries/civicrm/packages/kcfinder/[a-z_]+\.php$
RewriteCond %{REQUEST_URI} !/libraries/civicrm/packages/kcfinder/.*/[a-z_]+\.php$
# Deny access to any other PHP files that do not match the rules above.
# Specifically, disallow autoload.php from being served directly.
RewriteRule "^(.+/.*|autoload)\.php($|/)" - [F]
```https://lab.civicrm.org/dev/core/-/issues/2986PCP: Account creation profile does not support contact image2023-10-28T05:03:27ZKurund JalmiPCP: Account creation profile does not support contact image## Steps to replicate
* Create a profile with contact image field (Image URL)
* Include this profile as a part of PCP account creation process
* Contact image is never uploaded.## Steps to replicate
* Create a profile with contact image field (Image URL)
* Include this profile as a part of PCP account creation process
* Contact image is never uploaded.https://lab.civicrm.org/dev/core/-/issues/2985Original value is displayed after setting custom event field blank2023-01-21T04:07:10ZBobSOriginal value is displayed after setting custom event field blankOverview
----------------------------------------
Attempting to change the value of a custom event field from non-blank to blank results in the original value being populated in the form after the form is saved. If the form is subsequent...Overview
----------------------------------------
Attempting to change the value of a custom event field from non-blank to blank results in the original value being populated in the form after the form is saved. If the form is subsequently saved again, then the original non-blank value is written to the DB.
Reproduction steps
----------------------------------------
1. Create a custom event fieldset.
1. Add a custom field of type Alphanumeric, Single line input field. Accept all default settings.
1. Edit an event.
1. Set the new custom field to a non-blank value and save the event.
1. Set the new custom field to a blank value and save the event.
Current behaviour
----------------------------------------
The custom field displays the non-blank value. If the form is saved, the blank value currently in the DB is overwritten with the non-blank value.
Expected behaviour
----------------------------------------
The custom field should display the last saved value: blank.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ Chrome Version 96.0
* __CiviCRM:__ _5.45.alpha1_ <!-- If this problem relates to an upgrade, then specify both old and new versions -->
* __PHP:__ _7.4_
* __CMS:__ _Drupal 7.82_
* __Database:__ _MariaDB 10.4_
* __Web Server:__ _Apache 2.4_
Comments
----------------------------------------
Confirmed on dmaster.demo.civicrm.org5.59.0https://lab.civicrm.org/dev/core/-/issues/2984APIv4: Confusing error when calling CaseType API2021-12-09T21:08:05ZtottenAPIv4: Confusing error when calling CaseType APIOverview
----------------------------------------
`CaseType.get` API has a confusing disposition.
Reproduction steps
----------------------------------------
1. Disable CiviCase
1. Call `cv api4 CaseType.get`
Current behavior
--------...Overview
----------------------------------------
`CaseType.get` API has a confusing disposition.
Reproduction steps
----------------------------------------
1. Disable CiviCase
1. Call `cv api4 CaseType.get`
Current behavior
----------------------------------------
The call to `CaseType.get` begins executing - but it fails in the middle of query-building.
```
cv api4 CaseType.get -v
Entity: CaseType
Action: get
Params: {
"version": 4,
"checkPermissions": false
}
[Symfony\Component\Debug\Exception\FatalThrowableError]
Type error: array_merge(): Argument #1 must be of type array, null given
Exception trace:
() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:223
array_merge() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:223
Civi\Api4\Query\Api4SelectQuery->buildSelectClause() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:149
Civi\Api4\Query\Api4SelectQuery->getSql() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/Civi/Api4/Query/Api4SelectQuery.php:165
Civi\Api4\Query\Api4SelectQuery->run() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/Civi/Api4/Generic/DAOGetAction.php:111
Civi\Api4\Generic\DAOGetAction->getObjects() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/Civi/Api4/Generic/DAOGetAction.php:99
Civi\Api4\Generic\DAOGetAction->_run() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/Civi/Api4/Provider/ActionObjectProvider.php:68
Civi\Api4\Provider\ActionObjectProvider->invoke() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/Civi/API/Kernel.php:149
Civi\API\Kernel->runRequest() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/Civi/Api4/Generic/AbstractAction.php:234
Civi\Api4\Generic\AbstractAction->execute() at /home/m/bknix/build/tmp/web/sites/all/modules/civicrm/api/api.php:85
civicrm_api4() at phar:///home/m/bknix/bin/cv/src/Command/Api4Command.php:145
Civi\Cv\Command\Api4Command->execute() at phar:///home/m/bknix/bin/cv/vendor/symfony/console/Command/Command.php:257
Symfony\Component\Console\Command\Command->run() at phar:///home/m/bknix/bin/cv/vendor/symfony/console/Application.php:850
Symfony\Component\Console\Application->doRunCommand() at phar:///home/m/bknix/bin/cv/vendor/symfony/console/Application.php:193
Symfony\Component\Console\Application->doRun() at phar:///home/m/bknix/bin/cv/src/Application.php:46
Civi\Cv\Application->doRun() at phar:///home/m/bknix/bin/cv/vendor/symfony/console/Application.php:124
Symfony\Component\Console\Application->run() at phar:///home/m/bknix/bin/cv/src/Application.php:15
Civi\Cv\Application::main() at phar:///home/m/bknix/bin/cv/bin/cv:27
require() at /home/m/bknix/bin/cv:14
```
Expected behavior
----------------------------------------
Either:
* Allow `CaseType` API calls to work. Return the list of case types.
* OR... Disallow `CaseType` API calls. Give a clear error; eg
* `CaseType.get API is unavailable.`
* `Cannot process request. CiviCase is disabled.`https://lab.civicrm.org/dev/core/-/issues/2983On "My Contact Dashboard", what is the Manage Case link in the relationships ...2024-02-26T05:03:22ZDaveDOn "My Contact Dashboard", what is the Manage Case link in the relationships section supposed to do?At civicrm/user, in the relationships section, if you have a relationship to a case there's a link under More at the right that says Manage Case. But clicking it does nothing. Also there's an error on the page because it's trying to crea...At civicrm/user, in the relationships section, if you have a relationship to a case there's a link under More at the right that says Manage Case. But clicking it does nothing. Also there's an error on the page because it's trying to create the link as CRM_Core_Action::NONE but there is no offset 0 in the $links array, but I'm not sure if that's the full reason it fails.
It looks like it's supposed to do a popup-y version of manage case? Except it looks like the link if it were working would be e.g. civicrm/case/ajax/details?caseId=1&cid=3&snippet=4, which just gives a little table.
I'm just wondering if it's worth trying to fix or to just remove it. I have no idea when it broke - never even knew it existed.https://lab.civicrm.org/dev/core/-/issues/2982Repeat Contributions CiviReport warnings: Formatting non-numeric values is no...2021-12-09T00:48:06ZDaveDRepeat Contributions CiviReport warnings: Formatting non-numeric values is no longer supportedWhole screen fills with ones like this:
`User deprecated function: Formatting non-numeric values is no longer supported: 100.00 (2) Caller: CRM_Utils_Money::format in CRM_Core_Error::deprecatedWarning() (line 1060 of /srv/buildkit/build...Whole screen fills with ones like this:
`User deprecated function: Formatting non-numeric values is no longer supported: 100.00 (2) Caller: CRM_Utils_Money::format in CRM_Core_Error::deprecatedWarning() (line 1060 of /srv/buildkit/build/dmaster/web/sites/all/modules/civicrm/CRM/Core/Error.php).`
It must be the `(2)` part it doesn't like. Perhaps that repeat count should be its own column.
It's also showing `<br/>` in the column heading, which I assume is from the recent smarty escaping PRs.5.46.0