CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-11-23T07:23:38Zhttps://lab.civicrm.org/dev/core/-/issues/4332Error in membership status after a) changing membership type followed by b) p...2023-11-23T07:23:38ZJoeMurrayError in membership status after a) changing membership type followed by b) payment failure## Overview
Using a pay later payment when renewing a membership can lead to problems with the membership status, membership end date and membership type being changed at the time of the renewal being initiated ; these fields are update...## Overview
Using a pay later payment when renewing a membership can lead to problems with the membership status, membership end date and membership type being changed at the time of the renewal being initiated ; these fields are updated without a payment being recorded. It is possible that a payment will never be received, or its processing may fail. It is not easy to revert the data to its former state, or what it would have become through time from date of update to when the correction is attempted. For example, a renewal with a delayed payment might change the status from Grace to Current, the End Date from May 14, 2023 to May 14, 2024, and the Membership Type from General to Student.
These are very old problems dating to at least 2013 I believe.
The problem only occurs when both membership types have the same parent organization, and only for paid memberships. It occurs whether the membership period is Rolling or Fixed, and whether a membership type is being changed or not.
## Proposal
Refactor the current implementation so that a second, temporary membership is created that can store the new information without overwriting the old information until a payment is received for it. The new temporary membership would have a status of Pending. The Pending contribution would be related to the temporary rather than existing membership. When the payment is received (status=complete), the Pending membership's information is used to update the permanent membership, and the temporary membership record is deleted.
## Relevant code
In the Contribution Confirmation page postProcess call [legacyProcessMembership](https://github.com/civicrm/civicrm-core/blob/master/CRM/Contribute/Form/Contribution/Confirm.php#L1623), various fields are updated before the doPayment call is processed [here](https://github.com/civicrm/civicrm-core/blob/master/CRM/Contribute/Form/Contribution/Confirm.php#LL1702C42-L1702C51). As a result, the existing membership record is updated with selected membership type/end date/status after user submits a payment but before the code makes payment request, e.g. to a payment processor. This works if the payment went successfully.
In the case of a payment failure such as for IPN payment processors like Paypal Standard (occasionally when there is a delay on getting IPN callback or if the IPN response is not handled properly like https://lab.civicrm.org/dev/core/-/issues/1931) or for a manual Pay Later payment that isn't received, it leaves the selected membership in current/active state with a changed end date and possibly a different membership type. There is no fallback code written to revert the membership state or set it to Pending, and it isn't easy to reconstruct the data.
## New Behaviour
1. When initiating the Payment for Membership Renewal, create a new membership record and link the contribution in pending status to it. Add a new field, renewing_membership_id, to civicrm_membership to hold a reference from this 'temporary' pending membership to the existing membership that is being renewed. The existing membership record remains unchanged.
2. When the contribution status of the related contribution changes to Complete, update the original membership with the information from the temporary membership and delete the temporary membership.
Recommendation: delay the creation of activities for Membership Renewal (id=8), Change Membership Status (id=35) and Change Membership Type (id=36) until the contribution is completed.
Recommendation: create a new Activity Type, Membership Renewal Pending, to be created when the renewal request is received. In its body, provide: "ID of Membership being renewed: xx, Number of Periods: yy, Membership Type: [Label of Membership Type".
## Implementation:
1. Modify [here](https://github.com/civicrm/civicrm-core/blob/master/CRM/Contribute/Form/Contribution/Confirm.php#L2900), CRM_Member_BAO_Membership::getContactMembership and [here](https://github.com/civicrm/civicrm-core/blob/master/CRM/Contribute/Form/Contribution/Confirm.php#L2939).
2. Add / update new / existing unit tests on these new scenarios.EdselopezEdselopezhttps://lab.civicrm.org/dev/core/-/issues/3783Make Recent Items available providers an option group so extensions can exten...2022-08-10T04:17:03ZherbdoolMake Recent Items available providers an option group so extensions can extend itOverview
----------------------------------------
I would like my custom entities to be available to the Recent Items block.
In `CRM_Utils_Recent::getProviders()` there's a comment to make the hard-coded list into an Open Group. I agree...Overview
----------------------------------------
I would like my custom entities to be available to the Recent Items block.
In `CRM_Utils_Recent::getProviders()` there's a comment to make the hard-coded list into an Open Group. I agree. Let's make it so.
Current behaviour
----------------------------------------
Hard-coded list of some core entities.
Proposed behaviour
----------------------------------------
Create an option group and call it from this method.5.53.0https://lab.civicrm.org/dev/core/-/issues/3627Only add closing quote to href token output when necessary2022-06-11T14:57:26ZMichael McAndrewOnly add closing quote to href token output when necessary[Lines 1345 to 1350 of CRM/Mailing/BAO/Mailing.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Mailing/BAO/Mailing.php#L1345-L1350) add a trailing quote to tokens of 'type' `embedded_url` as according to the code:
```
...[Lines 1345 to 1350 of CRM/Mailing/BAO/Mailing.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Mailing/BAO/Mailing.php#L1345-L1350) add a trailing quote to tokens of 'type' `embedded_url` as according to the code:
```
// add trailing quote since we've gobbled it up in a previous regex
// function getPatterns, line 431
```
In my testing, sometimes the trailing quote is not gobbled, e.g. in this example `<p><a href="https://example.org/communication-preferences?cid1={contact.contact_id}&{contact.checksum}">Click here to select your communication preferences</a></p>`.
As a fix, I am proposing to only add a closing quote if the url does not already have one.https://lab.civicrm.org/dev/core/-/issues/3429API4: Make CaseType a managed entity2024-01-24T14:08:18ZherbdoolAPI4: Make CaseType a managed entityOverview
----------------------------------------
Allow CaseType to be a managed entity so it can be exported via api4.
Example use-case
----------------------------------------
1. Go to `/civicrm/api4`
1. Select CaseType and Export
C...Overview
----------------------------------------
Allow CaseType to be a managed entity so it can be exported via api4.
Example use-case
----------------------------------------
1. Go to `/civicrm/api4`
1. Select CaseType and Export
Current behaviour
----------------------------------------
Cannot export and use as a managed entity.https://lab.civicrm.org/dev/core/-/issues/3181PHP 8.12023-07-21T14:03:03ZJoeMurrayPHP 8.1We should support PHP 8.1.
The deprecations and breaking changes are listed in the release notes at https://www.php.net/releases/8.1/en.php#deprecations_and_bc_breaks
Here are the backward compatibility issues:
1. [ ] Tentative return...We should support PHP 8.1.
The deprecations and breaking changes are listed in the release notes at https://www.php.net/releases/8.1/en.php#deprecations_and_bc_breaks
Here are the backward compatibility issues:
1. [ ] Tentative return types in PHP built-in class methods
1. [ ] HTML entity en/decode functions process single quotes and substitute by default
1. [ ] $GLOBALS variable restrictions.
1. [ ] MySQLi: Default error mode set to exceptions
I am most worried about HTML encode/decode.
[ ] create edge test setup for PHP 8.1seamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/2927Warning: is_dir(): open_basedir restriction in effect2022-02-16T14:47:09ZherbdoolWarning: is_dir(): open_basedir restriction in effectOverview
----------------------------------------
See https://github.com/civicrm/civicrm-core/pulls?q=basedir+is%3Aopen (saw a bunch of PRs as highlighted in the dev digest but no associated issue).
Happens with APIv3 Explorer (https:/...Overview
----------------------------------------
See https://github.com/civicrm/civicrm-core/pulls?q=basedir+is%3Aopen (saw a bunch of PRs as highlighted in the dev digest but no associated issue).
Happens with APIv3 Explorer (https://github.com/civicrm/civicrm-core/pull/21593), /CRM/Core/Smarty/plugins/ (https://github.com/civicrm/civicrm-core/pull/21589), etc.
Reproduction steps
----------------------------------------
When a server has open_basedir restrictions, error message logged repeatedly when using CiviCRM backend, especially with APIv3. PHP message: PHP Warning: is_dir(): open_basedir restriction in effect. File ... is not within the allowed path(s)
Comments
----------------------------------------
Also see https://lab.civicrm.org/dev/core/-/issues/2795 - the solution should not simply suppress the errors.5.48.0https://lab.civicrm.org/dev/core/-/issues/2741Group visibility not respected after profile edit submission2023-09-12T05:03:26ZandyburnsGroup visibility not respected after profile edit submissionWhen using a profile in edit mode, my groups show up correctly. However, once I submit and go to the view screen to see what I submitted, it then exposes all the groups I am in, even if they are not set to 'Public Pages'. It should not e...When using a profile in edit mode, my groups show up correctly. However, once I submit and go to the view screen to see what I submitted, it then exposes all the groups I am in, even if they are not set to 'Public Pages'. It should not expose 'User and User Admin' visibility groups. I was using checksum function in this case.
I have less than 10 Public Groups, below is way more than that.
![Untitled](/uploads/1ba9fad0f6c44842aa6dcc8fd10dea9e/Untitled.png)https://lab.civicrm.org/dev/core/-/issues/2280Price set options show nine decimal places.2023-06-10T05:03:21ZspalmstromPrice set options show nine decimal places.Overview
----------------------------------------
The price in a price set option shows nine decimal places and not two.
Reproduction steps
----------------------------------------
1. Click on **Events -> Manage Price Sets**.
1. Click o...Overview
----------------------------------------
The price in a price set option shows nine decimal places and not two.
Reproduction steps
----------------------------------------
1. Click on **Events -> Manage Price Sets**.
1. Click on **View and Edit Price Fields** for a given event.
Current behaviour
----------------------------------------
The price shows to nine decimal places.
![image](/uploads/1080ecb928c6f2601d014afa2128bb19/image.png)
Expected behaviour
----------------------------------------
The price should show two decimal places.
![image](/uploads/40d5279f5c7ef20c0d6ff830184ebc88/image.png)
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is necessary. -->
* __Browser:__ _Edge_ but probably not relevant
* __CiviCRM:__ _5.32.2_ <!-- If this problem relates to an upgrade, then specify both old and new versions -->
* __PHP:__ _7.4__ but probably not relevant
* __CMS:__ _Drupal 9.1/Joomla 3.9.6_ but may not be relevant
* __Database:__ _MySQL 8.0.22_ but may not relevant
* __Web Server:__ _IIS 10_ but probably not relevant.
Comments
----------------------------------------
_Anything else you would like the reviewer to note._
The expected behaviour comes from a 5.30.1 system under Joomla and MySQL 5.6,https://lab.civicrm.org/dev/core/-/issues/2201PHP 8.0 Compatibility2021-12-09T16:32:47ZJoeMurrayPHP 8.0 CompatibilityPHP 8.0.0 is scheduled for release Nov 26, 2020.
Backward incompatible changes are listed at https://github.com/php/php-src/blob/php-8.0.0RC5/UPGRADING#L20 for 8.0.0RC5. As one would expect for a major version, it is a long list.
1. [...PHP 8.0.0 is scheduled for release Nov 26, 2020.
Backward incompatible changes are listed at https://github.com/php/php-src/blob/php-8.0.0RC5/UPGRADING#L20 for 8.0.0RC5. As one would expect for a major version, it is a long list.
1. [Create edge CI for PHP8](https://lab.civicrm.org/dev/core/-/issues/2200)https://lab.civicrm.org/dev/core/-/issues/2092Dedupe SUBSTR scalability problems2023-05-13T05:03:22ZJoeMurrayDedupe SUBSTR scalability problemsOverview
----------------------------------------
_When length of field is used in dedupe rules, rules tend not to work even on medium size sites._
Reproduction steps
----------------------------------------
1. On a site with 70k contac...Overview
----------------------------------------
_When length of field is used in dedupe rules, rules tend not to work even on medium size sites._
Reproduction steps
----------------------------------------
1. On a site with 70k contacts, a dudupe rule was defined that included first name (length 1), last name, email, phone, postal code.
1. Check for dedupes times out after 20m. On different runs it also took down MySQL with out of memory, and caused restarts of php-fpm.
1. Changing rule to first name, last name, email, phone, postal code yields 18.5k potential duplicate pairs in 3 minutes.
1. FWIW, using the deduper extension from Eileen didn't help with batch limit of 1000, but did improve things with search limit of 1000.
Current behaviour
----------------------------------------
Timeouts, mysql restarts, php restarts when using length restrictions on fields.
Expected behaviour
----------------------------------------
No timeouts or process restarts on medium sized databases even when using length restrictions on fields.
Implementation Alternative
----------------------------------------
The problem seems to stem from subquery join on substring function, which is well known to be not optimizable by MySQL: ON (SUBSTR(t1.first_name, 1, 1) = SUBSTR(t2.first_name, 1, 1)). Instead of this approach, one can imagine using Temporary tables that include SUBSTR(t.first_name, 1, 1) as a field that has index, and then the join would be optimized. While we use temp tables in reports, they do have the potential of causing their own issues, but I believe they are much reduced from the current situation here.
We don't currently have funding to work on this. Nonetheless, I thought it would be useful to report the problem and ask if there is approval for the concept of using temp tables with indexed fields for dedupe search criteria based on substrings?
I've labelled this as both improvement and bug since it seems to straddle the divide.https://lab.civicrm.org/dev/core/-/issues/1839Notice error Contribution Aggregate by Relationship report2023-03-29T05:03:21ZJoeMurrayNotice error Contribution Aggregate by Relationship reportOn dmaster just now (5.28.alpha1):
Notice: Undefined variable: entryFound in CRM_Report_Form_Contribute_History->alterDisplay() (line 842 of /srv/buildkit/build/dmaster/web/sites/all/modules/civicrm/CRM/Report/Form/Contribute/History.php).On dmaster just now (5.28.alpha1):
Notice: Undefined variable: entryFound in CRM_Report_Form_Contribute_History->alterDisplay() (line 842 of /srv/buildkit/build/dmaster/web/sites/all/modules/civicrm/CRM/Report/Form/Contribute/History.php).https://lab.civicrm.org/dev/core/-/issues/1795Using Parent tag in search form doesn't pull contacts marked with child tag i...2021-04-01T11:40:23ZMonish DebUsing Parent tag in search form doesn't pull contacts marked with child tag in search form resultReproduction steps
----------------------------------------
Steps to replicate:
1. Create a parent tag A and child tag A1
2. Create/update contact and tag with A1
3. Go to 'Find Contacts' or 'Find Contributions' and search by Tag - A
...Reproduction steps
----------------------------------------
Steps to replicate:
1. Create a parent tag A and child tag A1
2. Create/update contact and tag with A1
3. Go to 'Find Contacts' or 'Find Contributions' and search by Tag - A
Current behaviour
----------------------------------------
```
1. In 'Find Contact' result, it doesn't show contact which is tagged with child tag A1
2. In 'Find Contributions' result, it doesn't pull contribution(s) of contact tagged with child tag A1
```
Expected behaviour
----------------------------------------
Searching using a parent tag should automatically include the contacts which are in parent tag A and in its n child tag(s) - An. In this use-case it should show contact tagged with child A1 tag.
Comments
----------------------------------------
ping @JoeMurray @eileen @seamuslee @lcdweb5.29.0Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/1774$is_pay_later is TRUE in contribution receipt even if payment is completed.2023-03-16T05:03:20Zjitendra$is_pay_later is TRUE in contribution receipt even if payment is completed.To replicate -
- Create a pay later payment in civicrm using a front end contribution page.
- Note that the email received has `Invoice` in the subject. This text is set when you choose to pay via `pay later`.
- Complete the payment usi...To replicate -
- Create a pay later payment in civicrm using a front end contribution page.
- Note that the email received has `Invoice` in the subject. This text is set when you choose to pay via `pay later`.
- Complete the payment using `completetransaction` API (or use API explorer from UI). This is used by payment processors, xero, etc to complete the payment in civicrm on receiving the confirmation.
```
$result = civicrm_api3('Contribution', 'completetransaction', [
'id' => <contribution_id>,
]);
```
- The email confirmation still has `Invoice` in the subject.https://lab.civicrm.org/dev/core/-/issues/1727Allow sites to not have Billing Address Location Type2023-03-09T05:03:25ZJoeMurrayAllow sites to not have Billing Address Location TypeOverview
----------------------------------------
Currently CiviCRM has a confusing array of default location types and flags. This issue proposes changes that would allow the Is Billing flag to be the only indication that a location is ...Overview
----------------------------------------
Currently CiviCRM has a confusing array of default location types and flags. This issue proposes changes that would allow the Is Billing flag to be the only indication that a location is the billing location rather than also having a Location Type of Billing. The intent is to allow instances to experiment with simpler approaches to Location Types in order to improve the usability of the product.
Example use-case
Add a hook to allow billing location to be determined by an alternative implementation. Sample LExIM extension approach would return the address which has isBilling==True if there is no LocationType of 'Billing' or if there is no address passed to function with LocationType of 'Billing' from https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Payment/BaseIPN.php#L491
In current places where code creates a 'Billing' type address one potential approach would assume it should be home location type for individual and household contacts and work for organizational contacts. In all cases the Is Billing would still be set. The objective of this proposal is to refactor and introduce hooks in order to allow a LExIM extension to work. More details about specific places in code that need to be changed and ideas on what exactly to do in a simpler approach to be provided if there is potential for this proposal to go forward.
Comments
----------------------------------------
Just testing water for potential support before developing more detailed proposal. Otherwise we'll likely just override core for this particular client.JoeMurrayJoeMurrayhttps://lab.civicrm.org/dev/core/-/issues/1726Allow financial types to not have Expense account defined2022-08-26T20:51:52ZJoeMurrayAllow financial types to not have Expense account definedThis is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require ...This is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require Expenses account for every financial type, nor create relations to Expense and Premium accounts by default when creating a financial type.
Example use-case
----------------------------------------
1. Click on **Administer > CiviContribute > Financial Types**.
1. Click **Add Financial Type**.
1. Enter **Name** and click **Save**.
1. In Financial Accounts, there are Banking Fees and Premiums accounts, which is **undesirable**.
1. Click **Accounts** on the new Financial Type row.
1. Beside the 'Expense Account is', click **Delete**, then confirm by clicking **Delete** again.
1. Click on **Contributions > New Contribution**.
1. Select the Financial Type created above that does not have an Expense Account set up for it anymore, fill in **Contributor** and **Total Amount**, and click **Save**.
1. Try to edit the contribution but not in a popup, for example, go to the contact's page, right click on the edit button for the contribution and Copy Link Address, then paste address into a new tab. You'll see "Sorry, due to an error, we are unable to fulfill your request at the moment. You may want to contact your administrator or service provider with more details about what action you were performing when this occurred.
One of parameters (value: ) is not of the type Integer". This is caused by missing Expense account, **even though it is not needed**.
Current behaviour
----------------------------------------
See above.
Proposed behaviour
----------------------------------------
On creation of Financial Type, no Expense or Premiums account relationship would be setup. On editing a contribution (with a line item) with a financial type without an Expense account relationship setup, no error would occur.
Comments
----------------------------------------
The expectation when this was implemented circa 2014 was that payment processors would all soon record banking fees. That hasn't been the case for a variety of reasons.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/1621cascade delete of civicrm_financial_type to civicrm_entity_financial_account2023-02-20T05:04:09ZJoeMurraycascade delete of civicrm_financial_type to civicrm_entity_financial_account1. Background
The civicrm_entity_financial_account table stores, amongst other things, the relationships between civicrm_financial_type and civicrm_financial_account records.
2. Current situation
When financial types are deleted, the...1. Background
The civicrm_entity_financial_account table stores, amongst other things, the relationships between civicrm_financial_type and civicrm_financial_account records.
2. Current situation
When financial types are deleted, there are stranded records in civicrm_entity_financial_account with no referent for their financial_type_id.
3. Proposed change
When a financial type with id=n is going to be deleted, first
```sql
DELETE FROM civicrm_entity_financial_account WHERE financial_type_id=n;
```
On upgrade, run something like
```sql
DELETE efa.* FROM civicrm_entity_financial_account efa LEFT JOIN civicrm_financial_type ft ON efa.financial_type_id=ft.id WHERE ft.id IS NULL;
```seamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/1540Add user friendly error message on merge error.2020-01-21T05:14:18ZjitendraAdd user friendly error message on merge error.When a user tries to merge 2 contacts from an Advanced search screen, the following error is displayed if there is no Supervised rule present on the site.
![image](/uploads/efbef3eb21d963a0e8affd6afec30d53/image.png)
Related SE - https...When a user tries to merge 2 contacts from an Advanced search screen, the following error is displayed if there is no Supervised rule present on the site.
![image](/uploads/efbef3eb21d963a0e8affd6afec30d53/image.png)
Related SE - https://civicrm.stackexchange.com/questions/17432/error-message-when-merging-contactsjitendrajitendrahttps://lab.civicrm.org/dev/core/-/issues/1147cruft: option_group for case type2023-02-02T05:03:33ZJoeMurraycruft: option_group for case typeIn the tarball, there is a civicrm_option_group record defined for name='case_type' (value=26) that has no corresponding option_value records. There is instead a CiviCRM_case_type table. Likely missed during refactoring to create that ta...In the tarball, there is a civicrm_option_group record defined for name='case_type' (value=26) that has no corresponding option_value records. There is instead a CiviCRM_case_type table. Likely missed during refactoring to create that table.
Task for a newbie: remove the creation of the option_group from the tarball, and on upgrade remove the record in civicrm_option_group table for it.https://lab.civicrm.org/dev/core/-/issues/1142Upgrade test infrastructure to support 'edge' versions of dependencies like M...2020-01-08T01:52:11ZJoeMurrayUpgrade test infrastructure to support 'edge' versions of dependencies like MySQL8@totten wrote:
> we currently check releases using a matrix with `min` and `max` columns
>
> if we expect civicrm to be compatible already (and/or that updates will be quick and straight-forward), then we can use KISS approach and just...@totten wrote:
> we currently check releases using a matrix with `min` and `max` columns
>
> if we expect civicrm to be compatible already (and/or that updates will be quick and straight-forward), then we can use KISS approach and just bump up max (`mysql57` => `mysql80`)
>
> otoh, if the process is going to take longer, then we should add a new column (`min`, `max`, `edge`)
>
> (note: adding an `edge` column could also be useful wrt to phasing-in support for `php73`)
>
> ... it would mean that reports of errors for infrastructure that is on the edge of getting support are not treated as blockers, and are identifiable as edge errors
Once an edge technology dependency like MySQL 8 or PHP 7.3 are deemed ready to be declared supported, the max column will change, and the edge column will change to NULL or perhaps the next higher version if people are prepared to start work on that.
This issue is focused on adding edge support to the matrix through a new column, and everything that entails in the build infrastructure and error reporting.
In order to do that, MySQL8 happens to be the first edge version to be incorporated. Issues specific to getting MySQL8.0 testing working on infrastructure will be managed at https://lab.civicrm.org/dev/core/issues/1144. Issues regarding MySQL8 compatibility are being managed via the meta issue https://lab.civicrm.org/dev/core/issues/392.
PHP73 is expected to be the next edge dependency.5.23.0tottentottenhttps://lab.civicrm.org/dev/core/-/issues/1122Performance issue when loading GroupContactCache in larger databases2022-11-22T05:03:55ZlolcodePerformance issue when loading GroupContactCache in larger databasesI am attaching a sample from our slow query logs that shows that group loading is a problem for some of our Smart groups. In particular the code that checks for removed contacts looks like it could benefit from performance improvement.
...I am attaching a sample from our slow query logs that shows that group loading is a problem for some of our Smart groups. In particular the code that checks for removed contacts looks like it could benefit from performance improvement.
The potential solution is similar to the other ticket on group performance that I have opened: https://lab.civicrm.org/dev/core/issues/930
I have a patch for a partial solution so far and it doesn't break any of the existing tests. However I could use some guidance on making sure the tests cover enough cases of saved searches.
[performance-problem-samples.txt](/uploads/346b3cc009f6371ea13cf49dd9d94c6a/performance-problem-samples.txt)