Development issueshttps://lab.civicrm.org/groups/dev/-/issues2022-08-25T21:35:03Zhttps://lab.civicrm.org/dev/core/-/issues/3154Custom tokens not working in Scheduled Reminders2022-08-25T21:35:03ZmartyCustom tokens not working in Scheduled RemindersOverview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute No...Overview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute Now option.
Reproduction steps
----------------------------------------
1. Create a custom token using hook_civicrm_container() and implement the civi.token.list and civi.token.eval event listeners.
1. Add a new Scheduled Reminder (I'm using membership end date) and include the custom token in the email message.
1. Create a Cron Job to run civicrm/bin/cron.php periodically (I run every 15 minutes).
1. Enable the Send Scheduled Reminders job and set to run Always.
1. Trigger the reminder appropriately (I create a new membership and set the end date to trigger).
1. Note the custom token is __not__ included in the resulting email after the cron run.
1. Now trigger a new reminder and click Execute Now on the Scheduled Reminders job (before the next cron run).
1. Note the custom token __is evaluated properly__ and included in the resulting email message.
Current behaviour
----------------------------------------
Custom token not included in Scheduled Reminder email when initiated by Cron Job
Expected behaviour
----------------------------------------
Custom token should be included in Scheduled Reminder email when initiated by Cron Job.
Environment information
----------------------------------------
* __CiviCRM:__ _5.47.2_
* __PHP:__ _7.4.28_
* __CMS:__ _WordPress 5.9.2_
* __Database:__ _MySQL_
* __Web Server:__ _Apache_https://lab.civicrm.org/dev/core/-/issues/3711Permissions reset on upgrade or configuration change2022-08-25T21:41:06ZAdam WoodPermissions reset on upgrade or configuration changeApplies to CiviCRM running on Joomla.
The permission "See CiviCRM is installed" keeps resetting by itself. This definitely occurs whenever CiviCRM is upgraded (issue observed up to and including 5.50.4) and/or an extension is installed,...Applies to CiviCRM running on Joomla.
The permission "See CiviCRM is installed" keeps resetting by itself. This definitely occurs whenever CiviCRM is upgraded (issue observed up to and including 5.50.4) and/or an extension is installed, enabled/disabled, updated etc, and may occur at other times. I cannot discern the pattern!
This means that CiviCRM disappears from the 'Components' administrator menu unless you are logged in as Super Administrator.
Since CiviGrant was migrated to an extension, the same issue now applies to "access CiviGrant", "edit grants" and "delete in CiviGrant" - I have to keep re-applying these after each upgrade.https://lab.civicrm.org/dev/core/-/issues/719can't delete profile pre- post- field help on multilinguage sites2022-08-26T00:20:44ZJoeMurraycan't delete profile pre- post- field help on multilinguage sitesIf you add field pre and post help to profiles on a single language site, you can then delete it and save successfully. On a site with more than 1 language, trying to delete the help leads to it being resaved (I believe from the alternat...If you add field pre and post help to profiles on a single language site, you can then delete it and save successfully. On a site with more than 1 language, trying to delete the help leads to it being resaved (I believe from the alternative language's copy of the help).
Verified on dmaster 5.12.alpha1.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/1726Allow financial types to not have Expense account defined2022-08-26T20:51:52ZJoeMurrayAllow financial types to not have Expense account definedThis is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require ...This is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require Expenses account for every financial type, nor create relations to Expense and Premium accounts by default when creating a financial type.
Example use-case
----------------------------------------
1. Click on **Administer > CiviContribute > Financial Types**.
1. Click **Add Financial Type**.
1. Enter **Name** and click **Save**.
1. In Financial Accounts, there are Banking Fees and Premiums accounts, which is **undesirable**.
1. Click **Accounts** on the new Financial Type row.
1. Beside the 'Expense Account is', click **Delete**, then confirm by clicking **Delete** again.
1. Click on **Contributions > New Contribution**.
1. Select the Financial Type created above that does not have an Expense Account set up for it anymore, fill in **Contributor** and **Total Amount**, and click **Save**.
1. Try to edit the contribution but not in a popup, for example, go to the contact's page, right click on the edit button for the contribution and Copy Link Address, then paste address into a new tab. You'll see "Sorry, due to an error, we are unable to fulfill your request at the moment. You may want to contact your administrator or service provider with more details about what action you were performing when this occurred.
One of parameters (value: ) is not of the type Integer". This is caused by missing Expense account, **even though it is not needed**.
Current behaviour
----------------------------------------
See above.
Proposed behaviour
----------------------------------------
On creation of Financial Type, no Expense or Premiums account relationship would be setup. On editing a contribution (with a line item) with a financial type without an Expense account relationship setup, no error would occur.
Comments
----------------------------------------
The expectation when this was implemented circa 2014 was that payment processors would all soon record banking fees. That hasn't been the case for a variety of reasons.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/3819Contribution type specific custom data not shown on Pledge Payment form2022-08-30T09:25:36ZyashodhaContribution type specific custom data not shown on Pledge Payment formSteps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial t...Steps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial type Donation, the field is loaded on form.
![donation_only](/uploads/2cf7222d745b53268d403ab4b082fe75/donation_only.png)
- Create a pledge of financial type Donation.
- Record pledge payment and it should take to contribution with financial type Donation pre-filled
![new_pledge_payment](/uploads/c6952707b12344500e3252281f687a11/new_pledge_payment.png)
- The custom data fields (donation only) are missing though on load. On change of financial type, the custom data loads. It should work on on load as well.https://lab.civicrm.org/dev/core/-/issues/3818Trigger based logging - improve archivability2022-08-30T10:59:52ZeileenTrigger based logging - improve archivabilityWe would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ ...We would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2015-11-09 | Initialize |8|Bob|
| 2018-09-09 | Update |8|Robert|
| 2022-10-09 | Update |8|John|
In each case the rows are logged to have the status of the updated value. So on 2022-10-09 the value in `civicrm_contact` is "John"
If the change was made in error the change can be rolled back to the last change - ie 'Robert'
However, if we say that we don't want to retain logging data older than 4 years and we have just deleted all rows older than than the after the change the table looks like this
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2022-10-09 | Update |8|John|
And we can no longer roll back the change
The options seem to be
- Have an archive routine that adds an `initialization ` of current value when we truncate
- Switch the mechanism to log the PREVIOUS value not the current value on each update (this means the live table would need to be considered when calculating any diffs)https://lab.civicrm.org/dev/core/-/issues/3821Cron and smartgroups fail when SearchKit is unable to lookup a saved search2022-08-30T20:35:19Zjoshjosh@civicrm.orgCron and smartgroups fail when SearchKit is unable to lookup a saved searchAfter having corrected a previous issue with SearchKit unique to a specific instance, the site experienced cron and smartgroup failures. The resulting error presented:
` Argument 1 passed to CRM_Contact_BAO_GroupContactCache::getQueryOb...After having corrected a previous issue with SearchKit unique to a specific instance, the site experienced cron and smartgroup failures. The resulting error presented:
` Argument 1 passed to CRM_Contact_BAO_GroupContactCache::getQueryObjectSQL() must be of the type array, null given, called in /home/site/public_html/administrator/components/com_civicrm/civicrm/CRM/Contact/BAO/GroupContactCache.php on line 814
`
After troubleshooting, it was determined that SearchKit was looking for a smartgroup/saved search that no longer existed. As a result, crons failed entirely and existing smart group contacts would not display properly. New smart groups and related functionality functioned as expected.
Ideally, SearchKit would not hang up in this situation and/or cause issues with other areas of the system.
CC @deepak.srivastavahttps://lab.civicrm.org/dev/core/-/issues/1481Revisiting deadlocks2022-09-01T10:27:33ZeileenRevisiting deadlocksSome time ago we added handling to the DataObject class to catch and retry deadlocks. This seems to work and be helpful in some cases but in others it doesn't work and is possibly harmful. I want to re-open the discussion on how & where ...Some time ago we added handling to the DataObject class to catch and retry deadlocks. This seems to work and be helpful in some cases but in others it doesn't work and is possibly harmful. I want to re-open the discussion on how & where we catch them & roll them back.
**Why do we have deadlocks**
Deadlocks are a 'natural' part of mysql scalability. Mysql tries to manage contention under load to the extent that can be done at the DB layer, allowing tables & rows to be locked & queuing transactions that need to use those resources to complete after that. However, in some cases mysql cannot resolve it and it returns a deadlock error. The expectation is that the application layer will handle & retry.
For example the query in this PR was causing deadlocks - https://github.com/civicrm/civicrm-core/pull/16080 - here is how I think the flow worked - note the queries are not slow but this is under high volume.
1) contact create is called by 3 different processes in pretty quick succession
2) The query to update the employer name field is called by all 3
3) the first query 'gets the lock' - the other 2 get shared locks while they wait
4) the first query finishes. Neither of the other 2 can get the exclusive lock as they both have shared locks
5) one is reverted & a deadlock is thrown
6) the deadlock is caught in packages/DB/Dataobject
7) the Dataobject code retries the query and succeeds. The transaction succeeds.
**When don't we do a good job with deadlocks**
The above scenario is good because once the deadlock was resolved we were able to retry and it worked. However, it turns out that mysql often rolls back more than just the query it was doing when it decided there was a deadlock. Per https://dev.mysql.com/doc/refman/8.0/en/innodb-deadlock-detection.html " InnoDB automatically detects transaction deadlocks and rolls back a transaction or transactions to break the deadlock."
We are seeing a consistent pattern where under high volume we see deadlocks on ```INSERT INTO civicrm_email``` - via contact.create api (called in turn from a job api by drush). The INSERT email fails but on retry it hits a constraint error - because unknown to the php layer the ```INSERT INTO civicrm_contact ``` statement was also rolled back.
In this case the DB state is hopefully still consistent as the failure should have triggered more rollbacks but it's at least theoretically possible to have lost data in the rollback & then have the query succeed when retried - so the database is in an inconsistent state.
My high level conclusion is that we should be catching deadlocks & retrying higher up the stack - but I'm still trying to figure out where.https://lab.civicrm.org/dev/core/-/issues/3817Question/Discussion: Inconsistencies between "access CiviCRM" and "access AJA...2022-09-01T11:12:34ZJohn TwymanQuestion/Discussion: Inconsistencies between "access CiviCRM" and "access AJAX API" permission grants?Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications....Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications.
Users of our client applications do not need, and should not have, the "access CiviCRM" permission. So we have been building our apps on the basis of using the "access AJAX API" permission instead.
Unfortunately, we have discovered that almost every API call involving core entities assumes "access CiviCRM" as the baseline permission for use. `Group.get`, `Participant.get`, etc., [as defined in CRM/Core/Permission.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Permission.php#L968).
Perhaps we have mistakenly assumed that "access AJAX API" was designed as a functionally equivalent permission to "access CiviCRM", minus the CiviCRM UI access.
Should the "access AJAX API" permission have the same baseline (API) permissions as "access CiviCRM"?
(I see a bigger challenge for us here in terms of the range of permissions required for certain calls, eg. Participant.get requires 'access civicrm', 'access civievents', view all participants', but one problem at a time)
Reproduction steps
----------------------------------------
1. Set up a user with an API key, etc., and a role that does _not_ have the `access civicrm` permission but does have the `acess ajax api` one
2. Query the API v3 REST endpoint with the user's credentials; eg. Group.get, Contact.get, etc.
3. Get an API permissions error response
Current behaviour
----------------------------------------
Many/most API calls (at least Entity.get calls) made by users with only the 'access ajax api' call return a permissions denied error: 'require "access civicrm"
Expected behaviour
----------------------------------------
Many/most API calls made by these users should return results
Environment information
----------------------------------------
* __CiviCRM:__ 5.49.5
* __PHP:__ 7.4/
* __CMS:__ Drupal 7.91/
* __Database:__ MariaDB 10.4.21_https://lab.civicrm.org/dev/core/-/issues/3139Badgelayouts cannot be edited with PHP warning2022-09-01T13:49:35ZBradley TaylorBadgelayouts cannot be edited with PHP warning_Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit scr..._Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit screen should be pre-filled with the values entered initially.
**Actual outcome**
Each field is blank, a PHP warning is shown:
![Screenshot_2022-03-27_at_10.18.33](/uploads/105dc6046cbb9b569207500da40cf77f/Screenshot_2022-03-27_at_10.18.33.png)
**Technical explanation**
The bug was introduced in https://github.com/civicrm/civicrm-core/commit/873bfeb503caa413f17460dbe450b74fac3d6dbf.
The commit above added a new tokens:
```
'{event.start_date|crmDate:"%B %E%f"}' => ts('Event Start Date'),
'{event.end_date|crmDate:"%B %E%f"}' => ts('Event End Date'),`
```
The data for badge layouts is stored as encoded JSON. This means that the quote marks in these two tokens are being wrapped in double-quotes for the string, causing something like `"{event.start_date|crmDate:"%B %E%f"}"`. As such the JSON is not valid and cannot be `json_decode`ed.
The actual fix could be straightforward enough: Switch the tokens to use single instead of double quotes. However, I'm not sure what the correct solution is for any broken JSON which is now stored in CiviCRM databases. Some sort of upgrade script might be required to find/replace the known broken JSON.
Pinging @eileen who did a lot of work on tokens last year.https://lab.civicrm.org/dev/core/-/issues/2864Meta - token usage 5.43 standardisation effort2022-09-02T01:38:26ZeileenMeta - token usage 5.43 standardisation effortWe are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can ...We are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can be around the one release - documentation is [here](https://lab.civicrm.org/-/ide/project/documentation/docs/sysadmin/tree/case/-/docs/upgrade/version-specific.md/)
The goal is to consolidate such that
1. All token rendering is done through the token processor, token hooks are redundant and deprecated
2. All functions on CRM_Utils_Token are unused in core and deprecated except for those in the legacy BAO mailing classes.
3. We freeze, deprecate and phase out in legacy BAO classes in favour of flexmailer
4. All core classes that render tokens do so using consistent token naming & output conventions (rather than dates being formatted in various ways and fields sometimes outputting labels and other times values or name fields https://lab.civicrm.org/dev/core/-/issues/2650)
5. As much as possible we render via tokens rather than smarty in workflow message templates (that couples them with the entity rather than any form flow)
6. pdf & email & scheduled reminder tasks consistently offer the same entity-specific, contact & domain tokens
7. We can start to define obvious missing tokens such as `{domain.now}` (done), `{contribution.balance}` and formatted addresses
Task list / streams - this gives some idea of where things are progressing or blocked
- [x] standardise contact tokens to support/ advertise label style (e.g `{contact.prefix_id:label}`
- [x] make Contact tokens work with partial contact inputs - currently they only work with no contact array passed in, or an array with all possible values - currently blocked on a caching bug https://github.com/civicrm/civicrm-core/pull/21790
- [x] split contact tokens into own class & extend entity tokens like the other classes(blocked on stdise contact tokens)
- [x] switch contact tokens to use apiv4
- [x] work through standardising location token loading (this could go further - ie I had thought about adding loading for billing
- [x] date formatting standardisation [documentation](https://docs.civicrm.org/user/en/latest/common-workflows/tokens-and-mail-merge/#date) https://lab.civicrm.org/dev/core/-/issues/2746
- [x] locale sensitive standardised money formatting https://lab.civicrm.org/dev/core/-/issues/2638 & [pr](https://github.com/civicrm/civicrm-core/pull/21783)
- [x] switch badge tokens to use token processor
- [x] event tokens cleanup & loading fixes + remove unindexed query, move the `fee_amount` and `balance` tokens to the participant tokens class, fix event class to listen to either 'participantId' or 'eventId'. Participant class should run first & load eventId if required. Events should be cached in a static cache as it would be very rare to message thousands of events in one run.
- [x] Recurring contribution tokens work (currently availble in only the recurring_edit template with a pr to extend to cancelled recurring https://test.civicrm.org/job/CiviCRM-Core-PR/44632/
- [x] pdf letter is sane (https://lab.civicrm.org/dev/core/-/issues/2790) enough to extend to deliver tokens for other entities (participant https://lab.civicrm.org/dev/core/-/issues/780)
- [x] email letter extend to deliver tokens for other entities (membership & participant) (https://lab.civicrm.org/dev/core/-/issues/2862)
- [x] scheduled reminders supports participant tokens https://lab.civicrm.org/dev/core/-/issues/779
- [x] domain tokens are consistently available & render https://lab.civicrm.org/dev/core/-/issues/2838 - done but a gap on case tasks as they prioritise being able to filter tokens by case type
- [x] install flexmailer on new installs https://lab.civicrm.org/dev/core/-/issues/2836
- [ ] fix remaining places that process tokens other than via the payment processor
- [x] eliminate replaceGreeting - currently [blocked on apiv4 caching bug](https://github.com/civicrm/civicrm-core/pull/21790)
- [ ] fix CRM_Contact_BAO_Contact_Utils::updateGreeting to not call `getTokenDetails`
- [ ] confirm civicrm_api3_mailing_preview is replaced by flexmailer
- [x] remove getTokenDetails call from CRM_Contribute_Form_Task_PDFLetter - note this appears to be an expensive way to call the contact api & tokens are unused https://github.com/civicrm/civicrm-core/pull/21805
- [x] remove extra rendering in BAO_Pledge https://github.com/civicrm/civicrm-core/pull/21789
- [x] removed getTokenDetails from updatePledgeStatus https://github.com/civicrm/civicrm-core/pull/21847
- [x] remove getTokenDetails from transitionParticipants
- [ ] remove getTokenDetails from sendCancellation & cancelParticipant in selfsvc flow
- [ ] deliverGroup - replaced by flexmailer?
- [ ] MailingPage::preview - replaced by flexmailer?
- [ ] remove hook:;tokenValues from label task
- [ ] getAnonymousTokenDetails ???
- [ ] confirm BAO_Mailing::compose is only via flexmailer
- [x] communicate rc status via dev list
- [x] communicate changes via blog
- [ ] upgrade message https://lab.civicrm.org/dev/core/-/issues/2871
- [ ] docs update to these instructions / check it can still all be done https://docs.civicrm.org/dev/en/latest/step-by-step/create-custom-case-token/
**Currenly out of scope**
- [ ] Token.getlist api - until we have this we have an issue with audience-filtering and with switching tokens depending on the schema. Also case tasks still have to render tokens weirdly without this because they currently filter by case type with partial success https://lab.civicrm.org/dev/core/-/issues/2788
- [ ] Clarify how extensions should render tokens https://lab.civicrm.org/dev/core/-/issues/2863
- [ ] encourage flexmailer with status checks on existing installs https://lab.civicrm.org/dev/core/-/issues/2836
Analysis of current token rendering it core:
Method | Used in | Status
-- | -- | --
Use the token processor | scheduled reminders, civimail with flexmailer in use, activity pdf task | This is the method we are consolidating all paths to use.
Use the functions in CRM_Utils_Token | PDF tasks other than activity | Migrated (PRs on [membership](https://github.com/civicrm/civicrm-core/pull/21521) and [contribution](https://github.com/civicrm/civicrm-core/pull/21524). Also work on fixing up the [pdf task class](https://lab.civicrm.org/dev/core/-/issues/2790) supporting[add participant tokens](https://lab.civicrm.org/dev/core/-/issues/780) (once we’ve finished [the standardisation of those](https://github.com/civicrm/civicrm-core/pull/21587))
| Email tasks | Migrated to token processor. Added tokens for other entities (participant, membership https://lab.civicrm.org/dev/core/-/issues/1396) [also see](https://lab.civicrm.org/dev/core/-/issues/2862)
| Export tasks (when merging contacts in export) | Migrated to token processor
| Profile link rendering | Migrated to token processor
| Storing contact greetings (eg. email_greeting_display) | In progress. Currently resolving inconsistent token syntax and load issues as this needs to be able to use partial pre-loads
Use random ad hoc code | Event badges | [migrated](https://github.com/civicrm/civicrm-core/pull/21587) - this is the only place outside of scheduled reminders where event or participant tokens were used in core
| Send SMS task (BAO_Activity::sendSMS) | migrated
| Labels task CRM_Contact_Form_Task_Label | argh
| CRM_Event_Form_SelfSvcTransfer | Still pretty argh - but this is a workflow message template so the fix is a bit different
| Transition participants | Also a workflow - possibly redundant as seems to be mostly about contact tokens which are resolved in the send function anyway
| Pledge acknowledgements | Also appear to be redundant
| CiviMail with flexmailer not used (many functions & classes) | Push the switch to flexmailer.
| Various unsubscribe / resubscribe type actions | Probably out of scope for this roundhttps://lab.civicrm.org/dev/drupal/-/issues/182Drupal 7 modules not ported to Drupal 9 yet2022-09-09T10:08:56Zluke.stewartDrupal 7 modules not ported to Drupal 9 yetCurrently there are a number of Drupal 7 modules that have not been ported to Drupal 9 as a - comes bundled with CiviCRM - code located within the CiviCRM codebase on install.
This can be seen by comparing
https://github.com/civicrm/civ...Currently there are a number of Drupal 7 modules that have not been ported to Drupal 9 as a - comes bundled with CiviCRM - code located within the CiviCRM codebase on install.
This can be seen by comparing
https://github.com/civicrm/civicrm-drupal/tree/7.x-master/modules
and
https://github.com/civicrm/civicrm-drupal-8/tree/master/modules/
Now a number of these modules it doesn't make sense to run in D9:
Views, CiviCRM OG Sync CiviCRM Engage and possibly others?
Some have a D9 module:
CiviCRM Group Roles has been set up as a stand alone project
https://www.drupal.org/project/civicrm_group_roles
CiviCRM Theme has been set up as part of the civicrm-drupal-8 codebase
CiviCRM Member Roles has been proposed as a PR to the drupal-8 repo but this has been stalled for a while now.
https://github.com/civicrm/civicrm-drupal-8/pull/66
I feel like options for modules here are now:
- A) 🍎 Set up as a stand alone project leave it to Site Builders/Maintainers as to whether to install or not.
- B) 🍌 Set up as a stand alone project but require as a dependency via composer so code gets installed when installing CiviCRM
Do we have criteria for deciding which option in general?
Do we have criteria for deciding which option should be applied in the case of CiviCRM Member Roles?
- C) 🥕 Include within the CiviCRM Drupal codebase via the civicrm/civicrm-drupal-8 project
Do we need a general criteria before deciding approach for CiviCRM Member Roles?
I think probably we would not want to use B unless we wanted something in a separate module that was required for the vast majority of installs of CiviCRM to run?
Given that there has been little appetite for CiviCRM Member Roles perhaps the approach is to split this into it's own module on the basis that most D9 sites to date are probably not using - therefore perhaps this code should not be required to be present on all CiviCRM installs.
However - we still have a lot of sites running D7 - and perhaps the sites that would use Member Roles are mainly running D7?https://lab.civicrm.org/dev/core/-/issues/3830API4 naming for custom field tokens (:name / :label)2022-09-09T10:15:16Zmagnolia61API4 naming for custom field tokens (:name / :label)Overview
----------------------------------------
With all the work on standardization of token processing we have come a long way. One missing element I think is the availability of api4 naming of the custom field tokens. In https://lab...Overview
----------------------------------------
With all the work on standardization of token processing we have come a long way. One missing element I think is the availability of api4 naming of the custom field tokens. In https://lab.civicrm.org/dev/core/-/issues/2650 this is out of scope. I would like to get this on the radar. Maybe it is an easy improvement.
Current behaviour
----------------------------------------
For core tokens :label and :name are supported to provide the raw value of label of the field. (eg. `{contact.communication_style_id:name} / | {contact.communication_style_id:label}`)
For custom fields this is not available:
`{contact.custom_1173:name} / {contact.custom_1173:name} `
The label of the field/token could be: "This is a long label" and the value: "long")
Proposed behaviour
----------------------------------------
For consistency and for easy usage in smarty conditionals it would be great if api4 naming availability for custom field tokens would be supported.
Comments
----------------------------------------
I can help test and sometimes I can code something myself if it an easy needle and I know where to look in the haystack
The overall token issue can be found here : https://lab.civicrm.org/dev/core/-/issues/2864https://lab.civicrm.org/dev/financial/-/issues/207Currency codes for São Tome and Principe Dobra, Zambian Kwacha and Zimbabwe D...2022-09-09T10:17:04ZBradley TaylorCurrency codes for São Tome and Principe Dobra, Zambian Kwacha and Zimbabwe Dollar incorrectEsentially the same issue as dev/financial#184.
- São Tome and Principe Dobra: STN since 1 January 2018
- Zambian Kwacha: ZMW since December 31, 2012
- Zimbabwe Dollar: ZWL since June 2009
I think we need to copy what we did for https:...Esentially the same issue as dev/financial#184.
- São Tome and Principe Dobra: STN since 1 January 2018
- Zambian Kwacha: ZMW since December 31, 2012
- Zimbabwe Dollar: ZWL since June 2009
I think we need to copy what we did for https://github.com/civicrm/civicrm-core/pull/21751/files, but for these 3 additional currencies:
- Update the SQL files for new installs
- Add a migration to update any historical payments.
These is possibly an argument for not migrating the data for old transactions (as in all cases the ISO code changed to reflect some change to the currency, for example a redenomination). Doing the migration fits with what happened last time, and probably saves headaches with needing to support old currency codes, which will no longer exist in libraries like Brick\Money. However, as the data _was_ migrated for Ghana and Belarus I think it probably makes sense to continue with that pattern this time around.https://lab.civicrm.org/dev/financial/-/issues/200Contribution status’s are ambiguous (WIP)2022-09-10T05:42:18ZJamie Novick - CompucoContribution status’s are ambiguous (WIP)**Background:**
Contribution status’s are ambiguous in CiviCRM and can lead to confusion over the current state of an invoice or contribution and whether there are amounts owed or owing to the contributor. This leads to a wealth of prob...**Background:**
Contribution status’s are ambiguous in CiviCRM and can lead to confusion over the current state of an invoice or contribution and whether there are amounts owed or owing to the contributor. This leads to a wealth of problems about what the intention of the user really is in a number of scenarios when they change the contribution status.
Examples:
| Example | Narrative | Video |
| ------ | ------ | ------ |
| Example 1a: Overpayment / pending refund | A contribution which is created for $100 donation, but pending. We then record an overpayment of $120. Hypothetically the contribution is overpaid by $20 and should be pending refund. This is not shown in CiviCRM. There is nothing showing the user that there is an overpayment of $20.| ![1-Overpayment](/uploads/74eca492a8d044aa17033dd681922c4a/1-Overpayment.webm)
| Example 1b: Change contribution status to refunded | We can further demonstrate this by taking the same contribution and then changing the status of the contribution to refunded. This raises the question of what this means? Does this reflect the users intention to: Cancel the original debt of $100 (i.e. to credit note the original amounts owed and clear the balance Or reflect the fact a refund was made for the payment of $120, but that they are still owed the original $100 Or both? CiviCRM has refunded the original value of the invoice, i.e. $100, even though the payment of $120 was received. As such we still owe $20 but this is not shown in CiviCRM.|![2-Change_status_to_refunded](/uploads/bf9eae010687771d440b4baa0d528d1e/2-Change_status_to_refunded.webm)
| Example 2: A typo entered in the wrong contribution amount | Add a contribution for $100 and set as completed. Realised that actually the contribution was $120. System will automatically add the additional $20 payment. Note: If this was a change in the line items of an invoice, it would not be safe to assume we received the additional payment. The aim of the user might be to state the obligation to pay was $120. Also it probably wasn’t 2 separate payments and hence we should reverse out the first payment (-100) and then make a second payment for the full amount (+120) | ![Updating_a_contribution_amount_error](/uploads/bda0cc37c9c5911b54bb429367bdffbf/Updating_a_contribution_amount_error.webm)
Much of this stems from the contribution status field being both be the driver of changes to the contribution (i.e. by changing the status to completed or refunded) or changed itself (i.e. the result of changes to the position of the contribution) by recording payments or refunds.
As such this is a proposal to simplify this workflow so that the the contribution status is only the result of the status of the contribution, and can no longer be used to change the status of the contribution.
We would retain the ability to make changes to the contribution status via the API for backwards compatibility until extensions can adopt the new model.
**Further ramblings (feel free to skip this part if you are not too interested in accounting principles)**
Accounting has simple and sounds principles for dealing with all of this.
“Obligations” are managed by 2 documents: Invoices and Credit notes. An invoice would stipulate what is obligated. This can be reduced by creating a credit note, which would then reduce the obligated amount.
It’s worth noting that for sales tax (for example VAT) purposes invoices should be sequentially numbered and should not be edited once they are generated. Hence usually you would not just simply delete an invoice to reduce the obligated amount, you would create a credit note for some or all of the amount, apply it to the invoice and send that to the customer to show the reduce in their balance owed to you.
As such I think in CiviCRM we need to fully enforce the split of concepts of the “obligation” and the actual payments (or refunded) amounts. Currently we have started on the journey towards that where Contributions (and line items on them) represent the obligations in most cases, and payments (or financial transactions of type payment) are the payments.
The contribution status should be a reflection of the net of that position. Does the customer owe us money (= pending or partially paid*) or is the balance settled (= completed) or do we owe them money (= pending refund)?
</details>
**Proposed changes:**
1. Recalculate the contribution status whenever a contribution is created or updated
We should recalculate the contribution status whenever a contribution is created or updated based on the following rules:
| Ref | Calculation | Status |
| ------ | ------ | ------ |
| 1 | If there are no payments or refunds | Pending (arguably there would be a better label for this but in order to avoid making actual changes to the contribution status options this would seem to be the best fit) |
| 2 | If the sum of the line items (including any tax line items) > (sum of payments received - sum of refunds paid out) | Partially paid |
| 3 | If the sum of the line items (including any tax line items) < (sum of payments received - sum of refunds paid out) | Pending refund |
| 4 | If the sum of the line items (including any tax line items) = (sum of payments received - sum of refunds paid out) | Completed |
2. Changes to the User interface:
| Screen | How it looks currently | How it will work | Changes required |
| ------ | ------ | ------ |------ |
| New contribution screen | ![image](/uploads/568a11ebd7cc0b44edc36077d06f62d8/image.png) | ![Create_new_contribution](/uploads/cb028a89f3c3cc2c3798128efaa4e0d6/Create_new_contribution.png) | We split the "obligation" with the "payment". As such we remove the contribution status field from this screen and instead allow people to decide whether they want to record the "payment" at the same time as recording the "obligation" by checking the "Record payment" checkbox. The contribution status would then be calculated using the logic above. i. |
| View contribution screen (one line item) | ![Screenshot_2022-08-29_at_11.26.55](/uploads/02854f20d41d8ffec5159fa1977627e0/Screenshot_2022-08-29_at_11.26.55.png) | | No Changes |
| View contribution screen (multiple line items) || | |
| Edit contribution screen (one line item) | ![Screenshot_2022-08-29_at_11.30.42](/uploads/6faaed62b313698c1a4346956f48e719/Screenshot_2022-08-29_at_11.30.42.png)| ![Screenshot_2022-08-29_at_11.37.56](/uploads/68293bdc8d48c548606793c6a64d159d/Screenshot_2022-08-29_at_11.37.56.png)| Status field becomes read only and we introduce a button to cancel the contribution. See below for new screen for cancelling a contribution|
| Edit contribution screen (multiple line items) || | |
| New membership screen | ![image](/uploads/568a11ebd7cc0b44edc36077d06f62d8/image.png) | cell | ------ |
| View membership screen (one line item) | | ||
| View membership screen (multiple line items) | | ||
| Edit membership screen (one line item) | | ||
| Edit membership screen (multiple line items) | | ||
| New event participant screen | ![image](/uploads/568a11ebd7cc0b44edc36077d06f62d8/image.png) | ![New_participant](/uploads/7837cf2271b10bb5824d11483221037a/New_participant.png) | Change the checkbox "Record Payment" to "Record Contribution" OR have a toggle to allow the admin to decide whether they are recording a "Paid" or "Free ticket". We only need the financial type and contribution date if we are only recording the contribution (note in https://lab.civicrm.org/dev/core/-/issues/1403 we have agreed to change label of contribution received date to contribution date). The amounts / lines should come from above. |
| View event participant screen (one line item) | | ||
| View event participant screen (multiple line items) | | ||
| Edit event participant screen (one line item) | | ||
| Edit event participant screen (multiple line items) | | ||
- Removehttps://lab.civicrm.org/dev/financial/-/issues/201Stop adding 'In Progress' and 'Overdue' statuses to civicrm_contribution.cont...2022-09-15T21:47:08ZeileenStop adding 'In Progress' and 'Overdue' statuses to civicrm_contribution.contribution_status_id option groupThis has been around as a back burner project for a while but didn't have it's own gitlab ... tada
A long long time ago contribution, contribution recur, payments, pledges and pledge payments shared the same option group for their statu...This has been around as a back burner project for a while but didn't have it's own gitlab ... tada
A long long time ago contribution, contribution recur, payments, pledges and pledge payments shared the same option group for their status & some nasty magic massaged values in and out. Overdue and In progress 'leaked' from these other entities onto the contribution in a long-ago regression but was never intended as a contribution status option value in core.
The plan is to stop adding those statuses on NEW installs. (We might add a message on non-new installs that have them but we don't really want to remove them).
Unfortunately this has been a slow-burner as the tests found code places referencing the wrong option group list in many cases - I've slowly been whittling these down & as of writing three tests are failing
https://github.com/civicrm/civicrm-core/pull/23074
Note that this was notified in a long-ago dev-digest and steps have been taken in the Sepa extension (the only one identified as using these statuses) to ensure they are presenthttps://lab.civicrm.org/dev/core/-/issues/2646Enable/Disable Extension Receives API Error2022-09-16T00:12:46ZpbarmakEnable/Disable Extension Receives API ErrorOverview
----------------------------------------
Anytime we enable or disable an extension, we get an API error with the error message of: API error: Value already exists in the database on ReportTemplate.create. This is true with eithe...Overview
----------------------------------------
Anytime we enable or disable an extension, we get an API error with the error message of: API error: Value already exists in the database on ReportTemplate.create. This is true with either the Admin extension UI screen or using cv.
Reproduction steps
----------------------------------------
1. Run cv ext:disable uk.co.vedaconsulting.mosaico
2. Get the error message
3. The extension still gets disabled (or enabled, depending on what we run)
Current behaviour
----------------------------------------
Here is the full error message from cv:
```
Disabling extension "uk.co.vedaconsulting.mosaico"
Error: API Call Failed: Array
(
[entity] => Extension
[action] => disable
[params] => Array
(
[keys] => Array
(
[0] => uk.co.vedaconsulting.mosaico
)
[debug] => 1
[version] => 3
)
[result] => Array
(
[trace] => #0 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/ManagedEntities.php(241): CRM_Core_ManagedEntities->onApiError('ReportTemplate', 'create', Array, Array)
#1 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/ManagedEntities.php(184): CRM_Core_ManagedEntities->insertNewEntity(Array)
#2 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/ManagedEntities.php(146): CRM_Core_ManagedEntities->reconcileEnabledModule(Object(CRM_Core_Module), Array)
#3 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/ManagedEntities.php(127): CRM_Core_ManagedEntities->reconcileEnabledModules()
#4 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/Invoke.php(400): CRM_Core_ManagedEntities->reconcile()
#5 /var/www/civicrm/sites/all/modules/civicrm/CRM/Extension/Manager.php(420): CRM_Core_Invoke::rebuildMenuAndCaches(true)
#6 /var/www/civicrm/sites/all/modules/civicrm/api/v3/Extension.php(150): CRM_Extension_Manager->disable(Array)
#7 /var/www/civicrm/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_extension_disable(Array)
#8 /var/www/civicrm/sites/all/modules/civicrm/Civi/API/Kernel.php(149): Civi\API\Provider\MagicFunctionProvider->invoke(Array)
#9 /var/www/civicrm/sites/all/modules/civicrm/Civi/API/Kernel.php(81): Civi\API\Kernel->runRequest(Array)
#10 /var/www/civicrm/sites/all/modules/civicrm/api/api.php(22): Civi\API\Kernel->runSafe('Extension', 'disable', Array)
#11 phar:///usr/local/bin/cv/src/Command/BaseCommand.php(49): civicrm_api('Extension', 'disable', Array)
#12 phar:///usr/local/bin/cv/src/Command/ExtensionDisableCommand.php(56): Civi\Cv\Command\BaseCommand->callApiSuccess(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput), 'Extension', 'disable', Array)
#13 phar:///usr/local/bin/cv/vendor/symfony/console/Command/Command.php(257): Civi\Cv\Command\ExtensionDisableCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 phar:///usr/local/bin/cv/vendor/symfony/console/Application.php(850): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 phar:///usr/local/bin/cv/vendor/symfony/console/Application.php(193): Symfony\Component\Console\Application->doRunCommand(Object(Civi\Cv\Command\ExtensionDisableCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 phar:///usr/local/bin/cv/src/Application.php(46): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 phar:///usr/local/bin/cv/vendor/symfony/console/Application.php(124): Civi\Cv\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 phar:///usr/local/bin/cv/src/Application.php(15): Symfony\Component\Console\Application->run()
#19 phar:///usr/local/bin/cv/bin/cv(27): Civi\Cv\Application::main('phar:///usr/loc...')
#20 /usr/local/bin/cv(14): require('phar:///usr/loc...')
#21 {main}
[is_error] => 1
[error_message] => API error: Value already exists in the database on ReportTemplate.create
)
)
```
Expected behaviour
----------------------------------------
No error message
Environment information
----------------------------------------
* __CiviCRM:5.38.0
* __PHP:7.3.28
* __CMS:Drupal 7
* __Web Server: Apache 2.4https://lab.civicrm.org/dev/financial/-/issues/181Update billing details fails when no State/Province entered2022-09-16T21:08:32ZlarsssandergreenUpdate billing details fails when no State/Province enteredUpdate billing details gives "Page could not be loaded" on submit if the user has not entered a State/Province. In some cases, it is valid for a user to not enter a State/Province if they live in a country that does not have states. This...Update billing details gives "Page could not be loaded" on submit if the user has not entered a State/Province. In some cases, it is valid for a user to not enter a State/Province if they live in a country that does not have states. This works as expected on contribution pages, etc, but not on the update billing details page. Tested on 5.35.2.
I will have a look at how this is handled for contribution pages and port the solution to update billing details, at some point.https://lab.civicrm.org/dev/core/-/issues/3204Add recurring contributions to contribution reports2022-09-16T21:11:08ZlarsssandergreenAdd recurring contributions to contribution reportsIt would be very convenient to be able to do filter by recurring contributions for reports, add recurring contributions as a column or even group by.
[Here's a PR that adds this for the Contribution Summary report.](https://github.com/c...It would be very convenient to be able to do filter by recurring contributions for reports, add recurring contributions as a column or even group by.
[Here's a PR that adds this for the Contribution Summary report.](https://github.com/civicrm/civicrm-core/pull/20168) I will also implement in other relevant reports once this has been reviewed.https://lab.civicrm.org/dev/core/-/issues/2899DB error on updating multiple cases from the search form2022-09-20T10:50:03ZjitendraDB error on updating multiple cases from the search formTo replicate -
1. Navigate to https://dmaster.demo.civicrm.org/civicrm/case/search?reset=1
2. Search for cases. Select all results (ts_all radio button) => Update multiple cases action
![image](/uploads/fcbb716228f064c5cde2f8231a534e15...To replicate -
1. Navigate to https://dmaster.demo.civicrm.org/civicrm/case/search?reset=1
2. Search for cases. Select all results (ts_all radio button) => Update multiple cases action
![image](/uploads/fcbb716228f064c5cde2f8231a534e15/image.png)
3. Select Profile on the next step and hit Continue;
![image](/uploads/cb5fb171efd98f5014e5c822f367f717/image.png)