Development issueshttps://lab.civicrm.org/groups/dev/-/issues2022-09-02T01:38:26Zhttps://lab.civicrm.org/dev/core/-/issues/2864Meta - token usage 5.43 standardisation effort2022-09-02T01:38:26ZeileenMeta - token usage 5.43 standardisation effortWe are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can ...We are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can be around the one release - documentation is [here](https://lab.civicrm.org/-/ide/project/documentation/docs/sysadmin/tree/case/-/docs/upgrade/version-specific.md/)
The goal is to consolidate such that
1. All token rendering is done through the token processor, token hooks are redundant and deprecated
2. All functions on CRM_Utils_Token are unused in core and deprecated except for those in the legacy BAO mailing classes.
3. We freeze, deprecate and phase out in legacy BAO classes in favour of flexmailer
4. All core classes that render tokens do so using consistent token naming & output conventions (rather than dates being formatted in various ways and fields sometimes outputting labels and other times values or name fields https://lab.civicrm.org/dev/core/-/issues/2650)
5. As much as possible we render via tokens rather than smarty in workflow message templates (that couples them with the entity rather than any form flow)
6. pdf & email & scheduled reminder tasks consistently offer the same entity-specific, contact & domain tokens
7. We can start to define obvious missing tokens such as `{domain.now}` (done), `{contribution.balance}` and formatted addresses
Task list / streams - this gives some idea of where things are progressing or blocked
- [x] standardise contact tokens to support/ advertise label style (e.g `{contact.prefix_id:label}`
- [x] make Contact tokens work with partial contact inputs - currently they only work with no contact array passed in, or an array with all possible values - currently blocked on a caching bug https://github.com/civicrm/civicrm-core/pull/21790
- [x] split contact tokens into own class & extend entity tokens like the other classes(blocked on stdise contact tokens)
- [x] switch contact tokens to use apiv4
- [x] work through standardising location token loading (this could go further - ie I had thought about adding loading for billing
- [x] date formatting standardisation [documentation](https://docs.civicrm.org/user/en/latest/common-workflows/tokens-and-mail-merge/#date) https://lab.civicrm.org/dev/core/-/issues/2746
- [x] locale sensitive standardised money formatting https://lab.civicrm.org/dev/core/-/issues/2638 & [pr](https://github.com/civicrm/civicrm-core/pull/21783)
- [x] switch badge tokens to use token processor
- [x] event tokens cleanup & loading fixes + remove unindexed query, move the `fee_amount` and `balance` tokens to the participant tokens class, fix event class to listen to either 'participantId' or 'eventId'. Participant class should run first & load eventId if required. Events should be cached in a static cache as it would be very rare to message thousands of events in one run.
- [x] Recurring contribution tokens work (currently availble in only the recurring_edit template with a pr to extend to cancelled recurring https://test.civicrm.org/job/CiviCRM-Core-PR/44632/
- [x] pdf letter is sane (https://lab.civicrm.org/dev/core/-/issues/2790) enough to extend to deliver tokens for other entities (participant https://lab.civicrm.org/dev/core/-/issues/780)
- [x] email letter extend to deliver tokens for other entities (membership & participant) (https://lab.civicrm.org/dev/core/-/issues/2862)
- [x] scheduled reminders supports participant tokens https://lab.civicrm.org/dev/core/-/issues/779
- [x] domain tokens are consistently available & render https://lab.civicrm.org/dev/core/-/issues/2838 - done but a gap on case tasks as they prioritise being able to filter tokens by case type
- [x] install flexmailer on new installs https://lab.civicrm.org/dev/core/-/issues/2836
- [ ] fix remaining places that process tokens other than via the payment processor
- [x] eliminate replaceGreeting - currently [blocked on apiv4 caching bug](https://github.com/civicrm/civicrm-core/pull/21790)
- [ ] fix CRM_Contact_BAO_Contact_Utils::updateGreeting to not call `getTokenDetails`
- [ ] confirm civicrm_api3_mailing_preview is replaced by flexmailer
- [x] remove getTokenDetails call from CRM_Contribute_Form_Task_PDFLetter - note this appears to be an expensive way to call the contact api & tokens are unused https://github.com/civicrm/civicrm-core/pull/21805
- [x] remove extra rendering in BAO_Pledge https://github.com/civicrm/civicrm-core/pull/21789
- [x] removed getTokenDetails from updatePledgeStatus https://github.com/civicrm/civicrm-core/pull/21847
- [x] remove getTokenDetails from transitionParticipants
- [ ] remove getTokenDetails from sendCancellation & cancelParticipant in selfsvc flow
- [ ] deliverGroup - replaced by flexmailer?
- [ ] MailingPage::preview - replaced by flexmailer?
- [ ] remove hook:;tokenValues from label task
- [ ] getAnonymousTokenDetails ???
- [ ] confirm BAO_Mailing::compose is only via flexmailer
- [x] communicate rc status via dev list
- [x] communicate changes via blog
- [ ] upgrade message https://lab.civicrm.org/dev/core/-/issues/2871
- [ ] docs update to these instructions / check it can still all be done https://docs.civicrm.org/dev/en/latest/step-by-step/create-custom-case-token/
**Currenly out of scope**
- [ ] Token.getlist api - until we have this we have an issue with audience-filtering and with switching tokens depending on the schema. Also case tasks still have to render tokens weirdly without this because they currently filter by case type with partial success https://lab.civicrm.org/dev/core/-/issues/2788
- [ ] Clarify how extensions should render tokens https://lab.civicrm.org/dev/core/-/issues/2863
- [ ] encourage flexmailer with status checks on existing installs https://lab.civicrm.org/dev/core/-/issues/2836
Analysis of current token rendering it core:
Method | Used in | Status
-- | -- | --
Use the token processor | scheduled reminders, civimail with flexmailer in use, activity pdf task | This is the method we are consolidating all paths to use.
Use the functions in CRM_Utils_Token | PDF tasks other than activity | Migrated (PRs on [membership](https://github.com/civicrm/civicrm-core/pull/21521) and [contribution](https://github.com/civicrm/civicrm-core/pull/21524). Also work on fixing up the [pdf task class](https://lab.civicrm.org/dev/core/-/issues/2790) supporting[add participant tokens](https://lab.civicrm.org/dev/core/-/issues/780) (once we’ve finished [the standardisation of those](https://github.com/civicrm/civicrm-core/pull/21587))
| Email tasks | Migrated to token processor. Added tokens for other entities (participant, membership https://lab.civicrm.org/dev/core/-/issues/1396) [also see](https://lab.civicrm.org/dev/core/-/issues/2862)
| Export tasks (when merging contacts in export) | Migrated to token processor
| Profile link rendering | Migrated to token processor
| Storing contact greetings (eg. email_greeting_display) | In progress. Currently resolving inconsistent token syntax and load issues as this needs to be able to use partial pre-loads
Use random ad hoc code | Event badges | [migrated](https://github.com/civicrm/civicrm-core/pull/21587) - this is the only place outside of scheduled reminders where event or participant tokens were used in core
| Send SMS task (BAO_Activity::sendSMS) | migrated
| Labels task CRM_Contact_Form_Task_Label | argh
| CRM_Event_Form_SelfSvcTransfer | Still pretty argh - but this is a workflow message template so the fix is a bit different
| Transition participants | Also a workflow - possibly redundant as seems to be mostly about contact tokens which are resolved in the send function anyway
| Pledge acknowledgements | Also appear to be redundant
| CiviMail with flexmailer not used (many functions & classes) | Push the switch to flexmailer.
| Various unsubscribe / resubscribe type actions | Probably out of scope for this roundhttps://lab.civicrm.org/dev/core/-/issues/3139Badgelayouts cannot be edited with PHP warning2022-09-01T13:49:35ZBradley TaylorBadgelayouts cannot be edited with PHP warning_Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit scr..._Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit screen should be pre-filled with the values entered initially.
**Actual outcome**
Each field is blank, a PHP warning is shown:
![Screenshot_2022-03-27_at_10.18.33](/uploads/105dc6046cbb9b569207500da40cf77f/Screenshot_2022-03-27_at_10.18.33.png)
**Technical explanation**
The bug was introduced in https://github.com/civicrm/civicrm-core/commit/873bfeb503caa413f17460dbe450b74fac3d6dbf.
The commit above added a new tokens:
```
'{event.start_date|crmDate:"%B %E%f"}' => ts('Event Start Date'),
'{event.end_date|crmDate:"%B %E%f"}' => ts('Event End Date'),`
```
The data for badge layouts is stored as encoded JSON. This means that the quote marks in these two tokens are being wrapped in double-quotes for the string, causing something like `"{event.start_date|crmDate:"%B %E%f"}"`. As such the JSON is not valid and cannot be `json_decode`ed.
The actual fix could be straightforward enough: Switch the tokens to use single instead of double quotes. However, I'm not sure what the correct solution is for any broken JSON which is now stored in CiviCRM databases. Some sort of upgrade script might be required to find/replace the known broken JSON.
Pinging @eileen who did a lot of work on tokens last year.https://lab.civicrm.org/dev/core/-/issues/3817Question/Discussion: Inconsistencies between "access CiviCRM" and "access AJA...2022-09-01T11:12:34ZJohn TwymanQuestion/Discussion: Inconsistencies between "access CiviCRM" and "access AJAX API" permission grants?Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications....Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications.
Users of our client applications do not need, and should not have, the "access CiviCRM" permission. So we have been building our apps on the basis of using the "access AJAX API" permission instead.
Unfortunately, we have discovered that almost every API call involving core entities assumes "access CiviCRM" as the baseline permission for use. `Group.get`, `Participant.get`, etc., [as defined in CRM/Core/Permission.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Permission.php#L968).
Perhaps we have mistakenly assumed that "access AJAX API" was designed as a functionally equivalent permission to "access CiviCRM", minus the CiviCRM UI access.
Should the "access AJAX API" permission have the same baseline (API) permissions as "access CiviCRM"?
(I see a bigger challenge for us here in terms of the range of permissions required for certain calls, eg. Participant.get requires 'access civicrm', 'access civievents', view all participants', but one problem at a time)
Reproduction steps
----------------------------------------
1. Set up a user with an API key, etc., and a role that does _not_ have the `access civicrm` permission but does have the `acess ajax api` one
2. Query the API v3 REST endpoint with the user's credentials; eg. Group.get, Contact.get, etc.
3. Get an API permissions error response
Current behaviour
----------------------------------------
Many/most API calls (at least Entity.get calls) made by users with only the 'access ajax api' call return a permissions denied error: 'require "access civicrm"
Expected behaviour
----------------------------------------
Many/most API calls made by these users should return results
Environment information
----------------------------------------
* __CiviCRM:__ 5.49.5
* __PHP:__ 7.4/
* __CMS:__ Drupal 7.91/
* __Database:__ MariaDB 10.4.21_https://lab.civicrm.org/dev/core/-/issues/1481Revisiting deadlocks2022-09-01T10:27:33ZeileenRevisiting deadlocksSome time ago we added handling to the DataObject class to catch and retry deadlocks. This seems to work and be helpful in some cases but in others it doesn't work and is possibly harmful. I want to re-open the discussion on how & where ...Some time ago we added handling to the DataObject class to catch and retry deadlocks. This seems to work and be helpful in some cases but in others it doesn't work and is possibly harmful. I want to re-open the discussion on how & where we catch them & roll them back.
**Why do we have deadlocks**
Deadlocks are a 'natural' part of mysql scalability. Mysql tries to manage contention under load to the extent that can be done at the DB layer, allowing tables & rows to be locked & queuing transactions that need to use those resources to complete after that. However, in some cases mysql cannot resolve it and it returns a deadlock error. The expectation is that the application layer will handle & retry.
For example the query in this PR was causing deadlocks - https://github.com/civicrm/civicrm-core/pull/16080 - here is how I think the flow worked - note the queries are not slow but this is under high volume.
1) contact create is called by 3 different processes in pretty quick succession
2) The query to update the employer name field is called by all 3
3) the first query 'gets the lock' - the other 2 get shared locks while they wait
4) the first query finishes. Neither of the other 2 can get the exclusive lock as they both have shared locks
5) one is reverted & a deadlock is thrown
6) the deadlock is caught in packages/DB/Dataobject
7) the Dataobject code retries the query and succeeds. The transaction succeeds.
**When don't we do a good job with deadlocks**
The above scenario is good because once the deadlock was resolved we were able to retry and it worked. However, it turns out that mysql often rolls back more than just the query it was doing when it decided there was a deadlock. Per https://dev.mysql.com/doc/refman/8.0/en/innodb-deadlock-detection.html " InnoDB automatically detects transaction deadlocks and rolls back a transaction or transactions to break the deadlock."
We are seeing a consistent pattern where under high volume we see deadlocks on ```INSERT INTO civicrm_email``` - via contact.create api (called in turn from a job api by drush). The INSERT email fails but on retry it hits a constraint error - because unknown to the php layer the ```INSERT INTO civicrm_contact ``` statement was also rolled back.
In this case the DB state is hopefully still consistent as the failure should have triggered more rollbacks but it's at least theoretically possible to have lost data in the rollback & then have the query succeed when retried - so the database is in an inconsistent state.
My high level conclusion is that we should be catching deadlocks & retrying higher up the stack - but I'm still trying to figure out where.https://lab.civicrm.org/dev/core/-/issues/3821Cron and smartgroups fail when SearchKit is unable to lookup a saved search2022-08-30T20:35:19Zjoshjosh@civicrm.orgCron and smartgroups fail when SearchKit is unable to lookup a saved searchAfter having corrected a previous issue with SearchKit unique to a specific instance, the site experienced cron and smartgroup failures. The resulting error presented:
` Argument 1 passed to CRM_Contact_BAO_GroupContactCache::getQueryOb...After having corrected a previous issue with SearchKit unique to a specific instance, the site experienced cron and smartgroup failures. The resulting error presented:
` Argument 1 passed to CRM_Contact_BAO_GroupContactCache::getQueryObjectSQL() must be of the type array, null given, called in /home/site/public_html/administrator/components/com_civicrm/civicrm/CRM/Contact/BAO/GroupContactCache.php on line 814
`
After troubleshooting, it was determined that SearchKit was looking for a smartgroup/saved search that no longer existed. As a result, crons failed entirely and existing smart group contacts would not display properly. New smart groups and related functionality functioned as expected.
Ideally, SearchKit would not hang up in this situation and/or cause issues with other areas of the system.
CC @deepak.srivastavahttps://lab.civicrm.org/dev/core/-/issues/3818Trigger based logging - improve archivability2022-08-30T10:59:52ZeileenTrigger based logging - improve archivabilityWe would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ ...We would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2015-11-09 | Initialize |8|Bob|
| 2018-09-09 | Update |8|Robert|
| 2022-10-09 | Update |8|John|
In each case the rows are logged to have the status of the updated value. So on 2022-10-09 the value in `civicrm_contact` is "John"
If the change was made in error the change can be rolled back to the last change - ie 'Robert'
However, if we say that we don't want to retain logging data older than 4 years and we have just deleted all rows older than than the after the change the table looks like this
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2022-10-09 | Update |8|John|
And we can no longer roll back the change
The options seem to be
- Have an archive routine that adds an `initialization ` of current value when we truncate
- Switch the mechanism to log the PREVIOUS value not the current value on each update (this means the live table would need to be considered when calculating any diffs)https://lab.civicrm.org/dev/core/-/issues/3819Contribution type specific custom data not shown on Pledge Payment form2022-08-30T09:25:36ZyashodhaContribution type specific custom data not shown on Pledge Payment formSteps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial t...Steps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial type Donation, the field is loaded on form.
![donation_only](/uploads/2cf7222d745b53268d403ab4b082fe75/donation_only.png)
- Create a pledge of financial type Donation.
- Record pledge payment and it should take to contribution with financial type Donation pre-filled
![new_pledge_payment](/uploads/c6952707b12344500e3252281f687a11/new_pledge_payment.png)
- The custom data fields (donation only) are missing though on load. On change of financial type, the custom data loads. It should work on on load as well.https://lab.civicrm.org/dev/core/-/issues/1726Allow financial types to not have Expense account defined2022-08-26T20:51:52ZJoeMurrayAllow financial types to not have Expense account definedThis is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require ...This is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require Expenses account for every financial type, nor create relations to Expense and Premium accounts by default when creating a financial type.
Example use-case
----------------------------------------
1. Click on **Administer > CiviContribute > Financial Types**.
1. Click **Add Financial Type**.
1. Enter **Name** and click **Save**.
1. In Financial Accounts, there are Banking Fees and Premiums accounts, which is **undesirable**.
1. Click **Accounts** on the new Financial Type row.
1. Beside the 'Expense Account is', click **Delete**, then confirm by clicking **Delete** again.
1. Click on **Contributions > New Contribution**.
1. Select the Financial Type created above that does not have an Expense Account set up for it anymore, fill in **Contributor** and **Total Amount**, and click **Save**.
1. Try to edit the contribution but not in a popup, for example, go to the contact's page, right click on the edit button for the contribution and Copy Link Address, then paste address into a new tab. You'll see "Sorry, due to an error, we are unable to fulfill your request at the moment. You may want to contact your administrator or service provider with more details about what action you were performing when this occurred.
One of parameters (value: ) is not of the type Integer". This is caused by missing Expense account, **even though it is not needed**.
Current behaviour
----------------------------------------
See above.
Proposed behaviour
----------------------------------------
On creation of Financial Type, no Expense or Premiums account relationship would be setup. On editing a contribution (with a line item) with a financial type without an Expense account relationship setup, no error would occur.
Comments
----------------------------------------
The expectation when this was implemented circa 2014 was that payment processors would all soon record banking fees. That hasn't been the case for a variety of reasons.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/719can't delete profile pre- post- field help on multilinguage sites2022-08-26T00:20:44ZJoeMurraycan't delete profile pre- post- field help on multilinguage sitesIf you add field pre and post help to profiles on a single language site, you can then delete it and save successfully. On a site with more than 1 language, trying to delete the help leads to it being resaved (I believe from the alternat...If you add field pre and post help to profiles on a single language site, you can then delete it and save successfully. On a site with more than 1 language, trying to delete the help leads to it being resaved (I believe from the alternative language's copy of the help).
Verified on dmaster 5.12.alpha1.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/3711Permissions reset on upgrade or configuration change2022-08-25T21:41:06ZAdam WoodPermissions reset on upgrade or configuration changeApplies to CiviCRM running on Joomla.
The permission "See CiviCRM is installed" keeps resetting by itself. This definitely occurs whenever CiviCRM is upgraded (issue observed up to and including 5.50.4) and/or an extension is installed,...Applies to CiviCRM running on Joomla.
The permission "See CiviCRM is installed" keeps resetting by itself. This definitely occurs whenever CiviCRM is upgraded (issue observed up to and including 5.50.4) and/or an extension is installed, enabled/disabled, updated etc, and may occur at other times. I cannot discern the pattern!
This means that CiviCRM disappears from the 'Components' administrator menu unless you are logged in as Super Administrator.
Since CiviGrant was migrated to an extension, the same issue now applies to "access CiviGrant", "edit grants" and "delete in CiviGrant" - I have to keep re-applying these after each upgrade.https://lab.civicrm.org/dev/core/-/issues/3154Custom tokens not working in Scheduled Reminders2022-08-25T21:35:03ZmartyCustom tokens not working in Scheduled RemindersOverview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute No...Overview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute Now option.
Reproduction steps
----------------------------------------
1. Create a custom token using hook_civicrm_container() and implement the civi.token.list and civi.token.eval event listeners.
1. Add a new Scheduled Reminder (I'm using membership end date) and include the custom token in the email message.
1. Create a Cron Job to run civicrm/bin/cron.php periodically (I run every 15 minutes).
1. Enable the Send Scheduled Reminders job and set to run Always.
1. Trigger the reminder appropriately (I create a new membership and set the end date to trigger).
1. Note the custom token is __not__ included in the resulting email after the cron run.
1. Now trigger a new reminder and click Execute Now on the Scheduled Reminders job (before the next cron run).
1. Note the custom token __is evaluated properly__ and included in the resulting email message.
Current behaviour
----------------------------------------
Custom token not included in Scheduled Reminder email when initiated by Cron Job
Expected behaviour
----------------------------------------
Custom token should be included in Scheduled Reminder email when initiated by Cron Job.
Environment information
----------------------------------------
* __CiviCRM:__ _5.47.2_
* __PHP:__ _7.4.28_
* __CMS:__ _WordPress 5.9.2_
* __Database:__ _MySQL_
* __Web Server:__ _Apache_https://lab.civicrm.org/dev/financial/-/issues/205CiviFinancial Blue Sky Dreaming2022-08-25T21:31:40ZJoeMurrayCiviFinancial Blue Sky DreamingThere has been talk over the years by different people about a LExIM shift to a new paradigm/implementation for CiviAccount. I have been approached by someone who could rustle up a tiny starting stake for such a huge effort (I'm not conf...There has been talk over the years by different people about a LExIM shift to a new paradigm/implementation for CiviAccount. I have been approached by someone who could rustle up a tiny starting stake for such a huge effort (I'm not confident there would be an ability to get to 50% needed to launch an MIH that will likely cost in the six figures at CT billable rates). Here is a blue sky dreaming of how a CiviFinancials LExIM might succeed.
1. We find funding for integrating payments into Form Builder - this seems likely to be possible and occur, yet is still a big lift. Imagine all the financial aspects of webform_civicrm.
2. When implementing the new interface for contribution pages and event reg pages, etc., we have some 'dependency injection' that allows calls to be made either to the existing financial APIs or to a new set.
3. We work to implement the guts of CiviAccounting in a new way. Discussion is needed here, but I would be in favour of some significant breaking changes like simplifying the Financial Type/Financial Account model into much more standard accounting, preferrably with simple, sensible defaults.
1. We could design our own CiviAccounts v2 from scratch.
2. We could aim to implement an interface with an existing open source accounting system. The aim is to store all the CiviCRM financial transactions in a standard, validated way without having the development and support burden of making it ourselves. We'd only need to keep the integration going strong. Two candidates I have found that might be feasible, and there are many others, are Front Accounting and GnuCash. https://sourceforge.net/projects/frontaccounting/files/FrontAccounting%202.4/stats/timeline?dates=2019-08-01+to+2022-08-01 has the same technical stack as CiviCRM (PHP and MySQL) but seems to be trending down in installs. https://sourceforge.net/projects/gnucash/files/gnucash%20%28stable%29/stats/timeline?dates=2019-08-01+to+2022-08-01 is much more popular and losing momentum more slowly. It is built in C and C++, so it would be more complex to integrate.
3. If there is a fully open source implementation via CiviAccounts v2 or integrating an open source accounting package, then I would be okay with there being integrations with closed source accounting systems that are popular like Xero and QuickBooks Online. I suppose we could allow the existing implementation of CiviAccounts to be the open source option. While more financially realistic, it might be bad for our reputation.https://lab.civicrm.org/dev/core/-/issues/3703Confirmation emails are not sent if it is a recurring payment2022-08-25T17:04:50ZMariaVConfirmation emails are not sent if it is a recurring payment**Current behavior apparently:**
One-time donations: A confirmation email comes right after you fill out the donation form
Recurring donations: A confirmation email comes each time after the payment is processed
This works quite well ...**Current behavior apparently:**
One-time donations: A confirmation email comes right after you fill out the donation form
Recurring donations: A confirmation email comes each time after the payment is processed
This works quite well for online payment processors like PayPal, where the first payment is made within seconds of submitting the donation form.
With "asynchronous payment processors" such as SEPA, however, this approach works quite poorly, because it can take several weeks before the next payment run happens (usually once a month).
Currently this does not work at all.
Moreover, from a fundraiser's point of view, it is a real disadvantage if these confirmations are sent with every payment: People will be reminded every time that they still haven't cancelled their recurring donations.
**In contrast, the behavior most of us would like to see would be:**
One-time and recurring donations: A confirmation email comes right after you fill out the donation form
Further confirmations for recurring donations will only come if this has been set in a configuration setting.
**My suggestion:**
There could be an additional radio button:
One option that will be choosed by default which explains that the mail is only sent for recurring donations when a payment has actually been received (and for one-time donations directly). And for this purpose, there will be another option that says that a confirmation will be sent once directly after the form has been submitted - regardless of whether it is a one-time or recurring donation.
This will ensure that this function will continue to work for everyone who actually wants to use it that way - even if there are probably not many.https://lab.civicrm.org/dev/financial/-/issues/121Financial Type ACLs don't work on soft credits2022-08-17T22:07:10ZJonGoldFinancial Type ACLs don't work on soft creditsTo replicate:
* Create a soft credit on any contribution. Note the financial type.
* Enable Financial ACLs.
* Log in as a user who doesn't have permission to view the financial type noted above.
* If you're using `civicrm-buildkit`, an...To replicate:
* Create a soft credit on any contribution. Note the financial type.
* Enable Financial ACLs.
* Log in as a user who doesn't have permission to view the financial type noted above.
* If you're using `civicrm-buildkit`, any non-administrative user who can view CiviCRM (e.g. with `CiviCRM Webtest User` role) qualifies.
* View the contact with the soft credit.
### Expected Result
* The soft credit is not visible.
### Actual Result
* The soft credit is visible. Clicking `View` to view the original contribution returns a "permission denied", which is correct (but bad UX).JonGoldJonGoldhttps://lab.civicrm.org/dev/joomla/-/issues/16[Joomla 4.0] Warnings when CiviCRM is uninstalled2022-08-12T05:14:12ZAndrew Thompson[Joomla 4.0] Warnings when CiviCRM is uninstalledTwo PHP warnings are displayed when uninstalling CiviCRM from Joomla 4.0 alpha 11:
```
Warning: require_once(CRM/Utils/String.php): failed to open stream: No such file or directory in /var/www/html/j4/administrator/components/com_civic...Two PHP warnings are displayed when uninstalling CiviCRM from Joomla 4.0 alpha 11:
```
Warning: require_once(CRM/Utils/String.php): failed to open stream: No such file or directory in /var/www/html/j4/administrator/components/com_civicrm/script.civicrm.php on line 226
```
```
Fatal error: require_once(): Failed opening required 'CRM/Utils/String.php' (include_path='.:/usr/share/pear:/usr/share/php') in /var/www/html/j4/administrator/components/com_civicrm/script.civicrm.php on line 226
```Joomla 4 Integrationhttps://lab.civicrm.org/dev/financial/-/issues/202Unable to update recurring amount when using Sales Tax2022-08-03T15:19:05ZKarinGUnable to update recurring amount when using Sales TaxHitting Edit on the template contribution (with or without line item editor extension - June 2022 version) -> saving it [multiple times] -> does not update the amount in the recurring series -> note the mismatches of amounts:
![image](/...Hitting Edit on the template contribution (with or without line item editor extension - June 2022 version) -> saving it [multiple times] -> does not update the amount in the recurring series -> note the mismatches of amounts:
![image](/uploads/8280ff94ddb4e66ba1368e64c1d4b72a/image.png)https://lab.civicrm.org/dev/financial/-/issues/87Partial Refunds2022-06-14T18:38:44Zmattwiremjw@mjwconsult.co.ukPartial RefundsStripe supports partial refunds via the Stripe Dashboard. I think other processors support similar.
To record a partial refund in CiviCRM we can record a negative payment on the contribution using API `Payment.create`.
Currently nothin...Stripe supports partial refunds via the Stripe Dashboard. I think other processors support similar.
To record a partial refund in CiviCRM we can record a negative payment on the contribution using API `Payment.create`.
Currently nothing changes on the contribution:
* Total amount still shows the full amount but payments are recorded correctly so the sum of payments does not equal the contribution total amount.
* Contribution status remains Completed.
What should happen(?):
* A partial refund means that the total_amount paid, tax_amount and the fee_amount paid may be reduced.
* The contribution status is no longer "Completed" and should be `Partially refunded` (does not exist)? `Partially paid` (already exists).
* The `Financial Type` should be displayed correctly for the refund payment.
Currently, viewing a contribution, either in the summary or detail gives no indication of the actual amount paid:
![image](/uploads/39a50dd48d946207feda438321850a39/image.png)
![image](/uploads/63cc3abf7df50ac1babcbff85f2d4c86/image.png)
@JoeMurray @eileen @artfulrobot @ayduns Thoughts please?https://lab.civicrm.org/dev/financial/-/issues/144Figure out if core processors actually work.....2022-05-19T22:31:00ZeileenFigure out if core processors actually work.....The processors we currently ship with core are
| Processor | status|
| ------ | ------ |
| Authorize.net | works/tested |
| Elavon| broken? |
| Eway| works/tested/ converted to an extension |
| FirstData| broken? (Reported as working by...The processors we currently ship with core are
| Processor | status|
| ------ | ------ |
| Authorize.net | works/tested |
| Elavon| broken? |
| Eway| works/tested/ converted to an extension |
| FirstData| broken? (Reported as working by Brian Civi Version not mentioned) |
| PayflowPro| broken? (Reported as working as of 5.19 by Nicholas) (As of 5.38 Migrated to a core extension with unit tests) |
| PaymentExpress| definitely broken / removed|
| PayJunction| broken? |
| Paypal| works/tested |
| Realex| likely works, have seen gitlabs |
I think we should try to find out about the ones that might be broken with a view to either
1) bring them under CI & move to an extension (per Eway) OR
2) remove them from core
Other than pinging people like @lcdweb to see if they use any my other idea is to add a check or preUpgrade message to try to get people to let us know.
@seamusleehttps://lab.civicrm.org/dev/drupal/-/issues/177Cannot resolve path using "cms.root.url"2022-04-29T09:11:35ZalmadorxCannot resolve path using "cms.root.url"Hi! I'm having the issue with
```
In Paths.php line 140:
Cannot resolve path using "cms.root.url"
```
I'm using Drupal 9 and the last version of CiviCRM
I've tried the solution from here
([regression `cv` fails on CiviCRM 5.15.0](htt...Hi! I'm having the issue with
```
In Paths.php line 140:
Cannot resolve path using "cms.root.url"
```
I'm using Drupal 9 and the last version of CiviCRM
I've tried the solution from here
([regression `cv` fails on CiviCRM 5.15.0](https://lab.civicrm.org/dev/drupal/-/issues/75))
\vendor\civicrm\civicrm-core\CRM\Utils\System\Drupal8.php:
` public function getCurrentLanguage() {
// Drupal might not be bootstrapped if being called by the REST API.
if (!class_exists('Drupal') || !\Drupal::hasContainer()) {
return NULL;`
I've replaced _return NULL;_ with _return $url;_ but that doesn't solved the issue.https://lab.civicrm.org/dev/release/-/issues/18Scheduling/workflow for security updates of dependencies2022-04-29T08:58:12ZtottenScheduling/workflow for security updates of dependencies# Synopsis
The workflow for *first-party/in-house* security updates and the workflow for *third-party/upstream* security updates are qualitatively different
The question of this issue is: Do we keep one general policy/schedule for both...# Synopsis
The workflow for *first-party/in-house* security updates and the workflow for *third-party/upstream* security updates are qualitatively different
The question of this issue is: Do we keep one general policy/schedule for both kinds of security issues, or do we have a more nuanced policy that distinguishes between them?
# Background
CiviCRM's policy for scheduling/workflow on security updates has a few key elements:
* Report and discuss vulnerabilities privately
* Release updates on a designated release window (the first/third Wed of each month)
* Make an effort to pre-announce (often 1-4 weeks in advance)
Those bullets are based on the premise that we control the process for disclosure/development/etc. This is true and appropriate for the common case where the security vulnerability originates in code maintained directly by CiviCRM.
But there is another common case: *dependencies* ("libraries", "packages", "subpackages", etc) used by CiviCRM and maintained by another group. These break the bullet-points from above:
* The purpose of upstream's public advisory is *to notify people like us*. The issue is necessarily public when we get the information.
* There are several different upstreams. Their release scheduling is (on the whole) fluid - some have release-windows; some don't; some make pre-announcements; some don't; each of those policies may change over time.
* The vulnerability is public. Delaying the release (in service of a pre-announcement/spin-up period) exacerbates the risk exposure. We don't want our scheduling to add extra exposure.
The security updates of a dependency affect CiviCRM in a few ways. Anyone reading this probably has some understanding already. But just to be complete, those effects include:
* Dependency-updates require some correlated change in how we use that dependency. In the best case, that just means metadata (eg `composer.json`, `composer.lock`) - but it can also be much more involved. It varies case-by-case.
* Several artifacts need to be republished when a dependency changes (notably the tarballs/zipballs for WP/D7/BD/J - but also any images published via docker, etc).
# Proposal
Security fixes _that have been previously published by an upstream vendor_ should be assimilated through CiviCRM's public development channels (Gitlab/Github/Mattermost/etc). The process should closely match the process for patch-releases that fix recent-regressions:
* Like a regular patch-release...
* Any patches/PRs should be submitted to the RC's public queue.
* After approving the RC PR, then backport to stable/ESR. (Only backport if we believe it to be "likely" exploitable.)
* Discussion about testing, `r-run`, compatibility, etc can happen in the public PR.
* We do not need to assign a CIVI-SA-* identifier or write an "advisory" record.
* In addition, there are extra bits...
* We'll send a mailblast when the stable/ESR updates are published.
* Release notes should highlight the "Security" issue as distinct from any other "Bug fix" issues. They should link to upstream's advisory (in addition to the usual Github/Gitlab links for Civi).
* In the public media, don't discuss how to specifically exploit the vulnerability. If that requires discussion, go to private Mattermost (`security` and/or direct-message). The public PR may have general claims (eg "I have successfully exploited this on my local system"; or "Alice, Bob, and Carol discussed on security channel - and all felt it is probably exploitable.")
* Backports for stable and ESR will be done in parallel. (They may be done by different people).
All other security issues (ie *first-party vulnerabilities; unpublished third-party vulnerabilities; uninvestigated vulnerabilities*) should continue through the current (private) workflows.
We should update https://civicrm.org/security to indicate this distinction.
# Rationale
* If a black hat is motivated enough to monitor CiviCRM's issue/PR queue for heads-up about CiviCRM vulnerabilities, then they can just as easily monitor the official release feeds for `dompdf`, `ckeditor`, etc.
* Github's "dependabot" is already likely to post public PRs when there's a published vulnerability affecting a CiviCRM dependency.
* Pro-active contributors will find it natural to raise these issues publicly (because they're already public).
* This change should reduce typical turn-around-time / duration-of-exposure for this type of issue. (*Compare: 2 weeks vs 0-3 days*)
* Routing dependency-updates through the private security medium adds noise to the private tracker without adding much security benefit.
# Other thoughts
Microsoft made "Patch Tuesday" famous. But they generally own all their dependencies.
Drupal has landed on "third Wed" as their release-window. However, they appear to make an exception when a third-party library publishes outside their preferred schedule (ex: https://www.drupal.org/sa-core-2022-006).
If we relax the scheduling on dependency updates, then we don't need to keep 1st Wed on the books. CiviCRM-specific fixes could be like Drupal -- strictly third Wed.
Anecdotally, I feel upstream announcements land on a weekday (esp Tue/Wed/Thu) -- and this lines up the interest of deployers. We could lean into this (eg dependency updates only happen on weekdays).
Note: Backdrop's release-window is _any Wed_. AFAICS, WordPress, Joomla, and PHP don't have formal release-windows. Based on skimming advisories, Joomla has a strong Tue bias. WP+PHP float around. (Between them, I skimmed ~20 prior releases, and there was only one that landed on a weekend.)