Development issueshttps://lab.civicrm.org/groups/dev/-/issues2022-10-11T10:18:48Zhttps://lab.civicrm.org/dev/core/-/issues/3848Multisite: dashlet irregular behavior2022-10-11T10:18:48ZandyburnsMultisite: dashlet irregular behaviorWithin a multisite instance, the `civicrm_dashboard_contact` behavior is a bit odd with a new twist with SK dashlets switching domains in `civicrm_dashboard`. Examining the behavior of both CiviReport dashlets and Search Kit, below:
**C...Within a multisite instance, the `civicrm_dashboard_contact` behavior is a bit odd with a new twist with SK dashlets switching domains in `civicrm_dashboard`. Examining the behavior of both CiviReport dashlets and Search Kit, below:
**CiviReport Dashlets**
If you add a CiviReport to via "Available for Dashboard?", observe that it adds a row in `civicrm_dashboard_contact` for all users of multisite, not just those in your domain, even though CiviReports is domain-aware (this also occurs with SK dashlets). This occurs on all domains generating dashboards.
Within `civicrm_dashboard`, 1 entry exists for the domain it was created on, so there is no linkage for the dashlet to show on any of the domains even though it is added in `civicrm_dashboard_contact`. Therefore, it does not show for those users.
**Search Kit Dashlets**
One feature that would be powerful to have is to have global reports in multisite. Good news is Search Kit is close to achieving this, and I currently use a version of this today. You can create a display and link the page with a relative URL in your CMS nav for instance (e.g. /wp-admin/admin.php?page=CiviCRM&q=civicrm%2Fsome-display) and it will be accessible to any domain on your multisite and respect ACL's, etc. However, as with CiviReport, the `civicrm_dashboard` is generated for 1 domain. I have done a test of inserting the SK dashboard (cloning in MySQL) on all domains so users now see dashlet in available dashlets. Once they drag it as an active dashlet, it generates the entry in `civicrm_dashboard_contact.` That's good :)
However, we observe that domain ID 1 where the dashlet was created with Form Builder, switches the domain ID to the last in the array of domains of the civicrm.settings.php file. This occurs before and after generating an entry for all domains. In this case, See below from the `log_civicrm_dashboard` as evidence:
![image](/uploads/238a4a1b9f1f1a052207c16cf5785ce6/image.png)
I have been switching it back to domain ID 1, manually. Observe below that even with domain 57 having an entry for the dashboard, it still switched domain ID 1 dashlet to 57, resulting in duplicate dashlets.
![image](/uploads/fca03a597d44747e818c230c89a04c6e/image.png)
For Search Kit dashlets, the behavior that is not clear cut. It can vary and is dependent on if it is intended to be 1) a dashlet for it's domain only, or 2) a global dashlet for all domains. With this in mind, it would NOT be desirous to generate an entry in `civicrm_dashboard` for all domains, I think this is best left to inserting directly to MySQL if you intend to make a dashboard for more than 1 domain, or some paid development to enable this. It is similar to how `civicrm_navigation` needs to be built when generating a new domain on multisite.
Noting that `uf_match` table, adds an entry for domain ID 1 for people that are not users of domain ID 1. Perhaps this i why it is generating dashlets for these. @haystack just noting this may be related to uf_match behavior. I note that currently I have 352 domain ID 1 `uf_match` entries but not anywhere near that or ever had a cumulative user count near that number, however it is lower than the total users on all other domains:
![image](/uploads/14707cebad86bab3c9f27533f46c3e63/image.png)
Evidence below of 2 entries per user:
![image](/uploads/fb942aa4ad5f8d53159c262920577622/image.png)
Summary of issues:
- [ ] CiviReport / Search Kit dashlets should not generate an entry in `civicrm_dashboard_contact` for users not on it's domain (it's harmless, but also nonsensical)
- [ ] Search Kit Dashlets via Form Builder switch domain ID (High Priority)https://lab.civicrm.org/dev/core/-/issues/3841Core (etc.) extensions should be categorized and filterable2022-10-11T10:11:26ZJonGoldCore (etc.) extensions should be categorized and filterableThe extensions page is getting harder to navigate, and isn't alphabetized by description. Finding an extension can take a while.
We should take a cue from Drupal and add `category` to the `info.xml`, then put extensions into accordions ...The extensions page is getting harder to navigate, and isn't alphabetized by description. Finding an extension can take a while.
We should take a cue from Drupal and add `category` to the `info.xml`, then put extensions into accordions by category.
Like Drupal, I don't think we need a set taxonomy, and even if only core extensions had this, it would be a good UX improvement.
I also realize that we're probably near the point where the Extensions page is built in Form Builder, but the first step is buy-in on the concept, implementation can wait a bit.JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3836Searching by phone in Advanced Search with a non-default Search Profile retur...2022-11-14T19:16:04ZbrienneSearching by phone in Advanced Search with a non-default Search Profile returns DB Error: no such fieldOverview
----------------------------------------
If a user tries to search by a phone number via Advanced Search and uses a non-default Search Profile that includes a phone number and at least one location field (i.e. city), then the se...Overview
----------------------------------------
If a user tries to search by a phone number via Advanced Search and uses a non-default Search Profile that includes a phone number and at least one location field (i.e. city), then the search produces an error with the message `DB error: no such field`.
Reproduction steps
----------------------------------------
1. Go to **Administer > Custom Data and Screens > Profiles**
1. Click **Add Profile**
1. Enter a Profile Name and check the box next/to the left of *Search Views*
1. Add fields to your Profile. At a minimum, be sure to add **Contacts > Phone > Primary > Phone** and one address field, such as **Contacts > City > Primary**. (I also included first and last name as fields)
1. Go to **Administer > Custom Data and Screens > Search Preferences**. Select the name of your test Profile from the drop down menu for the **Default Contact Search Profile** (roughly halfway down on the settings page).
1. Hover over **Search** on the navbar and click **Advanced Search**
1. Enter a phone number (or even just a number/partial number) and click **Search**
1. At this point, the user will receive an error that states `DB error: no such field`
Current behaviour
----------------------------------------
Instead of returning the results when a user searches by phone number when a non-default Search Profile includes at least one address field, the user receives an error message that the field does not exist. From the SQL displayed in the error message, this happens because the query does not include a statement to join on the address table, but references it in another join statement in the `ON` clause.
```
| Type | DB_Error |
|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Code | -19 |
| Message | DB Error: no such field |
| Mode | 16 |
| UserInfo | SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name FROM civicrm_contact contact_a LEFT JOIN civicrm_phone `1-phone-1` ON contact_a.id = `1-phone-1`.contact_id AND ( `1-phone-1`.phone_type_id = '1' OR `1-phone-1`.phone_type_id IS NULL ) AND ( `1-phone-1`.is_primary = 1 ) LEFT JOIN civicrm_phone ON (contact_a.id = civicrm_phone.contact_id AND civicrm_phone.is_primary = 1) LEFT JOIN civicrm_location_type `1-location_type` ON ( ( `1-address`.location_type_id = `1-location_type`.id ) ) WHERE ( civicrm_phone.phone_numeric LIKE '%301%' ) AND ( 1 ) AND (contact_a.is_deleted = 0) [nativecode=1054 ** Unknown column '1-address.location_type_id' in 'on clause'] |
| DebugInfo | SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name FROM civicrm_contact contact_a LEFT JOIN civicrm_phone `1-phone-1` ON contact_a.id = `1-phone-1`.contact_id AND ( `1-phone-1`.phone_type_id = '1' OR `1-phone-1`.phone_type_id IS NULL ) AND ( `1-phone-1`.is_primary = 1 ) LEFT JOIN civicrm_phone ON (contact_a.id = civicrm_phone.contact_id AND civicrm_phone.is_primary = 1) LEFT JOIN civicrm_location_type `1-location_type` ON ( ( `1-address`.location_type_id = `1-location_type`.id ) ) WHERE ( civicrm_phone.phone_numeric LIKE '%301%' ) AND ( 1 ) AND (contact_a.is_deleted = 0) [nativecode=1054 ** Unknown column '1-address.location_type_id' in 'on clause'] |
```
Expected behaviour
----------------------------------------
When a user searches by phone number via Advanced Search they should not receive an error and instead be directed to the results page of the search.
Environment information
----------------------------------------
* __CiviCRM:__ _Master/5.54.alpha1
Comments
----------------------------------------
[PR #24450](https://github.com/civicrm/civicrm-core/pull/24450) proposes a fix for this issuehttps://lab.civicrm.org/dev/core/-/issues/3835Proposal: Deprecate profile standalone listing mode2023-11-30T17:03:40ZJonGoldProposal: Deprecate profile standalone listing modeProfile standalone listing code is deeply entangled with the query BAOs, and produces weird and wonderful bugs like #3834.
I don't think they're widely used in modern CiviCRM, they seem to have been a stopgap before Views integration, a...Profile standalone listing code is deeply entangled with the query BAOs, and produces weird and wonderful bugs like #3834.
I don't think they're widely used in modern CiviCRM, they seem to have been a stopgap before Views integration, and later for non-Drupal users to get a Views-esque experience. But Search Kit has effectively replaced them - they have feature parity and then some.
I think we can improve the query code if we no longer needed to worry about standalone listing mode.
Proposal:
* Remove the ability to create standalone listings on new installations of CiviCRM.
* Create a check for profiles using standalone listing mode, warning admins if they use it.
We can deprecate this over a long period of time, so the impact should be gentle.
If this proposal is approved, we may also want to consider a function that converts profile listings to SK searches. This will have added utility as we deprecate other uses of profiles (e.g. once SK replaces Advanced Search, Search Profiles will go away).https://lab.civicrm.org/dev/core/-/issues/3833PHP 8.2 Dynamic Properties are deprecated2024-03-22T12:22:59ZBradley TaylorPHP 8.2 Dynamic Properties are deprecatedSee https://wiki.php.net/rfc/deprecate_dynamic_properties.
I see this as a big one, which could require a significant number of fixes for CiviCRM. The RFC has a good example showing what has changed:
```
class User {
public $name;
...See https://wiki.php.net/rfc/deprecate_dynamic_properties.
I see this as a big one, which could require a significant number of fixes for CiviCRM. The RFC has a good example showing what has changed:
```
class User {
public $name;
}
$user = new User;
// Assigns declared property User::$name.
$user->name = "foo";
// Oops, a typo:
$user->nane = "foo";
// PHP <= 8.1: Silently creates dynamic $user->nane property.
// PHP 8.2: Raises deprecation warning, still creates dynamic property.
// PHP 9.0: Throws Error exception.
```
There are many uses of dynamic properties in CiviCRM core which fall into multiple categories:
1. Known undeclared properties. For example `CRM_Activity_Form_Task_Batch` repeatedly references `$_fields` which should just be defined.
2. Dynamic properties. The main example of this is `CRM_Core_DAO`. PHP allows for this use-case by either extending `stdClass` or using the new ` #[AllowDynamicProperties]` attribute. Where possible we should prefer extending `stdClass` which is likely to have better long-term support (i.e. into the PHP 9.x series)
3. Typo's etc. Cases where the property is defined, but later referenced with a different name. These are easy; we just fix the typo (whilst ensuring we don't break any backwards compatiability for plugins using the wrong spelling)
I suggest CiviCRM developers should start to be made aware of this now, to avoid making the problem worse. Incrementally starting to fix the issue, a class or two at a time, seems like a good idea.
In many cases it's not clear why dynamic properties (or declared properties for that matter) are being used instead of a standard variable. This might be a good chance to start annotating properties as either:
- `@api`; designed to be used within hooks, on singletons etc by CiviCRM extensions
- `@internal`; not designed to be used by extensions, and liable to be removed without deprecation.https://lab.civicrm.org/dev/core/-/issues/3832PHP 8.2 utf8_encode and utf8_decode functions deprecated2024-01-11T18:31:16ZBradley TaylorPHP 8.2 utf8_encode and utf8_decode functions deprecatedIn PHP 8.2 (planned to be released this November) `utf8_encode` and `utf8_decode` functions will be deprecated.
https://wiki.php.net/rfc/remove_utf8_decode_and_utf8_encode
I don't see this causing too many problems for CiviCRM, and mos...In PHP 8.2 (planned to be released this November) `utf8_encode` and `utf8_decode` functions will be deprecated.
https://wiki.php.net/rfc/remove_utf8_decode_and_utf8_encode
I don't see this causing too many problems for CiviCRM, and most uses are in upstream packages and libraries. However there are some uses in core itself (especially `utf8_decode`) which should be replaced.https://lab.civicrm.org/dev/core/-/issues/3830API4 naming for custom field tokens (:name / :label)2022-09-09T10:15:16Zmagnolia61API4 naming for custom field tokens (:name / :label)Overview
----------------------------------------
With all the work on standardization of token processing we have come a long way. One missing element I think is the availability of api4 naming of the custom field tokens. In https://lab...Overview
----------------------------------------
With all the work on standardization of token processing we have come a long way. One missing element I think is the availability of api4 naming of the custom field tokens. In https://lab.civicrm.org/dev/core/-/issues/2650 this is out of scope. I would like to get this on the radar. Maybe it is an easy improvement.
Current behaviour
----------------------------------------
For core tokens :label and :name are supported to provide the raw value of label of the field. (eg. `{contact.communication_style_id:name} / | {contact.communication_style_id:label}`)
For custom fields this is not available:
`{contact.custom_1173:name} / {contact.custom_1173:name} `
The label of the field/token could be: "This is a long label" and the value: "long")
Proposed behaviour
----------------------------------------
For consistency and for easy usage in smarty conditionals it would be great if api4 naming availability for custom field tokens would be supported.
Comments
----------------------------------------
I can help test and sometimes I can code something myself if it an easy needle and I know where to look in the haystack
The overall token issue can be found here : https://lab.civicrm.org/dev/core/-/issues/2864https://lab.civicrm.org/dev/financial/-/issues/207Currency codes for São Tome and Principe Dobra, Zambian Kwacha and Zimbabwe D...2022-09-09T10:17:04ZBradley TaylorCurrency codes for São Tome and Principe Dobra, Zambian Kwacha and Zimbabwe Dollar incorrectEsentially the same issue as dev/financial#184.
- São Tome and Principe Dobra: STN since 1 January 2018
- Zambian Kwacha: ZMW since December 31, 2012
- Zimbabwe Dollar: ZWL since June 2009
I think we need to copy what we did for https:...Esentially the same issue as dev/financial#184.
- São Tome and Principe Dobra: STN since 1 January 2018
- Zambian Kwacha: ZMW since December 31, 2012
- Zimbabwe Dollar: ZWL since June 2009
I think we need to copy what we did for https://github.com/civicrm/civicrm-core/pull/21751/files, but for these 3 additional currencies:
- Update the SQL files for new installs
- Add a migration to update any historical payments.
These is possibly an argument for not migrating the data for old transactions (as in all cases the ISO code changed to reflect some change to the currency, for example a redenomination). Doing the migration fits with what happened last time, and probably saves headaches with needing to support old currency codes, which will no longer exist in libraries like Brick\Money. However, as the data _was_ migrated for Ghana and Belarus I think it probably makes sense to continue with that pattern this time around.https://lab.civicrm.org/dev/core/-/issues/3824Possible for historic scheduled reminder to be sent multiple times if cron jo...2022-11-28T08:14:03ZAdam WoodPossible for historic scheduled reminder to be sent multiple times if cron job does not complete properlyWhen the `Send Scheduled Reminders` job runs, the process fails with errors like the following (reported by our Cron Daemon):
```
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 245760 bytes) in /ho...When the `Send Scheduled Reminders` job runs, the process fails with errors like the following (reported by our Cron Daemon):
```
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 245760 bytes) in /home/cses_org_uk/public_html/administrator/components/com_civicrm/civicrm/CRM/Utils/Cache/SqlGroup.php on line 184
```
If attempting to invoke the job manually from the scheduled jobs admin page, we get an HTTP 500 error and the following entries in the error_log:
```
[Thu Aug 25 19:51:26.801469 2022] [fcgid:warn] [pid 27274] [client 79.69.229.183:51994] mod_fcgid: read data timeout in 301 seconds, referer: https://cses.org.uk/administrator/?option=com_civicrm&task=civicrm/admin/job&action=view&reset=1&context=joblog&id=9
[Thu Aug 25 19:51:26.819905 2022] [core:error] [pid 27274] [client 79.69.229.183:51994] End of script output before headers: index.php, referer: https://cses.org.uk/administrator/?option=com_civicrm&task=civicrm/admin/job&action=view&reset=1&context=joblog&id=9
```
We can see that the job is starting but never finishing from the log:
![image](/uploads/f30bd5bc2f0df34579eb3fa9247545b9/image.png)
_**The scheduled reminders do seem to be sending successfully, however. Other cron jobs also seem to complete successfully.**_
This issue seemed to start recently with an unplanned, emergency server migration. It's a virtual server so notionally everything is the same, but there may be some other factor at play here (CiviCRM upgrade, change to one of the reminder templates etc). I have double-checked PHP resource limits (256M memory, 300 second timeout i.e. more than enough) and MySQL settings and can't find anything amiss.
The error occurs at the following line in `SQLGroup.php`:
```php
private function reobjectify($value) {
return is_object($value) ? unserialize(serialize($value)) : $value; // This line fails
}
```
It seems as though `serialize()` is failing: whatever value is at fault, it was previously `unserialize()`d successfully to read it out of the cache in the first place. What does this function actually do - why the round trip??
I attempted to debug with some judicious `error_log()` immediately before and after this line to see which cache value was causing the memory leak, but to no avail. No error_log was written before the crash (although many previous successful calls to the function were logged).
This is on CiviCRM 5.50.4. I will attempt an upgrade soon to see if this fixes it, and will continue to investigate / try to narrow down / work out how to reproduce.https://lab.civicrm.org/dev/core/-/issues/3821Cron and smartgroups fail when SearchKit is unable to lookup a saved search2022-08-30T20:35:19Zjoshjosh@civicrm.orgCron and smartgroups fail when SearchKit is unable to lookup a saved searchAfter having corrected a previous issue with SearchKit unique to a specific instance, the site experienced cron and smartgroup failures. The resulting error presented:
` Argument 1 passed to CRM_Contact_BAO_GroupContactCache::getQueryOb...After having corrected a previous issue with SearchKit unique to a specific instance, the site experienced cron and smartgroup failures. The resulting error presented:
` Argument 1 passed to CRM_Contact_BAO_GroupContactCache::getQueryObjectSQL() must be of the type array, null given, called in /home/site/public_html/administrator/components/com_civicrm/civicrm/CRM/Contact/BAO/GroupContactCache.php on line 814
`
After troubleshooting, it was determined that SearchKit was looking for a smartgroup/saved search that no longer existed. As a result, crons failed entirely and existing smart group contacts would not display properly. New smart groups and related functionality functioned as expected.
Ideally, SearchKit would not hang up in this situation and/or cause issues with other areas of the system.
CC @deepak.srivastavahttps://lab.civicrm.org/dev/core/-/issues/3819Contribution type specific custom data not shown on Pledge Payment form2022-08-30T09:25:36ZyashodhaContribution type specific custom data not shown on Pledge Payment formSteps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial t...Steps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial type Donation, the field is loaded on form.
![donation_only](/uploads/2cf7222d745b53268d403ab4b082fe75/donation_only.png)
- Create a pledge of financial type Donation.
- Record pledge payment and it should take to contribution with financial type Donation pre-filled
![new_pledge_payment](/uploads/c6952707b12344500e3252281f687a11/new_pledge_payment.png)
- The custom data fields (donation only) are missing though on load. On change of financial type, the custom data loads. It should work on on load as well.https://lab.civicrm.org/dev/core/-/issues/3818Trigger based logging - improve archivability2022-08-30T10:59:52ZeileenTrigger based logging - improve archivabilityWe would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ ...We would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2015-11-09 | Initialize |8|Bob|
| 2018-09-09 | Update |8|Robert|
| 2022-10-09 | Update |8|John|
In each case the rows are logged to have the status of the updated value. So on 2022-10-09 the value in `civicrm_contact` is "John"
If the change was made in error the change can be rolled back to the last change - ie 'Robert'
However, if we say that we don't want to retain logging data older than 4 years and we have just deleted all rows older than than the after the change the table looks like this
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2022-10-09 | Update |8|John|
And we can no longer roll back the change
The options seem to be
- Have an archive routine that adds an `initialization ` of current value when we truncate
- Switch the mechanism to log the PREVIOUS value not the current value on each update (this means the live table would need to be considered when calculating any diffs)https://lab.civicrm.org/dev/core/-/issues/3817Question/Discussion: Inconsistencies between "access CiviCRM" and "access AJA...2022-09-01T11:12:34ZJohn TwymanQuestion/Discussion: Inconsistencies between "access CiviCRM" and "access AJAX API" permission grants?Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications....Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications.
Users of our client applications do not need, and should not have, the "access CiviCRM" permission. So we have been building our apps on the basis of using the "access AJAX API" permission instead.
Unfortunately, we have discovered that almost every API call involving core entities assumes "access CiviCRM" as the baseline permission for use. `Group.get`, `Participant.get`, etc., [as defined in CRM/Core/Permission.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Permission.php#L968).
Perhaps we have mistakenly assumed that "access AJAX API" was designed as a functionally equivalent permission to "access CiviCRM", minus the CiviCRM UI access.
Should the "access AJAX API" permission have the same baseline (API) permissions as "access CiviCRM"?
(I see a bigger challenge for us here in terms of the range of permissions required for certain calls, eg. Participant.get requires 'access civicrm', 'access civievents', view all participants', but one problem at a time)
Reproduction steps
----------------------------------------
1. Set up a user with an API key, etc., and a role that does _not_ have the `access civicrm` permission but does have the `acess ajax api` one
2. Query the API v3 REST endpoint with the user's credentials; eg. Group.get, Contact.get, etc.
3. Get an API permissions error response
Current behaviour
----------------------------------------
Many/most API calls (at least Entity.get calls) made by users with only the 'access ajax api' call return a permissions denied error: 'require "access civicrm"
Expected behaviour
----------------------------------------
Many/most API calls made by these users should return results
Environment information
----------------------------------------
* __CiviCRM:__ 5.49.5
* __PHP:__ 7.4/
* __CMS:__ Drupal 7.91/
* __Database:__ MariaDB 10.4.21_https://lab.civicrm.org/dev/core/-/issues/3816Saving event intermittently throws 500 error2023-02-08T21:37:50ZBobSSaving event intermittently throws 500 errorOverview
----------------------------------------
Saving an event intermittently results in a hang with:
* Greyed out tab contents
* Spinning cursor
* Error in the browser console: "VM6153:1 POST https://dmaster.demo.civicrm.org/civicrm/...Overview
----------------------------------------
Saving an event intermittently results in a hang with:
* Greyed out tab contents
* Spinning cursor
* Error in the browser console: "VM6153:1 POST https://dmaster.demo.civicrm.org/civicrm/ajax/inline?class_name=CRM_UF_Form_Inline_PreviewById&id=14&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer 500 (Internal Server Error)"
Upon a page reload following a hang, the following popup is displayed:
```We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.<br /><br />Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.<br /><br />Error type: Could not find a valid session key.```
Problem confirmed on https://dmaster.demo.civicrm.org.
Reproduction steps
----------------------------------------
1. Go to https://dmaster.demo.civicrm.org
1. Click **Events** | **Manage Events**.
1. **Rain-forest Cup Youth Soccer Tournament**: Click **Configure** | **Online Registration**.
1. Click **Save**
Current behaviour
----------------------------------------
About 50% of the time it will hang as described above.
Oddly, the failures do not seem to be randomly distributed. It seems that it will tend to fail consistently for several minutes, and then to save successfully for several minutes. Or maybe I'm doing something subtly different that I have not noticed which makes the problem appear.
Problem has been observed upon saving various tabs (Info and Settings, Event Location, Online Registration)
Problem has not been observed upon clicking __Save and Done__.
__Stress Testing__
After a page load and successful save, it appears that every subsequent save will be successful. To stress test, therefore, the following sequence must be repeated:
1. Refresh page
1. Save
__GET/POST Requests__
When saving the Online Registration tab, the following sequence is observed in the browser's Network tab for a successful save:
1. POST https://dmaster.demo.civicrm.org/civicrm/event/manage/registration?action=update&id=3&component=event&qfKey=CRMEventFormManageEventRegistrationlkkd3ym1nsgogsggw8scokgo8kw04w0cw00wsg8csgwkc4gcw_4298&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer
1. GET https://dmaster.demo.civicrm.org/civicrm/event/manage/registration?reset=1&action=update&id=3&component=event&qfKey=CRMEventFormManageEventRegistrationlkkd3ym1nsgogsggw8scokgo8kw04w0cw00wsg8csgwkc4gcw_4298&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer
1. GET https://dmaster.demo.civicrm.org/civicrm/ajax/inline?class_name=CRM_UF_Form_Inline_PreviewById&id=12&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer
When the save fails, however, only the third request is observed, and it throws a 500 error.
__Civi Log__
```
Aug 20 14:07:08 [error]
$Fatal Error Details = array:3 [
"message" => "We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.<br /><br />Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.<br /><br />Error type: Could not find a valid session key."
"code" => null
"exception" => CRM_Core_Exception {#1777
-errorData: array:1 [
"error_code" => 0
]
#cause: null
-_trace: null
#message: "We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.<br /><br />Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.<br /><br />Error type: Could not find a valid session key."
#code: 0
#file: "/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php"
#line: 861
trace: {
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:861 {
› $msg = ts("We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.") . '<br /><br />' . ts('Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.') . '<br /><br />' . ts('Error type: Could not find a valid session key.');
› throw new CRM_Core_Exception($msg);
› }
}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:856 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:316 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:193 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller/Simple.php:49 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Utils/Wrapper.php:62 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Page/AJAX.php:63 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php:285 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php:69 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php:36 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/drupal/civicrm.module:471 { …}
/var/www/clients/client1/web3/web/drupal/includes/menu.inc:527 { …}
/var/www/clients/client1/web3/web/drupal/index.php:197 { …}
}
}
]
Aug 20 14:07:08 [debug] $backTrace = #0 /var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Error.php(441): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php(39): CRM_Core_Error::handleUnhandledException(Object(CRM_Core_Exception))
#2 /var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/drupal/civicrm.module(471): CRM_Core_Invoke::invoke((Array:3))
#3 /var/www/clients/client1/web3/web/drupal/includes/menu.inc(527): civicrm_invoke("ajax", "inline")
#4 /var/www/clients/client1/web3/web/drupal/index.php(197): menu_execute_active_handler()
#5 {main}
```
__Possibly Related__
Upon loading any of the event tabs, the following error is always displayed in the console:
```
angular.js:15697 TypeError: Cannot read properties of undefined (reading 'id')
at angular-modules.c3b61f5213a83292c272ff9114e7a62a.js?rgx685:2098:66
at m.$digest (angular.js:19270:23)
at angular.js:19562:15
at Yg.completeTask (angular.js:21403:7)
at angular.js:6879:7
```
Expected behaviour
----------------------------------------
The Save should complete without error.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _Chrome 104.0.5112.101_
* __CiviCRM:__ _5.52_, _5.54.alpha1_
* __PHP:__ _7.4_
* __CMS:__ _Drupal 7.91_
* __Database:__ _MariaDB 10.4_
* __Web Server:__ _Apache 2.4_
Comments
----------------------------------------
_Anything else you would like the reviewer to note._seamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/3813Performance issue on Redis with arrays2022-11-15T04:01:55ZeileenPerformance issue on Redis with arraysBoth WMF and AUG have achieved a significant performance improvement with [this patch](https://github.com/civicrm/civicrm-core/pull/24156) which removes unnecessary calls to the underlying cache provider. However, in both cases we are us...Both WMF and AUG have achieved a significant performance improvement with [this patch](https://github.com/civicrm/civicrm-core/pull/24156) which removes unnecessary calls to the underlying cache provider. However, in both cases we are using `Redis` and `Redis` itself is performing well.
I'm left with the conclusion that the issue is the way in which the php layer is handling arrays when using `redis`.
On digging we are specifically calling [`serialize`](https://github.com/civicrm/civicrm-core/blob/53a48c5aacea52dd6507a45e111a00ccbed0fdaa/CRM/Utils/Cache/Redis.php#L116) and [`unserialize`](https://github.com/civicrm/civicrm-core/blob/53a48c5aacea52dd6507a45e111a00ccbed0fdaa/CRM/Utils/Cache/Redis.php#L138) in `CRM_Utils_Cache_Redis`.
Discussion on the interweb thingee suggests that `json_encode` & `json_decode` *may* be faster but that is a moving target over php versions .... https://stackoverflow.com/questions/22718903/storing-an-array-of-data-using-redis-from-laravel
Another option appears to be the `h` methods within the `Redis php` class - I note in our `PrevNextRedisCache` implementation we do use some of these hash specific methods and I wonder if using them here would help
https://stackoverflow.com/questions/22718903/storing-an-array-of-data-using-redis-from-laravelhttps://lab.civicrm.org/dev/core/-/issues/3809[ext:message_admin] It's not obvious there's a preview function, and even whe...2024-03-28T00:43:21ZDaveD[ext:message_admin] It's not obvious there's a preview function, and even when told you might have trouble finding itIt should be a button on the message template edit page, down with the other buttons, where preview buttons normally are.It should be a button on the message template edit page, down with the other buttons, where preview buttons normally are.https://lab.civicrm.org/dev/core/-/issues/3808[ext:message_admin] Available preview choices should be based on the site's c...2024-03-26T18:25:12ZDaveD[ext:message_admin] Available preview choices should be based on the site's configIt has fixed choices. And formatting like the comma/dot separators it uses may or may not make sense for the selected preview.It has fixed choices. And formatting like the comma/dot separators it uses may or may not make sense for the selected preview.https://lab.civicrm.org/dev/core/-/issues/3807Download invoice button on contribution fails to attach it to activity on win...2024-03-28T00:43:35ZDaveDDownload invoice button on contribution fails to attach it to activity on windowsEverything else works, but the attachment doesn't get attached to the activity.
In https://github.com/civicrm/civicrm-core/blob/2aaf9d423288b5acb322379eb135324f380faca4/CRM/Core/BAO/File.php#L117-L130, it uses DIRECTORY_SEPARATOR when i...Everything else works, but the attachment doesn't get attached to the activity.
In https://github.com/civicrm/civicrm-core/blob/2aaf9d423288b5acb322379eb135324f380faca4/CRM/Core/BAO/File.php#L117-L130, it uses DIRECTORY_SEPARATOR when it should just always use '/', and in the one place it does need to use DIRECTORY_SEPARATOR it only checks '/':
```php
$path = explode('/', $data); // <--- THIS IS WRONG
$filename = $path[count($path) - 1];
// rename this file to go into the secure directory
if ($entitySubtype) {
$directoryName = $config->customFileUploadDir . $entitySubtype . DIRECTORY_SEPARATOR . $entityID;
}
else {
$directoryName = $config->customFileUploadDir;
}
CRM_Utils_File::createDir($directoryName);
if (!rename($data, $directoryName . DIRECTORY_SEPARATOR . $filename)) {
```https://lab.civicrm.org/dev/financial/-/issues/204Editing a line item on a membership extends the term of the membership2022-10-11T12:27:28ZkcristianoEditing a line item on a membership extends the term of the membershipCiviCRM 5.50+
Tested on WP Master
Required Line item editor extension and membership created from a price set.
wpcvmaster - 5.53
Steps to reproduce:
- Add a recurring membership
![image](/uploads/6bcad85c842288f49b37d665d4cf02a3...CiviCRM 5.50+
Tested on WP Master
Required Line item editor extension and membership created from a price set.
wpcvmaster - 5.53
Steps to reproduce:
- Add a recurring membership
![image](/uploads/6bcad85c842288f49b37d665d4cf02a3/image.png)
![image](/uploads/79381fa3d7cc4976f3df819fdb79c54e/image.png)
* Need to update rate for the next recurring membership payment
- click more on the recurring transaction that is in progress
![image](/uploads/3b1e39119d0eeb3ccd63ff6da8a72e67/image.png)
* view template
* then edit at bottom of screen
![image](/uploads/0c67dab579f9fd714b9fe3c7517e9750/image.png)
* edit line item
![image](/uploads/9eb56bec0334d8f389db8f5b68f0c33b/image.png)
* change amount
![image](/uploads/512f88dde2200e18efb8771bb3b73b7c/image.png)
- recurring is now updated:
![image](/uploads/4ffaae2bf8207c6b14f3f02d11e8f9aa/image.png)
Unintended consequence: Membership extended:
![image](/uploads/da6e71e2d4d2fbc7f9de4c2709691931/image.png)
cc @seamuslee @JoeMurray This _may_ be related to the 2.5 releaseMonish DebMonish Debhttps://lab.civicrm.org/dev/financial/-/issues/203False-positive duplicate detection caused by webform not passing correct par...2024-01-16T19:36:50ZAllenShawFalse-positive duplicate detection caused by webform not passing correct parameters to Authorize.net - it needs to pass contributionID(Joinery internal reference: "Trello#142: Membership Application and Payment Process Not Working")
From what I understand so far, it seems there's a well-thought effort to use propertybag to standardize parameter names in payment proces...(Joinery internal reference: "Trello#142: Membership Application and Payment Process Not Working")
From what I understand so far, it seems there's a well-thought effort to use propertybag to standardize parameter names in payment processors. Here's one area where this effort seems incomplete and easily improved:
There are a few places in `CRM_Core_Payment_AuthorizeNet::doPayment()` where directly accessing `$params` could (and does) cause problems; it seems right to switch these to `$propertybag`. (BTW I think that usage of `propertybag` in the context of alterPaymentProcessorParams hook is still not appropriate, so I'm not suggesting changes to that usage within this method.)
**Specific observable problem:**
(I can create a more detailed and stripped-down recipe if needed, but hoping to avoid the effort.)
- On a site with: D7, civicrm 5.49.5, webform_civicrm.module, and contributiontransactlegacy extension
- We have a webform which accepts membership signups, payable through the core Authorize.net payment processor.
- Payments made through this webform are actually transacted in Authorize.net, but the user is given the error, 'It appears that this transaction is a duplicate. Have you already submitted the form once? If so there may have been a connection problem. Check your email for a receipt from Authorize.net. If you do not receive a receipt within 2 hours you can try your transaction again. If you continue to have problems please contact the site administrator.'
- FWIW I have observed that the below corrective measure fixes this problem for this site.
**Why this error happens:**
- `CRM_Core_Payment_AuthorizeNet::doPayment()` calls `$this->checkDupe($authorizeNetFields['x_invoice_num'], $params['contributionID'] ?? NULL)` (see [line 171](https://github.com/civicrm/civicrm-core/blob/a1278f71bd804c8013600793916a692b4345e4a2/CRM/Core/Payment/AuthorizeNet.php#L171)) to detect a duplicate transaction.
- However, `$params['contributionID']` is undefined. (On the other hand, `$params['contribution_id']` is defined, as is `$propertyBag->getContributionID( )`).
- Since `checkDup()` gets a NULL value for `$contributionId`, it always thinks the contribution is a duplicate, so we get the above error message.
**Corrective measueres**
1. Seems like the intent of `propertybag` is to clear up exctly this kind of naming inconsistency. I suggest changing Line 171 to use `$propertyBag->getContributionID( )` instaed of `$params['contributionID']`.
2. [Line 146](https://github.com/civicrm/civicrm-core/blob/a1278f71bd804c8013600793916a692b4345e4a2/CRM/Core/Payment/AuthorizeNet.php#L146), which tries to read `$params['is_recur']` and `$params['contributionRecurID']`, is probably also a good candidate for similar cleanup?
Should I go head with a PR or is more discussion needed?