Development issueshttps://lab.civicrm.org/groups/dev/-/issues2022-10-11T10:38:49Zhttps://lab.civicrm.org/dev/core/-/issues/3851Discussion: How to make past unsubscribes visible in the UI2022-10-11T10:38:49ZlarsssandergreenDiscussion: How to make past unsubscribes visible in the UIEdit: Title changed to reflect discussion, which has gone in a different (better) direction.
Sometimes, a contact unsubscribes from a group and is later re-added to the same group. After this happens, we no longer have any record of the...Edit: Title changed to reflect discussion, which has gone in a different (better) direction.
Sometimes, a contact unsubscribes from a group and is later re-added to the same group. After this happens, we no longer have any record of the contact unsubscribing as their group status has been updated to Added with the added date.
It would be useful to be able to check when a contact unsubscribed in order to troubleshoot (are contacts being re-added to groups they have unsubscribed from?), to confirm with the contact manually that they wish to be in a newsletter subscription group and generally to be able to say that the org has a record of the contact unsubscribing. Similarly, if a contact unsubscribes from a group that is later deleted, then there is no longer any record of their having unsubscribed.
So, I propose to add an activity whenever a contact unsubscribes from a group by email:
1. Add an activity type: Unsubscribe.
2. Whenever a contact unsubscribes by email, add an activity of this type with the subject indicating the group unsubscribed from and the mailing from which they unsubscribed: "Unsubscribed from GROUP via MAILING".
For similar record keeping purposes, I think it would make sense to do the same with opt outs, with a new Opt out activity type, with subject: "Opted out from bulk emails via MAILING".
In theory, these changes would be viewable in the change log, but that's only if you have logging turned on and are willing to trawl through many change log entries to try to find the relevant one. An activity would be much more usable and is in line with many other actions that are recorded as activities (like changes to recurring contributions, contributions themselves, membership status changes, event registrations and changes, etc).https://lab.civicrm.org/dev/core/-/issues/3848Multisite: dashlet irregular behavior2022-10-11T10:18:48ZandyburnsMultisite: dashlet irregular behaviorWithin a multisite instance, the `civicrm_dashboard_contact` behavior is a bit odd with a new twist with SK dashlets switching domains in `civicrm_dashboard`. Examining the behavior of both CiviReport dashlets and Search Kit, below:
**C...Within a multisite instance, the `civicrm_dashboard_contact` behavior is a bit odd with a new twist with SK dashlets switching domains in `civicrm_dashboard`. Examining the behavior of both CiviReport dashlets and Search Kit, below:
**CiviReport Dashlets**
If you add a CiviReport to via "Available for Dashboard?", observe that it adds a row in `civicrm_dashboard_contact` for all users of multisite, not just those in your domain, even though CiviReports is domain-aware (this also occurs with SK dashlets). This occurs on all domains generating dashboards.
Within `civicrm_dashboard`, 1 entry exists for the domain it was created on, so there is no linkage for the dashlet to show on any of the domains even though it is added in `civicrm_dashboard_contact`. Therefore, it does not show for those users.
**Search Kit Dashlets**
One feature that would be powerful to have is to have global reports in multisite. Good news is Search Kit is close to achieving this, and I currently use a version of this today. You can create a display and link the page with a relative URL in your CMS nav for instance (e.g. /wp-admin/admin.php?page=CiviCRM&q=civicrm%2Fsome-display) and it will be accessible to any domain on your multisite and respect ACL's, etc. However, as with CiviReport, the `civicrm_dashboard` is generated for 1 domain. I have done a test of inserting the SK dashboard (cloning in MySQL) on all domains so users now see dashlet in available dashlets. Once they drag it as an active dashlet, it generates the entry in `civicrm_dashboard_contact.` That's good :)
However, we observe that domain ID 1 where the dashlet was created with Form Builder, switches the domain ID to the last in the array of domains of the civicrm.settings.php file. This occurs before and after generating an entry for all domains. In this case, See below from the `log_civicrm_dashboard` as evidence:
![image](/uploads/238a4a1b9f1f1a052207c16cf5785ce6/image.png)
I have been switching it back to domain ID 1, manually. Observe below that even with domain 57 having an entry for the dashboard, it still switched domain ID 1 dashlet to 57, resulting in duplicate dashlets.
![image](/uploads/fca03a597d44747e818c230c89a04c6e/image.png)
For Search Kit dashlets, the behavior that is not clear cut. It can vary and is dependent on if it is intended to be 1) a dashlet for it's domain only, or 2) a global dashlet for all domains. With this in mind, it would NOT be desirous to generate an entry in `civicrm_dashboard` for all domains, I think this is best left to inserting directly to MySQL if you intend to make a dashboard for more than 1 domain, or some paid development to enable this. It is similar to how `civicrm_navigation` needs to be built when generating a new domain on multisite.
Noting that `uf_match` table, adds an entry for domain ID 1 for people that are not users of domain ID 1. Perhaps this i why it is generating dashlets for these. @haystack just noting this may be related to uf_match behavior. I note that currently I have 352 domain ID 1 `uf_match` entries but not anywhere near that or ever had a cumulative user count near that number, however it is lower than the total users on all other domains:
![image](/uploads/14707cebad86bab3c9f27533f46c3e63/image.png)
Evidence below of 2 entries per user:
![image](/uploads/fb942aa4ad5f8d53159c262920577622/image.png)
Summary of issues:
- [ ] CiviReport / Search Kit dashlets should not generate an entry in `civicrm_dashboard_contact` for users not on it's domain (it's harmless, but also nonsensical)
- [ ] Search Kit Dashlets via Form Builder switch domain ID (High Priority)https://lab.civicrm.org/dev/core/-/issues/3841Core (etc.) extensions should be categorized and filterable2022-10-11T10:11:26ZJonGoldCore (etc.) extensions should be categorized and filterableThe extensions page is getting harder to navigate, and isn't alphabetized by description. Finding an extension can take a while.
We should take a cue from Drupal and add `category` to the `info.xml`, then put extensions into accordions ...The extensions page is getting harder to navigate, and isn't alphabetized by description. Finding an extension can take a while.
We should take a cue from Drupal and add `category` to the `info.xml`, then put extensions into accordions by category.
Like Drupal, I don't think we need a set taxonomy, and even if only core extensions had this, it would be a good UX improvement.
I also realize that we're probably near the point where the Extensions page is built in Form Builder, but the first step is buy-in on the concept, implementation can wait a bit.JonGoldJonGoldhttps://lab.civicrm.org/dev/drupal/-/issues/182Drupal 7 modules not ported to Drupal 9 yet2022-09-09T10:08:56Zluke.stewartDrupal 7 modules not ported to Drupal 9 yetCurrently there are a number of Drupal 7 modules that have not been ported to Drupal 9 as a - comes bundled with CiviCRM - code located within the CiviCRM codebase on install.
This can be seen by comparing
https://github.com/civicrm/civ...Currently there are a number of Drupal 7 modules that have not been ported to Drupal 9 as a - comes bundled with CiviCRM - code located within the CiviCRM codebase on install.
This can be seen by comparing
https://github.com/civicrm/civicrm-drupal/tree/7.x-master/modules
and
https://github.com/civicrm/civicrm-drupal-8/tree/master/modules/
Now a number of these modules it doesn't make sense to run in D9:
Views, CiviCRM OG Sync CiviCRM Engage and possibly others?
Some have a D9 module:
CiviCRM Group Roles has been set up as a stand alone project
https://www.drupal.org/project/civicrm_group_roles
CiviCRM Theme has been set up as part of the civicrm-drupal-8 codebase
CiviCRM Member Roles has been proposed as a PR to the drupal-8 repo but this has been stalled for a while now.
https://github.com/civicrm/civicrm-drupal-8/pull/66
I feel like options for modules here are now:
- A) 🍎 Set up as a stand alone project leave it to Site Builders/Maintainers as to whether to install or not.
- B) 🍌 Set up as a stand alone project but require as a dependency via composer so code gets installed when installing CiviCRM
Do we have criteria for deciding which option in general?
Do we have criteria for deciding which option should be applied in the case of CiviCRM Member Roles?
- C) 🥕 Include within the CiviCRM Drupal codebase via the civicrm/civicrm-drupal-8 project
Do we need a general criteria before deciding approach for CiviCRM Member Roles?
I think probably we would not want to use B unless we wanted something in a separate module that was required for the vast majority of installs of CiviCRM to run?
Given that there has been little appetite for CiviCRM Member Roles perhaps the approach is to split this into it's own module on the basis that most D9 sites to date are probably not using - therefore perhaps this code should not be required to be present on all CiviCRM installs.
However - we still have a lot of sites running D7 - and perhaps the sites that would use Member Roles are mainly running D7?https://lab.civicrm.org/dev/core/-/issues/3836Searching by phone in Advanced Search with a non-default Search Profile retur...2022-11-14T19:16:04ZbrienneSearching by phone in Advanced Search with a non-default Search Profile returns DB Error: no such fieldOverview
----------------------------------------
If a user tries to search by a phone number via Advanced Search and uses a non-default Search Profile that includes a phone number and at least one location field (i.e. city), then the se...Overview
----------------------------------------
If a user tries to search by a phone number via Advanced Search and uses a non-default Search Profile that includes a phone number and at least one location field (i.e. city), then the search produces an error with the message `DB error: no such field`.
Reproduction steps
----------------------------------------
1. Go to **Administer > Custom Data and Screens > Profiles**
1. Click **Add Profile**
1. Enter a Profile Name and check the box next/to the left of *Search Views*
1. Add fields to your Profile. At a minimum, be sure to add **Contacts > Phone > Primary > Phone** and one address field, such as **Contacts > City > Primary**. (I also included first and last name as fields)
1. Go to **Administer > Custom Data and Screens > Search Preferences**. Select the name of your test Profile from the drop down menu for the **Default Contact Search Profile** (roughly halfway down on the settings page).
1. Hover over **Search** on the navbar and click **Advanced Search**
1. Enter a phone number (or even just a number/partial number) and click **Search**
1. At this point, the user will receive an error that states `DB error: no such field`
Current behaviour
----------------------------------------
Instead of returning the results when a user searches by phone number when a non-default Search Profile includes at least one address field, the user receives an error message that the field does not exist. From the SQL displayed in the error message, this happens because the query does not include a statement to join on the address table, but references it in another join statement in the `ON` clause.
```
| Type | DB_Error |
|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Code | -19 |
| Message | DB Error: no such field |
| Mode | 16 |
| UserInfo | SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name FROM civicrm_contact contact_a LEFT JOIN civicrm_phone `1-phone-1` ON contact_a.id = `1-phone-1`.contact_id AND ( `1-phone-1`.phone_type_id = '1' OR `1-phone-1`.phone_type_id IS NULL ) AND ( `1-phone-1`.is_primary = 1 ) LEFT JOIN civicrm_phone ON (contact_a.id = civicrm_phone.contact_id AND civicrm_phone.is_primary = 1) LEFT JOIN civicrm_location_type `1-location_type` ON ( ( `1-address`.location_type_id = `1-location_type`.id ) ) WHERE ( civicrm_phone.phone_numeric LIKE '%301%' ) AND ( 1 ) AND (contact_a.is_deleted = 0) [nativecode=1054 ** Unknown column '1-address.location_type_id' in 'on clause'] |
| DebugInfo | SELECT DISTINCT LEFT(contact_a.sort_name, 1) as sort_name FROM civicrm_contact contact_a LEFT JOIN civicrm_phone `1-phone-1` ON contact_a.id = `1-phone-1`.contact_id AND ( `1-phone-1`.phone_type_id = '1' OR `1-phone-1`.phone_type_id IS NULL ) AND ( `1-phone-1`.is_primary = 1 ) LEFT JOIN civicrm_phone ON (contact_a.id = civicrm_phone.contact_id AND civicrm_phone.is_primary = 1) LEFT JOIN civicrm_location_type `1-location_type` ON ( ( `1-address`.location_type_id = `1-location_type`.id ) ) WHERE ( civicrm_phone.phone_numeric LIKE '%301%' ) AND ( 1 ) AND (contact_a.is_deleted = 0) [nativecode=1054 ** Unknown column '1-address.location_type_id' in 'on clause'] |
```
Expected behaviour
----------------------------------------
When a user searches by phone number via Advanced Search they should not receive an error and instead be directed to the results page of the search.
Environment information
----------------------------------------
* __CiviCRM:__ _Master/5.54.alpha1
Comments
----------------------------------------
[PR #24450](https://github.com/civicrm/civicrm-core/pull/24450) proposes a fix for this issuehttps://lab.civicrm.org/dev/core/-/issues/3835Proposal: Deprecate profile standalone listing mode2023-11-30T17:03:40ZJonGoldProposal: Deprecate profile standalone listing modeProfile standalone listing code is deeply entangled with the query BAOs, and produces weird and wonderful bugs like #3834.
I don't think they're widely used in modern CiviCRM, they seem to have been a stopgap before Views integration, a...Profile standalone listing code is deeply entangled with the query BAOs, and produces weird and wonderful bugs like #3834.
I don't think they're widely used in modern CiviCRM, they seem to have been a stopgap before Views integration, and later for non-Drupal users to get a Views-esque experience. But Search Kit has effectively replaced them - they have feature parity and then some.
I think we can improve the query code if we no longer needed to worry about standalone listing mode.
Proposal:
* Remove the ability to create standalone listings on new installations of CiviCRM.
* Create a check for profiles using standalone listing mode, warning admins if they use it.
We can deprecate this over a long period of time, so the impact should be gentle.
If this proposal is approved, we may also want to consider a function that converts profile listings to SK searches. This will have added utility as we deprecate other uses of profiles (e.g. once SK replaces Advanced Search, Search Profiles will go away).https://lab.civicrm.org/dev/core/-/issues/3833PHP 8.2 Dynamic Properties are deprecated2024-03-22T12:22:59ZBradley TaylorPHP 8.2 Dynamic Properties are deprecatedSee https://wiki.php.net/rfc/deprecate_dynamic_properties.
I see this as a big one, which could require a significant number of fixes for CiviCRM. The RFC has a good example showing what has changed:
```
class User {
public $name;
...See https://wiki.php.net/rfc/deprecate_dynamic_properties.
I see this as a big one, which could require a significant number of fixes for CiviCRM. The RFC has a good example showing what has changed:
```
class User {
public $name;
}
$user = new User;
// Assigns declared property User::$name.
$user->name = "foo";
// Oops, a typo:
$user->nane = "foo";
// PHP <= 8.1: Silently creates dynamic $user->nane property.
// PHP 8.2: Raises deprecation warning, still creates dynamic property.
// PHP 9.0: Throws Error exception.
```
There are many uses of dynamic properties in CiviCRM core which fall into multiple categories:
1. Known undeclared properties. For example `CRM_Activity_Form_Task_Batch` repeatedly references `$_fields` which should just be defined.
2. Dynamic properties. The main example of this is `CRM_Core_DAO`. PHP allows for this use-case by either extending `stdClass` or using the new ` #[AllowDynamicProperties]` attribute. Where possible we should prefer extending `stdClass` which is likely to have better long-term support (i.e. into the PHP 9.x series)
3. Typo's etc. Cases where the property is defined, but later referenced with a different name. These are easy; we just fix the typo (whilst ensuring we don't break any backwards compatiability for plugins using the wrong spelling)
I suggest CiviCRM developers should start to be made aware of this now, to avoid making the problem worse. Incrementally starting to fix the issue, a class or two at a time, seems like a good idea.
In many cases it's not clear why dynamic properties (or declared properties for that matter) are being used instead of a standard variable. This might be a good chance to start annotating properties as either:
- `@api`; designed to be used within hooks, on singletons etc by CiviCRM extensions
- `@internal`; not designed to be used by extensions, and liable to be removed without deprecation.https://lab.civicrm.org/dev/core/-/issues/3832PHP 8.2 utf8_encode and utf8_decode functions deprecated2024-01-11T18:31:16ZBradley TaylorPHP 8.2 utf8_encode and utf8_decode functions deprecatedIn PHP 8.2 (planned to be released this November) `utf8_encode` and `utf8_decode` functions will be deprecated.
https://wiki.php.net/rfc/remove_utf8_decode_and_utf8_encode
I don't see this causing too many problems for CiviCRM, and mos...In PHP 8.2 (planned to be released this November) `utf8_encode` and `utf8_decode` functions will be deprecated.
https://wiki.php.net/rfc/remove_utf8_decode_and_utf8_encode
I don't see this causing too many problems for CiviCRM, and most uses are in upstream packages and libraries. However there are some uses in core itself (especially `utf8_decode`) which should be replaced.https://lab.civicrm.org/dev/core/-/issues/3830API4 naming for custom field tokens (:name / :label)2022-09-09T10:15:16Zmagnolia61API4 naming for custom field tokens (:name / :label)Overview
----------------------------------------
With all the work on standardization of token processing we have come a long way. One missing element I think is the availability of api4 naming of the custom field tokens. In https://lab...Overview
----------------------------------------
With all the work on standardization of token processing we have come a long way. One missing element I think is the availability of api4 naming of the custom field tokens. In https://lab.civicrm.org/dev/core/-/issues/2650 this is out of scope. I would like to get this on the radar. Maybe it is an easy improvement.
Current behaviour
----------------------------------------
For core tokens :label and :name are supported to provide the raw value of label of the field. (eg. `{contact.communication_style_id:name} / | {contact.communication_style_id:label}`)
For custom fields this is not available:
`{contact.custom_1173:name} / {contact.custom_1173:name} `
The label of the field/token could be: "This is a long label" and the value: "long")
Proposed behaviour
----------------------------------------
For consistency and for easy usage in smarty conditionals it would be great if api4 naming availability for custom field tokens would be supported.
Comments
----------------------------------------
I can help test and sometimes I can code something myself if it an easy needle and I know where to look in the haystack
The overall token issue can be found here : https://lab.civicrm.org/dev/core/-/issues/2864https://lab.civicrm.org/dev/financial/-/issues/207Currency codes for São Tome and Principe Dobra, Zambian Kwacha and Zimbabwe D...2022-09-09T10:17:04ZBradley TaylorCurrency codes for São Tome and Principe Dobra, Zambian Kwacha and Zimbabwe Dollar incorrectEsentially the same issue as dev/financial#184.
- São Tome and Principe Dobra: STN since 1 January 2018
- Zambian Kwacha: ZMW since December 31, 2012
- Zimbabwe Dollar: ZWL since June 2009
I think we need to copy what we did for https:...Esentially the same issue as dev/financial#184.
- São Tome and Principe Dobra: STN since 1 January 2018
- Zambian Kwacha: ZMW since December 31, 2012
- Zimbabwe Dollar: ZWL since June 2009
I think we need to copy what we did for https://github.com/civicrm/civicrm-core/pull/21751/files, but for these 3 additional currencies:
- Update the SQL files for new installs
- Add a migration to update any historical payments.
These is possibly an argument for not migrating the data for old transactions (as in all cases the ISO code changed to reflect some change to the currency, for example a redenomination). Doing the migration fits with what happened last time, and probably saves headaches with needing to support old currency codes, which will no longer exist in libraries like Brick\Money. However, as the data _was_ migrated for Ghana and Belarus I think it probably makes sense to continue with that pattern this time around.https://lab.civicrm.org/dev/core/-/issues/3824Possible for historic scheduled reminder to be sent multiple times if cron jo...2022-11-28T08:14:03ZAdam WoodPossible for historic scheduled reminder to be sent multiple times if cron job does not complete properlyWhen the `Send Scheduled Reminders` job runs, the process fails with errors like the following (reported by our Cron Daemon):
```
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 245760 bytes) in /ho...When the `Send Scheduled Reminders` job runs, the process fails with errors like the following (reported by our Cron Daemon):
```
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 245760 bytes) in /home/cses_org_uk/public_html/administrator/components/com_civicrm/civicrm/CRM/Utils/Cache/SqlGroup.php on line 184
```
If attempting to invoke the job manually from the scheduled jobs admin page, we get an HTTP 500 error and the following entries in the error_log:
```
[Thu Aug 25 19:51:26.801469 2022] [fcgid:warn] [pid 27274] [client 79.69.229.183:51994] mod_fcgid: read data timeout in 301 seconds, referer: https://cses.org.uk/administrator/?option=com_civicrm&task=civicrm/admin/job&action=view&reset=1&context=joblog&id=9
[Thu Aug 25 19:51:26.819905 2022] [core:error] [pid 27274] [client 79.69.229.183:51994] End of script output before headers: index.php, referer: https://cses.org.uk/administrator/?option=com_civicrm&task=civicrm/admin/job&action=view&reset=1&context=joblog&id=9
```
We can see that the job is starting but never finishing from the log:
![image](/uploads/f30bd5bc2f0df34579eb3fa9247545b9/image.png)
_**The scheduled reminders do seem to be sending successfully, however. Other cron jobs also seem to complete successfully.**_
This issue seemed to start recently with an unplanned, emergency server migration. It's a virtual server so notionally everything is the same, but there may be some other factor at play here (CiviCRM upgrade, change to one of the reminder templates etc). I have double-checked PHP resource limits (256M memory, 300 second timeout i.e. more than enough) and MySQL settings and can't find anything amiss.
The error occurs at the following line in `SQLGroup.php`:
```php
private function reobjectify($value) {
return is_object($value) ? unserialize(serialize($value)) : $value; // This line fails
}
```
It seems as though `serialize()` is failing: whatever value is at fault, it was previously `unserialize()`d successfully to read it out of the cache in the first place. What does this function actually do - why the round trip??
I attempted to debug with some judicious `error_log()` immediately before and after this line to see which cache value was causing the memory leak, but to no avail. No error_log was written before the crash (although many previous successful calls to the function were logged).
This is on CiviCRM 5.50.4. I will attempt an upgrade soon to see if this fixes it, and will continue to investigate / try to narrow down / work out how to reproduce.https://lab.civicrm.org/dev/financial/-/issues/205CiviFinancial Blue Sky Dreaming2022-08-25T21:31:40ZJoeMurrayCiviFinancial Blue Sky DreamingThere has been talk over the years by different people about a LExIM shift to a new paradigm/implementation for CiviAccount. I have been approached by someone who could rustle up a tiny starting stake for such a huge effort (I'm not conf...There has been talk over the years by different people about a LExIM shift to a new paradigm/implementation for CiviAccount. I have been approached by someone who could rustle up a tiny starting stake for such a huge effort (I'm not confident there would be an ability to get to 50% needed to launch an MIH that will likely cost in the six figures at CT billable rates). Here is a blue sky dreaming of how a CiviFinancials LExIM might succeed.
1. We find funding for integrating payments into Form Builder - this seems likely to be possible and occur, yet is still a big lift. Imagine all the financial aspects of webform_civicrm.
2. When implementing the new interface for contribution pages and event reg pages, etc., we have some 'dependency injection' that allows calls to be made either to the existing financial APIs or to a new set.
3. We work to implement the guts of CiviAccounting in a new way. Discussion is needed here, but I would be in favour of some significant breaking changes like simplifying the Financial Type/Financial Account model into much more standard accounting, preferrably with simple, sensible defaults.
1. We could design our own CiviAccounts v2 from scratch.
2. We could aim to implement an interface with an existing open source accounting system. The aim is to store all the CiviCRM financial transactions in a standard, validated way without having the development and support burden of making it ourselves. We'd only need to keep the integration going strong. Two candidates I have found that might be feasible, and there are many others, are Front Accounting and GnuCash. https://sourceforge.net/projects/frontaccounting/files/FrontAccounting%202.4/stats/timeline?dates=2019-08-01+to+2022-08-01 has the same technical stack as CiviCRM (PHP and MySQL) but seems to be trending down in installs. https://sourceforge.net/projects/gnucash/files/gnucash%20%28stable%29/stats/timeline?dates=2019-08-01+to+2022-08-01 is much more popular and losing momentum more slowly. It is built in C and C++, so it would be more complex to integrate.
3. If there is a fully open source implementation via CiviAccounts v2 or integrating an open source accounting package, then I would be okay with there being integrations with closed source accounting systems that are popular like Xero and QuickBooks Online. I suppose we could allow the existing implementation of CiviAccounts to be the open source option. While more financially realistic, it might be bad for our reputation.https://lab.civicrm.org/dev/core/-/issues/3821Cron and smartgroups fail when SearchKit is unable to lookup a saved search2022-08-30T20:35:19Zjoshjosh@civicrm.orgCron and smartgroups fail when SearchKit is unable to lookup a saved searchAfter having corrected a previous issue with SearchKit unique to a specific instance, the site experienced cron and smartgroup failures. The resulting error presented:
` Argument 1 passed to CRM_Contact_BAO_GroupContactCache::getQueryOb...After having corrected a previous issue with SearchKit unique to a specific instance, the site experienced cron and smartgroup failures. The resulting error presented:
` Argument 1 passed to CRM_Contact_BAO_GroupContactCache::getQueryObjectSQL() must be of the type array, null given, called in /home/site/public_html/administrator/components/com_civicrm/civicrm/CRM/Contact/BAO/GroupContactCache.php on line 814
`
After troubleshooting, it was determined that SearchKit was looking for a smartgroup/saved search that no longer existed. As a result, crons failed entirely and existing smart group contacts would not display properly. New smart groups and related functionality functioned as expected.
Ideally, SearchKit would not hang up in this situation and/or cause issues with other areas of the system.
CC @deepak.srivastavahttps://lab.civicrm.org/dev/core/-/issues/3819Contribution type specific custom data not shown on Pledge Payment form2022-08-30T09:25:36ZyashodhaContribution type specific custom data not shown on Pledge Payment formSteps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial t...Steps to replicate :
- Create custom data for entity Contribution of financial type Donation.
![custom_fields](/uploads/cc517d654e574b63686b40c640306c0b/custom_fields.png)
- Check when you create a contribution and set the financial type Donation, the field is loaded on form.
![donation_only](/uploads/2cf7222d745b53268d403ab4b082fe75/donation_only.png)
- Create a pledge of financial type Donation.
- Record pledge payment and it should take to contribution with financial type Donation pre-filled
![new_pledge_payment](/uploads/c6952707b12344500e3252281f687a11/new_pledge_payment.png)
- The custom data fields (donation only) are missing though on load. On change of financial type, the custom data loads. It should work on on load as well.https://lab.civicrm.org/dev/core/-/issues/3818Trigger based logging - improve archivability2022-08-30T10:59:52ZeileenTrigger based logging - improve archivabilityWe would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ ...We would like to time-limit our database logging but there is a challenge when deleting old rows from the `log_` tables.
Example rows from `log_civicrm_contact`
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2015-11-09 | Initialize |8|Bob|
| 2018-09-09 | Update |8|Robert|
| 2022-10-09 | Update |8|John|
In each case the rows are logged to have the status of the updated value. So on 2022-10-09 the value in `civicrm_contact` is "John"
If the change was made in error the change can be rolled back to the last change - ie 'Robert'
However, if we say that we don't want to retain logging data older than 4 years and we have just deleted all rows older than than the after the change the table looks like this
| log_date| log_action |id|first_name|
| ------ | ------ |------ |------ |
| 2022-10-09 | Update |8|John|
And we can no longer roll back the change
The options seem to be
- Have an archive routine that adds an `initialization ` of current value when we truncate
- Switch the mechanism to log the PREVIOUS value not the current value on each update (this means the live table would need to be considered when calculating any diffs)https://lab.civicrm.org/dev/core/-/issues/3817Question/Discussion: Inconsistencies between "access CiviCRM" and "access AJA...2022-09-01T11:12:34ZJohn TwymanQuestion/Discussion: Inconsistencies between "access CiviCRM" and "access AJAX API" permission grants?Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications....Overview
----------------------------------------
We are developing client applications that integrate with CiviCRM via its API and the AuthX extension. This allows us to query the API as the user, rather than as the client applications.
Users of our client applications do not need, and should not have, the "access CiviCRM" permission. So we have been building our apps on the basis of using the "access AJAX API" permission instead.
Unfortunately, we have discovered that almost every API call involving core entities assumes "access CiviCRM" as the baseline permission for use. `Group.get`, `Participant.get`, etc., [as defined in CRM/Core/Permission.php](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/Permission.php#L968).
Perhaps we have mistakenly assumed that "access AJAX API" was designed as a functionally equivalent permission to "access CiviCRM", minus the CiviCRM UI access.
Should the "access AJAX API" permission have the same baseline (API) permissions as "access CiviCRM"?
(I see a bigger challenge for us here in terms of the range of permissions required for certain calls, eg. Participant.get requires 'access civicrm', 'access civievents', view all participants', but one problem at a time)
Reproduction steps
----------------------------------------
1. Set up a user with an API key, etc., and a role that does _not_ have the `access civicrm` permission but does have the `acess ajax api` one
2. Query the API v3 REST endpoint with the user's credentials; eg. Group.get, Contact.get, etc.
3. Get an API permissions error response
Current behaviour
----------------------------------------
Many/most API calls (at least Entity.get calls) made by users with only the 'access ajax api' call return a permissions denied error: 'require "access civicrm"
Expected behaviour
----------------------------------------
Many/most API calls made by these users should return results
Environment information
----------------------------------------
* __CiviCRM:__ 5.49.5
* __PHP:__ 7.4/
* __CMS:__ Drupal 7.91/
* __Database:__ MariaDB 10.4.21_https://lab.civicrm.org/dev/core/-/issues/3816Saving event intermittently throws 500 error2023-02-08T21:37:50ZBobSSaving event intermittently throws 500 errorOverview
----------------------------------------
Saving an event intermittently results in a hang with:
* Greyed out tab contents
* Spinning cursor
* Error in the browser console: "VM6153:1 POST https://dmaster.demo.civicrm.org/civicrm/...Overview
----------------------------------------
Saving an event intermittently results in a hang with:
* Greyed out tab contents
* Spinning cursor
* Error in the browser console: "VM6153:1 POST https://dmaster.demo.civicrm.org/civicrm/ajax/inline?class_name=CRM_UF_Form_Inline_PreviewById&id=14&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer 500 (Internal Server Error)"
Upon a page reload following a hang, the following popup is displayed:
```We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.<br /><br />Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.<br /><br />Error type: Could not find a valid session key.```
Problem confirmed on https://dmaster.demo.civicrm.org.
Reproduction steps
----------------------------------------
1. Go to https://dmaster.demo.civicrm.org
1. Click **Events** | **Manage Events**.
1. **Rain-forest Cup Youth Soccer Tournament**: Click **Configure** | **Online Registration**.
1. Click **Save**
Current behaviour
----------------------------------------
About 50% of the time it will hang as described above.
Oddly, the failures do not seem to be randomly distributed. It seems that it will tend to fail consistently for several minutes, and then to save successfully for several minutes. Or maybe I'm doing something subtly different that I have not noticed which makes the problem appear.
Problem has been observed upon saving various tabs (Info and Settings, Event Location, Online Registration)
Problem has not been observed upon clicking __Save and Done__.
__Stress Testing__
After a page load and successful save, it appears that every subsequent save will be successful. To stress test, therefore, the following sequence must be repeated:
1. Refresh page
1. Save
__GET/POST Requests__
When saving the Online Registration tab, the following sequence is observed in the browser's Network tab for a successful save:
1. POST https://dmaster.demo.civicrm.org/civicrm/event/manage/registration?action=update&id=3&component=event&qfKey=CRMEventFormManageEventRegistrationlkkd3ym1nsgogsggw8scokgo8kw04w0cw00wsg8csgwkc4gcw_4298&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer
1. GET https://dmaster.demo.civicrm.org/civicrm/event/manage/registration?reset=1&action=update&id=3&component=event&qfKey=CRMEventFormManageEventRegistrationlkkd3ym1nsgogsggw8scokgo8kw04w0cw00wsg8csgwkc4gcw_4298&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer
1. GET https://dmaster.demo.civicrm.org/civicrm/ajax/inline?class_name=CRM_UF_Form_Inline_PreviewById&id=12&snippet=json&crmAngularModules=crmApp,crmProfileUtils,crmResource,crmUi,crmUtil,ngRoute,ngSanitize,volunteer
When the save fails, however, only the third request is observed, and it throws a 500 error.
__Civi Log__
```
Aug 20 14:07:08 [error]
$Fatal Error Details = array:3 [
"message" => "We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.<br /><br />Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.<br /><br />Error type: Could not find a valid session key."
"code" => null
"exception" => CRM_Core_Exception {#1777
-errorData: array:1 [
"error_code" => 0
]
#cause: null
-_trace: null
#message: "We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.<br /><br />Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.<br /><br />Error type: Could not find a valid session key."
#code: 0
#file: "/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php"
#line: 861
trace: {
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:861 {
› $msg = ts("We can't load the requested web page. This page requires cookies to be enabled in your browser settings. Please check this setting and enable cookies (if they are not enabled). Then try again. If this error persists, contact the site administrator for assistance.") . '<br /><br />' . ts('Site Administrators: This error may indicate that users are accessing this page using a domain or URL other than the configured Base URL. EXAMPLE: Base URL is http://example.org, but some users are accessing the page via http://www.example.org or a domain alias like http://myotherexample.org.') . '<br /><br />' . ts('Error type: Could not find a valid session key.');
› throw new CRM_Core_Exception($msg);
› }
}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:856 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:316 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller.php:193 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Controller/Simple.php:49 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Utils/Wrapper.php:62 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Page/AJAX.php:63 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php:285 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php:69 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php:36 { …}
/var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/drupal/civicrm.module:471 { …}
/var/www/clients/client1/web3/web/drupal/includes/menu.inc:527 { …}
/var/www/clients/client1/web3/web/drupal/index.php:197 { …}
}
}
]
Aug 20 14:07:08 [debug] $backTrace = #0 /var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Error.php(441): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/CRM/Core/Invoke.php(39): CRM_Core_Error::handleUnhandledException(Object(CRM_Core_Exception))
#2 /var/www/clients/client1/web3/web/drupal/sites/all/modules/civicrm/drupal/civicrm.module(471): CRM_Core_Invoke::invoke((Array:3))
#3 /var/www/clients/client1/web3/web/drupal/includes/menu.inc(527): civicrm_invoke("ajax", "inline")
#4 /var/www/clients/client1/web3/web/drupal/index.php(197): menu_execute_active_handler()
#5 {main}
```
__Possibly Related__
Upon loading any of the event tabs, the following error is always displayed in the console:
```
angular.js:15697 TypeError: Cannot read properties of undefined (reading 'id')
at angular-modules.c3b61f5213a83292c272ff9114e7a62a.js?rgx685:2098:66
at m.$digest (angular.js:19270:23)
at angular.js:19562:15
at Yg.completeTask (angular.js:21403:7)
at angular.js:6879:7
```
Expected behaviour
----------------------------------------
The Save should complete without error.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _Chrome 104.0.5112.101_
* __CiviCRM:__ _5.52_, _5.54.alpha1_
* __PHP:__ _7.4_
* __CMS:__ _Drupal 7.91_
* __Database:__ _MariaDB 10.4_
* __Web Server:__ _Apache 2.4_
Comments
----------------------------------------
_Anything else you would like the reviewer to note._seamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/3813Performance issue on Redis with arrays2022-11-15T04:01:55ZeileenPerformance issue on Redis with arraysBoth WMF and AUG have achieved a significant performance improvement with [this patch](https://github.com/civicrm/civicrm-core/pull/24156) which removes unnecessary calls to the underlying cache provider. However, in both cases we are us...Both WMF and AUG have achieved a significant performance improvement with [this patch](https://github.com/civicrm/civicrm-core/pull/24156) which removes unnecessary calls to the underlying cache provider. However, in both cases we are using `Redis` and `Redis` itself is performing well.
I'm left with the conclusion that the issue is the way in which the php layer is handling arrays when using `redis`.
On digging we are specifically calling [`serialize`](https://github.com/civicrm/civicrm-core/blob/53a48c5aacea52dd6507a45e111a00ccbed0fdaa/CRM/Utils/Cache/Redis.php#L116) and [`unserialize`](https://github.com/civicrm/civicrm-core/blob/53a48c5aacea52dd6507a45e111a00ccbed0fdaa/CRM/Utils/Cache/Redis.php#L138) in `CRM_Utils_Cache_Redis`.
Discussion on the interweb thingee suggests that `json_encode` & `json_decode` *may* be faster but that is a moving target over php versions .... https://stackoverflow.com/questions/22718903/storing-an-array-of-data-using-redis-from-laravel
Another option appears to be the `h` methods within the `Redis php` class - I note in our `PrevNextRedisCache` implementation we do use some of these hash specific methods and I wonder if using them here would help
https://stackoverflow.com/questions/22718903/storing-an-array-of-data-using-redis-from-laravelhttps://lab.civicrm.org/dev/core/-/issues/3809[ext:message_admin] It's not obvious there's a preview function, and even whe...2024-03-28T00:43:21ZDaveD[ext:message_admin] It's not obvious there's a preview function, and even when told you might have trouble finding itIt should be a button on the message template edit page, down with the other buttons, where preview buttons normally are.It should be a button on the message template edit page, down with the other buttons, where preview buttons normally are.https://lab.civicrm.org/dev/core/-/issues/3808[ext:message_admin] Available preview choices should be based on the site's c...2024-03-26T18:25:12ZDaveD[ext:message_admin] Available preview choices should be based on the site's configIt has fixed choices. And formatting like the comma/dot separators it uses may or may not make sense for the selected preview.It has fixed choices. And formatting like the comma/dot separators it uses may or may not make sense for the selected preview.