Development issueshttps://lab.civicrm.org/groups/dev/-/issues2022-04-22T15:52:59Zhttps://lab.civicrm.org/dev/core/-/issues/3258CiviReports issue with boolean fields2022-04-22T15:52:59ZMWestergaardCiviReports issue with boolean fieldsI recently noticed that boolean fields (Deceased, Do Not Email, Do Not SMS, etc) are all displayed as "Yes" on contact and membership reports, regardless of their actual values. The data itself is correct and is fine elsewhere, such as i...I recently noticed that boolean fields (Deceased, Do Not Email, Do Not SMS, etc) are all displayed as "Yes" on contact and membership reports, regardless of their actual values. The data itself is correct and is fine elsewhere, such as in search results and contact displays. The issue seems to be only on report display. The report filters work correctly on those fields, only the display is wrong. I checked my production system (CiviHosting), my dev system (personal Linux box), and the public WordPress demo system at wpmaster.demo.civicrm.org and I see the same thing. I tried the Drupal demo system at CiviHosting and did not see this issue. Perhaps it's an issue under WordPress. I was advised on StackExchange to submit as an issue.
My system is CiviCRM 5.2.2, WordPress 4.9.6, PHP 7.1.16, MySQL 5.7.21.5.3.0https://lab.civicrm.org/dev/core/-/issues/3259Add the possibility to use the hook_civicrm_alterReportVar('sql') in all reports2022-04-22T15:53:00ZrubofvilAdd the possibility to use the hook_civicrm_alterReportVar('sql') in all reportsSome reports have rewrited the method "buildQuery" from class CRM_Report_Form. Not in all reports is included the line to launch the hook `hook_civicrm_alterReportVar` -> `CRM_Utils_Hook::alterReportVar('sql', $this, $this);`
Example
h...Some reports have rewrited the method "buildQuery" from class CRM_Report_Form. Not in all reports is included the line to launch the hook `hook_civicrm_alterReportVar` -> `CRM_Utils_Hook::alterReportVar('sql', $this, $this);`
Example
https://github.com/civicrm/civicrm-core/blob/abc4afe890add20796585a2a997bc3d832815abe/CRM/Report/Form/Member/ContributionDetail.php#L552https://lab.civicrm.org/dev/core/-/issues/3260Icon in status message after saving a civireport is misleading2022-04-22T15:53:01ZDaveDIcon in status message after saving a civireport is misleadingIt's the 'alert' icon instead of success.
1. Go to any CiviReport.
1. From the Actions dropdown choose Save.
1. Icon indicates a warning but it's all good.
Also for "save a copy".It's the 'alert' icon instead of success.
1. Go to any CiviReport.
1. From the Actions dropdown choose Save.
1. Icon indicates a warning but it's all good.
Also for "save a copy".5.29.0https://lab.civicrm.org/dev/core/-/issues/3261Membership Detail Report fails when ACLs are enabled2022-04-22T15:53:02ZJonGoldMembership Detail Report fails when ACLs are enabledDuring a tidy-up on the Membership Detail Report, the `from()` method was modified to use `CRM_Report_Form::setFromBase()`, which concatenates `$this->aclFrom`. However, concatenating `$this->aclFrom` was never removed from the child cl...During a tidy-up on the Membership Detail Report, the `from()` method was modified to use `CRM_Report_Form::setFromBase()`, which concatenates `$this->aclFrom`. However, concatenating `$this->aclFrom` was never removed from the child class, so it's added to the SQL statement twice.5.3.0JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3262Search Kit and Form Builder, add ability to use the afform component in WordP...2022-04-22T15:53:04Zjustinfreeman (Agileware)Search Kit and Form Builder, add ability to use the afform component in WordPress with the CiviCRM shortcodeWhen using the Search Kit and Form Builder to implement a front-end Search Listing on WordPress, would be really useful to be able to embed the afform in a WordPress page using a short-code. This would be really beneficial:
1. Have the a...When using the Search Kit and Form Builder to implement a front-end Search Listing on WordPress, would be really useful to be able to embed the afform in a WordPress page using a short-code. This would be really beneficial:
1. Have the ability to use any URL to display the form, instead of just /civicrm/afform-name
2. Better design control
3. Control of the content before / after the form
I did try to inserting the CiviCRM shortcode, just in case it works: `[civicrm component="afsearchMemberDirectory"]`
And get this error: **Do not know how to handle this shortcode.**
Which is a bummer.
Agileware Ref: CIVICRM-1895https://lab.civicrm.org/dev/core/-/issues/3263Only in WORDPRESS: REST.php ---> Unable to decode supplied JSON2022-04-22T15:53:06ZRTaglientoOnly in WORDPRESS: REST.php ---> Unable to decode supplied JSONThis is my first post for CiviCRM.
Hi to all.
I have seen others post related and nowadays the problem remain.
Different from this post:
https://civicrm.stackexchange.com/questions/6585/rest-api-unable-to-decode-supplied-json
- with...This is my first post for CiviCRM.
Hi to all.
I have seen others post related and nowadays the problem remain.
Different from this post:
https://civicrm.stackexchange.com/questions/6585/rest-api-unable-to-decode-supplied-json
- with wordpress I got always the same response "**Unable to decode supplied JSON**".
I tried clear installation with Joomla and also Backdrop, and both works correctly.
I am not a good programmer but I tried to do some stuff myself, here you what I prepared on my little server:
WordPress+CiviCRM with the problem: [link clear installation](http://144.91.105.68/cms/wordpress/wp-content/plugins/civicrm/civicrm/extern/rest.php?entity=Contact&action=get&api_key=a42bfc2b33cf13184711bff14fe1fb19&key=a42bfc2b33cf13184711bff14fe1fb19&json={%22sequential%22:1})
WordPress+CiviCRM fixed myself: [custom](http://144.91.105.68/cms/cust/wordpress/wp-content/plugins/civicrm/civicrm/extern/rest.php?entity=Contact&action=get&api_key=a42bfc2b33cf13184711bff14fe1fb19&key=a42bfc2b33cf13184711bff14fe1fb19&json={%22sequential%22:1})
Here you what I modified to the file REST.php...
after line 326
```php
if (array_key_exists('json', $requestParams) && $requestParams['json'][0] == "{") {
$params = json_decode($requestParams['json'], TRUE);
```
my modification start here
```php
$new_requestParams = $requestParams['json'];
$new_requestParams = str_replace("\\", "", $new_requestParams);
$new_requestParams = str_replace("\\", "", $new_requestParams);
$params = json_decode($new_requestParams, TRUE);
```
I practically saw a lot of "\\" before the JSON DECODE.
So removing this "\\" now works regularly.
If you wish can check directly on both sites:
Clear ----> http://144.91.105.68/cms/wordpress
Custom ---> http://144.91.105.68/cms/cust/wordpress
The user and password are both same: **admin** / **admin**
And also siteKey and api_key are always: **a42bfc2b33cf13184711bff14fe1fb19**
Regards Roberto T.
https://lab.civicrm.org/dev/core/-/issues/3264Transaction Date filter in Bookkeeping Transactions report2022-04-22T15:53:09ZMWestergaardTransaction Date filter in Bookkeeping Transactions reportIn the Bookkeeping Transactions report, the filter for Transaction Date does not include time values and leads to missing data. For example, if today is February 10 and I filter Transaction Date on "Yesterday", the WHERE clause looks li...In the Bookkeeping Transactions report, the filter for Transaction Date does not include time values and leads to missing data. For example, if today is February 10 and I filter Transaction Date on "Yesterday", the WHERE clause looks like:
> ( financial_trxn.trxn_date >= '20190209') AND ( financial_trxn.trxn_date <= '20190209' )
Only transactions that occurred exactly at midnight are included. Contrast that with the filter for Date Received:
> ( contribution.receive_date >= '20190209000000') AND ( contribution.receive_date <= '20190209235959' )
I verified on dmaster and wpmaster.5.12.0https://lab.civicrm.org/dev/core/-/issues/3265APIv4-Based Reports2022-04-22T15:53:11Zjoshjosh@civicrm.orgAPIv4-Based ReportsThis project will allow report templates to be built with the Search-Builder UI. Presently, report templates must be implemented via PHP code. Prerequisite is the Search-Builder UI report#32
Tasks:
- Integrate report schema with Form B...This project will allow report templates to be built with the Search-Builder UI. Presently, report templates must be implemented via PHP code. Prerequisite is the Search-Builder UI report#32
Tasks:
- Integrate report schema with Form Builder UI
- Re-implement report system with APIv4 backend
- Implement API for creating report templates
- Bridge legacy report templates with the new systemcolemanwcolemanwhttps://lab.civicrm.org/dev/core/-/issues/3266Form-Builder-Based Custom Searches2022-04-22T15:53:12Zjoshjosh@civicrm.orgForm-Builder-Based Custom SearchesThis project will allow user-friendly search screens to be built with the Search-Builder UI. Presently, custom searches must be implemented via PHP code. Prerequisite is the Search Builder UI report#32
Tasks:
- Integrate custom search s...This project will allow user-friendly search screens to be built with the Search-Builder UI. Presently, custom searches must be implemented via PHP code. Prerequisite is the Search Builder UI report#32
Tasks:
- Integrate custom search schema with Form Builder UI
- Implement API for creating custom searches
- Bridge legacy custom searches with the new systemcolemanwcolemanwhttps://lab.civicrm.org/dev/core/-/issues/3267Case Detail 'Active Role?' filter excludes cases without relationships2022-04-22T15:53:13Zaydunsaidan.saunders@squiffle.ukCase Detail 'Active Role?' filter excludes cases without relationshipsThe 'Active Role?' filter in the Case Detail report checks the status of relationships.
The current implementation of the filter as a simple boolean requires relationships to exist, and therefore excludes cases without relationships whe...The 'Active Role?' filter in the Case Detail report checks the status of relationships.
The current implementation of the filter as a simple boolean requires relationships to exist, and therefore excludes cases without relationships whether the setting is Yes or No.
Problem also reported here: https://civicrm.stackexchange.com/questions/28836/case-details-report-active-case-role-filter5.22.0aydunsaidan.saunders@squiffle.ukaydunsaidan.saunders@squiffle.ukhttps://lab.civicrm.org/dev/core/-/issues/3268Deprecate `getBasicContactFields` in favor of `getColumns('Contact')`2022-04-22T15:53:15ZJonGoldDeprecate `getBasicContactFields` in favor of `getColumns('Contact')`@eileen has ported some of the cleaner code for defining report specs from Extended Reports to core in the form of `getColumns()` and `getContactColumns()`. Since this appears to be the direction we're headed in, I think it makes sense ...@eileen has ported some of the cleaner code for defining report specs from Extended Reports to core in the form of `getColumns()` and `getContactColumns()`. Since this appears to be the direction we're headed in, I think it makes sense to deprecate `getBasicContactColumns()`, which does a similar job. However, `getBasicContactColumns()` adds a bunch of fields to core reports that `getContactColumns` doesn't.
My PR adds all the missing fields to `getContactColumns` (except for "Organization Name"; this seems unnecessary since it will virtually always match the display name). I also mark `getBasicContactFields()` as deprecated so future cleanup can target reports using it.5.12.0JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3269Allow new custom field entities to be added to reports2022-04-22T15:53:16ZJonGoldAllow new custom field entities to be added to reportsNow that we allow custom fields on almost any entity, it makes sense that we should allow CiviReport to make use of these. E.g. contribution reports should allow the use of FinancialType custom fields; relationship reports should allow ...Now that we allow custom fields on almost any entity, it makes sense that we should allow CiviReport to make use of these. E.g. contribution reports should allow the use of FinancialType custom fields; relationship reports should allow RelationshipType custom fields, etc.
Ideally, this should be possible by adding the entity to the report's `$_customGroupExtends` array. However, this fails because [CRM_Report_Form::customDataFrom()](https://github.com/civicrm/civicrm-core/blob/master/CRM/Report/Form.php#L4001) calls [CRM_Core_BAO_CustomQuery::$extendsMap](https://github.com/civicrm/civicrm-core/blob/3291b804470947bc3e3d1f580ffbd057d26e7898/CRM/Core/BAO/CustomQuery.php#L110-L129), a hard-coded list of tables which various entities extend.
I propose replacing calls to `CRM_Core_BAO_CustomQuery::$extendsMap` with a new method `CRM_Core_BAO_CustomQuery::getExtendsMap()`, which will also pull in any entities defined in `civicrm_option_value`.
I'm happy to do this work, I would like concept approval first though.https://lab.civicrm.org/dev/core/-/issues/3270getAddressColumns doesn't fully support prefix2022-04-22T15:53:18ZJonGoldgetAddressColumns doesn't fully support prefixThere's one place where the table name is hard coded that should be `$tableAlias`. I wrote a fix + UT.
I also realized that the way odd/even street numbers are handled causes the address table to always get joined regardless of whether...There's one place where the table name is hard coded that should be `$tableAlias`. I wrote a fix + UT.
I also realized that the way odd/even street numbers are handled causes the address table to always get joined regardless of whether it's necessary, but that's out of scope for now.5.14.0JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3271Regression: Can't use operators to filter contact subtypes other than "Is One...2022-04-22T15:53:19ZJonGoldRegression: Can't use operators to filter contact subtypes other than "Is One Of"This is introduced by core#544, which introduces a hard-coded `LIKE` where the code should allow for different operators.
[A commit based on 5.14 is here](https://github.com/MegaphoneJon/civicrm-core/commit/625b0b06e64d02883614e31bb4ef7...This is introduced by core#544, which introduces a hard-coded `LIKE` where the code should allow for different operators.
[A commit based on 5.14 is here](https://github.com/MegaphoneJon/civicrm-core/commit/625b0b06e64d02883614e31bb4ef79825a1b8ad2#diff-d355cdb00cea3915a3cf306c6c08f6a6L2131) and hopefully I (or someone) will have time to merge and add tests in the near future.5.16.0https://lab.civicrm.org/dev/core/-/issues/3272Postal Code Suffix returns Postal Code value2022-04-22T15:53:21ZlcdwebPostal Code Suffix returns Postal Code valueWhen the postal code suffix column is displayed, it returns the postal code value instead of the suffix.When the postal code suffix column is displayed, it returns the postal code value instead of the suffix.lcdweblcdwebhttps://lab.civicrm.org/dev/core/-/issues/3273Performance issue when exporting to CSV from reports with custom fields2022-04-22T15:53:23ZJonGoldPerformance issue when exporting to CSV from reports with custom fields### User Story
As an end user, I want to run a report of my organization's alumni. This is about 16,000 contacts, and my report contains 6 custom fields. However, this report takes about 8.5 minutes to generate, far longer than the 300...### User Story
As an end user, I want to run a report of my organization's alumni. This is about 16,000 contacts, and my report contains 6 custom fields. However, this report takes about 8.5 minutes to generate, far longer than the 300 second timeout on processes.
### Profiling
I stepped through the code until I saw that `CRM_Core_BAO_CustomField::displayValue()` was the culprit, which is called ~96,000 times (# of rows * custom fields). I used a profiler and it reported that 97% of the time was spent in the `CRM_Core_BAO_CustomField::getOptions()` method.
### Improvements
* `CRM_Core_BAO_CustomField::getOptions()` is called on every custom field - even those that don't use it (e.g. text or money fields). By moving it so it's only called when needed, I immediately got a speedup.
* `getOptions()` should only be called 6 times in my example, not 96,000 times. I'm going to put a cache into `getOptions()`.5.21.0JonGoldJonGoldhttps://lab.civicrm.org/dev/core/-/issues/3274Filter contact sub-type by "is not one of" fails2022-04-22T15:53:25Zaydunsaidan.saunders@squiffle.ukFilter contact sub-type by "is not one of" failsWhen filtering by contact sub-type, "is not one of" excludes all those that do not have a sub-type.
Noted previously here: https://lab.civicrm.org/dev/report/issues/15#note_25491
For example, filtering by "is not one of" 'Student' shou...When filtering by contact sub-type, "is not one of" excludes all those that do not have a sub-type.
Noted previously here: https://lab.civicrm.org/dev/report/issues/15#note_25491
For example, filtering by "is not one of" 'Student' should just exclude those who do have a sub-type of Student. Currently, it also excludes all those who have a sub-type of Student, and also all those with no sub-type.5.21.0aydunsaidan.saunders@squiffle.ukaydunsaidan.saunders@squiffle.ukhttps://lab.civicrm.org/dev/core/-/issues/3275A mental health plan for search2022-04-22T15:53:29ZeileenA mental health plan for searchI've been thinking a lot about how to move forwards with search functionality of late. When I think about the sorts of bugs and regressions we were dealing with 2 years ago versus today there are a lot of areas of the code that were cons...I've been thinking a lot about how to move forwards with search functionality of late. When I think about the sorts of bugs and regressions we were dealing with 2 years ago versus today there are a lot of areas of the code that were constantly breaking or requiring changes 2 years ago that we now rarely touch. We have added barrage of unit tests & retrofitted sanity into the code & eventually reached the end of the line on the whackamole in a lot of areas. Not so much searches & reports....
Part of the problem is that we were late to the party getting searches and reports under unit testing - or at least some of the more obscure search functionality. But much more of the problem is the poor underlying code structure for searches and the in-built reports. In addition search builder and advanced search expose much more functionality than actually works well - creating a back door for new tricky functionality. Compounding this, the original search code had a lot of security holes in it which have sent us into various cycles of whackamole.
When we talk about search people want to
- be able to retrieve data
- take actions on that data
- create groups based on the data
In many cases they can use a report or some custom form to retrieve a group of contacts but then they can't take the actions they want on those contacts. In particular it's currently impossible to create a smart group or utilise our bank of actions without working off the search framework. The problem with getting away from our quickform search structure is that we have a lot of assets tied up with it and so creating a non-quickform search quickly runs into the fact that various actions cannot be accessed and the lift involved in getting past that is too much - so we wind up with 'yet another custom search' or another change to core to bad code.
I don't think we can break the situation where we are spending a tonne of sunk time on search functionality in the short-term. I think adding thises & thats to search & custom searches and reports will continue at least for another year. The [data processor extension](https://lab.civicrm.org/extensions/dataprocessor/tree/master) by @jaapjansma does a pretty good of retrofitting sanity onto the quickform search structure, but I think we should set a goal that in a year's time quickform search structure and BAO_Query object are out of active development and there is a viable alternative.
We already have a pretty good alternative for retrieving the data - apiv4. When you look at apiv4 explorer if 'feels like' all you need to turn that into a replacement for search builder is a 'create smart group function'
Many of the things that work badly in search builder already work well in the apiv4 explorer but it is missing the ability to leverage our 'action-bank'. Going down the apiv4-as-smart-groups path also sets us up to offer the ability to have custom apiv4 apis that can also create smart groups and leverage actions.
That was a pretty long intro but the point of this ticket was more to spell out a path that I think allows us to start chipping away at this and to divert some of our 'product maintenance effort' towards breaking the cycle. These are the steps I think need taking (not necessarily in this order). Any of the steps, but in particular 3 & 4 would be suitable as funded CT work & that would speed them up & this whole process considerably, but I think there are things we can be doing now towards the end result.
1) create the ability to programaticially create a smart group based on an apiv4 explorer 'get' action. I think this is actually not that tricky. We would need to define the format of the array saved to 'form_values' - something like ['module' => 'apiv4', 'entity' => 'x', 'where' => [], 'select' => [], 'orderBy => []]. Perhaps ->createSavedSearch() would be an action that can be taken from apiv4? We would need to hobble UI functionality related to the smart group that might not work - ie suppress actions that don't work & the 'edit smart group search criteria' link in the group settings page - we can be a bit clunky on this in this step since it would not be exposed unless a developer exposed it. I feel like this step is pretty manageable & could be done in the next couple of releases. Since I think we should start on this I have created https://lab.civicrm.org/dev/core/issues/1107 for specific discussion on this issue
2) consider extending the above to cover other api actions - we talked about the idea of it being possible to create a smart group off a report. This is mostly a conversation about what sort of api an extension needs to create for it to be available for a saved search & what metadata is required. It would also involved creating a protocol to configure a url for 'edit smart group search criteria'. This could be done in conjunction with making smart groups possible from extended reports via an apiv4 action as experimental functionality. I don't see this as ever being a feature in core reports as I hope that by the time we've matured all of this we'll be figuring out how to leave them behind, but it seems like a way of opening up the transition process to leaping by extension.
3) tackle the search actions - figure out how to make them less tied to the QF architecture they sit on - could they be apis? Or just a much more sensible php interface? This is probably the deeper dive of this work and it might be a big lift but I also feel it's the sort of thing that we routinely chip away at in the product maintenance sphere & we could do that here. I suspect @mattwire has more idea than I do of that code at the moment...
4) Copy & adapt (or extract & componentise) the api v4 explorer to be search oriented. Mostly tie the search part of the interface with an action selector. Alternatively look at something like https://querybuilder.js.org/ - I feel like this piece is something CT should do as a dedicated chunk of work & which could be done at any point in the process by soliciting funding.
@totten @mattwire @colemanw @seamuslee @jitendra @monish.deb @pradeep @yashodha @bgm @lcdweb @johntwyman @mfb @pfigel @DaveD @justinfreemanhttps://lab.civicrm.org/dev/core/-/issues/3276Unreleased regression - fee levels incorrectly show sold out (in code that wi...2022-04-22T15:53:33ZeileenUnreleased regression - fee levels incorrectly show sold out (in code that will be 5.16)Fees are incorrectly showing as sold out, blocking change fee selection.
This is an unexpected consequence of
https://github.com/civicrm/civicrm-core/pull/14244
The path is that because $this->_id is now set it gets assigned to the fo...Fees are incorrectly showing as sold out, blocking change fee selection.
This is an unexpected consequence of
https://github.com/civicrm/civicrm-core/pull/14244
The path is that because $this->_id is now set it gets assigned to the form
https://github.com/civicrm/civicrm-core/blob/5774b54f47445de233daa85672ce4f793dff347a/CRM/Event/Form/Participant.php#L265
Which then gets passed to the call to load the fee block
https://github.com/civicrm/civicrm-core/blob/c4145dedecb1f3157ecf8fd85421f562e8128e73/templates/CRM/Event/Form/Participant.tpl#L411
which results in $_pid being set & as a result online being set
https://github.com/civicrm/civicrm-core/blob/90b461f1623e75e94e0f472a3c6bf23e01defbc1/CRM/Event/Form/EventFees.php#L350
which leads to the element being frozen
https://github.com/civicrm/civicrm-core/blob/90b461f1623e75e94e0f472a3c6bf23e01defbc1/CRM/Event/Form/EventFees.php#L393
which is interpretted as 'sold out'
https://github.com/civicrm/civicrm-core/blob/6b83d5bdd0f2ca546924feae6aa42aeddb1d40cf/templates/CRM/Price/Form/PriceSet.tpl#L93
This is an example of a code antipattern which is too prevalent in our codebase - ie. hanging various assumptions off a parameter.
I *think* the right answer is to assign a variable of online & then pass that through & make appropriate decisions based on that.
The first question is - do we revert https://github.com/civicrm/civicrm-core/pull/14244 out of the rc and then re-commit into master to give us more time given how awful this code path is5.16.0https://lab.civicrm.org/dev/core/-/issues/3277When Merging Contacts Related Tags are Missing2022-04-22T15:53:35ZmenachemWhen Merging Contacts Related Tags are Missing