CiviCRM Core issueshttps://lab.civicrm.org/dev/core/-/issues2023-10-13T21:11:08Zhttps://lab.civicrm.org/dev/core/-/issues/2486Apiv4 entity parity2023-10-13T21:11:08ZeileenApiv4 entity parity## Here are the entities in APIv3 & not v4
_In some cases by design - others we should maybe add..._
# Prioritised
- [ ] Attachment.php - `get` [has been added to support viewing attachments in SearchKit](https://github.com/civicrm/civ...## Here are the entities in APIv3 & not v4
_In some cases by design - others we should maybe add..._
# Prioritised
- [ ] Attachment.php - `get` [has been added to support viewing attachments in SearchKit](https://github.com/civicrm/civicrm-core/pull/27379) but `create` is still missing
- [ ] Extension.php - various install, uninstall etc functions (support for some actions already implemented)
# Partially done - lets resolve
- [ ] MembershipLog.php https://github.com/civicrm/civicrm-core/pull/21106
# Standard crud - note that doing these involves checking all pseudoconstants are correctly defined
- [ ] SmsProvider.php
- [ ] Premium.php
- [ ] RecurringEntity.php
# Mailing related - in scope
- [ ] MailingAB.php
- [ ] MailingComponent.php
- [ ] MailingContact.php
- [ ] MailingEventResubscribe.php
- [ ] MailingRecipients.php
# From here on down we are out of scope for 'entity parity'
## Order related - needed when we get to afform payment forms - out of scope for entity parity
- [ ] Order.php
- [ ] Payment.php
## Varied utils - out of scope for 'entity parity' - ad hoc
- [ ] SystemLog.php
- [ ] User.php
- [ ] Logging.php
## Migration doubtful - to be done ad hoc if we decide to
- [ ] ReportTemplate.php
- [ ] ParticipantPayment.php - likely should not be brought over
- [ ] CxnApp.php
- [ ] Cxn.php
- [ ] Profile.php
## Do not migrate
- ~~ActivityType.php (by design)~~
- ~~CustomSearch.php (by design)~~
- ~~Constant.php (by design)~~
- ~~SurveyRespondant.php(by design)~~
- ~~MembershipPayment.php (by design - use Line Item)~~
## Done
- [x] ~~AclRole.php~~ ACLEntityRole https://github.com/civicrm/civicrm-core/pull/20474
- [x] Batch.php https://github.com/civicrm/civicrm-core/pull/19931
- [x] CaseContact.php https://github.com/civicrm/civicrm-core/pull/19907
- [x] Case.php https://github.com/civicrm/civicrm-core/pull/19907
- [x] CaseType.php
- [x] ContributionProduct.php https://lab.civicrm.org/dev/core/-/issues/2683
- [x] Dedupe.php
- [x] EntityBatch.php https://lab.civicrm.org/dev/core/-/issues/2682
- [x] EntityFinancialAccount.php - https://github.com/civicrm/civicrm-core/pull/19927
- [x] EntityFinancialTrxn.php - https://github.com/civicrm/civicrm-core/pull/19932
- [x] ~~Exception.php~~ DedupeException.php https://github.com/civicrm/civicrm-core/pull/20466
- [x] File.php (part of attachment) https://github.com/civicrm/civicrm-core/pull/21158
- [x] FinancialItem.php - https://github.com/civicrm/civicrm-core/pull/20433
- [x] Job.php (crud portion)
- [x] Job.php
- [x] JobLog.php
- [x] Mailing.php - there are a bunch of related entities - see further down)
- [x] MailingEventConfirm.php
- [x] MailingEventQueue.php
- [x] MailingEventSubscribe.php
- [x] MailingEventUnsubscribe.php
- [x] MailingGroup.php
- [x] MailingJob.php
- [x] Membership.php - see https://lab.civicrm.org/dev/core/-/issues/2634 https://github.com/civicrm/civicrm-core/pull/21106
- [x] MembershipBlock.php https://github.com/civicrm/civicrm-core/pull/21106
- [x] MembershipStatus.php https://github.com/civicrm/civicrm-core/pull/21106
- [x] ParticipantStatusType.php
- [x] PaymentToken.php
- [x] Pcp.php
- [x] PrintLabel.php https://github.com/civicrm/civicrm-core/pull/21500
- [x] Product.php - since 5.41
- [x] ReportInstance.php
- [x] ~~Rule.php~~ DedupeRule.php https://github.com/civicrm/civicrm-core/pull/20466
- [x] ~~RuleGroup.php~~ DedupeRuleGroup.php https://github.com/civicrm/civicrm-core/pull/20466
- [x] Survey.php https://github.com/civicrm/civicrm-core/pull/21355
- [x] WordReplacement.php https://github.com/civicrm/civicrm-core/pull/20559https://lab.civicrm.org/dev/core/-/issues/2382Allow disabling of confirmation page for paid events2022-10-16T01:24:38ZlarsssandergreenAllow disabling of confirmation page for paid eventsCurrently, a paid event must have a confirmation page. In simple paid event registration cases, it would be sometimes be preferable to skip the confirmation page (and potential drop-off of registrants or confusion from those who don't re...Currently, a paid event must have a confirmation page. In simple paid event registration cases, it would be sometimes be preferable to skip the confirmation page (and potential drop-off of registrants or confusion from those who don't realize that the confirmation page wasn't the end of the process) and simply register the participant directly.
Note that you can skip the confirmation page on a contribution page and this is a very similar situation (people are making a selection and filling in their payment details). We find our supporters are likely to be confused by multiple, verbose pages and the confirmation page definitely adds complexity that isn't needed in some cases. Are there any downsides to giving users the option to set up events without a confirmation page?https://lab.civicrm.org/dev/core/-/issues/2396Translation support for Afforms & Managed Search Displays2024-02-07T17:45:35ZseamusleeTranslation support for Afforms & Managed Search DisplaysAs per https://lab.civicrm.org/extensions/afform/-/issues/22 and https://lab.civicrm.org/extensions/afform/-/issues/8 looks like there needs to be some improvements to support multilingual sites in AfformAs per https://lab.civicrm.org/extensions/afform/-/issues/22 and https://lab.civicrm.org/extensions/afform/-/issues/8 looks like there needs to be some improvements to support multilingual sites in Afformhttps://lab.civicrm.org/dev/core/-/issues/2410Image custom field type2023-01-30T20:19:36ZMichael McAndrewImage custom field typeOverview
----------------------------------------
We want to add a custom field type of image that can be used to store images as part of a CiviCRM entity.
In terms of how this might be different to the file field type, I think the main...Overview
----------------------------------------
We want to add a custom field type of image that can be used to store images as part of a CiviCRM entity.
In terms of how this might be different to the file field type, I think the main consideration is how these fields would be displayed (resizing, thumbnails, etc.)
Example use-case
----------------------------------------
1. Store (one or more) images related to a contact as part of the contact record.
2. *** feel free to add more***
Current behaviour
----------------------------------------
At the moment, there is a single core field for a contact image (e.g. a headshot of an indiviual or a company logo). No other images can be added. Images can be added as files but are not displayed as one would expect an image to be displayed.
Proposed behaviour
----------------------------------------
People should be able to define image fields and upload images to them. The original should be retained and resized images should also be created (e.g. for thumbnails).
(Wireframes, mockups and more thinking required.)
Comments
----------------------------------------
Some relevant content (previous workarounds, etc.):
- https://drupal.stackexchange.com/questions/220599/how-to-add-image-field-in-civicrm
- https://civicrm.stackexchange.com/questions/15900/how-to-add-image-field-in-civicrmhttps://lab.civicrm.org/dev/core/-/issues/2414Deductible value treated as non-deductible if a non-deductible selection made...2023-11-23T07:09:32Zfreeform.stephDeductible value treated as non-deductible if a non-deductible selection made in the same contributionOriginally reported here: https://civicrm.stackexchange.com/questions/35284/non-deductible-value-incorrect-if-you-choose-a-nondeductible-membership-type-and
Possibly related to: https://lab.civicrm.org/dev/core/-/issues/1083
If you add...Originally reported here: https://civicrm.stackexchange.com/questions/35284/non-deductible-value-incorrect-if-you-choose-a-nondeductible-membership-type-and
Possibly related to: https://lab.civicrm.org/dev/core/-/issues/1083
If you add a donation field (financial type 'donation' which is deductible) to a membership price set that includes membership types that are nondeductible, and then use that price set for a membership contribution form, the non-deductible value on the contribution is saved as membership fee + donation amount instead of just the membership fee portion of the contribution. This means the non-deductible value can not be trusted.
We use the CDN Tax Receipts extension so to get around the issue we wiped out the nondeductible value from all relevant contributions and let the extension + a custom extension calculate the correct values.
How to replicate:
- add a test organization and test individual
- create a non-deductible financial type and a deductible financial type
- create 2 new membership types for your test organization with a fee and set one to the deductible type, and one to the non-deductible type
- create a price set with 2 fields:
- radio option with the 2 membership types
- radio option with a couple of donation values to choose from, financial type 'donation'
- create a new contribution form, default financial type your custom deductible financial type, and enable your price set on the memberships tab
- now use your new form to create a membership for your test individual and make sure to select the nondeductible membership type & a donation value.
- go to the individual's profile and pull-up the contribution on the contributions tab
To me, it would make sense if the non-deductible value included the donation amount if the contribution status were anything other than 'complete' (pending check for example, because we wouldn't want to issue a tax receipt for monies not receive), but once that status is set to complete then the nondeductible value should respect the financial types of the individual line items.![3X9oo](/uploads/71f07a5b7390c453e28a7ba9f94c54b6/3X9oo.png)seamusleeseamusleehttps://lab.civicrm.org/dev/core/-/issues/2435Save fields button on export screen is not permissioned2023-05-24T06:21:51ZandyburnsSave fields button on export screen is not permissionedI cannot find in the code where the _Save Fields_ button is and therefore what permission is needed to save export mappings.
![image](/uploads/e9be7ed19215554edc7e6397dde46518/image.png)
Currently there is no core permission on who ca...I cannot find in the code where the _Save Fields_ button is and therefore what permission is needed to save export mappings.
![image](/uploads/e9be7ed19215554edc7e6397dde46518/image.png)
Currently there is no core permission on who can export. There is [this extension](https://github.com/progressivetech/net.ourpowerbase.exportpermission/blob/master/README.md) that hides the seach action of export but does not effect the actual export mappings screen.
Currently, non-admins try to save a field mapping and it just spins causing user confusion. What permission is currently needed to save field mappings? I think the logical solution is to get a permission added on saving field mappings button so users that don't have that permission do not see it.https://lab.civicrm.org/dev/core/-/issues/2467[Meta] XLTS Angular JS longer-term-support2022-12-20T18:24:57Zhomotechsual[Meta] XLTS Angular JS longer-term-support@JoeMurray raised in chat that [AngularJS goes EoL in December](https://chat.civicrm.org/civicrm/pl/cm6jqmcex3n39ekf3zzx4tiree) there is a company [XLTS](https://xlts.dev/angularjs) providing extended LTS support through 2026. I've send ...@JoeMurray raised in chat that [AngularJS goes EoL in December](https://chat.civicrm.org/civicrm/pl/cm6jqmcex3n39ekf3zzx4tiree) there is a company [XLTS](https://xlts.dev/angularjs) providing extended LTS support through 2026. I've send them an enquiry asking how much they are likely to look to charge given the open-source nature of CiviCRM to see if, perhaps, they'd be willing to "donate" the XLTS to us.
We lose nothing by asking!JoeMurrayJoeMurrayhttps://lab.civicrm.org/dev/core/-/issues/2469Export of custom multiple choice options to CSV files limited to 32 characters2023-07-18T16:44:17ZbkeevilExport of custom multiple choice options to CSV files limited to 32 charactersA user is trying to export data from an advanced query on custom fields to a CSV file using the "Export contacts" action. This fails with a backtrace which is attached below. The error message is:
```
#6 /var/www/libertarian/vendor/pear...A user is trying to export data from an advanced query on custom fields to a CSV file using the "Export contacts" action. This fails with a backtrace which is attached below. The error message is:
```
#6 /var/www/libertarian/vendor/pear/db/DB/mysqli.php(936): DB_common->raiseError(-1, NULL, NULL, "\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...", "1406 ** Data too long for column 'custom_49' at row 13")
```
The `custom_49` field referred to in the error is a select list with a set of multiple choice options, some of which are up to 55 characters long.
The bug is that the temporary table that is being constructed to service the export to CSV is constructing the column as VARCHAR(32) while the text value of the multiple choice options in civicrm_option_value is a VARCHAR(512)
### Details
If I select the schema of the table `civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7` it reports the table structure as:
```
CREATE TABLE `civicrm_tmp_d_export_c3a74c8fb66b720f55e2661dab58716b` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`last_name` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL,
`first_name` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL,
`nick_name` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,
`street_address` varchar(96) COLLATE utf8_unicode_ci DEFAULT NULL,
`city` varchar(64) COLLATE utf8_unicode_ci DEFAULT NULL,
`email` varchar(254) COLLATE utf8_unicode_ci DEFAULT NULL,
`phone` varchar(32) COLLATE utf8_unicode_ci DEFAULT NULL,
`sort_name` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,
`custom_61` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`custom_50` varchar(32) COLLATE utf8_unicode_ci DEFAULT NULL,
`custom_51` varchar(16) COLLATE utf8_unicode_ci DEFAULT NULL,
`custom_68` varchar(32) COLLATE utf8_unicode_ci DEFAULT NULL,
`custom_58` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`custom_49` varchar(32) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_street_address` (`street_address`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
```
Indeed it is trying to insert the text value of an option into a temporary table and several of those text options are over 32 characters in length.
```
SELECT label,CHAR_LENGTH(label) as length FROM civicrm.civicrm_option_value WHERE (option_group_id=103) AND (CHAR_LENGTH(label) > 32)
```
returns
```
| label | length |
|---------------------------------------------------------|--------|
| 003 - Aurora–Oak Ridges–Richmond Hill | 37 |
| 005 - Barrie–Springwater–Oro-Medonte | 36 |
| 035 - Haliburton–Kawartha Lakes–Brock | 37 |
| 039 - Hamilton West–Ancaster–Dundas | 35 |
| 040 - Hastings–Lennox and Addington | 35 |
| 052 - Leeds–Grenville–Thousand Islands and Rideau Lakes | 55 |
| 061 - Mississauga East–Cooksville | 33 |
| 073 - Northumberland–Peterborough South | 39 |
| 102 - Stormont–Dundas–South Glengarry | 37 |
```
The full backtrace may be relevant:
```
backTrace
#0 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/Error.php(148): CRM_Core_Error::backtrace()
#1 /var/www/libertarian/vendor/pear/pear-core-minimal/src/PEAR.php(944): CRM_Core_Error::handle(Object(DB_Error))
#2 /var/www/libertarian/vendor/pear/db/DB.php(997): PEAR_Error->__construct("DB Error: unknown error", -1, 16, (Array:2), "\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...")
#3 /var/www/libertarian/vendor/pear/pear-core-minimal/src/PEAR.php(575): DB_Error->__construct(-1, 16, (Array:2), "\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...")
#4 /var/www/libertarian/vendor/pear/pear-core-minimal/src/PEAR.php(223): PEAR->_raiseError(Object(DB_mysqli), NULL, -1, 16, (Array:2), "\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...", "DB_Error", TRUE)
#5 /var/www/libertarian/vendor/pear/db/DB/common.php(1928): PEAR->__call("raiseError", (Array:7))
#6 /var/www/libertarian/vendor/pear/db/DB/mysqli.php(936): DB_common->raiseError(-1, NULL, NULL, "\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...", "1406 ** Data too long for column 'custom_49' at row 13")
#7 /var/www/libertarian/vendor/pear/db/DB/mysqli.php(406): DB_mysqli->mysqliRaiseError()
#8 /var/www/libertarian/vendor/pear/db/DB/common.php(1234): DB_mysqli->simpleQuery("\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...")
#9 /var/www/libertarian/vendor/civicrm/civicrm-packages/DB/DataObject.php(2696): DB_common->query("\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...")
#10 /var/www/libertarian/vendor/civicrm/civicrm-packages/DB/DataObject.php(1829): DB_DataObject->_query("\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...")
#11 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/DAO.php(457): DB_DataObject->query("\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...")
#12 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/DAO.php(1563): CRM_Core_DAO->query("\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...", TRUE)
#13 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Export/BAO/Export.php(365): CRM_Core_DAO::executeQuery("\nINSERT INTO civicrm_tmp_d_export_1d387c75256673080ad64346b3c095c7 (`id`, `l...")
#14 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Export/BAO/Export.php(199): CRM_Export_BAO_Export::writeDetailsToTable(Object(CRM_Export_BAO_ExportProcessor), (Array:100), (Array:14))
#15 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Export/Form/Map.php(142): CRM_Export_BAO_Export::exportComponents(TRUE, (Array:118), (Array:8), "`sort_name` asc", (Array:14), NULL, 1, " contact_a.id IN ( 1546,1415,641,27,1688,1504,1701,42,650,626,56,1509,1537,15...", "civicrm_contact", 0, 0, (Array:11), "AND")
#16 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/Form.php(513): CRM_Export_Form_Map->postProcess()
#17 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/StateMachine.php(144): CRM_Core_Form->mainProcess()
#18 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/QuickForm/Action/Next.php(43): CRM_Core_StateMachine->perform(Object(CRM_Contact_Export_Form_Map), "next", "Next")
#19 /var/www/libertarian/vendor/civicrm/civicrm-packages/HTML/QuickForm/Controller.php(203): CRM_Core_QuickForm_Action_Next->perform(Object(CRM_Contact_Export_Form_Map), "next")
#20 /var/www/libertarian/vendor/civicrm/civicrm-packages/HTML/QuickForm/Page.php(103): HTML_QuickForm_Controller->handle(Object(CRM_Contact_Export_Form_Map), "next")
#21 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/Controller.php(347): HTML_QuickForm_Page->handle("next")
#22 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/Invoke.php(312): CRM_Core_Controller->run((Array:4), (Array:0))
#23 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/Invoke.php(68): CRM_Core_Invoke::runItem((Array:13))
#24 /var/www/libertarian/vendor/civicrm/civicrm-core/CRM/Core/Invoke.php(36): CRM_Core_Invoke::_invoke((Array:4))
#25 /var/www/libertarian/web/modules/contrib/civicrm/src/Civicrm.php(88): CRM_Core_Invoke::invoke((Array:4))
#26 /var/www/libertarian/web/modules/contrib/civicrm/src/Controller/CivicrmController.php(80): Drupal\civicrm\Civicrm->invoke((Array:4))
#27 [internal function](): Drupal\civicrm\Controller\CivicrmController->main((Array:4), "")
#28 /var/www/libertarian/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(123): call_user_func_array((Array:2), (Array:2))
#29 /var/www/libertarian/web/core/lib/Drupal/Core/Render/Renderer.php(573): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
#30 /var/www/libertarian/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(124): Drupal\Core\Render\Renderer->executeInRenderContext(Object(Drupal\Core\Render\RenderContext), Object(Closure))
#31 /var/www/libertarian/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(97): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext((Array:2), (Array:2))
#32 /var/www/libertarian/vendor/symfony/http-kernel/HttpKernel.php(151): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
#33 /var/www/libertarian/vendor/symfony/http-kernel/HttpKernel.php(68): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#34 /var/www/libertarian/web/core/lib/Drupal/Core/StackMiddleware/Session.php(57): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, TRUE)
#35 /var/www/libertarian/web/core/lib/Drupal/Core/StackMiddleware/KernelPreHandle.php(47): Drupal\Core\StackMiddleware\Session->handle(Object(Symfony\Component\HttpFoundation\Request), 1, TRUE)
#36 /var/www/libertarian/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(106): Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object(Symfony\Component\HttpFoundation\Request), 1, TRUE)
#37 /var/www/libertarian/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(85): Drupal\page_cache\StackMiddleware\PageCache->pass(Object(Symfony\Component\HttpFoundation\Request), 1, TRUE)
#38 /var/www/libertarian/web/core/lib/Drupal/Core/StackMiddleware/ReverseProxyMiddleware.php(47): Drupal\page_cache\StackMiddleware\PageCache->handle(Object(Symfony\Component\HttpFoundation\Request), 1, TRUE)
#39 /var/www/libertarian/web/core/lib/Drupal/Core/StackMiddleware/NegotiationMiddleware.php(52): Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, TRUE)
#40 /var/www/libertarian/vendor/stack/builder/src/Stack/StackedHttpKernel.php(23): Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, TRUE)
#41 /var/www/libertarian/web/core/lib/Drupal/Core/DrupalKernel.php(708): Stack\StackedHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, TRUE)
#42 /var/www/libertarian/web/index.php(19): Drupal\Core\DrupalKernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#43 {main}
```https://lab.civicrm.org/dev/core/-/issues/2481Discussion drop support for php 7.2 and mysql 5.62023-01-04T18:01:36ZeileenDiscussion drop support for php 7.2 and mysql 5.6We have been dropping support for EOL php & mysql on ESR releases because that gives sites the option of delaying for another 6 months for just $120. So far we are not aware of any sites opting for the ESR for that reason but we are crea...We have been dropping support for EOL php & mysql on ESR releases because that gives sites the option of delaying for another 6 months for just $120. So far we are not aware of any sites opting for the ESR for that reason but we are creating the option.
With 5.39 coming up we should consider whether we want to drop support for
php 7.2 (EOL Dec 2020)
mysql 5.6 (EOL Feb 2021)
Note that historically supporting older versions of mysql has been low effort although some features are not supported. However, it might be holding us back from using new features
Supporting older versions of php has historically been more problematic as it has been at the expense of supporting newer versions. In particular we often find packages don't support latest and EOL versions.
So dropping support for older php versions is more helpful than older mysql versions.
Last ESR we considered dropping support for php 7.1 and mysql 5.6. We did the former and not the latter. At the time the stats were:
mysql 5.6 : 805
php 7.1 : 658.
Maria DB 10.1 : 943
Currently they are
mysql 5.6 : 603
php 7.2 : 2846
Maria DB 10.1 : 526
Based on those numbers I think it's clearly too early for php 7.2 and I would be OK targetting both for the NEXT ESR (5.45)https://lab.civicrm.org/dev/core/-/issues/2484Create apiv4 payment api2023-07-13T08:07:37ZeileenCreate apiv4 payment apiIn order to be able to search for payments we need to be able to search for payment rows (rows with is_payment = TRUE) in the financial_trxn table and for the entity bridge to be able to find it's way from there to the contribution table...In order to be able to search for payments we need to be able to search for payment rows (rows with is_payment = TRUE) in the financial_trxn table and for the entity bridge to be able to find it's way from there to the contribution table.
This gives us the ability to say find all contributions with a payment with a payment instrument of 'Check' or find the contribution where the payment's trxn value is y
The scope I would be looking for is
1) can use any of the fields in financial_trxn as a filter
2) can logically link to the contribution (probably through bridge entities)
3) forces 'is_payment' as a filter (only gets payments)
4) the create action calls Payment.create
5) the update & delete actions are blocked
This effectively gives us parity with the v3 api albeit with more options provided through the apiv4 layer itself.
Note that I think this is likely blocked on other feature requests which @mattwire will add notes onhttps://lab.civicrm.org/dev/core/-/issues/2494Search-kit wishlist - exposed join filters2022-11-07T04:17:02ZeileenSearch-kit wishlist - exposed join filtersSo on my effort to get a workable lybunt I find that what I really want is to be able to expose the join filter in the afform -ie
```
SELECT Contacts
WITH contributions > {{{total_amount}}} IN DATE RANGE {{select_date}}
WITHOUT contribu...So on my effort to get a workable lybunt I find that what I really want is to be able to expose the join filter in the afform -ie
```
SELECT Contacts
WITH contributions > {{{total_amount}}} IN DATE RANGE {{select_date}}
WITHOUT contributions IN DATE RANGE {{select_date}}
```
Obviously the other filter issue (https://lab.civicrm.org/dev/core/-/issues/2405) is a prerequisite on thishttps://lab.civicrm.org/dev/core/-/issues/2509Search kit: Links to case (and other) activities go to the wrong form2023-08-01T01:50:48ZDaveDSearch kit: Links to case (and other) activities go to the wrong formIf you do a search either purely on activities and the results happen to include case activities, or a search on cases and include the related contact activities, the links to the activities in the results list go to the regular activity...If you do a search either purely on activities and the results happen to include case activities, or a search on cases and include the related contact activities, the links to the activities in the results list go to the regular activity form not the case activity form. This matters because:
1. It's not all the right fields on the form.
2. Form hooks won't work properly.
3. You get errors on save.
@colemanwhttps://lab.civicrm.org/dev/core/-/issues/2510Search kit: Case search with related contact activities includes deleted acti...2023-07-19T13:26:07ZDaveDSearch kit: Case search with related contact activities includes deleted activities by defaultI don't know if it's so much of a bug as unintuitive since typically deleted things are left out by default and e.g. a plain activity search just on activities alone leaves out deleted ones. So it's something about the join maybe.I don't know if it's so much of a bug as unintuitive since typically deleted things are left out by default and e.g. a plain activity search just on activities alone leaves out deleted ones. So it's something about the join maybe.https://lab.civicrm.org/dev/core/-/issues/2515CiviCRM Membership, inherited membership remains active even when the primary...2022-10-10T11:36:56Zjustinfreeman (Agileware)CiviCRM Membership, inherited membership remains active even when the primary member has been marked as deletedCiviCRM Membership, inherited membership remains active even when the primary member has been marked as deleted. Reproduced on **CiviCRM 5.37.alpha1**. Observed since CiviCRM 5.33.2 and earlier.
Steps to reproduce:
1. Set up a Membershi...CiviCRM Membership, inherited membership remains active even when the primary member has been marked as deleted. Reproduced on **CiviCRM 5.37.alpha1**. Observed since CiviCRM 5.33.2 and earlier.
Steps to reproduce:
1. Set up a Membership Type which can be inherited using the Employer Of relationship, Membership A
2. Create Organisation contact, Org B
3. Assign Membership A to Org B
4. Create Organisation contact, Org C
5. Assign Membership A to Org C
6. Create Employer Of relationship for Individual D, E, F with Org B
7. Note that Individual D, E, F have inherited Membership A from Org B
8. Merge Org B (Duplicate) and Org C (Original)
9. During the Merge action, keep the merge Relationship option enabled. This problem can be replicated with either enable or disable of the Move Related Memberships.
10. The merge action will mark Org B (Duplicate) as deleted
11. Post merge, check Individual D, E, F
12. Note that the Employer Of relationship for Individual D, E, F has been changed to Org C, this is correct.
13. Note that Individual D, E, F still have inherited Membership A from Org B (the deleted contact) - this is incorrect, see below.
The desired result is that Individual D, E, F will have inherited Membership A from Org C. The Relationship has been transferred to Org C which should trigger then membership to be inherited from Org C, not Org B.
Workaround is to restore the deleted Org C from trash and delete the Membership record. Then for each of the Individual contacts, disable and re-enable the Relationship with Org B. This should restore the correct inherited relationship.
Agileware Ref: CIVICRM-906 CIVICRM-1922
Screenshots below of reproduced issue.
* Green Action School DUPLICATE (Org B)
* Green Action School ORIGINAL (Org C)
* The two Individual Contacts show the incorrect inherited membership, associated with the Green Action School DUPLICATE (Org B).
![screencapture-dmaster-demo-civicrm-org-civicrm-contact-view-2021-04-07-10_39_07](/uploads/461820778757e6c6c542d85f8a84d447/screencapture-dmaster-demo-civicrm-org-civicrm-contact-view-2021-04-07-10_39_07.png)
![screencapture-dmaster-demo-civicrm-org-civicrm-contact-view-2021-04-07-10_38_57](/uploads/d852c509295ef4a39c47d879bf2de006/screencapture-dmaster-demo-civicrm-org-civicrm-contact-view-2021-04-07-10_38_57.png)
![screencapture-dmaster-demo-civicrm-org-civicrm-contact-merge-2021-04-07-10_38_49](/uploads/0820651ba28063017d00d58573e33f18/screencapture-dmaster-demo-civicrm-org-civicrm-contact-merge-2021-04-07-10_38_49.png)
![screencapture-dmaster-demo-civicrm-org-civicrm-contact-view-2021-04-07-10_39_34](/uploads/4e5b57757b8982c324fb17a54a286dec/screencapture-dmaster-demo-civicrm-org-civicrm-contact-view-2021-04-07-10_39_34.png)
![screencapture-dmaster-demo-civicrm-org-civicrm-contact-view-2021-04-07-10_39_25](/uploads/d6e7e3074f38b70107188a41d2b7e41b/screencapture-dmaster-demo-civicrm-org-civicrm-contact-view-2021-04-07-10_39_25.png)https://lab.civicrm.org/dev/core/-/issues/2546Add Settings, Disable, Delete buttons to group contacts listing page2023-09-14T15:09:10ZlarsssandergreenAdd Settings, Disable, Delete buttons to group contacts listing pageCurrently, you can only change settings, disable or delete a group from the Manage Groups page. It would make sense to be able to change the settings at least on the page that lists all the contacts in a group (disable and delete are les...Currently, you can only change settings, disable or delete a group from the Manage Groups page. It would make sense to be able to change the settings at least on the page that lists all the contacts in a group (disable and delete are less important, but I think they might as well be added as well). I often find I click into a group when I want to make it a mailing list and then realize I have to go back to the Manage Groups page and click settings.
I think this would make the most sense at the top of the page, maybe right below the "Add Contacts to this group" button or alternately, just above select records. Relatedly, I would also propose to put the "Edit Smart Group Criteria for this group" and "Add Contacts to this group" buttons on the same line, as they are currently on two.
For reference:
![image](/uploads/b7ed25111565401ddf2ae295795f3b8c/image.png)
I'll give this a go if it is supported.https://lab.civicrm.org/dev/core/-/issues/2559Cannot get Auth Code in Oauth2 from Microsoft Azure Application2023-01-10T14:28:11ZspalmstromCannot get Auth Code in Oauth2 from Microsoft Azure ApplicationOverview
----------------------------------------
You cannot get an Auth Code in Oath2 from a single-tenant Microsoft Azure Application because the access token string is
```
https://login.microsoftonline.com/common/oauth2/v2.0/token
``...Overview
----------------------------------------
You cannot get an Auth Code in Oath2 from a single-tenant Microsoft Azure Application because the access token string is
```
https://login.microsoftonline.com/common/oauth2/v2.0/token
```
when it should be:
```
https://login.microsoftonline.com/<tenant ID>/oauth2/v2.0/token
```
Reproduction steps
----------------------------------------
1. Click on Admin -> Oauth2 Administration
1. Select Microsoft Exchange Online
1. Click on Add token and enter an MS account
Current behaviour
----------------------------------------
```
AADSTS50194: Application '226037fb-d13a-4f81-ba32-561601248bea'(MissionAssist Mail) is not configured as a multi-tenant application. Usage of the /common endpoint is not supported for such applications created after '10/15/2018'. Use a tenant-specific endpoint or configure the application to be multi-tenant.
```
Expected behaviour
----------------------------------------
A token should be added.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _Edge_ but probably not relevant
* __CiviCRM:__ _5.36.1_ <!-- If this problem relates to an upgrade, then specify both old and new versions -->
* __PHP:__ _7.4.16__ but probably not relevant
* __CMS:__ _Drupal 9.1.7_ but probably not relevant.
* __Database:__ _MySQL 8.0.24_ but probably not relevant
* __Web Server:__ _IIS_ but probably not relevant.
Comments
----------------------------------------
It would be good if the setup could prompt for the tenant ID>5.59.0https://lab.civicrm.org/dev/core/-/issues/2562Simplify when scheduled reminders are sent or not sent for events2023-08-07T15:14:25ZlarsssandergreenSimplify when scheduled reminders are sent or not sent for eventsFor event scheduled reminders, you can set emails to be sent on specific days or N hours or days before or after an event (or before or after registration ends or starts). For emails sent on a specific day, this is straightforward. They ...For event scheduled reminders, you can set emails to be sent on specific days or N hours or days before or after an event (or before or after registration ends or starts). For emails sent on a specific day, this is straightforward. They are sent on that day and anyone who registers after that day does not receive the email.
But for emails scheduled N hours or days before or after an event, the behaviour is less clear. My testing indicates contacts registered soon after an event has ended can still receive emails scheduled to go out 24 hours before the event start or 30 hours before the event end (but contacts registered somewhat later won't receive these emails). Similarly, contacts registered a day after an event ends will receive an email scheduled to go out an hour after the event start. I can't find any documentation that tells users what will happen when they schedule these reminders, though there are a few questions on Stack Exchange wondering about what actually does happen (opinions vary, which is a bad sign).
I think the clearest and best option would be to simply send scheduled reminders at the time they are scheduled and not afterwards. For one, this is simple and probably what users assume happens. It's certainly what our users think. Secondly, let's say you have an event with a confirmation email, one scheduled reminder a day before the event and one 2 hours before the event: with the current system a person who registers at the last minute will receive three emails in short order, which doesn't make sense. Thirdly, if a user registers someone in the backend for an event after the event has ended, just to record their attendance, they probably don't want that person to automatically receive emails previously scheduled to go out right after the event start or similar (this could be avoided by using a different status for the post-event registration and the scheduled reminder, but that's a pretty fine point that users won't think about).
In short, I think there are so many potential pitfalls with a system that sends scheduled reminders after the time set on the reminder has passed, especially when none of this behaviour is apparent to users when they set up the reminders. To me, this outweighs the potential convenience of having reminders sent to late registrants, so we should simply not send scheduled reminders whose time has passed.
I haven't dug into the code yet to see how this all works, but I'll do that and submit a PR as long as this is supported.
EDIT: removed aside on timing of scheduled reminders.https://lab.civicrm.org/dev/core/-/issues/2565HTML editor for additional event confirmation email text2023-08-02T21:16:19ZlarsssandergreenHTML editor for additional event confirmation email textIt would be nice to be able to have an HTML editor for the additional event confirmation text entered on the Manage Event Page that gets inserted into the System Template email template. The benefit would be to be able to easily add link...It would be nice to be able to have an HTML editor for the additional event confirmation text entered on the Manage Event Page that gets inserted into the System Template email template. The benefit would be to be able to easily add links, etc. I understand this will be a complicated one, but it does seem worthwhile as almost every other similar field is HTML, so this one really stands out. [See previous issue.](https://lab.civicrm.org/dev/core/-/issues/852#note_58464)
How it works now: Text from the event confirmation email text is added into the event confirmation message template with some minimal tags added. I see a <p> wrapper around the text and <br /> inserted for newlines, at a cursory glance.
Here's my guess at what needs to happen:
1. Add HTML editor to Event Configuration
2. Add HTML editor on the New Event Registration page (which is also Add Participant from events or Add Event Registration for a contact), so that the message can be edited before sending.
3. Inject content as HTML into Message Templates, i.e. remove code that adds tags, wrap code in div or something.
4. Convert all existing content in the DB with the same process now taken to inject the text into the Message Template or handle the content injection into the Message Template with two cases based on text/HTML.
The first three should be relatively simple, but the last one there seems like the tricky part. How has this has been handled in the past?https://lab.civicrm.org/dev/core/-/issues/2571Move reCAPTCHA to core extension2022-12-02T09:28:56Zmattwiremjw@mjwconsult.co.ukMove reCAPTCHA to core extensionThis is a tracking issue to track progress.
* https://github.com/civicrm/civicrm-core/pull/19967 - initial extension merged.
* https://github.com/civicrm/civicrm-core/pull/19999 - add recaptcha to distmaker - merged.
* https://github.co...This is a tracking issue to track progress.
* https://github.com/civicrm/civicrm-core/pull/19967 - initial extension merged.
* https://github.com/civicrm/civicrm-core/pull/19999 - add recaptcha to distmaker - merged.
* https://github.com/civicrm/civicrm-core/pull/20011 - move recaptcha library to extension - merged.
* https://github.com/civicrm/civicrm-core/pull/20019 - Use standard function to add reCAPTCHA to PCPAccount form - merged.
Need to cleanup and move all calls to add reCAPTCHA to forms to the extension.https://lab.civicrm.org/dev/core/-/issues/2572Prevent unintentional removal from group for anonymous users filing out profiles2023-09-01T19:44:06ZlarsssandergreenPrevent unintentional removal from group for anonymous users filing out profilesCurrently, if an anonymous contact submits a profile that contains a groups field and the profile submission is deduped on submission with an existing contact, the group selections in the profile overwrite the contact's previous groups, ...Currently, if an anonymous contact submits a profile that contains a groups field and the profile submission is deduped on submission with an existing contact, the group selections in the profile overwrite the contact's previous groups, for the groups that are included in the profile (i.e. the public groups). The result is that if someone is subscribed to a mailing list and fills in a profile anonymously and doesn't check the same group name, they will be unsubscribed from the group. This is definitely not what users expect! I think it's clear that checking the box will add you to the group, but I doubt many expect that they will be removed from a group if they don't check the box that they've checked in the past.
This is also not what happens with other profile fields that are left blank, nor is it what happens if two contacts are merged manually.
I propose to ignore group checkboxes that are not checked in a profile submitted anonymously. No change for profiles submitted with known cids (as these will have the appropriate checkboxes already checked, allowing the contact to unsubscribe by unchecking them if desired). Currently, all group ids in the profile are being [passed here](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Contact/BAO/GroupContact.php#L534), removing the contact from groups that were unchecked. Will submit PR if supported.
EDIT: Here is some discussion on StackExchange: [Unwanted unsubscribe in profile](https://civicrm.stackexchange.com/questions/39403/unwanted-unsubscribe-in-profile/39406), [Anonymous user filing out a profile unintentionally remove themselves from groups](https://civicrm.stackexchange.com/questions/29337/anonymous-user-filing-out-a-profile-unintentionally-remove-themselves-from-group)https://lab.civicrm.org/dev/core/-/issues/3204Add recurring contributions to contribution reports2022-09-16T21:11:08ZlarsssandergreenAdd recurring contributions to contribution reportsIt would be very convenient to be able to do filter by recurring contributions for reports, add recurring contributions as a column or even group by.
[Here's a PR that adds this for the Contribution Summary report.](https://github.com/c...It would be very convenient to be able to do filter by recurring contributions for reports, add recurring contributions as a column or even group by.
[Here's a PR that adds this for the Contribution Summary report.](https://github.com/civicrm/civicrm-core/pull/20168) I will also implement in other relevant reports once this has been reviewed.https://lab.civicrm.org/dev/core/-/issues/2581Cannot add case type in multilingual installation with MariaDB 10.2 strict mode2023-09-02T01:52:37ZhansrosselCannot add case type in multilingual installation with MariaDB 10.2 strict modeWhen adding a new case type via the UI at /en/civicrm/a/#/caseType/new in a multilingual site there is an error
```
"error_message": "DB Error: unknown error",
"mode": 16,
"debug_info": "INSERT INTO `civicrm_case_type_en_US` (`n...When adding a new case type via the UI at /en/civicrm/a/#/caseType/new in a multilingual site there is an error
```
"error_message": "DB Error: unknown error",
"mode": 16,
"debug_info": "INSERT INTO `civicrm_case_type_en_US` (`name` , `is_active` ) VALUES ('ExampleNew' , 1 ) [nativecode=1423 ** Field of view 'sitex.civicrm_case_type_en_US' underlying table doesn't have a default value]",
"type": "DB_Error",
"user_info": "INSERT INTO `civicrm_case_type_en_US` (`name` , `is_active` ) VALUES ('ExampleNew' , 1 ) [nativecode=1423 ** Field of view 'sitex.civicrm_case_type_en_US' underlying table doesn't have a default value]",
"to_string": "[db_error: message=\"DB Error: unknown error\" code=-1 mode=callback callback=CRM_Utils_REST::fatal prefix=\"\" info=\"INSERT INTO `civicrm_case_type_en_US` (`name` , `is_active` ) VALUES ('ExampleNew' , 1 ) [nativecode=1423 ** Field of view 'sitex.civicrm_case_type_en_US' underlying table doesn't have a default value]\"]",
```
Environment information
----------------------------------------
Drupal 7.80 / CiviCRM 5.36.1
MariaDB 10.2 with strict mode
Cause
----------------------------------------
MySQL having a strict mode set which won’t allow INSERT or UPDATE commands with empty fields where the schema doesn’t have a default value set.
Solution
----------------------------------------
I solved it with disabling strict mode by adding sql_mode=NO_ENGINE_SUBSTITUTION to /etc/my.cnf (and restarting mysql). Because the new Case Type was made from /en/civicrm/a/#/caseType/new I got first an empty label, but could after saving change the label to the desired case type human name.
I suppose there could also be a default value defined for these multilingual tables.5.66.0https://lab.civicrm.org/dev/core/-/issues/2595Change Log Tab Excrutiatingly Slow - Poorly Performing Query and Fix (from 8 ...2023-08-14T13:09:44ZgordanChange Log Tab Excrutiatingly Slow - Poorly Performing Query and Fix (from 8 minutes down to 4 seconds)The page takes about 8 minutes to return which is absurdly slow for anything expected to be remotely interactive. It is so slow that my client is referring to it as the "Triangle of Doom".
I tracked it down to this query:
```
INSERT IG...The page takes about 8 minutes to return which is absurdly slow for anything expected to be remotely interactive. It is so slow that my client is referring to it as the "Triangle of Doom".
I tracked it down to this query:
```
INSERT IGNORE INTO civicrm_tmp_e_logsummary_ffa3ac146126d178ade367f2a5d17bf5
SELECT activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM staging_civicrm.log_civicrm_activity_contact entity_log_civireport
JOIN civicrm_contact modified_contact_civireport ON (entity_log_civireport.contact_id = modified_contact_civireport.id )
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON (entity_log_civireport.log_user_id = altered_by_contact_civireport.id)
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
EXPLAIN shows:
```
```
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | extra_table | ALL | NULL | NULL | NULL | NULL | 3014020 | |
| 1 | SIMPLE | entity_log_civireport | ALL | NULL | NULL | NULL | NULL | 5518537 | Using where; Using join buffer (flat, BNL join) |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+---------------+---------+---------+---------------------------------------------------+---------+-------------------------------------------------+
```
The fix passes:
First pass:
```
ALTER TABLE log_civicrm_activity_contact ADD INDEX index_activity_id (activity_id);
```
With no further changes, this alone makes the above query go from 8 minutes to 1m45s.
Explain plain:
```
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | extra_table | ALL | NULL | NULL | NULL | NULL | 3014020 | Using where |
| 1 | SIMPLE | entity_log_civireport | ref | index_activity_id | index_activity_id | 5 | staging_civicrm.extra_table.id | 1 | Using where |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+-------------------+-------------------+---------+---------------------------------------------------+---------+---------------------------------+
```
Second pass:
```
ALTER TABLE log_civicrm_activity ADD INDEX index_id (id);
```
Change the JOIN order explicitly and add a hint for the query optimizer to not re-order the JOINs:
```
SELECT STRAIGHT_JOIN activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM civicrm_contact modified_contact_civireport
JOIN staging_civicrm.log_civicrm_activity_contact entity_log_civireport ON entity_log_civireport.contact_id = modified_contact_civireport.id
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON entity_log_civireport.log_user_id = altered_by_contact_civireport.id
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
```
New EXPLAIN:
```
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
| 1 | SIMPLE | modified_contact_civireport | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | entity_log_civireport | ALL | index_activity_id | NULL | NULL | NULL | 5518537 | Using where; Using filesort |
| 1 | SIMPLE | extra_table | ref | index_id | index_id | 5 | staging_civicrm.entity_log_civireport.activity_id | 1 | |
| 1 | SIMPLE | altered_by_contact_civireport | eq_ref | PRIMARY | PRIMARY | 4 | staging_civicrm.entity_log_civireport.log_user_id | 1 | Using where |
+------+-------------+-------------------------------+--------+-------------------+----------+---------+---------------------------------------------------+---------+---------------------------------+
```
This gets it down to 4 seconds!
Since the first index we started with is no longer getting used in the final variant, we can just not add it.
Summary:
To fix "Change Log" tab taking forever to load, the following fix is needed:
Add index:
```
ALTER TABLE log_civicrm_activity ADD INDEX index_id (id);
```
Make the code emit the query modified as follows:
```
INSERT IGNORE INTO civicrm_tmp_e_logsummary_ffa3ac146126d178ade367f2a5d17bf5
SELECT STRAIGHT_JOIN activity_id, IF (entity_log_civireport.log_action = 'Insert' AND extra_table.activity_type_id = 51 , GROUP_CONCAT(entity_log_civireport.contact_id), 1) , entity_log_civireport.log_action as log_civicrm_entity_log_action, 'log_civicrm_activity_contact' as log_civicrm_entity_log_type, entity_log_civireport.log_user_id as log_civicrm_entity_log_user_id, entity_log_civireport.log_date as log_civicrm_entity_log_date, modified_contact_civireport.display_name as log_civicrm_entity_altered_contact, modified_contact_civireport.id as log_civicrm_entity_altered_contact_id, entity_log_civireport.log_conn_id as log_civicrm_entity_log_conn_id, modified_contact_civireport.is_deleted as log_civicrm_entity_is_deleted, altered_by_contact_civireport.display_name as altered_by_contact_display_name
FROM civicrm_contact modified_contact_civireport
JOIN staging_civicrm.log_civicrm_activity_contact entity_log_civireport ON entity_log_civireport.contact_id = modified_contact_civireport.id
JOIN staging_civicrm.log_civicrm_activity extra_table ON extra_table.id = entity_log_civireport.activity_id
LEFT JOIN civicrm_contact altered_by_contact_civireport ON entity_log_civireport.log_user_id = altered_by_contact_civireport.id
WHERE modified_contact_civireport.id = 338520 AND
entity_log_civireport.log_action != 'Initialization'
GROUP BY entity_log_civireport.log_conn_id,
entity_log_civireport.log_user_id,
EXTRACT(DAY_MICROSECOND FROM entity_log_civireport.log_date),
entity_log_civireport.id
ORDER BY entity_log_civireport.log_date DESC;
```
Not only does this speed it up from 8 minutes down to 4 seconds, it also doesn't wreak havoc with row locking where every row scanned gets locked by the transaction engine, potentially resulting in a massive query pile-up in the database.https://lab.civicrm.org/dev/core/-/issues/2597Tabset for manage contribution pages or manage events requires Classname to e...2023-08-07T15:13:57ZlarsssandergreenTabset for manage contribution pages or manage events requires Classname to equal pathIf you add a tab to a tabset for Configure Event, it will work fine with the last word of the class name different from the last word of the path, except for trying to save, which will save, but redirect to Manage Events instead of stayi...If you add a tab to a tabset for Configure Event, it will work fine with the last word of the class name different from the last word of the path, except for trying to save, which will save, but redirect to Manage Events instead of staying on the same tab. This is because [code here](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Event/Form/ManageEvent.php#L377) expects lastWord from CRM_myExtension_Form_lastWord to be the same as the path, i.e. /civicrm/event/manage/lastword. Everything else works as expected if these are different, except Save.
Looks like this same issue exists for [Contribution Pages](https://github.com/civicrm/civicrm-core/blob/5db0bc3c1f54eaca4307f103a73bda596ae914d6/CRM/Contribute/Form/ContributionPage.php#L427
) and probably elsewhere.
For now, I've added a warning to the docs for [the tabset hook](https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/908). It would be good to fix this so the code linked above uses the path for the url instead of the class, at some point.https://lab.civicrm.org/dev/core/-/issues/2607How to handle removed files2023-12-15T15:29:01Zaydunsaidan.saunders@squiffle.ukHow to handle removed filesSee [this SE question](https://civicrm.stackexchange.com/questions/39531/database-upgrade-script-failing-call-to-undefined-method-crm-upgrade-incrementa/39546#39546)
A file was removed from the installation tarball but [the Joomla upgr...See [this SE question](https://civicrm.stackexchange.com/questions/39531/database-upgrade-script-failing-call-to-undefined-method-crm-upgrade-incrementa/39546#39546)
A file was removed from the installation tarball but [the Joomla upgrade process](https://docs.civicrm.org/sysadmin/en/latest/upgrade/joomla/#install-the-extension) installs into existing directories meaning the 'removed' file was still present. In this case, it caused the upgrade to fail and will cause problems for any Joomla upgrades following the documented method.
More generally, the same problem applies to any files no longer included in releases.
Some possibilities:
- in the release notes say that files have been removed so don't follow the normal upgrade process
- add an upgrade step that removes old files (but note this problem occurred early in the upgrade process)
- change the upgrade instructions for Joomlahttps://lab.civicrm.org/dev/core/-/issues/2611Change default_value column for custom fields from varchar to text2023-08-20T02:32:58ZlarsssandergreenChange default_value column for custom fields from varchar to textA custom field can be of many types, including note. For a custom field that is a note, you may want to use a default value that is longer than 255 characters, but currently the default value is stored as a varchar(255). It would make se...A custom field can be of many types, including note. For a custom field that is a note, you may want to use a default value that is longer than 255 characters, but currently the default value is stored as a varchar(255). It would make sense to change this column to text, so that longer values can be entered.
In the custom fields UI, this field would be changed to a textarea when note is selected.
For us, the use case would be storing the content of a short waiver, so that the exact value of the waiver at the time of signing can be stored on the participant for an event, but I can imagine there may be other use cases.
Will submit PR if supported.https://lab.civicrm.org/dev/core/-/issues/2630Improve performance of queries in CRM_Contact_Form_Search_Custom_Base2023-02-06T04:12:42ZErikHommelImprove performance of queries in CRM_Contact_Form_Search_Custom_BaseSee this SE post: https://civicrm.stackexchange.com/questions/39636/slow-query-fixes-msp-custom-searches
It might be worth implementing the suggested improvements?See this SE post: https://civicrm.stackexchange.com/questions/39636/slow-query-fixes-msp-custom-searches
It might be worth implementing the suggested improvements?https://lab.civicrm.org/dev/core/-/issues/3314Contribution pages with memberships don't validate correctly when using "quic...2024-02-27T01:10:05ZJonGoldContribution pages with memberships don't validate correctly when using "quick config" price fieldsIn `CRM_Contribute_Form_Contribution_Main::formRule`, there are multiple references to `$self->_useForMember`. `$self->_useForMember` is only `TRUE` if the price set is a "quick config" price set (and is explicitly used as such in the t...In `CRM_Contribute_Form_Contribution_Main::formRule`, there are multiple references to `$self->_useForMember`. `$self->_useForMember` is only `TRUE` if the price set is a "quick config" price set (and is explicitly used as such in the template). However, during `formRule` validation, it's used as a proxy for "this contribution page uses memberships". @mattwire correctly identified the problem on the confirmation class in [PR 15094](https://github.com/civicrm/civicrm-core/pull/15094).
As a result, several membership-specific validations are bypassed.
I attempted to move his methods to the shared `ContributionBase` class and make them protected, but I ran into deeper validation issues. I can't justify further work on this, but wanted to document for future folks who bang their heads against this.https://lab.civicrm.org/dev/core/-/issues/2641Users affected by ACLs intermittently lose access to contacts2023-08-22T04:01:30ZjhungerfordUsers affected by ACLs intermittently lose access to contactsOverview
----------------------------------------
When a user's permission to view Contacts is affected by multiple ACLs, they will intermittently lose access to some of them, especially directly after editing a record.
Reproduction ste...Overview
----------------------------------------
When a user's permission to view Contacts is affected by multiple ACLs, they will intermittently lose access to some of them, especially directly after editing a record.
Reproduction steps
----------------------------------------
1. Create a test User who is not an Administrator and does not have "view all contacts" or "edit all contacts" permission
2. Create two Groups (of type Access Control List), and add the test User to both groups
3. Create two Groups (not of ACL type) with distinct names like "Contact Group1" and "Contact Group2"
4. Create two Contacts and add one to each of the Groups created in step 3
5. Create two ACLs, each giving users in one of the ACL groups access to one of the Contact groups
6. Log in as the test User and repeatedly edit the Nickname (or any other data) on both of the test Contacts until you encounter a failure ("Error You do not have the necessary permission to view this contact").
It will rarely take more than ten edits before you encounter a failure, and the failures will never occur for one of the ACL groups.
Current behaviour
----------------------------------------
Sometimes it works, sometimes the user loses permission during the save or reload operation, resulting in the error message below. Our users reported that usually the data would be saved, but sometimes not (e.g. when saving some Activities on a Case). I have not been able to replicate the data loss scenario.
```
Error
You do not have the necessary permission to view this contact.
```
Expected behaviour
----------------------------------------
The data should be saved without error messages appearing (and this is what happens some of the time).
Environment information
----------------------------------------
* __Browser:__ Mostly tested on Chromium, most recently version 90.0.4430.93
* __CiviCRM:__ Tested on 5.33.5 and prior ESR versions
* __PHP:__ 7.3
* __CMS:__ Drupal 7.80
* __Database:__ 10.3.27-MariaDB
* __Web Server:__ Apache
Comments
----------------------------------------
We have mitigated the problem with this patch (minus the watchdog line with the FIXME comment):
https://github.com/AsylumSeekersCentre/civicrm-core/commit/030ecb6e31ed17b9fd01716cb772f13bb60bf455
With this in place, I've been unable to reproduce the error in manual testing. If it still happens, it's very rare. However, I'm not sure that this is a very "proper" solution so I'm opening the issue without a pull request for discussion.
The watchdog line from that patch usually produces output like this:
```
( ( `contact_a`.id IN ( SELECT contact_id FROM civicrm_group_contact WHERE group_id IN (5, 6) AND status = 'Added' ) ) ) AND (contact_a.is_deleted = 0)
```
On the occasions when the permission failure occurs, it instead looks like this:
```
( ( `contact_a`.id IN ( SELECT contact_id FROM civicrm_group_contact WHERE group_id IN (5) AND status = 'Added' ) ) ) AND (contact_a.is_deleted = 0)
```
It has somehow lost one of the groups it should be checking for. It always seems to retain the same one, which I think is why it works if the user is only affected by one ACL, and the bug only shows up if users are affected by multiple ACLs.https://lab.civicrm.org/dev/core/-/issues/2646Enable/Disable Extension Receives API Error2022-09-16T00:12:46ZpbarmakEnable/Disable Extension Receives API ErrorOverview
----------------------------------------
Anytime we enable or disable an extension, we get an API error with the error message of: API error: Value already exists in the database on ReportTemplate.create. This is true with eithe...Overview
----------------------------------------
Anytime we enable or disable an extension, we get an API error with the error message of: API error: Value already exists in the database on ReportTemplate.create. This is true with either the Admin extension UI screen or using cv.
Reproduction steps
----------------------------------------
1. Run cv ext:disable uk.co.vedaconsulting.mosaico
2. Get the error message
3. The extension still gets disabled (or enabled, depending on what we run)
Current behaviour
----------------------------------------
Here is the full error message from cv:
```
Disabling extension "uk.co.vedaconsulting.mosaico"
Error: API Call Failed: Array
(
[entity] => Extension
[action] => disable
[params] => Array
(
[keys] => Array
(
[0] => uk.co.vedaconsulting.mosaico
)
[debug] => 1
[version] => 3
)
[result] => Array
(
[trace] => #0 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/ManagedEntities.php(241): CRM_Core_ManagedEntities->onApiError('ReportTemplate', 'create', Array, Array)
#1 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/ManagedEntities.php(184): CRM_Core_ManagedEntities->insertNewEntity(Array)
#2 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/ManagedEntities.php(146): CRM_Core_ManagedEntities->reconcileEnabledModule(Object(CRM_Core_Module), Array)
#3 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/ManagedEntities.php(127): CRM_Core_ManagedEntities->reconcileEnabledModules()
#4 /var/www/civicrm/sites/all/modules/civicrm/CRM/Core/Invoke.php(400): CRM_Core_ManagedEntities->reconcile()
#5 /var/www/civicrm/sites/all/modules/civicrm/CRM/Extension/Manager.php(420): CRM_Core_Invoke::rebuildMenuAndCaches(true)
#6 /var/www/civicrm/sites/all/modules/civicrm/api/v3/Extension.php(150): CRM_Extension_Manager->disable(Array)
#7 /var/www/civicrm/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_extension_disable(Array)
#8 /var/www/civicrm/sites/all/modules/civicrm/Civi/API/Kernel.php(149): Civi\API\Provider\MagicFunctionProvider->invoke(Array)
#9 /var/www/civicrm/sites/all/modules/civicrm/Civi/API/Kernel.php(81): Civi\API\Kernel->runRequest(Array)
#10 /var/www/civicrm/sites/all/modules/civicrm/api/api.php(22): Civi\API\Kernel->runSafe('Extension', 'disable', Array)
#11 phar:///usr/local/bin/cv/src/Command/BaseCommand.php(49): civicrm_api('Extension', 'disable', Array)
#12 phar:///usr/local/bin/cv/src/Command/ExtensionDisableCommand.php(56): Civi\Cv\Command\BaseCommand->callApiSuccess(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput), 'Extension', 'disable', Array)
#13 phar:///usr/local/bin/cv/vendor/symfony/console/Command/Command.php(257): Civi\Cv\Command\ExtensionDisableCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 phar:///usr/local/bin/cv/vendor/symfony/console/Application.php(850): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 phar:///usr/local/bin/cv/vendor/symfony/console/Application.php(193): Symfony\Component\Console\Application->doRunCommand(Object(Civi\Cv\Command\ExtensionDisableCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 phar:///usr/local/bin/cv/src/Application.php(46): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 phar:///usr/local/bin/cv/vendor/symfony/console/Application.php(124): Civi\Cv\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 phar:///usr/local/bin/cv/src/Application.php(15): Symfony\Component\Console\Application->run()
#19 phar:///usr/local/bin/cv/bin/cv(27): Civi\Cv\Application::main('phar:///usr/loc...')
#20 /usr/local/bin/cv(14): require('phar:///usr/loc...')
#21 {main}
[is_error] => 1
[error_message] => API error: Value already exists in the database on ReportTemplate.create
)
)
```
Expected behaviour
----------------------------------------
No error message
Environment information
----------------------------------------
* __CiviCRM:5.38.0
* __PHP:7.3.28
* __CMS:Drupal 7
* __Web Server: Apache 2.4https://lab.civicrm.org/dev/core/-/issues/2651CivicrmBreadcrumbBuilder is getting invalid URLs and breaking2023-09-29T13:46:29ZananelsonCivicrmBreadcrumbBuilder is getting invalid URLs and breakingOverview
----------------------------------------
I ran into this exception while trying to open most any page in CiviCase (using compucorp's CiviCase but I don't think that matters).
InvalidArgumentException: The user-entered string '...Overview
----------------------------------------
I ran into this exception while trying to open most any page in CiviCase (using compucorp's CiviCase but I don't think that matters).
InvalidArgumentException: The user-entered string 'https://example.com/civicrm/case/a/?case_type_category=1#/case?case_type_category=1' must begin with a '/', '?', or '#'. in Drupal\Core\Url::fromUserInput() (line 213 of core/lib/Drupal/Core/Url.php).
I found that the breadcrumbs list contained:
- /
- /civicrm?reset=1
- https://example.com/civicrm/case/a/?case_type_category=1#/case?case_type_category=1
So somehow the items being added to breadcrumbs list weren't getting parsed correctly. I worked around this temporarily by modifying the build() function to only add valid items:
```
public function build(RouteMatchInterface $route_match) {
$breadcrumb = new Breadcrumb();
$breadcrumb->addCacheContexts(['url']);
$breadcrumb->addLink(Link::createFromRoute($this->t('Home'), '<front>'));
foreach ($this->civicrmPageState->getBreadcrumbs() as $name => $url) {
// All urls here have been passed trough CRM_Utils_System::url, so we have
// to parse and decode them before creating a drupal Url object.
> if (substr($url, 0, 1) == "/") {
$url = Url::fromUserInput(html_entity_decode($url));
$breadcrumb->addLink(new Link($name, $url));
> }
}
return $breadcrumb;
}
```
Since it's not a big deal to me to not have the current page in breadcrumbs, and this was an easy way to fix.
I'm not sure how the invalid url was added to the breadcrumbs, and whether that's happening in the extension or in core, but it does seem like a little checking to be sure an invalid breadcrumb doesn't cause the whole page to break would be useful in core.
Environment information
----------------------------------------
* __CiviCRM:__ _5.38.0_https://lab.civicrm.org/dev/core/-/issues/3325Using separate donation with free membership result into Pending (incomplete ...2023-11-10T01:45:09ZMonish DebUsing separate donation with free membership result into Pending (incomplete transaction) 0$ contributionSteps to replicate:
1. Create a 0$ fixed membership type
2. Configure a new/existing online contribution page, with separate donation and choose 0$ membership type under `Membership Fee`
3. Do a online contribution with $1 separate dona...Steps to replicate:
1. Create a 0$ fixed membership type
2. Configure a new/existing online contribution page, with separate donation and choose 0$ membership type under `Membership Fee`
3. Do a online contribution with $1 separate donation with 0$ membership
Bug:
Two donation are created, one with Completed 1$ contribution and other `Pending (Incomplete Transaction)` 0$ contribution linked with Pending free membership:
![Screenshot_2021-06-22_at_2.06.49_PM](/uploads/0cbd58fb74d7bc9467613571f70bb74a/Screenshot_2021-06-22_at_2.06.49_PM.png)
Expected result:
Complete 0$ membership's donation associated with `new` free membership
cc @eileen @JoeMurrayMonish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/2669Target contact missing for automatically generated case activities2023-02-23T16:00:45ZAndreasandreas.howiller@civiservice.deTarget contact missing for automatically generated case activitiesOverview
----------------------------------------
Some actions on cases leave automatically generated activities in the case activity log, but have no target contacts. Examples: Change case title, change case tags. As a result, they cann...Overview
----------------------------------------
Some actions on cases leave automatically generated activities in the case activity log, but have no target contacts. Examples: Change case title, change case tags. As a result, they cannot be edited through the user interface. While this is a minor issue, some users may want to leave notes about the activity (for example, why they changed a case title) and therefore need to edit it.
Reproduction steps
----------------------------------------
1. Open any case in CiviCRM and change the case title.
2. Tap edit for the corresponding activity in the activity timeline.
3. Tap save.
Current behaviour
----------------------------------------
The dialog will freeze while the CiviCRM logo keeps turning. A fatal error "DB Error: constraint violation" will be filed in the logs.
If you add any contact as a target contact in the database or in the edit screen the error won't appear anymore.
What is confusing, too: When editing the activity the client contact already wrongly shows up as target contact in this dialog and if you'd like to change it there you need to search the contact (the link to choose the client contact below the search field won't do the change here).
Expected behaviour
----------------------------------------
The Client contact is filed as target contact by default and connected problems disappear. E.g. editable activities can be saved after edit through UI now.
Environment information
----------------------------------------
* __CiviCRM:__ _5.35.2_ and _5.38.0_
* __CMS:__ _WordPress 5.7.2_https://lab.civicrm.org/dev/core/-/issues/2693Remove all calls to BAO_Contribution::completeOrder other than from Payment.c...2023-09-10T15:41:47ZeileenRemove all calls to BAO_Contribution::completeOrder other than from Payment.createThe goal is that we should either
1) call Payment.create to add a payment - which would itself call completeOrder
2) OR call repeatTransaction via the repeattransaction api to create a pending order - which we would then update to cance...The goal is that we should either
1) call Payment.create to add a payment - which would itself call completeOrder
2) OR call repeatTransaction via the repeattransaction api to create a pending order - which we would then update to cancelled or failed if need be or - if completed call payment.create
Note that currently Payment.create calls Contribution.completetransaction - the goal is to reverse this
I'm currently working on removing calls to completeOrder other than from the 2 api calls - at which point we can change the api calls to no longer call completeOrder directlyhttps://lab.civicrm.org/dev/core/-/issues/3552Set time limit for bounce types with hold_threshold > 12023-09-05T10:00:02ZMichael McAndrewSet time limit for bounce types with hold_threshold > 1Bounce types can be split into those with a hold_threshold = 1 where a single bounce should cause this contact to be put on hold (e.g. mailbox non existent), and those where more than one bounce is required (mailbox full).
We don't curr...Bounce types can be split into those with a hold_threshold = 1 where a single bounce should cause this contact to be put on hold (e.g. mailbox non existent), and those where more than one bounce is required (mailbox full).
We don't currently set a time limit on those where more than one bounce is required, which is an oversight. For example, the bounce threshold for Mailbox full is 3. This means that as soon as we have received more than 3 mailbox full messages for a contact, their email will be put on hold. This might make sense if it was 3 in the last year, but doesn't really make sense if it was three in the last 10 years.
Similar arguments can be made for other bounce types, like Away, which is set to 30. If I send a fair amount of email to people that are out of the office quite a lot, over time a significant chunk of these will send me > 30 out of the office messages.
Hence we should only consider messages received in a certain time period. I am going to suggest one year. I would be up for making it user configurable (number of days) if people feel strongly about it.
Let me know what you think.https://lab.civicrm.org/dev/core/-/issues/2759Intermittent cookies error after contribution2023-03-14T07:16:59ZandyburnsIntermittent cookies error after contributionWe experience a cookies error intermittently after submitting a donation. Currently on Civi 5.35.2 and WP 5.7.2. This is multisite so may be multisite related.
![118401618_379155863073198_2489825481780434605_n](/uploads/1125c66666dd23c3...We experience a cookies error intermittently after submitting a donation. Currently on Civi 5.35.2 and WP 5.7.2. This is multisite so may be multisite related.
![118401618_379155863073198_2489825481780434605_n](/uploads/1125c66666dd23c31102ac79e053f06f/118401618_379155863073198_2489825481780434605_n.png)
A contributor does not get to the thank you page but gets the error above. The donation does succeed. This issue has persisted for a long-time (2+ years) and is a probably the worst civi issue I've encountered seeing it has to do with contribution pages and has no pattern that we can find. Tried all the go to solutions posted on SE with no luck. I would estimate that it occurs ~10-20% of the time.
This is related: https://lab.civicrm.org/dev/core/-/issues/517 and says using PHP 7.2 resolved, but were on 7.2 and now on 7.3
Some steps I've done:
- Set `Clean-up Temporary Data and Files` scheduled job run frequency from Hourly to Daily. It is on every site.
- Changed cookie domain constant in wp-config.php
```
// define('COOKIE_DOMAIN', false);
to
define('COOKIE_DOMAIN', $_SERVER\['HTTP_HOST'\] );
```
- Enabled https://lab.civicrm.org/extensions/qfsessionwarning and set timeout at 120 minutes.
`$civicrm_setting['core']['secure_cache_timeout_minutes'] = 120;`
We have considered changing how garbage collection works as sessions will die if garbage collection is run (server does that every 15 minutes).
[This post](https://civicrm.stackexchange.com/questions/39156/payments-fail-with-paypal-standard-and-could-not-find-a-valid-session-key) seems to have found a solution using the Fatal Error Handler. Is this recommended and how do you implement? Seems like a workaround symptom fix 'solution'.
Also getting this for additional event participants all the time (usually) or not at all depending on when I test: https://lab.civicrm.org/dev/core/-/issues/2708
Help?https://lab.civicrm.org/dev/core/-/issues/2761Proposal: Change Pay Later Billing Address to look up a profile rather than a...2022-10-04T21:07:14ZBarijohnProposal: Change Pay Later Billing Address to look up a profile rather than a coded block.Overview
----------------------------------------
Currently on the pay later option when you use the option of Billing Address the fields are hard coded. This would be better to use a profile then it could be edited by the end user witho...Overview
----------------------------------------
Currently on the pay later option when you use the option of Billing Address the fields are hard coded. This would be better to use a profile then it could be edited by the end user without having to mess with the code.
Example use-case
----------------------------------------
Click on Pay Later Option and enable Billing Fields. This shows multiple fields that might not be required depending on the part of the world. So in that case there could be a reserved profile just for billing fields. This could have a help field to show where this profile is saved, so it would easily amended.
Current behaviour
----------------------------------------
Currently this is set in Core/Payment.php and /civicrm/templates/CRM/Core/BillingBlock.tpl (based on communications with our devs).
Proposed behaviour
----------------------------------------
Make this block look up a billing address profile instead.
Comments
----------------------------------------
Currently this field asks for information that isn't required in the UK and most of our clients would like to be able to edit this directly.https://lab.civicrm.org/dev/core/-/issues/2775Scheduled reminders fail silently if there is an error in the template2023-03-27T16:26:57ZwmortadaScheduled reminders fail silently if there is an error in the templateOverview
----------------------------------------
We have recently come across an issue whereby scheduled reminders weren't being sent but we were completely unaware this was happening. We managed to track it down to an error in the temp...Overview
----------------------------------------
We have recently come across an issue whereby scheduled reminders weren't being sent but we were completely unaware this was happening. We managed to track it down to an error in the template for one of the event reminders we had set up - the template contained CSS and this caused a fatal error in Smarty.
```
PHP Fatal error: Smarty error: [in string:<meta content="text/html; charset=utf-8" http-equiv="Content-Type" />
<title></title>
<style type="text/css">html { font-size: 100%;}
html, button, input, select, textarea { font-family: sans-serif; }
body, h1, h2, h3, h4, h5, h6, p { margin: 0; }
table { border-collapse: collapse; border-spacing: 0; }
td img { display: block; }
...
```
The issue is that this stopped _all_ scheduled reminders from being sent - not just the event reminder. This meant that none of our membership renewal reminder emails were being sent either. We were completely unaware of this because there was no visible error message in the UI.
Reproduction steps
----------------------------------------
1. Create a scheduled reminder that works correctly.
1. Check that this is sending emails as expected.
1. Create another scheduled reminder with CSS in the template (see above) so that it causes a Smarty error.
1. Note that none of the scheduled reminder emails are being sent.
Current behaviour
----------------------------------------
CiviCRM fails silently when there is an error in a template for a scheduled reminder. This stops all scheduled reminders from being sent.
Expected behaviour
----------------------------------------
There are several things that should happen if there is an error in a Smarty template:
1. CiviCRM alerts the user at the point of creating/editing a scheduled reminder with a Smarty error in the template.
1. CiviCRM alerts the user if there is scheduled reminders fail to be sent - see #1933.
1. CiviCRM should continue to send the other scheduled reminders as normal so only the reminder with an error fails, rather than no reminders being sent at all.
Environment information
----------------------------------------
* __Browser:__ N/A
* __CiviCRM:__ _5.33.5_
* __PHP:__ _7.3_
* __CMS:__ _WordPress 5.4.6_
* __Database:__ _Maria DB 10.3.23._
* __Web Server:__ _Nginx 1.15.0_
Comments
-----------------------------------------
See this post on StackExchange for info about the Smarty error: [SMARTY error with template](https://civicrm.stackexchange.com/questions/31866/smarty-error-with-template).https://lab.civicrm.org/dev/core/-/issues/2788Add tokenlist api2023-09-24T22:53:09ZeileenAdd tokenlist apiWe need an api to call via ajax to determine the available tokens for a given entity. An example is the Scheduled reminders UI which currently presents the tokens for all entities (available or not) but would ideally look up the tokens v...We need an api to call via ajax to determine the available tokens for a given entity. An example is the Scheduled reminders UI which currently presents the tokens for all entities (available or not) but would ideally look up the tokens via ajax once the entity is selected.
I think the hardest part of doing this is agreeing how it should look - eg. entity, action, parameters - eg
```
Civi\Api4\Tokens::list()
->setSchema(['contributionId', 'contactId')
->setWhere('contact_type', '=', 'Organization')
->execute();
```
Possible entity names
- Token
- Tokens
- TokenProcessor
- EntityTokens
Possible actions
-list
- getTokens
- listTokens
Schema - maybe this one is clear cut
And - how do we pass relatively free form tokens such as activity_type_id, contact_type or (possibly in the future) saved_search_id where search kit plays a rolehttps://lab.civicrm.org/dev/core/-/issues/2795PHP8: Use of error-suppression operator (@) behaves differently in error_hand...2023-12-12T19:51:20ZDaveDPHP8: Use of error-suppression operator (@) behaves differently in error_handler functions in php8Parking what's left in https://github.com/civicrm/civicrm-core/pull/21064 here for now since I've cleared out a bunch and am taking a break from it. The gist is that in php8 the error suppression operator (e.g. `@fopen('/nonexistent/file...Parking what's left in https://github.com/civicrm/civicrm-core/pull/21064 here for now since I've cleared out a bunch and am taking a break from it. The gist is that in php8 the error suppression operator (e.g. `@fopen('/nonexistent/file.txt', 'r')` works differently with respect to error_handler functions. In a stock install as of right now you won't notice any difference, just there will be more errors depending on what 3rd party/custom error_handlers you're using and may behave differently on different CMSs in future depending on how their error-handling may change.
### smarty default modifier
The smarty modifier `|default` is a tricky one because its whole purpose is to deal with the situation where the var might be missing, but the way smarty compiles it is something like `<?php $this->assign('to', ((is_array($_tmp=@$this->_tpl_vars['to'])) ? $this->_run_mod_handler('default', true, $_tmp, '_high') : smarty_modifier_default($_tmp, '_high'))); ?>` which uses the @ to try to silence the error. In addition to [templates/CRM/Core/DatePickerRange.tpl and templates/CRM/Core/DatePickerRangeWrapper.tpl](https://github.com/civicrm/civicrm-core/commit/f671f500b658aa772fedc682dbb320cf456bc904), the below are all instances of this. They could all be rewritten to not use the modifier, but the modifier itself is part of smarty so can't easily be prevented from being used in future.
* CRM_Core_InvokeTest::testInvokeDashboardForNonAdmin
* CRM/Contact/Page/DashboardDashlet.tpl - $communityMessages
* CRM_Core_SessionTest::testSetStatusWithTextOnly
* CRM/common/info.tpl
* CRM_Core_SessionTest::testSetStatusWithTitleOnly
* CRM/common/info.tpl
* CRM_Core_SessionTest::testSetStatusWithBoth
* CRM/common/info.tpl
### opendir
```
CRM_Utils_File::cleanDir
if ($dh = @opendir($target)) {
```
### unlink and mkdir
Various calls to `@unlink` and `@mkdir`
### httpclient
```
CRM_Utils_HttpClientTest::testFetchHttp_badOutFile
Exception: fopen(/ba/d/path/too/utput): Failed to open stream: No such file or directory
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:253
...\CRM\Utils\HttpClient.php:79 <---- see here
...\tests\phpunit\CRM\Utils\HttpClientTest.php:77
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:267
...\phpunit8:671
```
### angular changeset
Aside: The code in question here suggests phpQuery has its own debug setting which maybe could be turned on automatically in future if you have civi debugging enabled:
`$return = phpQuery::$debug === 2 ? this->document->loadHTML($markup) : @$this->document->loadHTML($markup);`
```
Civi\Angular\ChangeSetTest::testInsertAfter
Exception: DOMDocument::loadHTML(): htmlParseEntityRef: no name in Entity, line: 2
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:253
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\DOMDocumentWrapper.php:200
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\DOMDocumentWrapper.php:542
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\DOMDocumentWrapper.php:505
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\DOMDocumentWrapper.php:469
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\phpQueryObject.php:2066
...\vendor\electrolinux\phpquery\phpQuery\phpQuery\phpQueryObject.php:2010
...\tests\phpunit\Civi\Angular\ChangeSetTest.php:28
...\Civi\Angular\ChangeSet.php:59
...\Civi\Angular\ChangeSet.php:19
...\tests\phpunit\Civi\Angular\ChangeSetTest.php:39
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:267
...\phpunit8:671
```
* not ok 103 - Error: Civi\Angular\PartialSyntaxTest::testConsistencyExamples with data set 6 `('<div ng-if="a && b"></div>', '<div ng-if="a && b"></div>')`
* not ok 104 - Error: Civi\Angular\PartialSyntaxTest::testAllPartials
* These are the same issue as the changeset tests.
### testing of deprecations
These deliberately test deprecations
* not ok 223 - Error: Civi\Payment\PropertyBagTest::testSetContactIDLegacyWay
* not ok 227 - Error: Civi\Payment\PropertyBagTest::testSetCustomProp
### testGetfieldsHasTitle
```
api_v3_SyntaxConformanceTest::testGetfieldsHasTitle with data set 32 ('Entity')
Failure in api call for Entity getactions: include_once(api/v3/batch_create.php): failed to open stream: No such file or directory
0 ...\api\v3\Entity.php(18): CiviUnitTestCase->letssee(2, 'include_once(ap...', '\path\to\...', 18, Array)
1 ...\api\v3\Entity.php(18): include_once()
2 ...\api\v3\utils.php(2461): _civicrm_api3_entity_deprecation(Array)
3 ...\api\v3\utils.php(251): _civicrm_api3_deprecation_check('Entity', Array)
4 ...\Civi\API\Provider\ReflectionProvider.php(108): civicrm_api3_create_success(Array, Array, 'Entity', 'getactions')
5 ...\Civi\API\Kernel.php(149): Civi\API\Provider\ReflectionProvider->invoke(Array)
6 ...\Civi\API\Kernel.php(81): Civi\API\Kernel->runRequest(Array)
7 ...\api\api.php(22): Civi\API\Kernel->runSafe('Entity', 'getactions', Array)
8 ...\Civi\Test\Api3TestTrait.php(292): civicrm_api('Entity', 'getactions', Array)
9 ...\Civi\Test\Api3TestTrait.php(173): CiviUnitTestCase->civicrm_api('Entity', 'getactions', Array)
10 ...\tests\phpunit\api\v3\SyntaxConformanceTest.php(1662): CiviUnitTestCase->callAPISuccess('Entity', 'getactions', Array)
11 phar://.../phpunit8/phpunit/Framework/TestCase.php(1213): api_v3_SyntaxConformanceTest->testGetfieldsHasTitle('Entity')
12 ...\tests\phpunit\CiviTest\CiviUnitTestCase.php(267): PHPUnit\Framework\TestCase->runTest()
13 phar://.../phpunit8/phpunit/Framework/TestCase.php(889): CiviUnitTestCase->runTest()
14 phar://.../phpunit8/phpunit/Framework/TestResult.php(577): PHPUnit\Framework\TestCase->runBare()
15 phar://.../phpunit8/phpunit/Framework/TestCase.php(663): PHPUnit\Framework\TestResult->run(Object(api_v3_SyntaxConformanceTest))
16 phar://.../phpunit8/phpunit/Framework/TestSuite.php(481): PHPUnit\Framework\TestCase->run(Object(PHPUnit\Framework\TestResult))
17 phar://.../phpunit8/phpunit/Framework/TestSuite.php(481): PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult))
18 phar://.../phpunit8/phpunit/TextUI/TestRunner.php(462): PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult))
19 phar://.../phpunit8/phpunit/TextUI/Command.php(129): PHPUnit\TextUI\TestRunner->doRun(Object(PHPUnit\Framework\TestSuite), Array, Array, true)
20 phar://.../phpunit8/phpunit/TextUI/Command.php(101): PHPUnit\TextUI\Command->run(Array, true)
21 ...\phpunit8(671): PHPUnit\TextUI\Command::main()
22 {main}
Failed asserting that a integer is empty.
...\Civi\Test\Api3TestTrait.php:110
...\Civi\Test\Api3TestTrait.php:174
...\tests\phpunit\api\v3\SyntaxConformanceTest.php:1662
...\tests\phpunit\CiviTest\CiviUnitTestCase.php:267
...\phpunit8:671
```
### mailing testConcurrency
This doesn't run properly on windows even without the error_handler, and I don't feel like taking the time right now to deal with that:
not ok 1161 - Error: api_v3_JobProcessMailingTest::testConcurrency with data set #0 (array(20, 3, 10, 4, 1), array(2, 1), 4)https://lab.civicrm.org/dev/core/-/issues/2811Menu Angular error after upgrade2023-01-08T01:26:08ZStoobMenu Angular error after upgradeI do a fair amount of upgrades, and anecdotally after about a _half_ of recent upgrades, the CiviCRM menu fails to load. For half of those, a simple reload/refresh of the browser resolves the issue. For the other half, the civicrm/menu...I do a fair amount of upgrades, and anecdotally after about a _half_ of recent upgrades, the CiviCRM menu fails to load. For half of those, a simple reload/refresh of the browser resolves the issue. For the other half, the civicrm/menu/rebuild cache (or drush cc) must be cleared in order for the menu to appear. It is usually an Angular issue, seen attached.
Two questions, if these steps are required with such frequency can the upgrade process be improved by either:
1. clearing the cache automatically when the upgrade is finished
2. providing instructions and/or link on the upgrade screen to clear cache
![menu-load](/uploads/70b9bc8bcfbae067b5169d330bd16545/menu-load.png)
![upg](/uploads/19aa40afb6bda323d97cafe3c3cec5b3/upg.png)https://lab.civicrm.org/dev/core/-/issues/2846Improve validation of start and end dates for events, contribution pages and ...2023-11-23T07:09:32ZJKingsnorthImprove validation of start and end dates for events, contribution pages and more**Motivation**
There are holes in the validation of start and end dates on contribution pages.
eg:
- Create a new contribution page
- Enter a start date
- Enter an end 'time' but leave the date part empty:
![image](/uploads/e75411e618...**Motivation**
There are holes in the validation of start and end dates on contribution pages.
eg:
- Create a new contribution page
- Enter a start date
- Enter an end 'time' but leave the date part empty:
![image](/uploads/e75411e61813ee5ae55b808f49fa3637/image.png)
- Result is a fatal DB error.
This may seem like an edge-case, but it can also happen when you are 'scrolling' down the page, and accidentally 'scroll' on the time field. This sets a time.
The same is true for event pages event 'start' and 'end' dates.
The same is true for event registration open and close dates:
![image](/uploads/270044386b6b0895da079268d9147e8c/image.png)
The same is true for price fields, with the 'active on' and 'expire on', where setting a time without a date results in a fatal error and the endlessly spinning civi icon of doom.
And for price fields, there is currently no validation to ensure the 'expire on' is before the 'active on'.
Creating campaigns also:
![image](/uploads/c51922aa11942eafa7e58e6d153a5fae/image.png)
**Proposed solution**
I think the validation should really be in the date picker element somewhere, to ensure a date and time are both set - where we expect it.
But I also thought we could take this opportunity to standardise the validation of start and end dates in:
- Event pages (event start and end)
- Event pages (registration open/close)
- Contribution pages (start and end)
- Price fields (active on, expire on)
- Campaign (start, end dates)
I am working on this now.
**After**
![image](/uploads/968b1bafab430a8c7cbc2e5a684e1b01/image.png)
**Follow-up / part 2**
Validation at the API/BAO level?https://lab.civicrm.org/dev/core/-/issues/2848Activity tokens - case id - explain?2023-09-24T22:52:50ZeileenActivity tokens - case id - explain?@ayduns I recently fixed up some activity tokens that weren't working - but in the process @DaveD queried whether 'case_id' makes sense as a token on activity tokens since there could be more than one - just opening this put that to you....@ayduns I recently fixed up some activity tokens that weren't working - but in the process @DaveD queried whether 'case_id' makes sense as a token on activity tokens since there could be more than one - just opening this put that to you....
https://github.com/civicrm/civicrm-core/pull/21489#pullrequestreview-757039309https://lab.civicrm.org/dev/core/-/issues/2863Agree / document how tokens should be rendered from outside core2023-09-24T22:52:19ZeileenAgree / document how tokens should be rendered from outside coreIdeally we would have one recommended api way to do this. This would be a way that is usable via api explorer and all the ways we use the api and core code would be converted to do things using that same api method
There is [a documenta...Ideally we would have one recommended api way to do this. This would be a way that is usable via api explorer and all the ways we use the api and core code would be converted to do things using that same api method
There is [a documentation pr here](https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/962) - at the moment I don't think it meets this need.
**Permissions & non-workflow rendering**
There is a question as to whether the api would support non-workflow messaging (in which case the api would likely be more like `Message::render()`. The argument against is that the permissions are more complex. However some notes on that
1) the most common use case is probably when checkPermissions = FALSE in php code.
2) there is an appropriate permission already in core
```
render templates' => [
$prefix . ts('render templates'),
ts('Render open-ended template content. (Additional constraints may apply to autoloaded records and specific notations.)'),
],
```
3) currently the v3 MessageTemplate.send api allows access to the core function - it required a messageTemplateID to be set - but this can probably be faked to allow 'any text'https://lab.civicrm.org/dev/core/-/issues/2864Meta - token usage 5.43 standardisation effort2022-09-02T01:38:26ZeileenMeta - token usage 5.43 standardisation effortWe are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can ...We are doing a big token standardisation in 5.43 to solve multiple issues and blockers. By getting all these changes into 1 release we can also encourage people to specifically test that rc and any changes that they may need to make can be around the one release - documentation is [here](https://lab.civicrm.org/-/ide/project/documentation/docs/sysadmin/tree/case/-/docs/upgrade/version-specific.md/)
The goal is to consolidate such that
1. All token rendering is done through the token processor, token hooks are redundant and deprecated
2. All functions on CRM_Utils_Token are unused in core and deprecated except for those in the legacy BAO mailing classes.
3. We freeze, deprecate and phase out in legacy BAO classes in favour of flexmailer
4. All core classes that render tokens do so using consistent token naming & output conventions (rather than dates being formatted in various ways and fields sometimes outputting labels and other times values or name fields https://lab.civicrm.org/dev/core/-/issues/2650)
5. As much as possible we render via tokens rather than smarty in workflow message templates (that couples them with the entity rather than any form flow)
6. pdf & email & scheduled reminder tasks consistently offer the same entity-specific, contact & domain tokens
7. We can start to define obvious missing tokens such as `{domain.now}` (done), `{contribution.balance}` and formatted addresses
Task list / streams - this gives some idea of where things are progressing or blocked
- [x] standardise contact tokens to support/ advertise label style (e.g `{contact.prefix_id:label}`
- [x] make Contact tokens work with partial contact inputs - currently they only work with no contact array passed in, or an array with all possible values - currently blocked on a caching bug https://github.com/civicrm/civicrm-core/pull/21790
- [x] split contact tokens into own class & extend entity tokens like the other classes(blocked on stdise contact tokens)
- [x] switch contact tokens to use apiv4
- [x] work through standardising location token loading (this could go further - ie I had thought about adding loading for billing
- [x] date formatting standardisation [documentation](https://docs.civicrm.org/user/en/latest/common-workflows/tokens-and-mail-merge/#date) https://lab.civicrm.org/dev/core/-/issues/2746
- [x] locale sensitive standardised money formatting https://lab.civicrm.org/dev/core/-/issues/2638 & [pr](https://github.com/civicrm/civicrm-core/pull/21783)
- [x] switch badge tokens to use token processor
- [x] event tokens cleanup & loading fixes + remove unindexed query, move the `fee_amount` and `balance` tokens to the participant tokens class, fix event class to listen to either 'participantId' or 'eventId'. Participant class should run first & load eventId if required. Events should be cached in a static cache as it would be very rare to message thousands of events in one run.
- [x] Recurring contribution tokens work (currently availble in only the recurring_edit template with a pr to extend to cancelled recurring https://test.civicrm.org/job/CiviCRM-Core-PR/44632/
- [x] pdf letter is sane (https://lab.civicrm.org/dev/core/-/issues/2790) enough to extend to deliver tokens for other entities (participant https://lab.civicrm.org/dev/core/-/issues/780)
- [x] email letter extend to deliver tokens for other entities (membership & participant) (https://lab.civicrm.org/dev/core/-/issues/2862)
- [x] scheduled reminders supports participant tokens https://lab.civicrm.org/dev/core/-/issues/779
- [x] domain tokens are consistently available & render https://lab.civicrm.org/dev/core/-/issues/2838 - done but a gap on case tasks as they prioritise being able to filter tokens by case type
- [x] install flexmailer on new installs https://lab.civicrm.org/dev/core/-/issues/2836
- [ ] fix remaining places that process tokens other than via the payment processor
- [x] eliminate replaceGreeting - currently [blocked on apiv4 caching bug](https://github.com/civicrm/civicrm-core/pull/21790)
- [ ] fix CRM_Contact_BAO_Contact_Utils::updateGreeting to not call `getTokenDetails`
- [ ] confirm civicrm_api3_mailing_preview is replaced by flexmailer
- [x] remove getTokenDetails call from CRM_Contribute_Form_Task_PDFLetter - note this appears to be an expensive way to call the contact api & tokens are unused https://github.com/civicrm/civicrm-core/pull/21805
- [x] remove extra rendering in BAO_Pledge https://github.com/civicrm/civicrm-core/pull/21789
- [x] removed getTokenDetails from updatePledgeStatus https://github.com/civicrm/civicrm-core/pull/21847
- [x] remove getTokenDetails from transitionParticipants
- [ ] remove getTokenDetails from sendCancellation & cancelParticipant in selfsvc flow
- [ ] deliverGroup - replaced by flexmailer?
- [ ] MailingPage::preview - replaced by flexmailer?
- [ ] remove hook:;tokenValues from label task
- [ ] getAnonymousTokenDetails ???
- [ ] confirm BAO_Mailing::compose is only via flexmailer
- [x] communicate rc status via dev list
- [x] communicate changes via blog
- [ ] upgrade message https://lab.civicrm.org/dev/core/-/issues/2871
- [ ] docs update to these instructions / check it can still all be done https://docs.civicrm.org/dev/en/latest/step-by-step/create-custom-case-token/
**Currenly out of scope**
- [ ] Token.getlist api - until we have this we have an issue with audience-filtering and with switching tokens depending on the schema. Also case tasks still have to render tokens weirdly without this because they currently filter by case type with partial success https://lab.civicrm.org/dev/core/-/issues/2788
- [ ] Clarify how extensions should render tokens https://lab.civicrm.org/dev/core/-/issues/2863
- [ ] encourage flexmailer with status checks on existing installs https://lab.civicrm.org/dev/core/-/issues/2836
Analysis of current token rendering it core:
Method | Used in | Status
-- | -- | --
Use the token processor | scheduled reminders, civimail with flexmailer in use, activity pdf task | This is the method we are consolidating all paths to use.
Use the functions in CRM_Utils_Token | PDF tasks other than activity | Migrated (PRs on [membership](https://github.com/civicrm/civicrm-core/pull/21521) and [contribution](https://github.com/civicrm/civicrm-core/pull/21524). Also work on fixing up the [pdf task class](https://lab.civicrm.org/dev/core/-/issues/2790) supporting[add participant tokens](https://lab.civicrm.org/dev/core/-/issues/780) (once we’ve finished [the standardisation of those](https://github.com/civicrm/civicrm-core/pull/21587))
| Email tasks | Migrated to token processor. Added tokens for other entities (participant, membership https://lab.civicrm.org/dev/core/-/issues/1396) [also see](https://lab.civicrm.org/dev/core/-/issues/2862)
| Export tasks (when merging contacts in export) | Migrated to token processor
| Profile link rendering | Migrated to token processor
| Storing contact greetings (eg. email_greeting_display) | In progress. Currently resolving inconsistent token syntax and load issues as this needs to be able to use partial pre-loads
Use random ad hoc code | Event badges | [migrated](https://github.com/civicrm/civicrm-core/pull/21587) - this is the only place outside of scheduled reminders where event or participant tokens were used in core
| Send SMS task (BAO_Activity::sendSMS) | migrated
| Labels task CRM_Contact_Form_Task_Label | argh
| CRM_Event_Form_SelfSvcTransfer | Still pretty argh - but this is a workflow message template so the fix is a bit different
| Transition participants | Also a workflow - possibly redundant as seems to be mostly about contact tokens which are resolved in the send function anyway
| Pledge acknowledgements | Also appear to be redundant
| CiviMail with flexmailer not used (many functions & classes) | Push the switch to flexmailer.
| Various unsubscribe / resubscribe type actions | Probably out of scope for this roundhttps://lab.civicrm.org/dev/core/-/issues/2869Api4 explorer - Clicking the Debug checkbox on or off doesn't change the samp...2023-09-30T05:15:09ZDaveDApi4 explorer - Clicking the Debug checkbox on or off doesn't change the sample codeI think it should add setDebug()?
Issue needs debugging. I tried clicking the debug checkbox to ... oh, right.
FYI @colemanwI think it should add setDebug()?
Issue needs debugging. I tried clicking the debug checkbox to ... oh, right.
FYI @colemanwhttps://lab.civicrm.org/dev/core/-/issues/2796Regression? No way to change the tax rate and tax amount using buildamount hook2023-11-24T22:45:42ZPradeep Nayakpradpnayak@gmail.comRegression? No way to change the tax rate and tax amount using buildamount hookIn 5.35.x LineItem create api allowed one to set a different tax_amount in api params for a financial type that had tax enabled. The tax amount could then be calculated programatically based on region, location etc instead of using the c...In 5.35.x LineItem create api allowed one to set a different tax_amount in api params for a financial type that had tax enabled. The tax amount could then be calculated programatically based on region, location etc instead of using the config specified in the UI.
```
civicrm_api3('LineItem', 'create', [
'entity_id' => 284,
'qty' => 1,
'unit_price' => 100,
'line_total' => 100,
'entity_table' => "civicrm_contribution",
'tax_amount' => 20,
'financial_type_id' => "Donation",
'price_field_id' => "contribution_amount",
'price_field_value_id' => "single",
]);
```
Another way of calculating tax on online forms was using the [buildamount hook](https://docs.civicrm.org/dev/en/latest/hooks/hook_civicrm_buildAmount/)
```
function advanceinvoicing_civicrm_buildAmount($pageType, &$form, &$amount) {
$amount[$priceFieldId]['options'][$priceFieldValueId]['tax_rate'] = 10;
$amount[$priceFieldId]['options'][$priceFieldValueId]['tax_amount'] = 0.1;
}
```
A recent [change](https://github.com/civicrm/civicrm-core/commit/e967ce8fe2b#diff-a16d4d7449cf5f3a0616d1d282a32f27ab6d3f7d2726d076c02ad1d4d655af41R56) to the line item create function has stopped allowing tax amount to be set programmatically.
Is this intentional? Is there any way to support different tax rate for single financial type?https://lab.civicrm.org/dev/core/-/issues/2899DB error on updating multiple cases from the search form2022-09-20T10:50:03ZjitendraDB error on updating multiple cases from the search formTo replicate -
1. Navigate to https://dmaster.demo.civicrm.org/civicrm/case/search?reset=1
2. Search for cases. Select all results (ts_all radio button) => Update multiple cases action
![image](/uploads/fcbb716228f064c5cde2f8231a534e15...To replicate -
1. Navigate to https://dmaster.demo.civicrm.org/civicrm/case/search?reset=1
2. Search for cases. Select all results (ts_all radio button) => Update multiple cases action
![image](/uploads/fcbb716228f064c5cde2f8231a534e15/image.png)
3. Select Profile on the next step and hit Continue;
![image](/uploads/cb5fb171efd98f5014e5c822f367f717/image.png)https://lab.civicrm.org/dev/core/-/issues/2915Proposal: Remove the Group Visibility terminology "Expose Publicly" as it has...2024-02-08T23:31:11Zjustinfreeman (Agileware)Proposal: Remove the Group Visibility terminology "Expose Publicly" as it has unintended connotations and would be good to standardise the terminology to describe "Group Visibility" and its optionsThis is a proposal to:
1. Remove the **Group Visibility** terminology _Expose Publicly_ as it has unintended connotations (as pointed out to me today by a Lawyer) and
2. Would be good to standardise the terminology to describe **Group Vi...This is a proposal to:
1. Remove the **Group Visibility** terminology _Expose Publicly_ as it has unintended connotations (as pointed out to me today by a Lawyer) and
2. Would be good to standardise the terminology to describe **Group Visibility** and its options. Ideally making it more obvious as to the purpose of each option.
As described on https://docs.civicrm.org/user/en/latest/organising-your-data/groups-and-tags/#visibility
* **Group Visibility**: **Public pages / Expose Publicly** - if you want to allow contacts to join and remove themselves from this group via Registration and Account Profile forms
* **Group Visibility**: **User and User Admin Only** - if membership in this group is controlled only by authorised CiviCRM users
As I understand it, this option _only_ applies to **Mailing List, Group Types**.
# Expose Publicly
![expose-publicly](/uploads/8d44c054f82a86d8fe92f8f7cd535663/expose-publicly.png)
# Group Visibility and options
![visbility](/uploads/7c4069b330d413230996fab273d242f4/visbility.png)https://lab.civicrm.org/dev/core/-/issues/2928Expand Gender options?2024-01-16T16:07:07ZJoeMurrayExpand Gender options?Overview
----------------------------------------
Currently core has a Gender field with options in tarball of: Male, Female, Other. There has been a great deal of social change in this area with widespread use of additional terms. In ad...Overview
----------------------------------------
Currently core has a Gender field with options in tarball of: Male, Female, Other. There has been a great deal of social change in this area with widespread use of additional terms. In addition, it might be useful to have default questions and options for whether a contact is transgendered and what pronouns they would like.
Example use-case
----------------------------------------
1. Click on **Contacts -> New Individual**.
1. Enter **First Name** and **Last Name** and click **Save**.
Current behaviour
----------------------------------------
![2021-10-26_11-46-51](/uploads/f5985494ccec6ce0c0c143cc688014b2/2021-10-26_11-46-51.png)
Proposed behaviour
----------------------------------------
I'm going to break up the proposal into several parts that can be discussed and possibly approved separately.
1. Add to the list of gender options after Other: Prefer not to specify.
2. After Male and before Other, insert new gender options: Non-binary, Genderfluid.
2. Change from Female to Female/woman and from Male to Male/man.
3. Change the Gender field from single select to multiselect.
4. Add a new core field under Gender: Are you transgender? Yes, No, Prefer not to specify.
5. Add a new core field under the above: What pronouns do you use? He/him/his, She/her/hers, They/them/theirs, Zie/hir/hirs, Other.
6. Make the field for Pronouns multiselect.
An alternative that might make sense is to do some of this in an extension. Another alternative would be to assist with best practices by providing questions or options but defaulting to disabled.
Comments
----------------------------------------
Here is a note from an organization that works in this area in response to my query on behalf of a client for a longer list of genders:
> Coming up with a universal list of terminology to describe trans people can be difficult, since the exact terms used to describe the concepts involved varies dramatically from language to language and culture to culture. While trans communities in Canada and the US tend to use the same terminology, it can vary even with other majority Anglophone countries, to say nothing of the rest of the world.
>
> There is a lot of discussion out there in the data world about how best to record sexual orientation and gender identity (or “SOGI”) data, and best practices depend on the field in question. That said, a good starting point is to divide things into four questions framed like so:
>
> 1. What is your gender? (select all that apply)
> a. Female/woman
> b. Male/man
> c. Non-binary
> d. Genderfluid
> e. Other
> f. Prefer not to specify
>
> 2. Are you transgender?
> a. Yes
> b. No
> c. Prefer not to specify
>
> 3. What pronouns do you use? (select all that apply)
> a. He/him/his
> b. She/her/hers
> c. They/them/theirs
> d. Zie/hir/hirs
> e. Other
>
> Breaking things out like this has several advantages. First, it recognizes that “transgender” is not itself a gender identity, but is an umbrella category that includes certain men, women, and non-binary and genderfluid people. It simplifies the first question, keeping a shorter list for people to have to scroll through to find the proper term to describe themselves. It also allows organizations to easily modify the list to fit their specific use cases – an organization working in Samoa or with a significant Samoan population could add “fa’afafine” to the list while one dealing with a significant Native/First Nations population could add “two-spirit.”
>
> It also permits trans individuals to interact with the organization without either lying or outing themselves, by not forcing them to select either (for example) “cisgender woman” or “transgender woman.” (If you’re not familiar with the term, ‘cisgender’ is a word indicating a person who is not ‘transgender’.) In fact, it also allows the organization to sort their data in such a way that they don’t unnecessarily out trans people to staff and volunteers, by allowing them to pull a list of all women or all men that doesn’t automatically flag whether or not they are transgender.
>
> In addition, it reflects the fact that not all non-binary and genderfluid people consider themselves to fit under the “transgender” umbrella – this is a complicated issue that’s largely unknown outside of the larger LGBTQ community, and is often disregarded by organizations working with these communities.
>
> Finally, it flags the most useful information – what pronouns to use when referring to the person – into a separate easy to reference field that can be included on a display screen or other record without being specifically tied to someone’s transgender status.
>
> Of course, this is only scratching the surface of the issues that might be relevant to a specific organization’s needs. For instance, they should also consider having a separate “legal name” and “preferred name” field – the former is something that would only be referenced when writing checks or otherwise transacting business in the person’s legal name, but have the “preferred name” be what actually displays for all other interactions with the person. This would also allow cisgender people who primarily go by a short form, initials, or nickname to have that be displayed instead of their full name.
>
> A medical organization, meanwhile, might need to collect information on whether or not someone is intersex, or what their “gender assigned at birth” is, and there can be a lot of ways to capture this information based on how the organization interacts with its staff and patients.
>
> For additional discussion on collecting SOGI data, with multiple examples, please take a look at these resources:
>
> https://www.thetrevorproject.org/wp-content/uploads/2021/07/Measuring-Youth-Sexual-Orientation-and-Gender-Identity.pdf
>
> https://williamsinstitute.law.ucla.edu/quick-facts/survey-measures/JoeMurrayJoeMurrayhttps://lab.civicrm.org/dev/core/-/issues/2929Geocoding failures kill contributions2023-10-23T16:08:42ZandrewcormickdockeryGeocoding failures kill contributionsOverview
----------------------------------------
Last night between about 6pm and midnight AEDT, it appears that Nominatim (the Open Street Map API provider) had an intermittent outage. Some requests were receiving a 502 error. We hav...Overview
----------------------------------------
Last night between about 6pm and midnight AEDT, it appears that Nominatim (the Open Street Map API provider) had an intermittent outage. Some requests were receiving a 502 error. We have configured this as our Geocoding provider in CiviCRM under /civicrm/admin/setting/mapping?reset=1
People who were using CiviCRM in ways which caused the Geomapping provider to be called were receiving fatal errors as a result. This included people who were making donations (i.e. entering their address on the form which in turn was handed to the Geomapping provider for geocoding).
Current behaviour
----------------------------------------
When a request to the Geomapping provider fails (for example with a 502 error), the generating transaction within CiviCRM returns a fatal error and does not complete. This includes financial transactions, and indeed any administrative function that involves adding or updating a contact's address.
Expected behaviour
----------------------------------------
A Geomapping provider failure should never cause financial transactions to fail. I can't think of any reason why adding or updating an address should fail either. Fatal errors should only occur when an issue occurs which genuinely prevent a transaction from succeeding. A payment can still be processed regardless of whether that person's address can be geocoded or not. I expect a 502 Bad Gateway error from Nominatim would be logged, but never cause an address to fail to be added/updated. Obviously that would occur without the latitude/longitude being recorded. I can't even see the utility in informing the user. It's a system admin problem.
Environment information
----------------------------------------
* __CiviCRM:__ _5.35.2_
* __PHP:__ _7.3__
* __CMS:__ _Drupal 7.82_
* __Database:__ _MariaDB 10.4.21_
* __Web Server:__ _Nginx 1.10.3_
Comments
----------------------------------------
_Anything else you would like the reviewer to note._https://lab.civicrm.org/dev/core/-/issues/2935Print/Merge produces blank results when using a docx or odt template file2023-07-27T10:57:01ZChrisHardiePrint/Merge produces blank results when using a docx or odt template fileOverview
----------------------------------------
When trying to generate thank you letters or other printed/merged documents, and when uploading a .docx or .odt template file to use, CiviCRM generates blank output (e.g. a blank PDF file...Overview
----------------------------------------
When trying to generate thank you letters or other printed/merged documents, and when uploading a .docx or .odt template file to use, CiviCRM generates blank output (e.g. a blank PDF file, a blank .docx file). When generating the same document without uploading a template file, CiviCRM successfully generates output that contains the desired document body.
Reproduction steps
----------------------------------------
1. Find one or more contacts, or contributions, or other entities that can be printed/merged
1. Select an action that involves print/merge (e.g. thank you letters)
1. Upload a template document to use for the printed/merged output, in .odt or .docx format
1. Generate the merged document in any format (pdf, docx, html, odt)
1. Observe that the resulting document is blank
Expected behaviour
----------------------------------------
When a merged/generated document is produced, it should use the uploaded template and the specified document body, and not be blank.
Environment information
----------------------------------------
* __Browser:__ Attempted with Safari 15.1 and Chrome 96.0.4664.27 on macOS
* __CiviCRM:__ 5.42.0
* __PHP:__ 7.4
* __CMS:__ WordPress 5.8.1https://lab.civicrm.org/dev/core/-/issues/2940Remove requirement to have a bounce email account setup2022-09-30T20:45:40ZseamusleeRemove requirement to have a bounce email account setupWith some more modern SMTP providers bounce email accounts may not be necessary for CiviMail Bounces to work because e.g. with Amazon SES / Sendgrid you can get the bounces processed via webhooks instead.
However within the code base we...With some more modern SMTP providers bounce email accounts may not be necessary for CiviMail Bounces to work because e.g. with Amazon SES / Sendgrid you can get the bounces processed via webhooks instead.
However within the code base we assume we need a bounce account to generate stuff like [no-reply email address](https://github.com/civicrm/civicrm-core/blob/master/CRM/Core/BAO/Domain.php#L364) or when we are getting [message ids](https://github.com/civicrm/civicrm-core/blob/513f6a469927c5dc5a41905feb7c3cff2cf15f0b/CRM/Campaign/BAO/Petition.php#L548)
It would be nice to get rid of this reliance on it being the default mail account which is the bounce processing one https://github.com/civicrm/civicrm-core/blob/513f6a469927c5dc5a41905feb7c3cff2cf15f0b/CRM/Core/BAO/MailSettings.php#L59
cc @JoeMurrayMonish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/2945Make CRM_Core_Smarty::singleton() use Civi::statics instead of a static var2023-10-13T05:11:14ZDaveDMake CRM_Core_Smarty::singleton() use Civi::statics instead of a static varDuring unit tests, static vars can be problematic because they don't get reset between tests. There's a point within smarty where if a test fails at that point then its secure mode doesn't get reset. It's come up a couple times and then ...During unit tests, static vars can be problematic because they don't get reset between tests. There's a point within smarty where if a test fails at that point then its secure mode doesn't get reset. It's come up a couple times and then all the tests that use smarty on certain forms fail after it. It's not easy to track down if you haven't seen it before.
There's an argument civi code should be smarty secure-mode-compliant instead, but it isn't at the moment.https://lab.civicrm.org/dev/core/-/issues/2957Possibility to add a status message (alert box) after screen submission in Fo...2023-09-13T10:33:15ZErikHommelPossibility to add a status message (alert box) after screen submission in FormBuilderOverview
----------------------------------------
I would like to be able to send a status message (for example: Activity saved etc.) once a form that I have created with the FormBuilder is submitted.
Current behaviour
-----------------...Overview
----------------------------------------
I would like to be able to send a status message (for example: Activity saved etc.) once a form that I have created with the FormBuilder is submitted.
Current behaviour
----------------------------------------
At the moment I have created a form with FormBuilder which I use to add data, for example an activity. Right now once I have entered data and hit the "submit" button I am returned to whatever page I have entered at the Post Submit Page, but I do not get confirmation that my data has been saved.
Proposed behaviour
----------------------------------------
I would like to be able to specify that I want a status message when I have created, updated or deleted data.https://lab.civicrm.org/dev/core/-/issues/2963Export file empty when utilizing "Display results as" and searching for custo...2023-01-10T16:46:00Zfabian_SYSTOPIAExport file empty when utilizing "Display results as" and searching for custom dataOverview
----------------------------------------
When exporting data after an advanced search the exported file is empty if:
* "Display results as" was set to "related contacts" AND
* one search criterium was a custom field
The expor...Overview
----------------------------------------
When exporting data after an advanced search the exported file is empty if:
* "Display results as" was set to "related contacts" AND
* one search criterium was a custom field
The export preview shows all information correctly but the exported file does not contain any data.
Reproduction steps
----------------------------------------
1. Create a contact custom field (if you do not have one already, ex: Favorite Color: A/B/C)
1. Perform an advanced search, set "Display results as" to "related contacts" and any relationship type (ex: 'Employee Of')
1. Choose at least one custom filed as a search criterium (the search result must contain at least one contact (you may have to create/edit a test contact so that the Custom Field has a value)
1. Export the contact(s) from the search result
Current behaviour
----------------------------------------
The export file is empty although the export's preview shows all information correctly (in case you choose data to export and not export primary fields). Note that the export will work as intended if you do not set "Display results as" to "related contacts" or do not have at least one search criterium was a custom field.
Expected behaviour
----------------------------------------
Export file should contain data of selected contacts
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ all
* __CiviCRM:__ Master/5.35.x, probably also earlier versions
Comments
----------------------------------------
Possibly unrelated but referencing #2873 and assigning to @monish.deb as there is currently a fix for the export feature in progress.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/2966Duplicate contacts when custom field value is 'email', 'phone', etc.2023-03-18T04:08:01Zrita_compucorpDuplicate contacts when custom field value is 'email', 'phone', etc.**Issue**:
when existing contacts are donating (without being logged in) with the same details (email, first name, last name) that are registered in the system, most of the times there is a new contact created instead of merging it to t...**Issue**:
when existing contacts are donating (without being logged in) with the same details (email, first name, last name) that are registered in the system, most of the times there is a new contact created instead of merging it to the existing contact automatically.
**Investigation**:
When a custom field with a checkbox had its title and value set to ‘email’ or 'phone' and the unsupervised dedupe rule was prepared to check for email or phone as well causing the problem. So the problem occurs when:
- the unsupervised dedupe rule has email in it AND
- the custom field’s value is set to ‘email’
However not only the email option can cause duplications, but if you prepare an:
- unsupervised dedupe rule with phone ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24](/uploads/758592a6d5330c1133dcd1206e714daa/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24.png) AND
- the custom field’s value is set to ‘phone’ ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12](/uploads/df40ee69692fda55f51e946b2c841b9d/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12.png)
And even though the clients enter the same email/phone numbers on the donation form, the dedupe rule will be skipped and a new contact with the same information will be created. Probably this can be reproduced with many fields in the system, but only tried these two.
I checked on a few **different civi versions:**
- 4.7.26 - couldn’t reproduce
- 5.35.2 - issue reproducible
- 5.45.alpha1 (core demo civi) - issue reproducible
**Reproduction steps:**
1. create an unsupervised individual rule with the following details: ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24](/uploads/758592a6d5330c1133dcd1206e714daa/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_34_24.png)
2. create a custom field set for Contacts with checkboxes like this: ![screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12](/uploads/df40ee69692fda55f51e946b2c841b9d/screenshot-dmaster.demo.civicrm.org-2021.11.22-08_35_12.png)
3. add the custom field set that you created in previous step to a profile that can be added to a contribution page
4. create a new (or use an old) contribution page and add the profile that you created in previous step to it in the Profiles tab
5. create a new contact, and make sure it has a First name, Last name and Phone number
6. open the contribution page as anonymous and donate to it, using the exact same First name, Last name and Phone number that you entered in previous step AND select the ‘phone’ option at the custom field you created → _after you submit the donation, a new contact will be created instead of merging it to the existing one, which is wrong behaviour_
7. donate again on the same form, with the same details as before, but instead of selecting ‘phone’ select either the ‘email’ or ‘post’ options → _the contribution will be merged to the original contact, which is the correct behaviour. It is because email and post are not added to the unsupervised dedupe rule in this case._
I think most of the time civi admins create custom fields with values set as numbers, but in some cases they might configure it to be the same as the label name and in that case duplications might occur. I think it might be happening when the value of the custom field is the same as the name of the field in the database (just a thought, can't say for sure).https://lab.civicrm.org/dev/core/-/issues/2976Feature Request: Form Builder and Permissions - Provide ability to restrict a...2023-11-02T12:39:11Zjustinfreeman (Agileware)Feature Request: Form Builder and Permissions - Provide ability to restrict access to selected Roles rather than just specific Permission GrantsWould be great if the Form Builder, Permissions had the ability to restrict access to selected Roles rather than just specific Permission Grants.
Roles being far more flexible, can be automatically granted and removed for user accounts ...Would be great if the Form Builder, Permissions had the ability to restrict access to selected Roles rather than just specific Permission Grants.
Roles being far more flexible, can be automatically granted and removed for user accounts and this is a common pattern on Member websites. Whereas Permission Grants are not used in this way.
For example: When a user with the Member role views a Form Builder page, they are granted access to use the Form and view the Search Results because they have the Member role.
![Screenshot_20211202_130808](/uploads/25c2fbadece5eea59beb05d7b269e937/Screenshot_20211202_130808.png)
Agileware Ref: CIVICRM-1899https://lab.civicrm.org/dev/core/-/issues/2987API call Contribution.repeattransaction fails with recurring contributions th...2023-03-20T01:51:15ZandrewcormickdockeryAPI call Contribution.repeattransaction fails with recurring contributions that have a one-to-one custom group record attachedOverview
----------------------------------------
Re: https://chat.civicrm.org/civicrm/pl/i8fht46isjn5uj4dbdosrqpqmr
The API call Contribution.repeattransaction appears to write a custom group record twice when completing a transaction....Overview
----------------------------------------
Re: https://chat.civicrm.org/civicrm/pl/i8fht46isjn5uj4dbdosrqpqmr
The API call Contribution.repeattransaction appears to write a custom group record twice when completing a transaction. On the second attempt, a database error caused by the clashing unique key is generated. The transaction appears to end up being recorded correctly, however the fatal error means that processes which call this function repeatedly end up not completing. Our use case involves finding all Eway transactions whose next scheduled contribution date is today, and then calling Contribution.repeattransaction for each one.
Reproduction steps
----------------------------------------
1. Ensure a one-to-one custom group exists against contributions.
1. Ensure a recurring contribution exists with a record in that custom group against the transaction.
1. Run `drush cvapi Contribution.repeattransaction contribution_status_id=Completed total_amount=<some value> contribution_recur_id=<contribution_recur_id in previous step> trxn_id=<random unique number>`
Current behaviour
----------------------------------------
Failure message appears: `DB Error: already exists`; process terminates. Record appears to be correctly produced in CiviCRM.
Log details:
```
Dec 07 12:40:11 [error] $Fatal Error Details = Array
(
[callback] => Array
(
[0] => CRM_Core_Error
[1] => exceptionHandler
)
[code] => -5
[message] => DB Error: already exists
[mode] => 16
[debug_info] => INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_field_2`,`custom_field_3`,`custom_field_4`,`entity_id` ) VALUES ( 1,'Some Text','','22943426',123456 ) [nativecode=1062 ** Duplicate entry '123456' for key 'unique_entity_id']
[type] => DB_Error
[user_info] => INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_field_2`,`custom_field_3`,`custom_field_4`,`entity_id` ) VALUES ( 1,'Some Text','','22943426',123456 ) [nativecode=1062 ** Duplicate entry '123456' for key 'unique_entity_id']
[to_string] => [db_error: message="DB Error: already exists" code=-5 mode=callback callback=CRM_Core_Error::exceptionHandler prefix="" info="INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_field_2`,`custom_field_3`,`custom_field_4`,`entity_id` ) VALUES ( 1,'Some Text','','22943426',123456 ) [nativecode=1062 ** Duplicate entry '123456' for key 'unique_entity_id']"]
)
Dec 07 12:40:11 [debug] $backTrace = #0 /path/to/civi/root/civicrm/CRM/Core/Error.php(942): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 /path/to/civi/root/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php(922): CRM_Core_Error::exceptionHandler(Object(DB_Error))
#2 /path/to/civi/root/civicrm/vendor/pear/db/DB.php(997): PEAR_Error->__construct("DB Error: already exists", -5, 16, (Array:2), "INSERT INTO civicrm_value_contribution_custom_group_1 ( `hide_from_slider_182`,`custom_...")
#3 /path/to/civi/root/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php(575): DB_Error->__construct(-5, 16, (Array:2), "INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#4 /path/to/civi/root/civicrm/vendor/pear/pear-core-minimal/src/PEAR.php(223): PEAR->_raiseError(Object(DB_mysqli), NULL, -5, 16, (Array:2), "INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", "DB_Error", TRUE)
#5 /path/to/civi/root/civicrm/vendor/pear/db/DB/common.php(1928): PEAR->__call("raiseError", (Array:7))
#6 /path/to/civi/root/civicrm/vendor/pear/db/DB/mysqli.php(936): DB_common->raiseError(-5, NULL, NULL, "INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", "1062 ** Duplicate entry '123456' for key 'unique_entity_id'")
#7 /path/to/civi/root/civicrm/vendor/pear/db/DB/mysqli.php(406): DB_mysqli->mysqliRaiseError()
#8 /path/to/civi/root/civicrm/vendor/pear/db/DB/common.php(1234): DB_mysqli->simpleQuery("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#9 /path/to/civi/root/civicrm/packages/DB/DataObject.php(2696): DB_common->query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#10 /path/to/civi/root/civicrm/packages/DB/DataObject.php(1829): DB_DataObject->_query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#11 /path/to/civi/root/civicrm/CRM/Core/DAO.php(468): DB_DataObject->query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...")
#12 /path/to/civi/root/civicrm/CRM/Core/DAO.php(1621): CRM_Core_DAO->query("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", TRUE)
#13 /path/to/civi/root/civicrm/CRM/Core/BAO/CustomValueTable.php(271): CRM_Core_DAO::executeQuery("INSERT INTO civicrm_value_contribution_custom_group_1 ( `custom_field_1`,`custom_...", (Array:5))
#14 /path/to/civi/root/civicrm/CRM/Core/BAO/CustomValueTable.php(391): CRM_Core_BAO_CustomValueTable::create((Array:2), "create")
#15 /path/to/civi/root/civicrm/CRM/Core/BAO/CustomValueTable.php(411): CRM_Core_BAO_CustomValueTable::store((Array:5), "civicrm_contribution", 123456, "create")
#16 /path/to/civi/root/civicrm/CRM/Core/DAO.php(2022): CRM_Core_BAO_CustomValueTable::postProcess((Array:5), "civicrm_contribution", 123456, "Contribution", "create")
#17 /path/to/civi/root/civicrm/CRM/Contribute/BAO/Contribution.php(2394): CRM_Core_DAO->copyCustomFields(123455, 123456)
#18 /path/to/civi/root/civicrm/CRM/Contribute/BAO/Contribution.php(4000): CRM_Contribute_BAO_Contribution::repeatTransaction((Array:10), (Array:18))
#19 /path/to/civi/root/civicrm/api/v3/Contribution.php(687): CRM_Contribute_BAO_Contribution::completeOrder((Array:10), "2269", NULL, NULL)
#20 /path/to/civi/root/civicrm/api/v3/Contribution.php(635): _ipn_process_transaction((Array:7), Object(CRM_Contribute_BAO_Contribution), (Array:10), (Array:10))
#21 /path/to/civi/root/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_contribution_repeattransaction((Array:7))
#22 /path/to/civi/root/civicrm/Civi/API/Kernel.php(149): Civi\API\Provider\MagicFunctionProvider->invoke((Array:8))
#23 /path/to/civi/root/civicrm/Civi/API/Kernel.php(81): Civi\API\Kernel->runRequest((Array:8))
#24 /path/to/civi/root/civicrm/api/api.php(132): Civi\API\Kernel->runSafe("contribution", "repeattransaction", (Array:5))
#25 /path/to/civi/root/civicrm/drupal/drush/civicrm.drush.inc(1581): civicrm_api3("Contribution", "repeattransaction", Array:5))
#26 /usr/local/src/drush/includes/command.inc(422): drush_civicrm_api("Contribution.repeattransaction", "contribution_status_id=Completed", "total_amount=11.00", "contribution_recur_id=22663", "trxn_id=aa11bb22cc33dd47")
#27 /usr/local/src/drush/includes/command.inc(231): _drush_invoke_hooks((Array:38), (Array:5))
#28 /usr/local/src/drush/includes/command.inc(199): drush_command("Contribution.repeattransaction", "contribution_status_id=Completed", "total_amount=11.00", "contribution_recur_id=22663", "trxn_id=aa11bb22cc33dd47")
#29 /usr/local/src/drush/lib/Drush/Boot/BaseBoot.php(67): drush_dispatch((Array:38))
#30 /usr/local/src/drush/includes/preflight.inc(67): Drush\Boot\BaseBoot->bootstrap_and_dispatch()
#31 /usr/local/src/drush/drush.php(12): drush_main()
#32 {main}
```
Expected behaviour
----------------------------------------
Fatal error not raised; no attempt to write the exact same record twice.
I have mitigated this behaviour with the following patch, but this is clearly a temporary solution and the real root cause needs to be found:
```
diff --git a/CRM/Core/BAO/CustomValueTable.php b/CRM/Core/BAO/CustomValueTable.php
index 8556c4ec21..6773b994c4 100644
--- a/CRM/Core/BAO/CustomValueTable.php
+++ b/CRM/Core/BAO/CustomValueTable.php
@@ -55,7 +55,7 @@ class CRM_Core_BAO_CustomValueTable {
$hookOP = 'edit';
}
else {
- $sqlOP = "INSERT INTO $tableName ";
+ $sqlOP = "INSERT IGNORE INTO $tableName ";
$where = NULL;
$hookOP = 'create';
}
```
Environment information
----------------------------------------
* __Browser:__ _Firefox 94.0.2_
* __CiviCRM:__ _5.43.2_
* __PHP:__ _7.3_
* __CMS:__ _Drupal 7.82_
* __Database:__ _MariaDB 10.4.21_
* __Web Server:__ _Nginx 1.10.3_https://lab.civicrm.org/dev/core/-/issues/2990CiviCampaign - Move data into a single table2024-03-11T20:47:07ZcolemanwCiviCampaign - Move data into a single tableBackground
------------
CiviCampaign is an optional component (disabled by default on new installs). It allows contributions, events, etc. to be associated with a campaign.
Campaigns are basically like tags. The campaign itself contain...Background
------------
CiviCampaign is an optional component (disabled by default on new installs). It allows contributions, events, etc. to be associated with a campaign.
Campaigns are basically like tags. The campaign itself contains a bit more data (type, start & end date, description, goal) than a Tag, but the mechanism for linking an entity with a campaign is a very simple foreign key.
How it works now
-------------
"Tagging" an entity with a campaign works via a column on that entity's table. E.g. `civicrm_contribution_page.campaign_id`. There are currently 10 entities with such a column, allowing them to be "tagged" with a campaign_id.
The problem
-----------
This design involves tight coupling between CiviCampaign and other components, as it involves adding a column to their tables. It also limits the possibilities for which entities can have a campaign_id.
The solution
----------
There is another common pattern in CiviCRM, which is to add a small "bridge" table to join two entities, which is a good alternative to adding a column. `civicrm_entity_tag` is one example. It includes `tag_id`, `entity_id` and `entity_table` columns, and an OptionGroup `tag_used_for` which lists all the possible values for `entity_table`. Importantly, it's easily added-on by extensions; if an extension provides a new entity type which should be taggable, is just adds it to the option list.
The migration
------------
If CiviCampaign were a greenfield project designed today, we wouldn't think twice about structuring the data with a bridge table instead of littering our database with `campaign_id` columns. As a brownfield problem, fixing the structure may be more trouble than it's worth, but I thought I'd at least articulate how it might be possible:
- APIv4 has a new concept of "Extra" calculated fields, which could provide a faux `campaign_id` column for backward compatibility.
- CiviCampaign could implement a post-hook to save campaign_id if it's passed into "create" params. This would keep create and update working as if that column still exists.
- APIv3 would require some hacking to make a pseudo-field to fill in for a missing `campaign_id`.
Random musings
-----------
While researching this, I stumbled across a table called `civicrm_campaign_group`. I have no idea what it does but it looks very similar to how I imagined the new bridge table would be. It includes a `campaign_id`, `entity_id` and `entity_table` column. Why does this table exist??https://lab.civicrm.org/dev/core/-/issues/2991Proposal: rework name fields to make name entity2023-12-13T09:20:59ZAndie HuntProposal: rework name fields to make name entity## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handli...## Overview
A recent [discussion with Guy](https://chat.civicrm.org/civicrm/pl/wwqowz635fbpifio7on4e8r8dw) and #2883, plus the ongoing mayhem of organization names and personal experience with a variety of great and terrible name handling systems, have led me to think that the solution to names isn't creating more or better-labeled fields for more types of names but rather to make name its own entity akin to phone, email, address, website, etc.
Once name is an entity, each contact could have multiple names, one primary, but with a bunch of possible types:
- legal name
- trade name
- nickname
- former name
- name in other language/culture
- name in an external system
- common mistakes
There could also be a bunch of name renderings - ways the name is displayed or inserted:
- sort name
- display name
- name for greetings
- name for listings
The latter could generate the existing greetings and potentially others. In particular, I think there's a real gap for how you insert someone's name in emails where you don't want "Dear", "Hi", or any other greeting as such.
Finally, the duplicate matching process could look for any of the possible names, cutting down on duplicates.
## Example use-case
### Organization has multiple names
An organization has their corporate name, colloquial name, and the name of their main product. These all refer to the same contact, but employees might fill the current employer field as one of the various ways. As an admin, you want to avoid having to merge them all the time. As a routine user, you will want to know all the various ways an organization is termed. As a bookkeeper, you will want to send invoices, checks, receipts, etc. with the correct entity name.
In this case, there are three name entities:
| Organization name | Name type | Is primary |
| ------------------------ | --------- | ---------- |
| Widget Productions | General | true |
| Widget Productions, Inc. | Legal | false |
| WidgetPro | General | false |
| Widgetizer | Product | false |
There would be a couple of name renderings:
| Rendering type | Value |
| -------------- | ------------------------ |
| Display Name | Widget Productions |
| Sort Name | Widget Productions |
| Billing name | Widget Productions, Inc. |
### Minister has variety of prefixes
Someone's name is styled differently when it is listed versus when they are addressed. In this case, a minister is styled "The Reverend Sally Jones" or "Rev. Sally Jones" when listed somewhere but her salutation is "Ms. Jones".
I think she'd have only one entry as a name entity but several renderings:
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------------|------------|-------------|-----------|--------|-----------|------------|
| Ms. | The Reverend | Sally | Lynn | Jones | *null* | General | true |
These values are going to depend upon your sitewide preferences and choices for this contact, but a reasonable default might produce the following:
| Rendering type | Value |
|-----------------|--------------------------|
| Display Name | Sally Lynn Jones |
| Sort Name | Jones, Sally Lynn |
| Addressee | The Reverend Sally Jones |
| Email greeting | Dear Sally |
| Postal greeting | Dear Ms. Jones |
| Short name | Sally |
### An organization has a different name in another language
This should be self-explanatory:
| Organization name | Name type | Is primary |
|--------------------------|-----------|------------|
| Médecins Sans Frontières | General | true |
| Doctors Without Borders | Localized | false |
| Rendering type | Value |
|----------------|--------------------------|
| Display Name | Médecins Sans Frontières |
| Sort Name | Médecins Sans Frontières |
| English Name | Doctors Without Borders |
### Someone's family name comes before their given name
I'm on the fence about how this should be handled, but all I know is that we currently handle it terribly (except maybe in translations that assume it?) One option might be to call the fields first, middle, and last names according to position (*Ai* being the family name, but it comes first):
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Ai | *null* | Weiwei | *null* | General | true |
Alternatively, we could name the fields according to function:
| Prefix | Title | Given name | Additional name | Surname | Suffix | Name type | Is primary |
|--------|--------|------------|-----------------|---------|--------|-----------|------------|
| Mr. | *null* | Weiwei | *null* | Ai | *null* | General | true |
In either case, the renderings should be like
| Rendering type | Value |
|-----------------|---------------|
| Display Name | Ai Weiwei |
| Sort Name | Ai Weiwei |
| Addressee | Mr. Ai Weiwei |
| Email greeting | Dear Weiwei |
| Postal greeting | Dear Mr. Ai |
| Short name | Weiwei |
### Clinic needs to retain legal name for insurance billing
Someone's real name doesn't match their legal name, let alone their nickname. While most organizations wouldn't want or need to record a contact's dead name, a clinic might need it for billing correspondence. In this case, a patient is named Stephen, or Steve for short, but his legal name is Barbara.
| Prefix | Title | First name | Middle Name | Last name | Suffix | Name type | Is primary |
|--------|--------|------------|-------------|-----------|--------|-----------|------------|
| Mr. | *null* | Stephen | Wayne | Smith | *null* | General | true |
| Mr. | *null* | Steve | *null* | Smith | *null* | Nickname | false |
| *null* | *null* | Barbara | Wayne | Smith | *null* | Legal | false |
All of the name renderings would use the primary name or nickname, but the clinic could add an additional name rendering for how they refer to patients when billing to the insurance company.
| Rendering type | Value |
|-----------------|----------------------|
| Display Name | Stephen Wayne Smith |
| Sort Name | Smith, Stephen Wayne |
| Addressee | Mr. Stephen Smith |
| Email greeting | Dear Steve |
| Postal greeting | Dear Mr. Smith |
| Short name | Steve |
| Insurance name | Barbara Smith |
### Household name possibilities
The current household name field befuddles lots of people and, in my opinion, makes households less attractive to use. If a household can have multiple names, it might be more useful.
You might use a Household name field that has everyone all listed out, or you could use the Last name field.
| Household name | Last name | Name type | Is primary |
|-------------------------------------|----------------|-----------|------------|
| Katherine Williams and Kendra Green | *null* | General | true |
| *null* | Williams-Green | General | false |
| Rendering type | Value |
|-----------------|------------------------------------------|
| Display Name | Katherine Williams and Kendra Green |
| Sort Name | Katherine Williams and Kendra Green |
| Addressee | The Williams-Green Household |
| Email greeting | Dear Katherine Williams and Kendra Green |
| Postal greeting | Dear Katherine Williams and Kendra Green |
Or if everyone shares a last name, a single name entity could do it all:
| Given name | Surname | Name type | Is primary |
|---------------|---------|-----------|------------|
| Jack and Jill | Hill | General | true |
(If we don't feel the need to stick with the legacy name field names, there's really no reason that "Given name" couldn't replace Organization name and Household name, too, so this illustrates that.)
| Rendering type | Value |
|-----------------|-------------------------|
| Display Name | Jack and Jill Hill |
| Sort Name | Hill, Jack and Jill |
| Addressee | The Hill Household |
| Email greeting | Dear Jack and Jill |
| Postal greeting | Dear Jack and Jill Hill |
## Current behavior
Right now, each of these situations is pretty annoying. It might require a separate custom field, located away from the name fields. It might make it difficult to automatically and accurately generate greetings for many people.
Currently, there are fields in the `civicrm_contact` table for each name part, plus nickname and legal name.
The only name renderings are display name and sort name (set sitewide), and addressee and email and postal greetings (site-wide presets and defaults, with custom options).
There's no provision for additional renderings or any way to do fallbacks like pick a nickname and fall back to a first name.
There's a limited number of alternative names for a contact.
Alternative names are not compared against primary names when deduping.
## Proposed behavior
### Schema
The name fields would be gone from the contact entity, and a new `civicrm_contact_name` entity is added with one of the following sets of fields:
| Traditional | Radical |
|-------------------|-------------------|
| id | id |
| contact_id | contact_id |
| contact_name_type | contact_name_type |
| is_primary | is_primary |
| first_name | given_name |
| middle_name | additional_name |
| last_name | surname |
| prefix_id | prefix_id |
| suffix_id | suffix_id |
| formal_title | formal_title |
| household_name | |
| organization_name | |
The `contact_name_type` would have reserved options General, Nickname, Legal, and Localized, but more could be added.
A new `civicrm_contact_rendering` entity is added with the following set of fields:
| Fields |
|-------------------------------|
| id |
| contact_id |
| contact_rendering_type |
| contact_rendering_template_id |
| value |
The `contact_rendering_type` would have reserved options Display name, Sort name, Addressee, Email greeting, Postal greeting, and Short name, but more could be added.
The `contact_rendering_template_id` would be akin to the existing `addressee_id`, `email_greeting_id`, and `postal_greeting_id`, while `value` would be the actual generated or custom value.
### Upgrade
On upgrade, the `civicrm_contact_name` would be populated with the main name fields, with nickname and legal name as additional rows. On upgrade, the `civicrm_contact_rendering` would be populated with rows for each contact's display name, sort name, addressee, and greetings.
### Contact rendering templates
The contact rendering templates would have the option to specify a contact name type to provide the default value, with a fallback to the primary contact name. For example, email greeting might default to "Dear {nickname.first_name}", where it picks the value of `first_name` from a contact's name entry with the name type of Nickname, and if that isn't available, it picks the primary name entry's value for `first_name`.
### Dedupe
When matching contacts on any of the name fields, the dedupe process would compare against all rows in the `civicrm_contact_name` field, regardless of `contact_name_type` or if it's the primary name.
## Comments
This is all very early in the process: I think it's important to do something like this, but I don't have my heart on the specific mechanisms here. I think it would need to be evaluated for performance in a variety of ways, but I don't expect it would have too huge of an impact.https://lab.civicrm.org/dev/core/-/issues/2994Support oEmbed for external facing pages2024-03-21T18:50:44ZJoeMurraySupport oEmbed for external facing pagesAn important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different...An important evolution of CiviCRM is to better support remote sites exposing CiviCRM forms and content.
Many platforms including WordPress and Drupal core since 8.6 support oEmbed, which allows a site to pull in content from a different site that exposes its content as oEmbed.
There is a good list of oEmbed providers, and CiviCRM should aspire to join it: https://oembed.com/providers.json
We should expose the following 'pages' as oEmbed content, using appropriate standard markup for use in WordPress and Drupal 8.6+ sites, and support the workflow for confirmation, thank you, and search results as appropriate:
1. Phase 1
1. Contribution page
2. Event info page
3. Event registration page
2. Phase 2
1. Something like a profile create/edit page that does not involve a payment but allows submission of a form to create a contact and activity and perhaps more (eg parent-child relationship)
2. Search Kit page
3. Petition page
3. Phase 3
1. Personal Campaign Page
2. Pages from extensions, eg Grant Application Page
3. CiviSurvey pages
Note: I haven't done the research to understand the details of oEmbed and potential challenges. I think this could be done through extension(s).
I'm posting this in order to generate discussion for a possible Roadmap item.https://lab.civicrm.org/dev/core/-/issues/3008Load create cms user for checksum contact without uf_id2023-11-19T20:45:34ZjitendraLoad create cms user for checksum contact without uf_idTo replicate -
- Create a profile with Account creation required = 1.
- Add this profile to contribution or event page.
- Create a contact A without a CMS user record.
- Load the contribution page with checksum of contact A.
- The profi...To replicate -
- Create a profile with Account creation required = 1.
- Add this profile to contribution or event page.
- Create a contact A without a CMS user record.
- Load the contribution page with checksum of contact A.
- The profile wont load the username field even when the checksum contact does not have a cms user attached to it.
![image](/uploads/c6af3e369abe278101f21e831471ea98/image.png)https://lab.civicrm.org/dev/core/-/issues/3010authx: Users that are disabled/blocked in the cms can still log in, even if t...2023-11-07T10:19:00ZDaveDauthx: Users that are disabled/blocked in the cms can still log in, even if the `XXX_user` setting is set to "require"At https://docs.civicrm.org/dev/en/latest/framework/authx/#settings you have a choice for whether the cms user needs to exist in order to log in, but even if you set it to "require", it will let a disabled/blocked user log in. My expecta...At https://docs.civicrm.org/dev/en/latest/framework/authx/#settings you have a choice for whether the cms user needs to exist in order to log in, but even if you set it to "require", it will let a disabled/blocked user log in. My expectation is that it would work the same way as regular logins if I choose "require" as the policy.
Some cms's may also allow other ways to block users, such as throttling or temporary IP bans. I would argue it should respect those too - i.e. if the cms doesn't allow login then authx shouldn't (when using "require").https://lab.civicrm.org/dev/core/-/issues/3014Proposal: Explore adding hooks to allow plugins to track database queries, e....2023-04-23T20:13:36ZBradley TaylorProposal: Explore adding hooks to allow plugins to track database queries, e.g. to enable integration with plugins like Query MonitorOne of my favourite plugins in the WordPress space is [Query Monitor](https://wordpress.org/plugins/query-monitor/). Among other things it allows you to monitor at a glance what database calls have been made for a specific request, makin...One of my favourite plugins in the WordPress space is [Query Monitor](https://wordpress.org/plugins/query-monitor/). Among other things it allows you to monitor at a glance what database calls have been made for a specific request, making it easy to spot inefiencient code and queries.
Currently CiviCRM does not have anything analogous to this. The [CiviCRM docs recommend using the MySQL query log](https://docs.civicrm.org/dev/en/latest/tools/debugging/#viewing-a-query-log-from-mysql), but this isn't easy to view at a glance, and it doesn't make it easy to correlate queries to sepecific web requests. CiviCRM's database queries don't show in Query Monitor because CiviCRM does not use WordPress' `WPDB` class. Therefore, I've been looking to see if it would be possible to extend Query Monitor to know about CiviCRM's database queries. Its reasonably easy to make this work if you patch the `CRM_Core_DAO::query` method to the following:
```
public function query($query, $i18nRewrite = TRUE) {
// rewrite queries that should use $dbLocale-based views for multi-language installs
global $wpdb, $dbLocale, $_DB_DATAOBJECT;
if ( defined( 'SAVEQUERIES' ) && SAVEQUERIES ) {
$wpdb->timer_start();
}
if (empty($_DB_DATAOBJECT['CONNECTIONS'][$this->_database_dsn_md5])) {
// Will force connection to be populated per CRM-20541.
new CRM_Core_DAO();
}
$conn = &$_DB_DATAOBJECT['CONNECTIONS'][$this->_database_dsn_md5];
$orig_options = $conn->options;
$this->_setDBOptions($this->_options);
if ($i18nRewrite and $dbLocale) {
$query = CRM_Core_I18n_Schema::rewriteQuery($query);
}
$ret = parent::query($query);
if ( defined( 'SAVEQUERIES' ) && SAVEQUERIES ) {
$wpdb->num_queries++;
$wpdb->queries[] = array(
$query,
$wpdb->timer_stop(),
$wpdb->get_caller(),
$wpdb->time_start,
array(),
'trace' => new QM_Backtrace( array(
'ignore_frames' => 1,
)),
'result' => is_countable($ret) ? count($ret) : 1
);
}
$this->_setDBOptions($orig_options);
return $ret;
}
```
Esentially the `$wpdb->queries` array is being populated with knowledge of the CiviCRM database queries, which query monitor can then read.
With this in place you can start to see some interesting insights. For example, we can see that the new mailing screen contains a number of database queries that are reapeated 40 times each, which is probably ripe for optimisation:
![Screenshot_2022-01-02_at_14.13.41](/uploads/117036cbf7cb970e93c7399eeb436f64/Screenshot_2022-01-02_at_14.13.41.png)
The WordPress admin bar shows the total count of queries (243) combined from both CiviCRM and WordPress.
Hacking the core files works for my purposes, but obviously doesn't allow for this to be packaged into a distributable plugin. Therefore, I think it'd be cool if there were a generic hook at the end of `CRM_Core_DAO::query` which could be used to save the query information in a format ameanable to Query Monitor. Alternatively CiviCRM could copy WordPress idea of the `$wpdb->queries` array, creating an array of queries which plugins like Query Monitor could make use of (in WordPress this is behind a flag so there is no performance hit when not in a development environment).
I've not given too much thought to exactly how this hook should look, but before I take this further (e.g. to the Pull Request state) it'd be good to get some views from the community:
- Is this something others in the community would find useful if this were distributed as a plugin?
- Does anyone know of plugins similar to Query Monitor in the Drupal or Joomla space which may benifit from this type of solution?
- Would people find this sort of hook useful for other use cases - for example, for writing debugging tools native to CiviCRM.
Keen to get others views on this one. I'll be interested to know what people think.https://lab.civicrm.org/dev/core/-/issues/3027If you don't have permission to add contacts don't show ways to add contacts2022-11-15T17:49:39ZandyburnsIf you don't have permission to add contacts don't show ways to add contactsIn various places in Civi, you can either search for or add a contact when adding a contribution, participant, activity, etc. This works if you have the `add contacts` permission but not if you don't. The error is misleading too.
![add_...In various places in Civi, you can either search for or add a contact when adding a contribution, participant, activity, etc. This works if you have the `add contacts` permission but not if you don't. The error is misleading too.
![add_contact_form](/uploads/7d307dea4e7643e33db812398bc745e4/add_contact_form.png)
https://lab.civicrm.org/dev/core/-/blob/master/js/Common.js#L811https://lab.civicrm.org/dev/core/-/issues/30315.44+ Unsubscribe from Smart Group form generates excessive process load2023-01-31T13:58:20Ztommybobo5.44+ Unsubscribe from Smart Group form generates excessive process loadOverview
----------------------------------------
The recent change to email unsubscribe from Smart Group is overloading mysql. [This change ](https://github.com/civicrm/civicrm-core/pull/21176) by @mattwire is providing a check to see w...Overview
----------------------------------------
The recent change to email unsubscribe from Smart Group is overloading mysql. [This change ](https://github.com/civicrm/civicrm-core/pull/21176) by @mattwire is providing a check to see which smart group an unsubscribing user is in before unsubscribing them. On a large mailing, site with a lot of smart groups, or even one complex smart this code is a time bomb. Especially as we see a rise in automated link checking bots on large email providers.
Current behavior
----------------------------------------
When a email recipient clicks unsubscribe all smart groups are regenerated. When CRM_Contact_BAO_GroupContactCache is called all smart groups are regenerated regardless of the setting of cache time. Also this is running even if the email doesn't contain any smart groups. Then the cache table is Left Joined onto the query for a search. Which also can be refreshed by another user clicking the link in the 5-10 second process time of a large site with large smart groups.
This is an incredibly high process load before any action is actually taken in order to display a group title on page that 9 times out of 10 is a bot not an end user.
Proposed behavior
----------------------------------------
Members do not need to see the Group labels from which they are unsubscribing, and if they are unsubscribing from this Email list then it is reasonable to mark them as unsubscribed from a smart group used in the mailing even if they are not in it. If they in the future become a member of the smart group, they probably don't want email about it.
It is reasonable to run this process AFTER the unsubscribe button is clicked by the user. Running it on page load is just wasting a ton resources on bot traffic.
Comments
----------------------------------------
@mattwire 's solution is reasonable for many, but it just isn't scaling well on larger civicrm installs. We have either reverted the code or switched to Opt Out of all emails for those larger clients, but those are temporary solutions.https://lab.civicrm.org/dev/core/-/issues/3040UI allows creation of unresolvable failure for status check "Relationship Typ...2023-11-16T16:09:55ZAllenShawUI allows creation of unresolvable failure for status check "Relationship Type Internal Name Duplicates"Admittedly, "unresolvable" is a bit of an overstatement, but you can create this situation in the UI, and then the UI provides no way to resolve it (it is resolvable via the API).
**To reproduce:**
1. Visit https://dmaster.demo.civicrm....Admittedly, "unresolvable" is a bit of an overstatement, but you can create this situation in the UI, and then the UI provides no way to resolve it (it is resolvable via the API).
**To reproduce:**
1. Visit https://dmaster.demo.civicrm.org/civicrm/admin/reltype?reset=1
1. Create a new relationship type with Label A "Foobar" and Label B "Employee Of"
1. Visit https://dmaster.demo.civicrm.org/civicrm/a/#/status and observe error: "Relationship Type Internal Name Duplicates;
Relationship type Employee of has the same internal machine name as another type.": ![error](/uploads/6d2eb70ff9aac5ce16ca948faee6d077/error.png)
1. Visit https://dmaster.demo.civicrm.org/civicrm/admin/reltype?reset=1 and edit the offending relationship (created in step 1); you can edit the labels but not the name. Best you can do is change "Employee of" to something else, like "Foobar of".
1. Visit https://dmaster.demo.civicrm.org/civicrm/a/#/status and observe that the error persists, exactly the same as in Step 3.
There's no way to change the name in the UI; you must use the API to change it.
**Proposed solution:**
* Upon creating a relationship type (other than reserved types created upon installation), use centralized logic to derive unique name from label; i.e., opt-in to the new behavior provided by https://github.com/civicrm/civicrm-core/pull/22508.
Thoughts?https://lab.civicrm.org/dev/core/-/issues/3050Error thrown when 'remove'ing from CiviContribute Batch2022-11-02T00:48:39ZmarshError thrown when 'remove'ing from CiviContribute BatchOverview
----------------------------------------
Civi throws an error when working with Accounting Batches. It's kind of hard to pin down - the front end gives the default / standard / very helpful
No response from the server. Check yo...Overview
----------------------------------------
Civi throws an error when working with Accounting Batches. It's kind of hard to pin down - the front end gives the default / standard / very helpful
No response from the server. Check your internet connection and try reloading the page.
Reproduction steps
----------------------------------------
- add a new accounting Batch (Contributions > Accounting Batches > New Batch)
- Assign a contribution to the batch
- try to remove the contribution from the batch
- at this point the message is displayed:
message, although ConfigAndLog does give a bit more, i.e.:
$Fatal Error Details = array:3 [
"message" => "Cannot delete EntityBatch with no id."
"code" => null
"exception" => CRM_Core_Exception {#2039
-errorData: array:1 [
"error_code" => 0
]
#cause: null
-_trace: null
#message: "Cannot delete EntityBatch with no id."
#code: 0
#file: "/var/www/html/drupal7/sites/all/modules/civicrm/CRM/Core/DAO.php"
#line: 958
trace: {
/var/www/html/drupal7/sites/all/modules/civicrm/CRM/Core/DAO.php:958 {
› if (empty($record['id'])) {
› throw new CRM_Core_Exception("Cannot delete {$entityName} with no id.");
› }
}
/var/www/html/drupal7/sites/all/modules/civicrm/CRM/Batch/BAO/EntityBatch.php:39 { …}
/var/www/html/drupal7/sites/all/modules/civicrm/CRM/Financial/Page/AJAX.php:207 { …}
/var/www/html/drupal7/sites/all/modules/civicrm/CRM/Utils/REST.php:266 { …}
/var/www/html/drupal7/sites/all/modules/civicrm/CRM/Utils/REST.php:550 { …}
/var/www/html/drupal7/sites/all/modules/civicrm/CRM/Core/Invoke.php:279 { …}
/var/www/html/drupal7/sites/all/modules/civicrm/CRM/Core/Invoke.php:69 { …}
/var/www/html/drupal7/sites/all/modules/civicrm/CRM/Core/Invoke.php:36 { …}
/var/www/html/drupal7/sites/all/modules/civicrm/drupal/civicrm.module:458 { …}
/var/www/html/drupal7/includes/menu.inc:527 { …}
/var/www/html/drupal7/index.php:21 { …}
}
}
]
This has alr3eady been [raised on stackechange](https://civicrm.stackexchange.com/questions/41126/entitybatch-with-no-id-error-when-remobing-items-from-accounting-batches?noredirect=1#comment48386_41126)
Current behaviour
----------------------------------------
![image](/uploads/f3431d6145caa54bbd5c96dec58ac3cf/image.png)
![image](/uploads/4dae3c200fc6696f0ecd6f5b25fd6747/image.png)
Expected behaviour
----------------------------------------
The item is removed from the batch
Environment information
----------------------------------------
- reproduced on dmaster
- using Chromium locally on a Debian boxhttps://lab.civicrm.org/dev/core/-/issues/3067Improving performance of retreiving display names for multivalued custom data2022-11-25T15:39:20ZMichael McAndrewImproving performance of retreiving display names for multivalued custom data.Have been taking a look at https://github.com/civicrm/civicrm-core/pull/20764/files#diff-36ab608da0b5718996afd18c10a8c12653c2a1675e81fd3c4dc3e40c2ebb25d2R1040.
My hunch is that for a query that returns 50,000 rows, the 'group_concat fi....Have been taking a look at https://github.com/civicrm/civicrm-core/pull/20764/files#diff-36ab608da0b5718996afd18c10a8c12653c2a1675e81fd3c4dc3e40c2ebb25d2R1040.
My hunch is that for a query that returns 50,000 rows, the 'group_concat find_in_set' subquery is executed 50,000 times even if pagination is turned on since the limit is applied after it does all the subqueries
If you turn that subquery into a 'superquery', i.e. you do the 'group_concat find_in_set' hydration after the limit, it imrpoves the performance a lot.
I did a proof of concept below. For a query with 500 results it went from 37 seconds to 1.3 seconds.
And for 34,000 it went from never completing to 1.0 seconds.
Not sure how easy it would be to acheive this within the constraints of searchkit @colemanw but it would be great to get your initial thoughts on this.
Proof of concept changes to the query below...
Changing this query:
```sql
SELECT `a`.`id`, (
SELECT GROUP_CONCAT(
`display_name`
ORDER BY FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`Case_of_abuse_details_1`.`perpetrator_143`, '�', ','))
SEPARATOR '�'
)
FROM `civicrm_contact`
WHERE FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`Case_of_abuse_details_1`.`perpetrator_143`, '�', ','))
) AS `Case_of_abuse_details.Perpetrator.display_name`
FROM civicrm_activity a
LEFT JOIN `civicrm_value_case_of_abuse_68` `Case_of_abuse_details_1` ON a.id = Case_of_abuse_details_1.entity_id
WHERE ((`a`.`activity_type_id` IS NULL OR (`a`.`activity_type_id` IN (1, 2, 3, 4, 5, 9, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 40, 44, 45, 48, 49, 50, 51, 52, 53, 55, 57, 59, 61, 63, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74)))) AND (`a`.`activity_type_id` = "68") AND (`a`.`is_test` = "0") AND (`a`.`is_deleted` = "0")
GROUP BY `a`.`id`
LIMIT 50
OFFSET 0
```
into this query
```sql
SELECT id,(SELECT GROUP_CONCAT(
`display_name`
ORDER BY FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`perpetrator_143`, '�', ','))
SEPARATOR '�'
)
FROM `civicrm_contact`
WHERE FIND_IN_SET(`civicrm_contact`.`id`, REPLACE(`perpetrator_143`, '�', ','))
) AS `Case_of_abuse_details.Perpetrator.display_name` FROM (SELECT `a`.`id`, `Case_of_abuse_details_1`.`perpetrator_143`
FROM civicrm_activity a
LEFT JOIN `civicrm_value_case_of_abuse_68` `Case_of_abuse_details_1` ON a.id = Case_of_abuse_details_1.entity_id
WHERE ((`a`.`activity_type_id` IS NULL OR (`a`.`activity_type_id` IN (1, 2, 3, 4, 5, 9, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 40, 44, 45, 48, 49, 50, 51, 52, 53, 55, 57, 59, 61, 63, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74)))) AND (`a`.`activity_type_id` = "68") AND (`a`.`is_test` = "0") AND (`a`.`is_deleted` = "0")
GROUP BY `a`.`id`
LIMIT 50
OFFSET 0) AS alias
```https://lab.civicrm.org/dev/core/-/issues/3068PDF filename on Pay later contributions is receipt.pdf2023-11-23T07:47:01Zluke.stewartPDF filename on Pay later contributions is receipt.pdfOverview
----------------------------------------
Currently if attach PDF copies to receipts is enabled and a contact completes a "pay later" contribution an email is sent with a pdf named 'receipt.pdf'
As a quick fix it is trivial to...Overview
----------------------------------------
Currently if attach PDF copies to receipts is enabled and a contact completes a "pay later" contribution an email is sent with a pdf named 'receipt.pdf'
As a quick fix it is trivial to detect in CRM/Contribute/BAO/ContributionPage::sendMail whether the payment is pay later or not and use this to change the PDF name.
We also need to check other workflows that trigger pdfs receipts.
These look to be CRM/Contribute/BAO/ContributionSoft.php which should be a similar fix to above - and also consider workflows for resending "receipts" where this is hardcoded as well.
Long term this value wouldn't be hard coded in this file but exposed through the UI as either a global setting - or potentially having a per contribution page override.
Additionally as it is currently this is not a translateable string either.
Reproduction steps
----------------------------------------
1. Enable "Attach PDF copy to receipts" via Admin -> System Settings -> Misc
1. Create a contribution page with a pay later and payment processor (dummy processor is fine) and add a $1 payment.
1. Under the Receipt options -> "Email Receipt to Contributor?" set true.
Current behaviour
----------------------------------------
When submitting the contribution form with "Pay Later" the email has "Invoice" in the subject - but the attachment filename is "receipt.pdf"
Expected behaviour
----------------------------------------
When submitting the contribution form with "Pay Later" the email has "Invoice" in the subject - but the attachment filename is "receipt.pdf"
Environment information
----------------------------------------
* __CiviCRM:__ _Master ( and Current Stable 5.46.2.)https://lab.civicrm.org/dev/core/-/issues/3070Placeholders in forms2023-12-06T15:27:43ZshaneonabikePlaceholders in formsOverview
----------------------------------------
_Please describe your improvement in detail._
Example use-case
----------------------------------------
1. View a Contribution form
1. Fields such as _First name_ do not have a placehold...Overview
----------------------------------------
_Please describe your improvement in detail._
Example use-case
----------------------------------------
1. View a Contribution form
1. Fields such as _First name_ do not have a placeholder and only a label
Current behaviour
----------------------------------------
I think that in general providing placeholders is a good practice for all fields for the following reasons:
+ More accessible
+ Simplifies design (hidden labels) reduces the size of forms
Proposed behaviour
----------------------------------------
I was thinking there could be two ways to make this happen to start.
+ Provide placeholders that match the labels (Phase 1)
+ Provide ability on all fields with text entry to have a customizable placeholder (Phase 2)
Comments
----------------------------------------
Would there be any problem with me writing a patch that would allow the creation of placeholders for text holders that presently uses the label and copies that into a placeholder field?https://lab.civicrm.org/dev/core/-/issues/3191SearchKit not functional on Joomla 42023-05-24T05:20:40Zjoshjosh@civicrm.orgSearchKit not functional on Joomla 4The past two release of CiviCRM have resulted in SearchKit not functioning on Joomla 4. When browsing to SearchKit, the user is presented with what appears to be unrendered code:
![Screenshot_2022-02-21_12.13.32](/uploads/d52471e665bbfc...The past two release of CiviCRM have resulted in SearchKit not functioning on Joomla 4. When browsing to SearchKit, the user is presented with what appears to be unrendered code:
![Screenshot_2022-02-21_12.13.32](/uploads/d52471e665bbfcaf82f934243c6c9c19/Screenshot_2022-02-21_12.13.32.jpg)
Console presents the following error:
```
Error: [$injector:unpr] http://errors.angularjs.org/1.8.2/$injector/unpr?p0=savedSearchesProvider%20%3C-%20savedSearches%20%3C-%20searchList
at angular.min.js:7:168
at angular.min.js:46:468
at Object.d [as get] (angular.min.js:44:197)
at angular.min.js:47:29
at d (angular.min.js:44:197)
at e (angular.min.js:44:438)
at Object.instantiate (angular.min.js:45:333)
at angular.min.js:99:267
at Object.link (angular-modules.d0a616a75686afef23d3772fcee92a7f.js:5648:217)
at angular.min.js:17:134 '<div ng-view="" class="ng-scope">'
(anonymous) @ angular.js:15697
```
All other aspects of CiviCRM appear to be functional. The issue can be reproduced at https://cividemo.com and on fresh installs of just Joomla and CiviCRM (no extensions installed on either systems).Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/3083getNonDeductibleAmountFromPriceSet should take into account the financial typ...2023-12-05T21:36:19ZherbdoolgetNonDeductibleAmountFromPriceSet should take into account the financial type of the price field optionOverview
----------------------------------------
`getNonDeductibleAmountFromPriceSet()` gets called in a couple places to calculate the non-deductible amount of contributions. While the calling method will take into account the financi...Overview
----------------------------------------
`getNonDeductibleAmountFromPriceSet()` gets called in a couple places to calculate the non-deductible amount of contributions. While the calling method will take into account the financial type of the contribution when assigning the non-deductible amount, `getNonDeductibleAmountFromPriceSet()` does *not* take into account the financial type of the price set option. I believe it should be.
Reproduction steps
----------------------------------------
1. Create price set and add two fields: one with financial type Donation (deductible) = `$10` and other Event Fee (non-deductible) = `$20`. Make the Event Fee have a random non-deductible amount of say `$2.50`.
2. Add a contribution in the backend to a contact and choose this new price set.
3. Look at the resulting contribution.
Current behaviour
----------------------------------------
The contribution has non-deductible = `$2.50` for a `$30` contribution.
Expected behaviour
----------------------------------------
I expect the non-deducible to be `$20`, the full price of the Event Fee. The contribution should be setting the non-deductible to be equal that of the merchandise price field, regardless of the actual non-deductible amount for that field. If a price field is non-deductible then it is 100% non-deductible based on the financial type.https://lab.civicrm.org/dev/core/-/issues/3089Meta: Create list of items for moving a core component to an extension2024-03-08T05:07:35ZDaveDMeta: Create list of items for moving a core component to an extensionIt's never really been fully done before. Some of the Grant things are likely to come up again if it's not documented. Partial list just quickly compiled from tickets. Doesn't include the things that did work (which would still need doin...It's never really been fully done before. Some of the Grant things are likely to come up again if it's not documented. Partial list just quickly compiled from tickets. Doesn't include the things that did work (which would still need doing), and might be missing some PRs that had no tickets.
* https://lab.civicrm.org/dev/core/-/issues/3069
* https://lab.civicrm.org/dev/core/-/issues/3076
* https://lab.civicrm.org/dev/core/-/issues/3056
* https://lab.civicrm.org/dev/core/-/issues/3057
* https://lab.civicrm.org/dev/core/-/issues/3087
* https://lab.civicrm.org/dev/core/-/issues/3093
* https://lab.civicrm.org/dev/core/-/issues/3100
* https://github.com/civicrm/civicrm-buildkit/pull/677
* https://lab.civicrm.org/dev/core/-/issues/3101
* https://github.com/colemanw/webform_civicrm/pull/719
* extendedreports
* https://github.com/eileenmcnaughton/nz.co.fuzion.extendedreport/commit/6184de98c5b9f40475ab4164c993b19d68561240
* https://github.com/eileenmcnaughton/nz.co.fuzion.extendedreport/pull/503
* https://github.com/eileenmcnaughton/nz.co.fuzion.extendedreport/pull/506
* civicrm_entity
* https://github.com/eileenmcnaughton/civicrm_entity/pull/364
* an unresolved cache(?) issue
* https://github.com/twomice/com.joineryhq.jsumfields/pull/23
* https://github.com/civicrm/civicrm-drupal/pull/653
* https://github.com/civicrm/civicrm-drupal/pull/656
* https://github.com/civicrm/civicrm-core/pull/22905
* https://lab.civicrm.org/dev/core/-/issues/3118
* https://lab.civicrm.org/dev/core/-/issues/3119
* https://lab.civicrm.org/dev/core/-/issues/3159
* https://github.com/civicrm/civicrm-core/pull/23116
* https://github.com/civicrm/civicrm-core/pull/23118
* https://github.com/civicrm/civicrm-core/pull/23115
* https://lab.civicrm.org/dev/core/-/issues/3161
* https://github.com/civicrm/civicrm-core/pull/23336
* https://lab.civicrm.org/dev/core/-/issues/3485
* https://lab.civicrm.org/dev/core/-/issues/3492
* https://lab.civicrm.org/dev/core/-/issues/3503
* https://github.com/civicrm/civicrm-core/pull/24191
* https://github.com/civicrm/civicrm-core/pull/26118
Round 2:
* https://github.com/civicrm/civicrm-core/pull/26497
* https://github.com/civicrm/civicrm-core/pull/26499
Late to the party:
* https://lab.civicrm.org/dev/core/-/issues/5075https://lab.civicrm.org/dev/core/-/issues/3096Marking historical contact data - add on hold, hold date, reset date columns ...2023-01-06T22:02:03ZandyburnsMarking historical contact data - add on hold, hold date, reset date columns to civicrm_address and civicrm_phone to match civicrm_emailCurrently only civicrm_email has a mechanism to mark a bad email. This data model should be mirrored over to civicrm_address and civicrm_phone. We currently use the lightweight "location type" approach where we have an Invalid location t...Currently only civicrm_email has a mechanism to mark a bad email. This data model should be mirrored over to civicrm_address and civicrm_phone. We currently use the lightweight "location type" approach where we have an Invalid location type to designate a bad email/phone/address. However, it is limiting as you can only have 1 address per location type for a contact.
This was previously brought up here but was not pursued: https://issues.civicrm.org/jira/browse/CRM-13777.
The rationale is storing older data on these 3 entities helps prevent the creation of duplicates when bringing external sources of data (that may be older as well) upon import and therefore need to remain in the respective db table and not only sent to a log table.
The limitation of 1 address per location type would not apply to on hold addresses, therefore creating a history of addresses.
Setting to on hold would automatically set the hold date.
On hold entities would be shown in a "Former Communications Data" tab much like this extension (https://civicrm.org/extensions/former-communication-data) to prevent cluttering the Contact Summary Screen by only querying the entities that are on hold (and conversely, the contact's summary screen would only show those not on hold) and showing them in descending On Hold date order. Allow filtering by email/phone/address.
On hold entities would not be able to used throughout CiviCRM. E.g. a on hold mobile phone would not show Outbound SMS action, an on hold address would be added as a criteria to this checkbox upon export:
![image](/uploads/d481ef9470febc1620b895af2fb1baff/image.png)
There is obviously overlap with what detailed logging does and this so I'd greatly appreciate anyone's thoughts on how to approach / refine this feature request. We are willing to fund a solution.
Ref:
- https://civicrm.stackexchange.com/questions/681/what-are-best-practices-for-bad-postal-addresses
- https://civicrm.stackexchange.com/questions/20584/need-historical-address-informationhttps://lab.civicrm.org/dev/core/-/issues/3103Document contract for alterMailParams2023-11-23T17:28:30ZDaveDDocument contract for alterMailParamsThe data in $params for alterMailParams is inconsistent, and with recent token changes has changed in some cases.
Copied from https://github.com/civicrm/civicrm-core/pull/22878, which is specific to message template emails (there are al...The data in $params for alterMailParams is inconsistent, and with recent token changes has changed in some cases.
Copied from https://github.com/civicrm/civicrm-core/pull/22878, which is specific to message template emails (there are also other types of emails that trigger alterMailParams):
> Quote
* Add a class CRM_Event_WorkflowMessage_OfflineReceipt. This would look a lot like CRM_Contribute_WorkflowMessage_ContributionOfflineReceipt which describes contribution_offline_receipt. (Also, there's some draft docs in https://lab.civicrm.org/documentation/docs/dev/-/merge_requests/987.)
* Add a variant of hook_alterMailParams which gives the message as an object. Here's a sketch for firing+consuming hook_alterWorkflowMessage [gist](https://gist.github.com/totten/14178dabea79ea2a5a639a5b928262e2).https://lab.civicrm.org/dev/core/-/issues/3121The mysql session timezone that civi sets does not observe daylight saving (a...2023-12-09T05:04:30ZDaveDThe mysql session timezone that civi sets does not observe daylight saving (aka The Contact Logging civireport can show the wrong time)Pulling this out as a separate issue from #2122. It's also different from that issue in that here the value stored in the database is correct, just the display is wrong.
Mysql/mariadb often do not ship with named timezones populated (i....Pulling this out as a separate issue from #2122. It's also different from that issue in that here the value stored in the database is correct, just the display is wrong.
Mysql/mariadb often do not ship with named timezones populated (i.e. the mysql.time_zone table and related tables are empty). So when civi sets the mysql timezone for the session it calculates an offset instead (e.g. '+01:00'), based on "now". Offsets don't know about DST rules, so when it's converted for display based on that "now" offset, it will be wrong if "now" is on the other side of DST from the date that was stored in the database. Example:
Suppose log_civicrm_contact has a log_date with a value of March 8, 2022 12:34, created in timezone America/New_York. When viewed on March 9, 10, 11, etc it displays correctly.
When viewed on, say, March 16, civi will set the mysql timezone as '-04:00', which is different from what it was on March 8 ('-05:00'). Therefore when it displays, it shows 13:34 instead of 12:34.
This only affects timestamp columns. Datetime columns don't do any conversion based on session timezone.
The code where this happens is at https://github.com/civicrm/civicrm-core/blob/1754e1752eb8e516ec846a438005f6fcb55be247/CRM/Utils/System/Base.php#L760
Note also that getTimeZoneString() does different things for different CMSs, so you may experience the issue slightly differently depending on the install.https://lab.civicrm.org/dev/core/-/issues/3128Cases dashlet sometimes gives error when sort by subject2023-12-04T13:49:19ZDaveDCases dashlet sometimes gives error when sort by subjectPulling this out from https://lab.civicrm.org/dev/core/-/issues/1624 as a separate issue.
_Sometimes_, sorting by the subject on a cases dashlet will give a datatables error. The actual error if you look at the network response is `Colu...Pulling this out from https://lab.civicrm.org/dev/core/-/issues/1624 as a separate issue.
_Sometimes_, sorting by the subject on a cases dashlet will give a datatables error. The actual error if you look at the network response is `Column 'subject' in order clause is ambiguous`.
```
Error Field Error Value
Type DB_Error
Code -1
Message DB Error: unknown error
Mode 16
UserInfo SELECT civicrm_case.id as case_id, civicrm_case.subject as case_subject, civicrm_contact.id as contact_id, civicrm_contact.sort_name as sort_name, civicrm_phone.phone as phone, civicrm_contact.contact_type as contact_type, civicrm_contact.contact_sub_type as contact_sub_type, t_act.activity_type_id as activity_type_id, civicrm_case.case_type_id as case_type_id, civicrm_case.status_id as case_status_id, t_act.status_id as status_id, civicrm_case.start_date as case_start_date, GROUP_CONCAT(DISTINCT IF(case_relationship.contact_id_b = 2, case_relation_type.label_a_b, case_relation_type.label_b_a) SEPARATOR ', ') as case_role, t_act.activity_date_time as activity_date_time, t_act.id as activity_id, case_status.label AS case_status, civicrm_case_type.title AS case_type FROM civicrm_case INNER JOIN civicrm_case_contact ON civicrm_case.id = civicrm_case_contact.case_id INNER JOIN civicrm_contact ON civicrm_case_contact.contact_id = civicrm_contact.id LEFT JOIN civicrm_case_type ON civicrm_case.case_type_id = civicrm_case_type.id LEFT JOIN civicrm_option_group option_group_case_status ON ( option_group_case_status.name = 'case_status' ) LEFT JOIN civicrm_option_value case_status ON ( civicrm_case.status_id = case_status.value AND option_group_case_status.id = case_status.option_group_id ) LEFT JOIN civicrm_case_activity ca4 ON civicrm_case.id = ca4.case_id LEFT JOIN civicrm_activity t_act ON t_act.id = ca4.activity_id AND t_act.is_current_revision = 1 LEFT JOIN civicrm_phone ON civicrm_phone.contact_id = civicrm_contact.id AND civicrm_phone.is_primary = 1 LEFT JOIN civicrm_relationship case_relationship ON ((case_relationship.contact_id_a = civicrm_case_contact.contact_id AND case_relationship.contact_id_b = 2) OR (case_relationship.contact_id_b = civicrm_case_contact.contact_id AND case_relationship.contact_id_a = 2)) AND case_relationship.is_active AND case_relationship.case_id = civicrm_case.id LEFT JOIN civicrm_relationship_type case_relation_type ON case_relation_type.id = case_relationship.relationship_type_id AND case_relation_type.id = case_relationship.relationship_type_id WHERE (1) AND civicrm_case.is_deleted = 0 AND civicrm_contact.is_deleted <> 1 AND (case_relationship.contact_id_b = 2 OR case_relationship.contact_id_a = 2) AND case_relationship.is_active AND civicrm_case.status_id IN (1,3) GROUP BY case_id ORDER BY subject asc LIMIT 0, 10 [nativecode=1052 ** Column 'subject' in order clause is ambiguous]
DebugInfo SELECT civicrm_case.id as case_id, civicrm_case.subject as case_subject, civicrm_contact.id as contact_id, civicrm_contact.sort_name as sort_name, civicrm_phone.phone as phone, civicrm_contact.contact_type as contact_type, civicrm_contact.contact_sub_type as contact_sub_type, t_act.activity_type_id as activity_type_id, civicrm_case.case_type_id as case_type_id, civicrm_case.status_id as case_status_id, t_act.status_id as status_id, civicrm_case.start_date as case_start_date, GROUP_CONCAT(DISTINCT IF(case_relationship.contact_id_b = 2, case_relation_type.label_a_b, case_relation_type.label_b_a) SEPARATOR ', ') as case_role, t_act.activity_date_time as activity_date_time, t_act.id as activity_id, case_status.label AS case_status, civicrm_case_type.title AS case_type FROM civicrm_case INNER JOIN civicrm_case_contact ON civicrm_case.id = civicrm_case_contact.case_id INNER JOIN civicrm_contact ON civicrm_case_contact.contact_id = civicrm_contact.id LEFT JOIN civicrm_case_type ON civicrm_case.case_type_id = civicrm_case_type.id LEFT JOIN civicrm_option_group option_group_case_status ON ( option_group_case_status.name = 'case_status' ) LEFT JOIN civicrm_option_value case_status ON ( civicrm_case.status_id = case_status.value AND option_group_case_status.id = case_status.option_group_id ) LEFT JOIN civicrm_case_activity ca4 ON civicrm_case.id = ca4.case_id LEFT JOIN civicrm_activity t_act ON t_act.id = ca4.activity_id AND t_act.is_current_revision = 1 LEFT JOIN civicrm_phone ON civicrm_phone.contact_id = civicrm_contact.id AND civicrm_phone.is_primary = 1 LEFT JOIN civicrm_relationship case_relationship ON ((case_relationship.contact_id_a = civicrm_case_contact.contact_id AND case_relationship.contact_id_b = 2) OR (case_relationship.contact_id_b = civicrm_case_contact.contact_id AND case_relationship.contact_id_a = 2)) AND case_relationship.is_active AND case_relationship.case_id = civicrm_case.id LEFT JOIN civicrm_relationship_type case_relation_type ON case_relation_type.id = case_relationship.relationship_type_id AND case_relation_type.id = case_relationship.relationship_type_id WHERE (1) AND civicrm_case.is_deleted = 0 AND civicrm_contact.is_deleted <> 1 AND (case_relationship.contact_id_b = 2 OR case_relationship.contact_id_a = 2) AND case_relationship.is_active AND civicrm_case.status_id IN (1,3) GROUP BY case_id ORDER BY subject asc LIMIT 0, 10 [nativecode=1052 ** Column 'subject' in order clause is ambiguous]
PEAR_Exception: DB Error: unknown error in ...\sites\all\modules\civicrm\vendor\pear\pear-core-minimal\src\PEAR.php on line 922
- DB_Error: DB Error: unknown error in unknown on line unknown
Exception trace
# Function Location
0 CRM_Core_Error::exceptionHandler(Object(DB_Error)) ...\sites\all\modules\civicrm\vendor\pear\pear-core-minimal\src\PEAR.php:922
1 PEAR_Error->__construct('DB Error: unknow…', -1, 16, Array, 'SELECT civicrm_c…') ...\sites\all\modules\civicrm\vendor\pear\db\DB.php:997
2 DB_Error->__construct(-1, 16, Array, 'SELECT civicrm_c…') ...\sites\all\modules\civicrm\vendor\pear\pear-core-minimal\src\PEAR.php:575
3 PEAR::_raiseError(Object(DB_mysqli), null, -1, 16, Array, 'SELECT civicrm_c…', 'DB_Error', true) ...\sites\all\modules\civicrm\vendor\pear\pear-core-minimal\src\PEAR.php:223
4 PEAR->__call('raiseError', Array) ...\sites\all\modules\civicrm\vendor\pear\db\DB\common.php:1928
5 DB_common->raiseError(-1, null, null, 'SELECT civicrm_c…', '1052 ** Column '…') ...\sites\all\modules\civicrm\vendor\pear\db\DB\mysqli.php:936
6 DB_mysqli->mysqliRaiseError() ...\sites\all\modules\civicrm\vendor\pear\db\DB\mysqli.php:406
7 DB_mysqli->simpleQuery('SELECT civicrm_c…') ...\sites\all\modules\civicrm\vendor\pear\db\DB\common.php:1234
8 DB_common->query('SELECT civicrm_c…') ...\sites\all\modules\civicrm\packages\DB\DataObject.php:2696
9 DB_DataObject->_query('SELECT civicrm_c…') ...\sites\all\modules\civicrm\packages\DB\DataObject.php:1829
10 DB_DataObject->query('SELECT civicrm_c…') ...\sites\all\modules\civicrm\CRM\Core\DAO.php:472
11 CRM_Core_DAO->query('SELECT civicrm_c…', true) ...\sites\all\modules\civicrm\CRM\Core\DAO.php:1637
12 CRM_Core_DAO::executeQuery('SELECT civicrm_c…') ...\sites\all\modules\civicrm\CRM\Case\BAO\Case.php:583
13 CRM_Case_BAO_Case::getCases(false, Array) ...\sites\all\modules\civicrm\CRM\Case\Page\AJAX.php:186
14 CRM_Case_Page_AJAX::getCases() ...\sites\all\modules\civicrm\CRM\Core\Invoke.php:285
15 CRM_Core_Invoke::runItem(Array) ...\sites\all\modules\civicrm\CRM\Core\Invoke.php:69
16 CRM_Core_Invoke::_invoke(Array) ...\sites\all\modules\civicrm\CRM\Core\Invoke.php:36
17 CRM_Core_Invoke::invoke(Array) ...\sites\all\modules\civicrm\drupal\civicrm.module:471
18 civicrm_invoke('ajax', 'get-cases') ...\includes\menu.inc:527
19 menu_execute_active_handler() ...\index.php:21
20 {main}
Sorry, due to an error, we are unable to fulfill your request at the moment. You may want to contact your administrator or service provider with more details about what action you were performing when this occurred.
DB Error: unknown error
```https://lab.civicrm.org/dev/core/-/issues/3130Lack of hooks to detect when an Attachment is deleted2023-12-07T08:27:58ZhaystackLack of hooks to detect when an Attachment is deletedOverview
----------------------------------------
At present, when an Attachment is deleted via the CiviCRM UI (e.g. on "Edit Activity" dialogs/screens) only two CiviCRM hooks fire: the `civi.dao.preDelete` and `civi.dao.postDelete` Symf...Overview
----------------------------------------
At present, when an Attachment is deleted via the CiviCRM UI (e.g. on "Edit Activity" dialogs/screens) only two CiviCRM hooks fire: the `civi.dao.preDelete` and `civi.dao.postDelete` Symfony events triggered by `CRM_Core_BAO_EntityTag::del()`. All other deletions are done via direct SQL queries.
It would be great to:
* Have `hook_civicrm_pre` and `hook_civicrm_post` fire
* Have some way of retrieving the Attachment data prior to deletion
Reproduction steps
----------------------------------------
1. Set up CiviCRM to log all hooks
1. Click the "bin/trash" icon next to an Attachment (as per the screenshot below)
1. Inspect logged callbacks
![attachments-accordion](/uploads/9abded9e0e7f9cc602213b8b20de345d/attachments-accordion.png)
Current behaviour
----------------------------------------
The following callbacks are the only ones that will be seen (data trimmed for concision)
#### `civi.dao.preDelete`
```
[event] => Civi\Core\DAO\Event\PreDelete Object
(
[object] => CRM_Core_BAO_EntityTag Object
(
[id] =>
[entity_table] => civicrm_file
[entity_id] => 6
[tag_id] =>
)
)
```
#### `civi.dao.postDelete`
```
[event] => Civi\Core\DAO\Event\PostDelete Object
(
[object] => CRM_Core_BAO_EntityTag Object
(
[id] =>
[entity_table] => civicrm_file
[entity_id] => 6
[tag_id] =>
)
[result] => 0
)
```
It is possible to use the `entity_id` to retrieve the "File" data via the CiviCRM API. However, because of the order of deletions in [the `deleteEntityFile()` method](https://github.com/civicrm/civicrm-core/blob/6769e8bf8556701b81f914d30fc1bb913f8ed2ce/CRM/Core/BAO/File.php#L250), it is _not_ possible to retrieve the compound "Attachment" data via the API (because the "Entity File" data has already been deleted) and it is therefore not possible to find out which Entity/Entities the File was attached to.
It should also be noted that when the "Delete All Attachment(s)" checkbox is used, the `civi.dao.preDelete` and `civi.dao.postDelete` Symfony events fire *before* `hook_civicrm_pre` is fired for the Activity. To me, it would make more sense for `hook_civicrm_pre` to fire *before* the File deletion process is initiated so that `Activity.pre` and `Activity.post` "wrap" the entire process.
Expected behaviour
----------------------------------------
I would expect to be able to receive callbacks from `hook_civicrm_pre` and `hook_civicrm_post`, where `$op = "delete"` and `$objectName = "File"` along with some more contextual data about the "File" and the "Attachment" being deleted.
Comments
----------------------------------------
I'm going to open a PR that addresses the issue of retrieving the "Attachment" data via the API when using the Symfony hook, since that's pretty straightforward.
I'd appreciate any guidance from those more familiar with this class on the most sensible place(s) to add hooks during the deletion process - or whether deletion should more properly be done via `BAO` objects instead of direct queries.https://lab.civicrm.org/dev/core/-/issues/3137FormBuilder - setting a field as 'required' does not implement any validation...2023-02-28T19:52:15ZUpperholmeFormBuilder - setting a field as 'required' does not implement any validation checkCreating a submission form using Form Builder, it is possible to set any given field to be 'Required'.
I'm assuming that by doing this the form could not then be submitted if the required field is empty. A validation error should be res...Creating a submission form using Form Builder, it is possible to set any given field to be 'Required'.
I'm assuming that by doing this the form could not then be submitted if the required field is empty. A validation error should be resented assisting the user to complete the required field/s.
In practice the form can be submitted with no alert or apparent validation check taking place.
Using CiviCRM 5.47.2 with Wordpress.Kurund JalmiKurund Jalmihttps://lab.civicrm.org/dev/core/-/issues/3139Badgelayouts cannot be edited with PHP warning2022-09-01T13:49:35ZBradley TaylorBadgelayouts cannot be edited with PHP warning_Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit scr..._Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit screen should be pre-filled with the values entered initially.
**Actual outcome**
Each field is blank, a PHP warning is shown:
![Screenshot_2022-03-27_at_10.18.33](/uploads/105dc6046cbb9b569207500da40cf77f/Screenshot_2022-03-27_at_10.18.33.png)
**Technical explanation**
The bug was introduced in https://github.com/civicrm/civicrm-core/commit/873bfeb503caa413f17460dbe450b74fac3d6dbf.
The commit above added a new tokens:
```
'{event.start_date|crmDate:"%B %E%f"}' => ts('Event Start Date'),
'{event.end_date|crmDate:"%B %E%f"}' => ts('Event End Date'),`
```
The data for badge layouts is stored as encoded JSON. This means that the quote marks in these two tokens are being wrapped in double-quotes for the string, causing something like `"{event.start_date|crmDate:"%B %E%f"}"`. As such the JSON is not valid and cannot be `json_decode`ed.
The actual fix could be straightforward enough: Switch the tokens to use single instead of double quotes. However, I'm not sure what the correct solution is for any broken JSON which is now stored in CiviCRM databases. Some sort of upgrade script might be required to find/replace the known broken JSON.
Pinging @eileen who did a lot of work on tokens last year.https://lab.civicrm.org/dev/core/-/issues/3145CiviCase: if contact is an Organization or Household, cannot change Case Coor...2023-11-23T13:15:30ZAllenShawCiviCase: if contact is an Organization or Household, cannot change Case Coordinator (or, must edit Case Coordinator relationship type)# To reproduced on https://dmaster.demo.civicrm.org/ as of today ("Powered by CiviCRM 5.49.alpha1"):
1. Observe that the Homeless Services Coordinator role is defined as in a default installation (contact a type and contact b type are b...# To reproduced on https://dmaster.demo.civicrm.org/ as of today ("Powered by CiviCRM 5.49.alpha1"):
1. Observe that the Homeless Services Coordinator role is defined as in a default installation (contact a type and contact b type are both 'individual')
2. Create a new "Housing Support" case with an Organization contact for the Client.
3. Open Manage Case page for this case
4. Under Roles accordion, click the pencil icon to edit the "Homeless Services Coordinator is (Case Manager)" role
5. Observe pop-up overlay "Reassign Homeless Services Coordinator" where you would select an individual for this role
5. At this point:
* Expected behavior: This pop-up contains a contact reference field allowing me to type a contact name or email address
* Actual behavior: This pop-up may be displayed partially outside of the viewport, with no way to scroll to see all of it; in any case, you can see -- if you're able to use Developer Tools to make the thing display within the viewport -- that the expected contact reference field is just a plain text field; also the Save and Cancel buttons are non-functional, and the only thing I can do is to click the X control to close the pop-up:
![popup](/uploads/92a2aa68c67897f1da5b8ee4ee3e2a7d/popup.png)
# Workaround
1. Edit the Homeless Services Coordinator role to allow Organizations in the Contact A position.
2. Repease the repro steps above and observe expected behavior.
# Other thoughts:
* Testing on client sites indicates his is not limited to the "Homeless Services Coordinator" relationship type; instead, it's happening for any relationship type which is confiugured as the type for the Case Coordinator role.
* Not sure about how best to design a fix from the UX perspective. Prevent case creation? Warn in System Status?https://lab.civicrm.org/dev/core/-/issues/3148Dedupe with multi-select custom fields can trigger IDS2023-03-18T04:20:40ZJonGoldDedupe with multi-select custom fields can trigger IDSWhen deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a che...When deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a checkbox). Note that you need several of these to trigger the "kick" on the IDS (3, I think).
* Create two contacts that are duplicates.
* Find and merge the records.
### Expected result
Contacts merged successfully.
### Actual result
"Your activity is a bit suspicious, hence aborting"
The issue is the POST request, which is passing arguments like `move_custom_12` with the `VALUE_SEPARATOR` control character. This triggers the IDS filter labeled "Detects nullbytes and other dangerous characters".
I'm really not certain what the correct answer is here - I can exempt users from the IDS, and maybe that's the solution to pursue, but it seems like there should be another solution available. Is it possible to exempt certain paths from the IDS, or use an alternate set of rules for a certain path?
Keyword: Intrusion Detection Systemhttps://lab.civicrm.org/dev/core/-/issues/3152Data stored in universal time does not handle DST consistently2023-11-30T05:00:20ZtottenData stored in universal time does not handle DST consistently[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Dayl...[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Daylight Savings Time (DST): if the current-date and the target-date sit on different sides of the DST-switch, then the time may present as +/- 1 hour.
This bug was one of the major subissues identified in https://lab.civicrm.org/dev/core/-/issues/2122. Although that particular feature was rolled-back/deferred from 5.47, the DST bug still exists -- it's just less obvious.
Example use-case
----------------------------------------
1. Create a contact record.
2. View the contact record. Note the creation time (`civicrm_contact.created_date`).
3. Change the system clock - set to a date where DST differs (eg if today is March 30, then go to December 5).
4. View the contact record. Note the creation time (`civicrm_contact.created_date`).
Current behavior
----------------------------------------
The displayed value of `civicrm_contact.created_date` _appears_ to change by +/- 1 hour, depending on when you view it.
> (Viewed on Mar 31, 2022)
>
> ![Screen_Shot_2022-03-31_at_12.40.53_AM](/uploads/8a2f5a0e4fb6ec884aede78e30d31b1a/Screen_Shot_2022-03-31_at_12.40.53_AM.png)
> (Viewed on Dec 5, 2022)
>
> ![Screen_Shot_2022-12-05_at_12.44.57_AM](/uploads/2df387488ee029a949010da163bf2ea8/Screen_Shot_2022-12-05_at_12.44.57_AM.png)
Why? CiviCRM sends a note to MySQL about the current user's timezone (`SET time_zone = '...'`). However, it doesn't identify the timezone effectively. It gives [the current numeric offset (at the moment of viewing)](https://github.com/civicrm/civicrm-core/blob/5.47.3/CRM/Utils/System/Base.php#L758-L762) - but (in locales with DST) the offsets fluctuate over time.
(_Ex: On Mar 31, the offset in California is `-0700`. Under current/long-standing law, the offset will be `-0800` on Dec 5. Of course, the US Congress is reconsidering this law... so we don't really know what the offset will be!_)
Proposed behavior 1: Fix MySQL timezones
----------------------------------------
CiviCRM should send the timezone as a symbolic name, such as `Europe/Helsinki`, `America/Los_Angeles`, or `Australia/Sydney`. These symbolic-names have an underlying database which allows them adjust automatically based on DST-rules/target-dates/current-law. On the surface, the fix is extremely simple:
```diff
diff --git a/CRM/Utils/System/Base.php b/CRM/Utils/System/Base.php
index a4660834c5..8e40f6da35 100644
--- a/CRM/Utils/System/Base.php
+++ b/CRM/Utils/System/Base.php
@@ -755,10 +755,9 @@ abstract class CRM_Utils_System_Base {
* Set timezone in mysql so that timestamp fields show the correct time.
*/
public function setMySQLTimeZone() {
- $timeZoneOffset = $this->getTimeZoneOffset();
- if ($timeZoneOffset) {
- $sql = "SET time_zone = '$timeZoneOffset'";
- CRM_Core_DAO::executequery($sql);
+ $timeZone = $this->getTimeZoneString();
+ if ($timeZone) {
+ CRM_Core_DAO::executequery('SET time_zone = %1', [1 => [$timeZone, 'String']]);
}
}
```
There are a couple of catches.
* __Timezone rules change__ (occasionally). Any software that supports timezones ultimately needs a _data feed_ with current rules. The good news: IANA publishes a free/open feed (https://www.iana.org/time-zones; aka `tzdata`; aka `zoneinfo`), most Linux/Unix distros have this feed, and MySQL can read it (`mysql_tzinfo_to_sql`). It usually requires one command (which could run during system-config, system-startup, and/or cron):
```bash
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql mysql
```
The problem: we have no measures for (a) how many CiviCRM deployments actually subscribe to this feed and (b) how many could subscribe, if they chose to.
* __Timezone names may be inconsistent__ (occasionally). For example, in different contexts, it's been fashionable to refer to California's timezone as `America/Los_Angeles`, `US/Pacific`, and `PST8PDT`. (The current+official fashion is `America/Los_Angeles` - the others are deprecated.) However, since Civi integrates with various layers (different CMSs; PHP APIs; MySQL APIs), there are edge-case where the layers may choose different names. (*I'm not super-concerned, but we should raise sensible warnings when names are invalid or mismatched.*)
The central issue is - how to cope when data isn't available? This comes to mind:
* (Status check) If the active TZ (`getTimeZoneString()`) has a deprecated name (eg `PST8PDT`) or an offset (eg `-0700`), show a warning.
* (Status check) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, show a warning.
* (Runtime) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, fallback to sending offset.
Proposed behavior 2: Change format. Use only PHP TZs.
----------------------------------------
(_This expands on one of @haystack's suggestions in dev/core#2122._)
If you assume that MySQL time services aren't available - what else would you do? You could use PHP time services.
The astute observer will note the status-quo (using both PHP and MySQL time-services) creates two points-of-failure. If either PHP _or_ MySQL has bad/incomplete/old timezone data, then you'll get mis-calculations _somewhere_. Consolidating on PHP time-services would reduce the #dependencies.
Both Drupal and WordPress take this approach. (I suspect this is extremely useful for maximizing compatibility with heterogeneous web-hosts.) They each do it a bit differently, but some central concepts are the same:
* In Drupal, PHP processes read+write temporal data in universal time -- as an `INT` (Unix-style, seconds-since-epoch).
* In WordPress, PHP processes read+write temporal data in universal time -- as a `DATETIME` with a `_gmt` suffix (eg `post_modified_gmt`).
* Hypothetically, you could hardcode MySQL to `SET time_zone='+0:00'`. PHP processes would read+write temporal data in universal time -- as a `TIMESTAMP`.
In all those cases, the onus is on the PHP devs to convert to/from universal-time when implementing functionality (eg "find records from March 1 - March 15" or "find records from this afternoon" or "extract the hour:minute component").
But there is a catch here: Civi already relies on several MySQL time-services. The schema works a certain way; the reports/searches/UIs/APIs expect the schema to work a certain way; etc.
The central issue is - how do you manage/QA all the changes (in schema+logic) required to change time-service?
Comments
----------------------------------------
* I haven't tested, but I'm fairly certain there will be another manifestation in CiviMail scheduling. Ex:
* You live in a timezone where DST changes on March 16.
* On March 10, you schedule a mail-blast for 2:00pm on March 20. It stores the schedule with the wrong offset.
* When March 20 comes, the mailing actually goes out at 3:00pm (or maybe 1:00pm).https://lab.civicrm.org/dev/core/-/issues/3154Custom tokens not working in Scheduled Reminders2022-08-25T21:35:03ZmartyCustom tokens not working in Scheduled RemindersOverview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute No...Overview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute Now option.
Reproduction steps
----------------------------------------
1. Create a custom token using hook_civicrm_container() and implement the civi.token.list and civi.token.eval event listeners.
1. Add a new Scheduled Reminder (I'm using membership end date) and include the custom token in the email message.
1. Create a Cron Job to run civicrm/bin/cron.php periodically (I run every 15 minutes).
1. Enable the Send Scheduled Reminders job and set to run Always.
1. Trigger the reminder appropriately (I create a new membership and set the end date to trigger).
1. Note the custom token is __not__ included in the resulting email after the cron run.
1. Now trigger a new reminder and click Execute Now on the Scheduled Reminders job (before the next cron run).
1. Note the custom token __is evaluated properly__ and included in the resulting email message.
Current behaviour
----------------------------------------
Custom token not included in Scheduled Reminder email when initiated by Cron Job
Expected behaviour
----------------------------------------
Custom token should be included in Scheduled Reminder email when initiated by Cron Job.
Environment information
----------------------------------------
* __CiviCRM:__ _5.47.2_
* __PHP:__ _7.4.28_
* __CMS:__ _WordPress 5.9.2_
* __Database:__ _MySQL_
* __Web Server:__ _Apache_https://lab.civicrm.org/dev/core/-/issues/4452Message Admin doesn't show which message templates have been edited2023-09-18T12:15:28ZlarsssandergreenMessage Admin doesn't show which message templates have been edited~~It looks like Message Admin is now enabled by default, but~~ I've noticed that there is no indication in Message Admin of which templates have been edited or not. This is pretty important info for admins and should be available in the ...~~It looks like Message Admin is now enabled by default, but~~ I've noticed that there is no indication in Message Admin of which templates have been edited or not. This is pretty important info for admins and should be available in the UI.
Compare the old - very clear which has been edited:
![image](/uploads/43963f9cbbf0e08c5f1679056e4a2c35/image.png)
to the new - no indication which has been edited:
![image](/uploads/4a394422a26c4a1536306e96d3c77a7c/image.png)
I can't see anything on the edit page to indicate this either.
Edit: There is also no longer any way to revert a message to the default, other than by manually copying and pasting the template content. I think having the "Revert to default" link is useful, if you made a small change and have decided you no longer need it because there has been an upgrade.
~~Also, I guess the drafting process isn't implemented yet (or not accessible from the UI), so maybe it would make sense to hide the Draft column until that's ready.~~https://lab.civicrm.org/dev/core/-/issues/3789Site language set back to English after update to 5.51.02023-06-16T02:58:58Zpal_urSite language set back to English after update to 5.51.0Overview
----------------------------------------
I've got a CiviCRM site running on Drupal7, with Hungarian language settings.
There are two available languages, English and Hungarian.
I updated my CiviCRM 5.50.3 to 5.51.0, than to ...Overview
----------------------------------------
I've got a CiviCRM site running on Drupal7, with Hungarian language settings.
There are two available languages, English and Hungarian.
I updated my CiviCRM 5.50.3 to 5.51.0, than to 5.52.0, and I lost my site language: it's set back to English, and I'm not able to set back to Hungarian. My translation file is in /sites/all/modules/civicrm/l10n/hu_HU, and the site language is set to hungarian, and in the `civicrm_setting` table there are the following values: `contact_default_language`: `s:5:"hu_HU";`, `lcMessages` `s:5:"hu_HU";`, `uiLanguages` `a:2:{i:0;s:5:"en_US";i:1;s:5:"hu_HU";}`, `format_locale` `s:5:"hu_HU";`.
Reproduction steps
----------------------------------------
Site works with English language.
Current behaviour
----------------------------------------
Expected behaviour
----------------------------------------
Because of the settings I'd like to use my site in Hungarian.
Environment information
----------------------------------------
* __CiviCRM:__ _Master/5.51.0/5.52.0_
* __PHP:__ _PHP 7.4.30 (cli)__
* __CMS:__ Drupal 7.91_
* __Database:__ _10.3.34-MariaDB-0+deb10u1 - Debian 10_
* __Web Server:__ _Apache/2.4.38 (Debian)_
Comments
----------------------------------------https://lab.civicrm.org/dev/core/-/issues/3185Tags for attachments are not properly assigned to the attachment2023-10-07T21:15:50ZDaveDTags for attachments are not properly assigned to the attachmentI'm not sure if this is recent or where it's happening. What happens is that civicrm_entity_tag.entity_id is one higher than the appropriate id from civicrm_file, e.g.
civicrm_file:
| id | file_type_id | mime_type | uri ...I'm not sure if this is recent or where it's happening. What happens is that civicrm_entity_tag.entity_id is one higher than the appropriate id from civicrm_file, e.g.
civicrm_file:
| id | file_type_id | mime_type | uri |
|------|--------------|-------------------|-------------------------------------------------|
|__28__| NULL | text/plain | abc_aef1644a7b96451b6c15b7e34b862f5d.txt |
civicrm_entity_tag:
| id | entity_table | entity_id | tag_id |
|----|------------------|---------------|--------|
| 57 | civicrm_file | --> __29__ <--| 31 |
1. Create a tag set for attachments
1. Create some tags for the set
1. Create an activity
1. In the attachments section add a file and choose a tag
1. When you go back to view or edit the activity, the tag isn't displayed. Check the db and you'll see the entity_id doesn't match up.
1. Manually edit the entity_id in the db and then go back to view the activity. Now the tag is displayed.https://lab.civicrm.org/dev/core/-/issues/3440Check for matching contact on contact add form sends hardcoded fields to dupl...2023-07-06T07:25:40ZdarrickCheck for matching contact on contact add form sends hardcoded fields to duplicatecheck api callOverview
----------------------------------------
If a custom Supervised rule is created using any field not in ['first_name', 'last_name', 'nick_name', 'household_name', 'organization_name', 'email'] then clicking on the **Check for Mat...Overview
----------------------------------------
If a custom Supervised rule is created using any field not in ['first_name', 'last_name', 'nick_name', 'household_name', 'organization_name', 'email'] then clicking on the **Check for Matching Contact** button will return no results.
Reproduction steps
----------------------------------------
1. Click on **Contacts -> Find and Merge Duplicate Contacts**.
2. Click on **Add Individual Rule**
3. Add a rule with field phone, weight 10, threshold 10
4. Click on **Change rule** and set to **Supervised**
5. Click on **Contacts -> New Individual**
6. Add a contact with First Name: Bob, Last Name: Dobbs and phone 666.666.6666
7. Save the contact.
8. Click on **Contacts -> New Individual**
9. Add a contact with First Name: Bob, Last Name: Dobbs and phone 666.666.6666
10. Click on **Check for Matching Contact**
Current behaviour
----------------------------------------
A popup displays "Similar contact if found" after entering the First Name and after entering the Last Name in step 9.
After entering the duplicate phone number nothing happens.
After clicking on **Check for Matching Contact** nothing happens.
Expected behavior
----------------------------------------
After entering either First Name or Last Name nothing should happen unless the entered field is included in the Supervised rule.
A popup displaying "Similar contact if found" should happen after the phone number is entered and also after clicking on **Check for Matching Contact**
Comments
----------------------------------------
I ran across this while looking to see if I could fix any other outstanding bugs related dedupe. Was working on this one: (https://lab.civicrm.org/dev/core/-/issues/2966) I wasn't able to reproduce their issue.
It may still be useful to hard code those fields so by default the form always matches on name or email when entering those fields but then the additional fields will also be searched when added using the custom rule. As any fields not needed for the custom rule will just be ignored.https://lab.civicrm.org/dev/core/-/issues/3441When generating emails from Search results, activity is recorded as a Print P...2023-03-06T20:39:37ZStoobWhen generating emails from Search results, activity is recorded as a Print PDF DocumentSteps to reproduce:
1. use Search Kit to generate results as _Contributions_
2. choose to send emails receipts
3. emails are sent, but activity is recorded "Print/Merge Document" rather than Type "Email".
#searchkitSteps to reproduce:
1. use Search Kit to generate results as _Contributions_
2. choose to send emails receipts
3. emails are sent, but activity is recorded "Print/Merge Document" rather than Type "Email".
#searchkithttps://lab.civicrm.org/dev/core/-/issues/3586Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process ...2023-09-12T02:13:33Zjustinfreeman (Agileware)Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process any emails if a single email is sent to the bounce mailbox with an invalid returnPathCiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service ema...CiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service email" causing all other emails to remain unprocessed. CiviCRM mail reports are therefore incorrect and ultimately the CiviCRM Fetch Bounces scheduled job fails to perform the job it was designed to do.
Example invalid returnPaths as observed in production:
```
x2108-arfsfufwyfgkvni2x7ww5wvac3eifc2wx3vr1fcwbbysieqppis4ytnyip09xcohck9s1vrm77nbg4xh431tthdsmtmfvfpa@rspf.mxthunderstruck.net
```
```
SRS0=3Ypr33=QM=crm.nothappyjohn.org.au=bounce+b.25211.6039846.eb7e7e8ba282f9d0@airquotes.com.au
```
This problem has been highlighted by the introduction of "Scheduled Job Failures" feature which raises the visibility of this type of problem.
Bounce mailboxes have been observed with 10k to 50k unprocessed emails, depending on the time when the "Denial of service email" was received.
Agileware Ref: CIVICRM-1970https://lab.civicrm.org/dev/core/-/issues/3584Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially w...2023-11-09T13:25:31Zjustinfreeman (Agileware)Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and mailing will not sendCiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to ...CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to complete command, DNS not available or timed out 451 Domain of sender address does not resolve_
When the CiviCRM Mailing Job executes again, the process repeats, the same invalid email addresses are used, the same errors are returned and the mailing again will not send.
This process will repeat until someone intervenes, cancelling the mailing, locating and removing the contacts which have the invalid email addresses.
The CiviCRM Mailing Job does not currently check the actual SMTP error code and just treats all SMTP errors as "SMTP Connection Errors". Counting up to 5 and then aborting the job when the threshold is reached.
I think it's fair to call this a "denial of service" because a bad actor could sign up a bunch of fake Contacts to a CiviCRM mailing list with either invalid email domains or valid email domains which are then rendered invalid - and thus cause that mailing list to cease processing.
**Possible solutions**
Change the CiviCRM Mailing Job to check the return SMTP error code and only increment the count for valid "SMTP Connection Errors".
Change the CiviCRM Mailing Job to continue with the mailing, regardless of the 5 errors encountered, which will also skip sending the emails to those Contacts with an invalid email domain. As shown in the patch below.
```
Index: CRM/Mailing/BAO/MailingJob.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/CRM/Mailing/BAO/MailingJob.php b/CRM/Mailing/BAO/MailingJob.php
--- a/CRM/Mailing/BAO/MailingJob.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/CRM/Mailing/BAO/MailingJob.php (date 1620794367592)
@@ -672,20 +672,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryGroup = TRUE;
+ CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $this->writeToDB(
- $deliveredParams,
- $targetParams,
- $mailing,
- $job_date
- );
-
- CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- CRM_Utils_System::civiExit();
}
// Register the bounce event.
Index: ext/flexmailer/src/Listener/DefaultSender.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/ext/flexmailer/src/Listener/DefaultSender.php b/ext/flexmailer/src/Listener/DefaultSender.php
--- a/ext/flexmailer/src/Listener/DefaultSender.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/ext/flexmailer/src/Listener/DefaultSender.php (date 1620794390628)
@@ -76,15 +76,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryBatch = TRUE;
+ \CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $job->writeToDB($deliveredParams, $targetParams, $mailing, $job_date);
-
- \CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- \CRM_Utils_System::civiExit();
}
else {
$this->recordBounce($job, $task, $result->getMessage());
```
Agileware Ref: CIVICRM-1728https://lab.civicrm.org/dev/core/-/issues/3559CiviCRM bounce processing, processed emails are never deleted from the mailbo...2023-09-08T22:34:57Zjustinfreeman (Agileware)CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little valueCiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox,...CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox, or
1. A new Scheduled Job process should be implemented which deletes all processed email which is older than 30 days
Relates to https://lab.civicrm.org/dev/mail/-/issues/108
Agileware Ref: CIVICRM-1977https://lab.civicrm.org/dev/core/-/issues/3453Afform - relationships fill from other entity2022-12-07T01:43:54ZsamuelsovAfform - relationships fill from other entityFollow-up from https://github.com/civicrm/civicrm-core/pull/23296#issuecomment-1119014194
Let say we want to do a form that allow an employer to update :
- the main contact of the organization
- all the employees
Thanks to https://lab....Follow-up from https://github.com/civicrm/civicrm-core/pull/23296#issuecomment-1119014194
Let say we want to do a form that allow an employer to update :
- the main contact of the organization
- all the employees
Thanks to https://lab.civicrm.org/dev/core/-/issues/3117 it's now possible to do a form that will allow to create in one step :
- the organization
- the main contact as Individual 1 with a relationship between the organization and Individual 1
- the employees as Individual 2 with a relationship between the organization and Invividual 2 (which is multiple)
However, there is no way to have an edit mode for such a form. It's possible to add the organization id as an argument but we also need a way to pre-populate the main contacts / list of employees as an option based on the relationships definition.colemanwcolemanwhttps://lab.civicrm.org/dev/core/-/issues/3462Event location search2024-01-31T10:18:23Zaydunsaidan.saunders@squiffle.ukEvent location searchOverview
----------------------------------------
Following on from #2103 - when configuring an event and reusing a location, Civi shows a message like 'This location is used by 2 other events' but does not indicate which events.
It w...Overview
----------------------------------------
Following on from #2103 - when configuring an event and reusing a location, Civi shows a message like 'This location is used by 2 other events' but does not indicate which events.
It would be useful to show a list of those locations, or include a link to search for them.
Note that SearchKit displays now handle LocBlocks (see #2676)https://lab.civicrm.org/dev/core/-/issues/3470Search Kit: Mailing labels don't work2022-12-19T15:47:10ZJonGoldSearch Kit: Mailing labels don't workMailing labels don't work with Search Kit.
**Steps to Replicate**
* Do a search for contacts in search kit (with a small number of contacts to avoid hitting #3222).
* Select 1 or more contacts, select **Mailing Labels** from **Actions**...Mailing labels don't work with Search Kit.
**Steps to Replicate**
* Do a search for contacts in search kit (with a small number of contacts to avoid hitting #3222).
* Select 1 or more contacts, select **Mailing Labels** from **Actions**.
* Select a label and press **Make Mailing Labels**.
**Expected Result**
Mailing labels.
**Actual Result**
Spins indefinitely.
This doesn't happen when making PDFs, or if you grab the last HTTP request from the devtools Network tab and open it in a new window.
It happens because TCPDF isn't expecting to be in a "snippet" context at this point - it's just spitting out raw HTML response headers direct to the browser (`CRM_Contact_Form_Task_Label::createLabel()` calls `$pdf->Output()`). "Make documents" uses the more modern `CRM_Utils_PDF_Utils::html2pdf(), which uses DOMPDF or wkHTMLtoPDF, and knows how to handle this. Unfortunately, TCPDF has the mailing label functionality, so we're stuck with it.
**How to proceed**
I could probably make a fix that involves saving the file to disk first - but I think a better fix is to modify the Search Kit "Actions" JS to open the labels in a new window, but I can't find where that happens.https://lab.civicrm.org/dev/core/-/issues/3478Afform - email delivery2023-06-29T15:47:01Zaydunsaidan.saunders@squiffle.ukAfform - email deliveryOverview
----------------------------------------
Add an action to enable scheduled Email Delivery of SearchKit results.
Reports provides settings to enable scheduled Email Delivery of reports. The lack of an option to do the same dete...Overview
----------------------------------------
Add an action to enable scheduled Email Delivery of SearchKit results.
Reports provides settings to enable scheduled Email Delivery of reports. The lack of an option to do the same deters some people from using SearchKit - see https://civicrm.stackexchange.com/q/41966/225
Comments
----------------------------------------
Not a high priority item but noting for parity with Reports functionality.https://lab.civicrm.org/dev/core/-/issues/3490Upgrader - Apply extension updates after core updates2023-02-08T19:08:30ZtottenUpgrader - Apply extension updates after core updatesOverview
----------------------------------------
Functionality in `civicrm-core` is increasingly organized with internal extensions (`civicrm-core:ext/*`). The main upgrade workflow should incorporate the upgrade-steps for extensions.
...Overview
----------------------------------------
Functionality in `civicrm-core` is increasingly organized with internal extensions (`civicrm-core:ext/*`). The main upgrade workflow should incorporate the upgrade-steps for extensions.
Example use-case
----------------------------------------
1. Extract tarball with a new version of `civicrm-core`, including updated `ext/*`
1. Run the main upgrader (`civicrm/upgrade?reset=1`, `cv core:upgrade`, `drush civicrm-upgrade-db`, or `wp civicrm upgrade-db`)
Current behaviour
----------------------------------------
It updates the schema for things like `civicrm_contact` (eg `CRM/Upgrade/*`) but not things like `civicrm_search_segment` (eg `ext/search_kit/CRM/Search/Upgrader.php`).
As a sysadmin, you must run the upgrades separately.
Most are not habituated to running these together. This leads to confusing user-stories like https://civicrm.stackexchange.com/questions/42094/civimail-issue-db-error-no-such-table-when-sending-mailing (where the first upgrade seemed to work - but you get random errors because the second upgrade hasn't run).
Proposed behaviour
----------------------------------------
Run both.
The core-upgrade should delegate/incorporate the ext-upgrade, and it should preserve essential elements of the ext-upgrade contract. Specifically: ext-upgrades can use features/services/APIs from core. This implies that (1) ext-upgrades run after core-upgrades and (2) when it gets to ext-upgrades, it should follow a normal/open dispatch-policy.
Comments
----------------------------------------
Filing issue after some Mattermost discussion with rudanpal, @colemanw, @bgm, @totten: https://chat.civicrm.org/civicrm/pl/b76y4gy517yxdet331ueqrxgrw
I posted a draft at https://github.com/civicrm/civicrm-core/compare/5.50...totten:run-ext-upgrades?expand=1
As mentioned in PHP comments+Mattermost, there's an issue with the current draft -- when it gets to ext-upgrade tasks, it would (unexpectedly) continue to apply the limited dispatch-policy used for core-upgrades (`upgrade.main` vs `upgrade.finish`). Some ways to address that:
* __Runner state__: While the upgrader goes through phases (*ex: 1-core upgrade, 2-ext upgrade, 3-new exts*), keep a stored value naming the phase. this stored value determines the active dispatch-policy
* ex: `Civi::settings()->set('upgrade_phase', '...')`
* __Task wrapping/task tagging__: When taking ext-upgrade tasks and putting them into the core-upgrade queue, put some wrapper or tag on each one to indicate the phase/policy that it relies on
* wrapper example: `CRM_Upgrade_Form::wrappedTask(string $dispatchPolicy, array $realCallback)
* tag example: `$queue->createItem($task, ['dispatchPolicy' => string])`https://lab.civicrm.org/dev/core/-/issues/3493Afform: Failed to find key by ID or tag (SIGN)2022-12-19T21:18:09ZandyburnsAfform: Failed to find key by ID or tag (SIGN)I created a simple afform submitting a name and it just spins and shows this console error. On Civi 5.50.0, WP 5.8.4 and multisite. I had this issue on 5.46.3 as well. It does submit the contact even though it appears not to.
![image](/...I created a simple afform submitting a name and it just spins and shows this console error. On Civi 5.50.0, WP 5.8.4 and multisite. I had this issue on 5.46.3 as well. It does submit the contact even though it appears not to.
![image](/uploads/98a9537c12d29cda91345959e268d8a0/image.png)
Sometimes only the angular error:
![image](/uploads/4e77c2cbad1b88909dc62e7b4e3d4500/image.png)
No error reported to civi logs.