Development issueshttps://lab.civicrm.org/groups/dev/-/issues2022-09-01T13:49:35Zhttps://lab.civicrm.org/dev/core/-/issues/3139Badgelayouts cannot be edited with PHP warning2022-09-01T13:49:35ZBradley TaylorBadgelayouts cannot be edited with PHP warning_Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit scr..._Reproduced on dmaster and locally on WordPress_
**Steps to reproduce**
1. Navigate to "Administer CiviCRM", "Event Name Badge Layouts".
2. Create a new name badge
3. Edit the newly created name badge.
**Expected outcode**
The edit screen should be pre-filled with the values entered initially.
**Actual outcome**
Each field is blank, a PHP warning is shown:
![Screenshot_2022-03-27_at_10.18.33](/uploads/105dc6046cbb9b569207500da40cf77f/Screenshot_2022-03-27_at_10.18.33.png)
**Technical explanation**
The bug was introduced in https://github.com/civicrm/civicrm-core/commit/873bfeb503caa413f17460dbe450b74fac3d6dbf.
The commit above added a new tokens:
```
'{event.start_date|crmDate:"%B %E%f"}' => ts('Event Start Date'),
'{event.end_date|crmDate:"%B %E%f"}' => ts('Event End Date'),`
```
The data for badge layouts is stored as encoded JSON. This means that the quote marks in these two tokens are being wrapped in double-quotes for the string, causing something like `"{event.start_date|crmDate:"%B %E%f"}"`. As such the JSON is not valid and cannot be `json_decode`ed.
The actual fix could be straightforward enough: Switch the tokens to use single instead of double quotes. However, I'm not sure what the correct solution is for any broken JSON which is now stored in CiviCRM databases. Some sort of upgrade script might be required to find/replace the known broken JSON.
Pinging @eileen who did a lot of work on tokens last year.https://lab.civicrm.org/dev/core/-/issues/3145CiviCase: if contact is an Organization or Household, cannot change Case Coor...2023-11-23T13:15:30ZAllenShawCiviCase: if contact is an Organization or Household, cannot change Case Coordinator (or, must edit Case Coordinator relationship type)# To reproduced on https://dmaster.demo.civicrm.org/ as of today ("Powered by CiviCRM 5.49.alpha1"):
1. Observe that the Homeless Services Coordinator role is defined as in a default installation (contact a type and contact b type are b...# To reproduced on https://dmaster.demo.civicrm.org/ as of today ("Powered by CiviCRM 5.49.alpha1"):
1. Observe that the Homeless Services Coordinator role is defined as in a default installation (contact a type and contact b type are both 'individual')
2. Create a new "Housing Support" case with an Organization contact for the Client.
3. Open Manage Case page for this case
4. Under Roles accordion, click the pencil icon to edit the "Homeless Services Coordinator is (Case Manager)" role
5. Observe pop-up overlay "Reassign Homeless Services Coordinator" where you would select an individual for this role
5. At this point:
* Expected behavior: This pop-up contains a contact reference field allowing me to type a contact name or email address
* Actual behavior: This pop-up may be displayed partially outside of the viewport, with no way to scroll to see all of it; in any case, you can see -- if you're able to use Developer Tools to make the thing display within the viewport -- that the expected contact reference field is just a plain text field; also the Save and Cancel buttons are non-functional, and the only thing I can do is to click the X control to close the pop-up:
![popup](/uploads/92a2aa68c67897f1da5b8ee4ee3e2a7d/popup.png)
# Workaround
1. Edit the Homeless Services Coordinator role to allow Organizations in the Contact A position.
2. Repease the repro steps above and observe expected behavior.
# Other thoughts:
* Testing on client sites indicates his is not limited to the "Homeless Services Coordinator" relationship type; instead, it's happening for any relationship type which is confiugured as the type for the Case Coordinator role.
* Not sure about how best to design a fix from the UX perspective. Prevent case creation? Warn in System Status?https://lab.civicrm.org/dev/core/-/issues/3148Dedupe with multi-select custom fields can trigger IDS2023-03-18T04:20:40ZJonGoldDedupe with multi-select custom fields can trigger IDSWhen deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a che...When deduping contacts that have multi-select custom fields, and selecting to move the custom fields to the new contact, the IDS is triggered.
### Steps to replicate
* Create a custom field that allows saving multiple values (e.g. a checkbox). Note that you need several of these to trigger the "kick" on the IDS (3, I think).
* Create two contacts that are duplicates.
* Find and merge the records.
### Expected result
Contacts merged successfully.
### Actual result
"Your activity is a bit suspicious, hence aborting"
The issue is the POST request, which is passing arguments like `move_custom_12` with the `VALUE_SEPARATOR` control character. This triggers the IDS filter labeled "Detects nullbytes and other dangerous characters".
I'm really not certain what the correct answer is here - I can exempt users from the IDS, and maybe that's the solution to pursue, but it seems like there should be another solution available. Is it possible to exempt certain paths from the IDS, or use an alternate set of rules for a certain path?
Keyword: Intrusion Detection Systemhttps://lab.civicrm.org/dev/core/-/issues/3152Data stored in universal time does not handle DST consistently2023-11-30T05:00:20ZtottenData stored in universal time does not handle DST consistently[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Dayl...[[_TOC_]]
Overview
----------------------------------------
CiviCRM has a number of `TIMESTAMP` columns -- these are stored in universal time (UTC) and displayed in the user's timezone. However, there is a subtle error in handling Daylight Savings Time (DST): if the current-date and the target-date sit on different sides of the DST-switch, then the time may present as +/- 1 hour.
This bug was one of the major subissues identified in https://lab.civicrm.org/dev/core/-/issues/2122. Although that particular feature was rolled-back/deferred from 5.47, the DST bug still exists -- it's just less obvious.
Example use-case
----------------------------------------
1. Create a contact record.
2. View the contact record. Note the creation time (`civicrm_contact.created_date`).
3. Change the system clock - set to a date where DST differs (eg if today is March 30, then go to December 5).
4. View the contact record. Note the creation time (`civicrm_contact.created_date`).
Current behavior
----------------------------------------
The displayed value of `civicrm_contact.created_date` _appears_ to change by +/- 1 hour, depending on when you view it.
> (Viewed on Mar 31, 2022)
>
> ![Screen_Shot_2022-03-31_at_12.40.53_AM](/uploads/8a2f5a0e4fb6ec884aede78e30d31b1a/Screen_Shot_2022-03-31_at_12.40.53_AM.png)
> (Viewed on Dec 5, 2022)
>
> ![Screen_Shot_2022-12-05_at_12.44.57_AM](/uploads/2df387488ee029a949010da163bf2ea8/Screen_Shot_2022-12-05_at_12.44.57_AM.png)
Why? CiviCRM sends a note to MySQL about the current user's timezone (`SET time_zone = '...'`). However, it doesn't identify the timezone effectively. It gives [the current numeric offset (at the moment of viewing)](https://github.com/civicrm/civicrm-core/blob/5.47.3/CRM/Utils/System/Base.php#L758-L762) - but (in locales with DST) the offsets fluctuate over time.
(_Ex: On Mar 31, the offset in California is `-0700`. Under current/long-standing law, the offset will be `-0800` on Dec 5. Of course, the US Congress is reconsidering this law... so we don't really know what the offset will be!_)
Proposed behavior 1: Fix MySQL timezones
----------------------------------------
CiviCRM should send the timezone as a symbolic name, such as `Europe/Helsinki`, `America/Los_Angeles`, or `Australia/Sydney`. These symbolic-names have an underlying database which allows them adjust automatically based on DST-rules/target-dates/current-law. On the surface, the fix is extremely simple:
```diff
diff --git a/CRM/Utils/System/Base.php b/CRM/Utils/System/Base.php
index a4660834c5..8e40f6da35 100644
--- a/CRM/Utils/System/Base.php
+++ b/CRM/Utils/System/Base.php
@@ -755,10 +755,9 @@ abstract class CRM_Utils_System_Base {
* Set timezone in mysql so that timestamp fields show the correct time.
*/
public function setMySQLTimeZone() {
- $timeZoneOffset = $this->getTimeZoneOffset();
- if ($timeZoneOffset) {
- $sql = "SET time_zone = '$timeZoneOffset'";
- CRM_Core_DAO::executequery($sql);
+ $timeZone = $this->getTimeZoneString();
+ if ($timeZone) {
+ CRM_Core_DAO::executequery('SET time_zone = %1', [1 => [$timeZone, 'String']]);
}
}
```
There are a couple of catches.
* __Timezone rules change__ (occasionally). Any software that supports timezones ultimately needs a _data feed_ with current rules. The good news: IANA publishes a free/open feed (https://www.iana.org/time-zones; aka `tzdata`; aka `zoneinfo`), most Linux/Unix distros have this feed, and MySQL can read it (`mysql_tzinfo_to_sql`). It usually requires one command (which could run during system-config, system-startup, and/or cron):
```bash
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql mysql
```
The problem: we have no measures for (a) how many CiviCRM deployments actually subscribe to this feed and (b) how many could subscribe, if they chose to.
* __Timezone names may be inconsistent__ (occasionally). For example, in different contexts, it's been fashionable to refer to California's timezone as `America/Los_Angeles`, `US/Pacific`, and `PST8PDT`. (The current+official fashion is `America/Los_Angeles` - the others are deprecated.) However, since Civi integrates with various layers (different CMSs; PHP APIs; MySQL APIs), there are edge-case where the layers may choose different names. (*I'm not super-concerned, but we should raise sensible warnings when names are invalid or mismatched.*)
The central issue is - how to cope when data isn't available? This comes to mind:
* (Status check) If the active TZ (`getTimeZoneString()`) has a deprecated name (eg `PST8PDT`) or an offset (eg `-0700`), show a warning.
* (Status check) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, show a warning.
* (Runtime) If the active TZ (`getTimeZoneString()`) isn't supported by MySQL, fallback to sending offset.
Proposed behavior 2: Change format. Use only PHP TZs.
----------------------------------------
(_This expands on one of @haystack's suggestions in dev/core#2122._)
If you assume that MySQL time services aren't available - what else would you do? You could use PHP time services.
The astute observer will note the status-quo (using both PHP and MySQL time-services) creates two points-of-failure. If either PHP _or_ MySQL has bad/incomplete/old timezone data, then you'll get mis-calculations _somewhere_. Consolidating on PHP time-services would reduce the #dependencies.
Both Drupal and WordPress take this approach. (I suspect this is extremely useful for maximizing compatibility with heterogeneous web-hosts.) They each do it a bit differently, but some central concepts are the same:
* In Drupal, PHP processes read+write temporal data in universal time -- as an `INT` (Unix-style, seconds-since-epoch).
* In WordPress, PHP processes read+write temporal data in universal time -- as a `DATETIME` with a `_gmt` suffix (eg `post_modified_gmt`).
* Hypothetically, you could hardcode MySQL to `SET time_zone='+0:00'`. PHP processes would read+write temporal data in universal time -- as a `TIMESTAMP`.
In all those cases, the onus is on the PHP devs to convert to/from universal-time when implementing functionality (eg "find records from March 1 - March 15" or "find records from this afternoon" or "extract the hour:minute component").
But there is a catch here: Civi already relies on several MySQL time-services. The schema works a certain way; the reports/searches/UIs/APIs expect the schema to work a certain way; etc.
The central issue is - how do you manage/QA all the changes (in schema+logic) required to change time-service?
Comments
----------------------------------------
* I haven't tested, but I'm fairly certain there will be another manifestation in CiviMail scheduling. Ex:
* You live in a timezone where DST changes on March 16.
* On March 10, you schedule a mail-blast for 2:00pm on March 20. It stores the schedule with the wrong offset.
* When March 20 comes, the mailing actually goes out at 3:00pm (or maybe 1:00pm).https://lab.civicrm.org/dev/core/-/issues/3154Custom tokens not working in Scheduled Reminders2022-08-25T21:35:03ZmartyCustom tokens not working in Scheduled RemindersOverview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute No...Overview
----------------------------------------
Custom tokens are not evaluated for Scheduled Reminders when initiated by Cron Job. The tokens are evaluated properly when the Scheduled Reminders job is run manually using the Execute Now option.
Reproduction steps
----------------------------------------
1. Create a custom token using hook_civicrm_container() and implement the civi.token.list and civi.token.eval event listeners.
1. Add a new Scheduled Reminder (I'm using membership end date) and include the custom token in the email message.
1. Create a Cron Job to run civicrm/bin/cron.php periodically (I run every 15 minutes).
1. Enable the Send Scheduled Reminders job and set to run Always.
1. Trigger the reminder appropriately (I create a new membership and set the end date to trigger).
1. Note the custom token is __not__ included in the resulting email after the cron run.
1. Now trigger a new reminder and click Execute Now on the Scheduled Reminders job (before the next cron run).
1. Note the custom token __is evaluated properly__ and included in the resulting email message.
Current behaviour
----------------------------------------
Custom token not included in Scheduled Reminder email when initiated by Cron Job
Expected behaviour
----------------------------------------
Custom token should be included in Scheduled Reminder email when initiated by Cron Job.
Environment information
----------------------------------------
* __CiviCRM:__ _5.47.2_
* __PHP:__ _7.4.28_
* __CMS:__ _WordPress 5.9.2_
* __Database:__ _MySQL_
* __Web Server:__ _Apache_https://lab.civicrm.org/dev/drupal/-/issues/177Cannot resolve path using "cms.root.url"2022-04-29T09:11:35ZalmadorxCannot resolve path using "cms.root.url"Hi! I'm having the issue with
```
In Paths.php line 140:
Cannot resolve path using "cms.root.url"
```
I'm using Drupal 9 and the last version of CiviCRM
I've tried the solution from here
([regression `cv` fails on CiviCRM 5.15.0](htt...Hi! I'm having the issue with
```
In Paths.php line 140:
Cannot resolve path using "cms.root.url"
```
I'm using Drupal 9 and the last version of CiviCRM
I've tried the solution from here
([regression `cv` fails on CiviCRM 5.15.0](https://lab.civicrm.org/dev/drupal/-/issues/75))
\vendor\civicrm\civicrm-core\CRM\Utils\System\Drupal8.php:
` public function getCurrentLanguage() {
// Drupal might not be bootstrapped if being called by the REST API.
if (!class_exists('Drupal') || !\Drupal::hasContainer()) {
return NULL;`
I've replaced _return NULL;_ with _return $url;_ but that doesn't solved the issue.https://lab.civicrm.org/dev/translation/-/issues/75Translatable fields within Extension table2022-04-21T20:39:46ZseamusleeTranslatable fields within Extension tableAt present the CiviCRM i18n widget code and also the i10n schema code depend on the functions in the schema structure file returning an array of fields or html widgets. The Schema Structure file does not contain any information about tab...At present the CiviCRM i18n widget code and also the i10n schema code depend on the functions in the schema structure file returning an array of fields or html widgets. The Schema Structure file does not contain any information about tables other than core tables so for example in the JMA Grant Applications extension if we wanted to make the title or the introductory text field translatable we would have to hack the schema structure file.
I would like to propose that we create 3 new Events or similar to handle altering the fields, indices and widgets functions in the schema structure file.https://lab.civicrm.org/dev/core/-/issues/3185Tags for attachments are not properly assigned to the attachment2023-10-07T21:15:50ZDaveDTags for attachments are not properly assigned to the attachmentI'm not sure if this is recent or where it's happening. What happens is that civicrm_entity_tag.entity_id is one higher than the appropriate id from civicrm_file, e.g.
civicrm_file:
| id | file_type_id | mime_type | uri ...I'm not sure if this is recent or where it's happening. What happens is that civicrm_entity_tag.entity_id is one higher than the appropriate id from civicrm_file, e.g.
civicrm_file:
| id | file_type_id | mime_type | uri |
|------|--------------|-------------------|-------------------------------------------------|
|__28__| NULL | text/plain | abc_aef1644a7b96451b6c15b7e34b862f5d.txt |
civicrm_entity_tag:
| id | entity_table | entity_id | tag_id |
|----|------------------|---------------|--------|
| 57 | civicrm_file | --> __29__ <--| 31 |
1. Create a tag set for attachments
1. Create some tags for the set
1. Create an activity
1. In the attachments section add a file and choose a tag
1. When you go back to view or edit the activity, the tag isn't displayed. Check the db and you'll see the entity_id doesn't match up.
1. Manually edit the entity_id in the db and then go back to view the activity. Now the tag is displayed.https://lab.civicrm.org/dev/release/-/issues/18Scheduling/workflow for security updates of dependencies2022-04-29T08:58:12ZtottenScheduling/workflow for security updates of dependencies# Synopsis
The workflow for *first-party/in-house* security updates and the workflow for *third-party/upstream* security updates are qualitatively different
The question of this issue is: Do we keep one general policy/schedule for both...# Synopsis
The workflow for *first-party/in-house* security updates and the workflow for *third-party/upstream* security updates are qualitatively different
The question of this issue is: Do we keep one general policy/schedule for both kinds of security issues, or do we have a more nuanced policy that distinguishes between them?
# Background
CiviCRM's policy for scheduling/workflow on security updates has a few key elements:
* Report and discuss vulnerabilities privately
* Release updates on a designated release window (the first/third Wed of each month)
* Make an effort to pre-announce (often 1-4 weeks in advance)
Those bullets are based on the premise that we control the process for disclosure/development/etc. This is true and appropriate for the common case where the security vulnerability originates in code maintained directly by CiviCRM.
But there is another common case: *dependencies* ("libraries", "packages", "subpackages", etc) used by CiviCRM and maintained by another group. These break the bullet-points from above:
* The purpose of upstream's public advisory is *to notify people like us*. The issue is necessarily public when we get the information.
* There are several different upstreams. Their release scheduling is (on the whole) fluid - some have release-windows; some don't; some make pre-announcements; some don't; each of those policies may change over time.
* The vulnerability is public. Delaying the release (in service of a pre-announcement/spin-up period) exacerbates the risk exposure. We don't want our scheduling to add extra exposure.
The security updates of a dependency affect CiviCRM in a few ways. Anyone reading this probably has some understanding already. But just to be complete, those effects include:
* Dependency-updates require some correlated change in how we use that dependency. In the best case, that just means metadata (eg `composer.json`, `composer.lock`) - but it can also be much more involved. It varies case-by-case.
* Several artifacts need to be republished when a dependency changes (notably the tarballs/zipballs for WP/D7/BD/J - but also any images published via docker, etc).
# Proposal
Security fixes _that have been previously published by an upstream vendor_ should be assimilated through CiviCRM's public development channels (Gitlab/Github/Mattermost/etc). The process should closely match the process for patch-releases that fix recent-regressions:
* Like a regular patch-release...
* Any patches/PRs should be submitted to the RC's public queue.
* After approving the RC PR, then backport to stable/ESR. (Only backport if we believe it to be "likely" exploitable.)
* Discussion about testing, `r-run`, compatibility, etc can happen in the public PR.
* We do not need to assign a CIVI-SA-* identifier or write an "advisory" record.
* In addition, there are extra bits...
* We'll send a mailblast when the stable/ESR updates are published.
* Release notes should highlight the "Security" issue as distinct from any other "Bug fix" issues. They should link to upstream's advisory (in addition to the usual Github/Gitlab links for Civi).
* In the public media, don't discuss how to specifically exploit the vulnerability. If that requires discussion, go to private Mattermost (`security` and/or direct-message). The public PR may have general claims (eg "I have successfully exploited this on my local system"; or "Alice, Bob, and Carol discussed on security channel - and all felt it is probably exploitable.")
* Backports for stable and ESR will be done in parallel. (They may be done by different people).
All other security issues (ie *first-party vulnerabilities; unpublished third-party vulnerabilities; uninvestigated vulnerabilities*) should continue through the current (private) workflows.
We should update https://civicrm.org/security to indicate this distinction.
# Rationale
* If a black hat is motivated enough to monitor CiviCRM's issue/PR queue for heads-up about CiviCRM vulnerabilities, then they can just as easily monitor the official release feeds for `dompdf`, `ckeditor`, etc.
* Github's "dependabot" is already likely to post public PRs when there's a published vulnerability affecting a CiviCRM dependency.
* Pro-active contributors will find it natural to raise these issues publicly (because they're already public).
* This change should reduce typical turn-around-time / duration-of-exposure for this type of issue. (*Compare: 2 weeks vs 0-3 days*)
* Routing dependency-updates through the private security medium adds noise to the private tracker without adding much security benefit.
# Other thoughts
Microsoft made "Patch Tuesday" famous. But they generally own all their dependencies.
Drupal has landed on "third Wed" as their release-window. However, they appear to make an exception when a third-party library publishes outside their preferred schedule (ex: https://www.drupal.org/sa-core-2022-006).
If we relax the scheduling on dependency updates, then we don't need to keep 1st Wed on the books. CiviCRM-specific fixes could be like Drupal -- strictly third Wed.
Anecdotally, I feel upstream announcements land on a weekday (esp Tue/Wed/Thu) -- and this lines up the interest of deployers. We could lean into this (eg dependency updates only happen on weekdays).
Note: Backdrop's release-window is _any Wed_. AFAICS, WordPress, Joomla, and PHP don't have formal release-windows. Based on skimming advisories, Joomla has a strong Tue bias. WP+PHP float around. (Between them, I skimmed ~20 prior releases, and there was only one that landed on a weekend.)https://lab.civicrm.org/dev/backdrop/-/issues/9Shoreditch styling2023-01-18T16:43:38ZlarynShoreditch stylingWould it be appropriate to add some CSS and/or functionality that cleans up the Shoreditch display out of the box for Backdrop? (Is Shoreditch still slated to become the default theme at some point?)
For example:
- https://github.com/...Would it be appropriate to add some CSS and/or functionality that cleans up the Shoreditch display out of the box for Backdrop? (Is Shoreditch still slated to become the default theme at some point?)
For example:
- https://github.com/civicrm/org.civicrm.shoreditch/issues/539
- I've also noticed the Backdrop-specific CSS clobbers a little too hard and overrides in Shoreditch as well:
https://github.com/civicrm/civicrm-backdrop/blob/1.x-master/civicrm_backdrop.css#L5-L10
(We may be able to tweak those to hit a sweet spot where it overrides Backdrop styles as desired, but not Shoreditch tab styles).https://lab.civicrm.org/dev/drupal/-/issues/180Naming: Drupal "8" references are anachronistic on Drupal 9/102022-11-18T15:01:20ZtottenNaming: Drupal "8" references are anachronistic on Drupal 9/10If you install CiviCRM-Drupal on D9/D10, some of the technical-names have anachronistic references to D8, eg
* The git repo name (`civicrm-drupal-8.git`)
* The composer package name (`civicrm/civicrm-drupal-8`)
* The UF name (`Drupal8`)...If you install CiviCRM-Drupal on D9/D10, some of the technical-names have anachronistic references to D8, eg
* The git repo name (`civicrm-drupal-8.git`)
* The composer package name (`civicrm/civicrm-drupal-8`)
* The UF name (`Drupal8`)
* Any classes based on the UF name (`CRM_Utils_System_{$UF}`, `CRM_Utils_Hook_{$UF}`, `CRM_Core_Permission_{$UF}`)
This is an off-shoot from discussion with @AlanDixon on https://lab.civicrm.org/dev/drupal/-/issues/178#note_74093https://lab.civicrm.org/dev/core/-/issues/3440Check for matching contact on contact add form sends hardcoded fields to dupl...2023-07-06T07:25:40ZdarrickCheck for matching contact on contact add form sends hardcoded fields to duplicatecheck api callOverview
----------------------------------------
If a custom Supervised rule is created using any field not in ['first_name', 'last_name', 'nick_name', 'household_name', 'organization_name', 'email'] then clicking on the **Check for Mat...Overview
----------------------------------------
If a custom Supervised rule is created using any field not in ['first_name', 'last_name', 'nick_name', 'household_name', 'organization_name', 'email'] then clicking on the **Check for Matching Contact** button will return no results.
Reproduction steps
----------------------------------------
1. Click on **Contacts -> Find and Merge Duplicate Contacts**.
2. Click on **Add Individual Rule**
3. Add a rule with field phone, weight 10, threshold 10
4. Click on **Change rule** and set to **Supervised**
5. Click on **Contacts -> New Individual**
6. Add a contact with First Name: Bob, Last Name: Dobbs and phone 666.666.6666
7. Save the contact.
8. Click on **Contacts -> New Individual**
9. Add a contact with First Name: Bob, Last Name: Dobbs and phone 666.666.6666
10. Click on **Check for Matching Contact**
Current behaviour
----------------------------------------
A popup displays "Similar contact if found" after entering the First Name and after entering the Last Name in step 9.
After entering the duplicate phone number nothing happens.
After clicking on **Check for Matching Contact** nothing happens.
Expected behavior
----------------------------------------
After entering either First Name or Last Name nothing should happen unless the entered field is included in the Supervised rule.
A popup displaying "Similar contact if found" should happen after the phone number is entered and also after clicking on **Check for Matching Contact**
Comments
----------------------------------------
I ran across this while looking to see if I could fix any other outstanding bugs related dedupe. Was working on this one: (https://lab.civicrm.org/dev/core/-/issues/2966) I wasn't able to reproduce their issue.
It may still be useful to hard code those fields so by default the form always matches on name or email when entering those fields but then the additional fields will also be searched when added using the custom rule. As any fields not needed for the custom rule will just be ignored.https://lab.civicrm.org/dev/core/-/issues/3441When generating emails from Search results, activity is recorded as a Print P...2023-03-06T20:39:37ZStoobWhen generating emails from Search results, activity is recorded as a Print PDF DocumentSteps to reproduce:
1. use Search Kit to generate results as _Contributions_
2. choose to send emails receipts
3. emails are sent, but activity is recorded "Print/Merge Document" rather than Type "Email".
#searchkitSteps to reproduce:
1. use Search Kit to generate results as _Contributions_
2. choose to send emails receipts
3. emails are sent, but activity is recorded "Print/Merge Document" rather than Type "Email".
#searchkithttps://lab.civicrm.org/dev/core/-/issues/3586Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process ...2023-09-12T02:13:33Zjustinfreeman (Agileware)Denial of service - CiviCRM Fetch Bounces scheduled job will fail to process any emails if a single email is sent to the bounce mailbox with an invalid returnPathCiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service ema...CiviCRM bounce processing will fail to process any emails if a single email ("Denial of service email") is sent to the bounce mailbox with an invalid returnPath.
CiviCRM will repeatedly try and fail to process the "Denial of service email" causing all other emails to remain unprocessed. CiviCRM mail reports are therefore incorrect and ultimately the CiviCRM Fetch Bounces scheduled job fails to perform the job it was designed to do.
Example invalid returnPaths as observed in production:
```
x2108-arfsfufwyfgkvni2x7ww5wvac3eifc2wx3vr1fcwbbysieqppis4ytnyip09xcohck9s1vrm77nbg4xh431tthdsmtmfvfpa@rspf.mxthunderstruck.net
```
```
SRS0=3Ypr33=QM=crm.nothappyjohn.org.au=bounce+b.25211.6039846.eb7e7e8ba282f9d0@airquotes.com.au
```
This problem has been highlighted by the introduction of "Scheduled Job Failures" feature which raises the visibility of this type of problem.
Bounce mailboxes have been observed with 10k to 50k unprocessed emails, depending on the time when the "Denial of service email" was received.
Agileware Ref: CIVICRM-1970https://lab.civicrm.org/dev/core/-/issues/3584Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially w...2023-11-09T13:25:31Zjustinfreeman (Agileware)Denial of service - CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and mailing will not sendCiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to ...CiviCRM Mailing Job, if there are 5 errors sequentially which are caused by an invalid email domain. The Mailing Job will abort and the mailing will not send.
An invalid email domain returns an SMTP error 451: _SMTP error 451 Unable to complete command, DNS not available or timed out 451 Domain of sender address does not resolve_
When the CiviCRM Mailing Job executes again, the process repeats, the same invalid email addresses are used, the same errors are returned and the mailing again will not send.
This process will repeat until someone intervenes, cancelling the mailing, locating and removing the contacts which have the invalid email addresses.
The CiviCRM Mailing Job does not currently check the actual SMTP error code and just treats all SMTP errors as "SMTP Connection Errors". Counting up to 5 and then aborting the job when the threshold is reached.
I think it's fair to call this a "denial of service" because a bad actor could sign up a bunch of fake Contacts to a CiviCRM mailing list with either invalid email domains or valid email domains which are then rendered invalid - and thus cause that mailing list to cease processing.
**Possible solutions**
Change the CiviCRM Mailing Job to check the return SMTP error code and only increment the count for valid "SMTP Connection Errors".
Change the CiviCRM Mailing Job to continue with the mailing, regardless of the 5 errors encountered, which will also skip sending the emails to those Contacts with an invalid email domain. As shown in the patch below.
```
Index: CRM/Mailing/BAO/MailingJob.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/CRM/Mailing/BAO/MailingJob.php b/CRM/Mailing/BAO/MailingJob.php
--- a/CRM/Mailing/BAO/MailingJob.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/CRM/Mailing/BAO/MailingJob.php (date 1620794367592)
@@ -672,20 +672,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryGroup = TRUE;
+ CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $this->writeToDB(
- $deliveredParams,
- $targetParams,
- $mailing,
- $job_date
- );
-
- CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- CRM_Utils_System::civiExit();
}
// Register the bounce event.
Index: ext/flexmailer/src/Listener/DefaultSender.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/ext/flexmailer/src/Listener/DefaultSender.php b/ext/flexmailer/src/Listener/DefaultSender.php
--- a/ext/flexmailer/src/Listener/DefaultSender.php (revision 20a0376709a4616e32689a821a7e95ca255ed85f)
+++ b/ext/flexmailer/src/Listener/DefaultSender.php (date 1620794390628)
@@ -76,15 +76,9 @@
if ($smtpConnectionErrors <= 5) {
$mailer->disconnect();
$retryBatch = TRUE;
+ \CRM_Core_Error::debug_log_message("More than 5 consecutive SMTP Socket Errors. Re-starting mailer.");
continue;
}
-
- // seems like we have too many of them in a row, we should
- // write stuff to disk and abort the cron job
- $job->writeToDB($deliveredParams, $targetParams, $mailing, $job_date);
-
- \CRM_Core_Error::debug_log_message("Too many SMTP Socket Errors. Exiting");
- \CRM_Utils_System::civiExit();
}
else {
$this->recordBounce($job, $task, $result->getMessage());
```
Agileware Ref: CIVICRM-1728https://lab.civicrm.org/dev/core/-/issues/3559CiviCRM bounce processing, processed emails are never deleted from the mailbo...2023-09-08T22:34:57Zjustinfreeman (Agileware)CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little valueCiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox,...CiviCRM bounce processing, processed emails are never deleted from the mailbox which uses up disk space and provides very little value.
Possible solutions:
1. Once a bounce email has been processed it should be deleted from the mailbox, or
1. A new Scheduled Job process should be implemented which deletes all processed email which is older than 30 days
Relates to https://lab.civicrm.org/dev/mail/-/issues/108
Agileware Ref: CIVICRM-1977https://lab.civicrm.org/dev/core/-/issues/3453Afform - relationships fill from other entity2022-12-07T01:43:54ZsamuelsovAfform - relationships fill from other entityFollow-up from https://github.com/civicrm/civicrm-core/pull/23296#issuecomment-1119014194
Let say we want to do a form that allow an employer to update :
- the main contact of the organization
- all the employees
Thanks to https://lab....Follow-up from https://github.com/civicrm/civicrm-core/pull/23296#issuecomment-1119014194
Let say we want to do a form that allow an employer to update :
- the main contact of the organization
- all the employees
Thanks to https://lab.civicrm.org/dev/core/-/issues/3117 it's now possible to do a form that will allow to create in one step :
- the organization
- the main contact as Individual 1 with a relationship between the organization and Individual 1
- the employees as Individual 2 with a relationship between the organization and Invividual 2 (which is multiple)
However, there is no way to have an edit mode for such a form. It's possible to add the organization id as an argument but we also need a way to pre-populate the main contacts / list of employees as an option based on the relationships definition.colemanwcolemanwhttps://lab.civicrm.org/dev/core/-/issues/3462Event location search2024-01-31T10:18:23Zaydunsaidan.saunders@squiffle.ukEvent location searchOverview
----------------------------------------
Following on from #2103 - when configuring an event and reusing a location, Civi shows a message like 'This location is used by 2 other events' but does not indicate which events.
It w...Overview
----------------------------------------
Following on from #2103 - when configuring an event and reusing a location, Civi shows a message like 'This location is used by 2 other events' but does not indicate which events.
It would be useful to show a list of those locations, or include a link to search for them.
Note that SearchKit displays now handle LocBlocks (see #2676)https://lab.civicrm.org/dev/core/-/issues/3470Search Kit: Mailing labels don't work2022-12-19T15:47:10ZJonGoldSearch Kit: Mailing labels don't workMailing labels don't work with Search Kit.
**Steps to Replicate**
* Do a search for contacts in search kit (with a small number of contacts to avoid hitting #3222).
* Select 1 or more contacts, select **Mailing Labels** from **Actions**...Mailing labels don't work with Search Kit.
**Steps to Replicate**
* Do a search for contacts in search kit (with a small number of contacts to avoid hitting #3222).
* Select 1 or more contacts, select **Mailing Labels** from **Actions**.
* Select a label and press **Make Mailing Labels**.
**Expected Result**
Mailing labels.
**Actual Result**
Spins indefinitely.
This doesn't happen when making PDFs, or if you grab the last HTTP request from the devtools Network tab and open it in a new window.
It happens because TCPDF isn't expecting to be in a "snippet" context at this point - it's just spitting out raw HTML response headers direct to the browser (`CRM_Contact_Form_Task_Label::createLabel()` calls `$pdf->Output()`). "Make documents" uses the more modern `CRM_Utils_PDF_Utils::html2pdf(), which uses DOMPDF or wkHTMLtoPDF, and knows how to handle this. Unfortunately, TCPDF has the mailing label functionality, so we're stuck with it.
**How to proceed**
I could probably make a fix that involves saving the file to disk first - but I think a better fix is to modify the Search Kit "Actions" JS to open the labels in a new window, but I can't find where that happens.https://lab.civicrm.org/dev/joomla/-/issues/405 broken/dated links in post-install success screen for Joomla2022-10-11T22:50:40Znicol5 broken/dated links in post-install success screen for JoomlaOn successfully installing CiviCRM on Joomla (I'm assuming not on the other CMSs), there's a welcome screen with four links to the old wiki/confluence (ie 'http://wiki.civicrm.org/confluence/display/CRMDOC/Installation+and+Upgrades') whi...On successfully installing CiviCRM on Joomla (I'm assuming not on the other CMSs), there's a welcome screen with four links to the old wiki/confluence (ie 'http://wiki.civicrm.org/confluence/display/CRMDOC/Installation+and+Upgrades') which would better point to new (https) docs pages. The register a site link is also broken.
I'm happy to make a PR for this but not sure where the file is located, I checked in https://github.com/civicrm/civicrm-core/tree/master/install & https://github.com/civicrm/civicrm-joomla but couldn't see it. Maybe it's not an actual smarty template but something that's passed to Joomla's UI as it looks natively styled in J3 and J4?
![image](/uploads/8b0a011ff0531bcfc719e6b4eb5e30b0/image.png)
New links needed for:
- 'Installation Guide' - I guess to this: https://docs.civicrm.org/user/en/latest/initial-set-up/installation-and-basic-set-up/
- Create front-end forms and searchable directories using Profiles - https://docs.civicrm.org/user/en/latest/organising-your-data/profiles/
- Create online contribution pages - https://docs.civicrm.org/user/en/latest/contributions/online-contributions/
- Create events with online event registration - https://docs.civicrm.org/user/en/latest/events/what-is-civievent/
- Register a site - https://civicrm.org/register-a-sitenicolnicol