Development issueshttps://lab.civicrm.org/groups/dev/-/issues2022-08-26T20:51:52Zhttps://lab.civicrm.org/dev/core/-/issues/1726Allow financial types to not have Expense account defined2022-08-26T20:51:52ZJoeMurrayAllow financial types to not have Expense account definedThis is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require ...This is a proposal for discussion and refinement.
Overview
----------------------------------------
Simplify accounting configuration to remove requirement for, and default creation of, widely unused stuff. In particular, don't require Expenses account for every financial type, nor create relations to Expense and Premium accounts by default when creating a financial type.
Example use-case
----------------------------------------
1. Click on **Administer > CiviContribute > Financial Types**.
1. Click **Add Financial Type**.
1. Enter **Name** and click **Save**.
1. In Financial Accounts, there are Banking Fees and Premiums accounts, which is **undesirable**.
1. Click **Accounts** on the new Financial Type row.
1. Beside the 'Expense Account is', click **Delete**, then confirm by clicking **Delete** again.
1. Click on **Contributions > New Contribution**.
1. Select the Financial Type created above that does not have an Expense Account set up for it anymore, fill in **Contributor** and **Total Amount**, and click **Save**.
1. Try to edit the contribution but not in a popup, for example, go to the contact's page, right click on the edit button for the contribution and Copy Link Address, then paste address into a new tab. You'll see "Sorry, due to an error, we are unable to fulfill your request at the moment. You may want to contact your administrator or service provider with more details about what action you were performing when this occurred.
One of parameters (value: ) is not of the type Integer". This is caused by missing Expense account, **even though it is not needed**.
Current behaviour
----------------------------------------
See above.
Proposed behaviour
----------------------------------------
On creation of Financial Type, no Expense or Premiums account relationship would be setup. On editing a contribution (with a line item) with a financial type without an Expense account relationship setup, no error would occur.
Comments
----------------------------------------
The expectation when this was implemented circa 2014 was that payment processors would all soon record banking fees. That hasn't been the case for a variety of reasons.Monish DebMonish Debhttps://lab.civicrm.org/dev/core/-/issues/1725Contact export is very slow, creates huge temporary tables2023-04-26T05:03:21ZBobSContact export is very slow, creates huge temporary tablesOverview
----------------------------------------
When exporting contacts with the default output fields, a query is performed which can generate huge temporary files, even for databases that are not very greatly populated. I was in fac...Overview
----------------------------------------
When exporting contacts with the default output fields, a query is performed which can generate huge temporary files, even for databases that are not very greatly populated. I was in fact alerted to this problem when an export of only 80 contacts was performed that maxed out our 40GB of spare disk space and crashed the entire server. The following is a study of this single query, which I'll call Query1, and how it can be optimized to eliminate the problem.
Exponential Growth of Temporary Tables
----------------------------------------
The issue is caused by the joins of multiple tables, each having multiple rows for each of the exported contacts, combined with the fact MySQL creates temporary tables if the queries contain GROUP_CONCAT (among other causes). Although the query returns only one row per exported contact, the temporary table that is built in the process has a row count which is a sum of products.
e.g. Assume a join of tables A, B and C and a WHERE clause which includes N contacts. The temporary table generated will contain
`(A1*B1*C1) + (A2*B2*C2) … + (AN*BN*CN)` rows, where A1, for example, is the number of rows in table A for contact 1. Each additional exported field having multiple rows per contact causes exponential growth in the temporary table size, while the number of exported contacts increases the table size arithmetically.
A Tale of Two Contacts
----------------------------------------
To illustrate a real-world case, the row counts for the tables of interest of two contacts in my organization are as follows:
| Table | Contact_2 | Contact_1016 |
| ------ | ------ | ------ |
| civicrm_group_contact| 91 | 39 |
| civicrm_group_contact_cache | 30 | 17 |
| civicrm_tag | 18 | 6 |
| civicrm_note | 6 | 11 |
| civicrm_email | 4 | 3 |
| civicrm_phone | 2 | 4 |
| civicrm_address | 2 | 3 |
| PRODUCT | 4,717,440 | 1,575,288 |
| SUM OF PRODUCTS | ... | 6,292,728 |
Query1, shown along with its slow query log in [Queries](#queries), was captured from an export operation with default output fields of these two contacts. That query joins each of these tables, among others. "SUM OF PRODUCTS" indicates the number of rows in the resulting temporary table.
An export of 2000 similar contacts rather than just these 2 would be expected to generate a temporary table 1000 times this size (6 billion rows)!
Query Optimization
----------------
Query1 was modified to create the functionally equivalent Query2 (also shown with its slow query log in [Queries](#queries)), employing three subqueries to eliminate the product terms due to the first three DB tables shown above. Performance was vastly improved as shown in the following table:
| Parameters from Slow Query Log | Query1 | Query2 |
| ------ | ------ | ------ |
| Execution Time (S) | 67 | 0.38 |
| Temp file size (MB) | 4,473 | 30 |
| Rows Examined | 29,579,813 | 138,786 |
Just to be clear, these measurements were made running the query for only the two contacts described above.
Recommended Action
-------------------------
When building queries for export operations, subqueries should be used in the join clause for any table which might return more that one row per contact to ensure that it contributes only one row per contact to the temporary table. At the very least, this should be done for those fields that might contain numerous rows per contact, such as groups (smart and non-smart), tags, and notes.
Additional Issues with Contact Export
--------------------------
I would like additionally to note the following secondary issues with contact export:
- Smart groups are not rebuilt upon an export with the 'groups' field. If civicrm_group_contact_cache has been cleared (e.g. due to adding a contact), before running an export, the export will not include any smart groups in the ‘groups’ column.
- The groups, tags and notes filed are each truncated to a length of 1024 characters due to the default limitation of GROUP_CONCAT. This limit could be increased by executing “SET SESSION group_concat_max_len = 100000;”, for example.
- With the default output fields, is_primary is not specified in the email/phone/address join clauses. The results returned are therefore indeterminate for each of these fields when more than one row exists per contact.
Test Environment
--------------------------
- Query1 was captured using CiviCRM 5.24.3.
- To eliminate tainted timing measurements due to possible throttling or over-allocation by my VPS service provider, the above measurements were made on a freshly built Ubuntu 18.04 virtual machine on my local PC where only MariaDB 10.4 was installed (no Apache, PHP, CMS), along with a copy of my site’s Drupal database. Increasing RAM to 12GB from 8GB and parameter tuning using MySQLTuner led to only modest improvements in the execution speed.
References
--------------------------
- [Internal Temporary Table Use in MySQL](https://dev.mysql.com/doc/refman/5.7/en/internal-temporary-tables.html)
- [Original report on civicrm.stackexchange.com](https://civicrm.stackexchange.com/questions/35255/contact-export-runs-very-slowly-creates-huge-temporary-db-files)
Queries
------------
<details>
<summary><b>Query1:</b> Slow query log and SQL</summary>
```sql
# Thread_id: 53 Schema: drupal QC_hit: No
# Query_time: 67.547173 Lock_time: 0.000418 Rows_sent: 2 Rows_examined: 29579813
# Rows_affected: 0 Bytes_sent: 12505
# Tmp_tables: 5 Tmp_disk_tables: 2 Tmp_table_sizes: 4472889344
# Full_scan: No Full_join: Yes Tmp_table: Yes Tmp_table_on_disk: Yes
# Filesort: Yes Filesort_on_disk: No Merge_passes: 0 Priority_queue: No
SELECT
contact_a.id AS contact_id,
contact_a.contact_type AS `contact_type`,
contact_a.contact_sub_type AS `contact_sub_type`,
contact_a.sort_name AS `sort_name`,
contact_a.display_name AS `display_name`,
contact_a.do_not_email AS `do_not_email`,
contact_a.do_not_phone AS `do_not_phone`,
contact_a.do_not_mail AS `do_not_mail`,
contact_a.do_not_sms AS `do_not_sms`,
contact_a.do_not_trade AS `do_not_trade`,
contact_a.is_opt_out AS `is_opt_out`,
contact_a.legal_identifier AS `legal_identifier`,
contact_a.external_identifier AS `external_identifier`,
contact_a.nick_name AS `nick_name`,
contact_a.legal_name AS `legal_name`,
contact_a.image_URL AS `image_URL`,
contact_a.preferred_communication_method AS `preferred_communication_method`,
contact_a.preferred_language AS `preferred_language`,
contact_a.preferred_mail_format AS `preferred_mail_format`,
contact_a.hash AS `hash`,
contact_a.source AS `contact_source`,
contact_a.first_name AS `first_name`,
contact_a.middle_name AS `middle_name`,
contact_a.last_name AS `last_name`,
contact_a.prefix_id AS `prefix_id`,
contact_a.suffix_id AS `suffix_id`,
contact_a.formal_title AS `formal_title`,
contact_a.communication_style_id AS `communication_style_id`,
contact_a.email_greeting_id AS email_greeting_id,
contact_a.postal_greeting_id AS postal_greeting_id,
contact_a.addressee_id AS addressee_id,
contact_a.job_title AS `job_title`,
contact_a.gender_id AS `gender_id`,
contact_a.birth_date AS `birth_date`,
contact_a.is_deceased AS `is_deceased`,
contact_a.deceased_date AS `deceased_date`,
contact_a.household_name AS `household_name`,
IF(contact_a.contact_type = 'Individual',
NULL,
contact_a.organization_name) AS organization_name,
contact_a.sic_code AS `sic_code`,
contact_a.user_unique_id AS `user_unique_id`,
contact_a.employer_id AS `current_employer_id`,
contact_a.is_deleted AS `contact_is_deleted`,
contact_a.created_date AS `created_date`,
contact_a.modified_date AS `modified_date`,
contact_a.addressee_display AS addressee_display,
contact_a.addressee_custom AS addressee_custom,
contact_a.email_greeting_display AS email_greeting_display,
contact_a.email_greeting_custom AS email_greeting_custom,
contact_a.postal_greeting_display AS postal_greeting_display,
contact_a.postal_greeting_custom AS postal_greeting_custom,
IF(contact_a.contact_type = 'Individual',
contact_a.organization_name,
NULL) AS current_employer,
civicrm_address.id AS address_id,
civicrm_location_type.id AS location_type_id,
civicrm_location_type.name AS `location_type`,
civicrm_address.street_address AS `street_address`,
civicrm_address.street_number AS `street_number`,
civicrm_address.street_number_suffix AS `street_number_suffix`,
civicrm_address.street_name AS `street_name`,
civicrm_address.street_unit AS `street_unit`,
civicrm_address.supplemental_address_1 AS `supplemental_address_1`,
civicrm_address.supplemental_address_2 AS `supplemental_address_2`,
civicrm_address.supplemental_address_3 AS `supplemental_address_3`,
civicrm_address.city AS `city`,
civicrm_address.postal_code_suffix AS `postal_code_suffix`,
civicrm_address.postal_code AS `postal_code`,
civicrm_address.geo_code_1 AS `geo_code_1`,
civicrm_address.geo_code_2 AS `geo_code_2`,
civicrm_address.manual_geo_code AS `manual_geo_code`,
civicrm_address.name AS `address_name`,
civicrm_address.master_id AS `master_id`,
civicrm_address.county_id AS county_id,
civicrm_address.state_province_id AS state_province_id,
civicrm_address.country_id AS country_id,
civicrm_phone.id AS phone_id,
civicrm_phone.phone_type_id AS `phone_type_id`,
civicrm_phone.phone AS `phone`,
civicrm_phone.phone_ext AS `phone_ext`,
civicrm_email.id AS email_id,
civicrm_email.email AS `email`,
civicrm_email.on_hold AS `on_hold`,
civicrm_email.is_bulkmail AS `is_bulkmail`,
civicrm_email.signature_text AS `signature_text`,
civicrm_email.signature_html AS `signature_html`,
civicrm_im.id AS im_id,
civicrm_im.provider_id AS `im_provider`,
civicrm_im.provider_id AS provider_id,
civicrm_im.name AS `im`,
civicrm_openid.id AS openid_id,
civicrm_openid.openid AS `openid`,
civicrm_worldregion.id AS worldregion_id,
civicrm_worldregion.name AS `world_region`,
civicrm_website.id AS website_id,
civicrm_website.url AS `url`,
CONCAT_WS(',',
GROUP_CONCAT(DISTINCT IF(civicrm_group_contact.status = 'Added',
civicrm_group_contact.group_id,
'')),
GROUP_CONCAT(DISTINCT civicrm_group_contact_cache.group_id)) AS `groups`,
GROUP_CONCAT(DISTINCT (civicrm_tag.name)) AS tags,
GROUP_CONCAT(DISTINCT (civicrm_note.note)) AS notes
FROM
civicrm_contact contact_a
LEFT JOIN
civicrm_address ON (contact_a.id = civicrm_address.contact_id)
LEFT JOIN
civicrm_country ON (civicrm_address.country_id = civicrm_country.id)
LEFT JOIN
civicrm_email ON (contact_a.id = civicrm_email.contact_id)
LEFT JOIN
civicrm_phone ON (contact_a.id = civicrm_phone.contact_id)
LEFT JOIN
civicrm_im ON (contact_a.id = civicrm_im.contact_id)
LEFT JOIN
civicrm_openid ON (civicrm_openid.contact_id = contact_a.id)
LEFT JOIN
civicrm_location_type ON civicrm_address.location_type_id = civicrm_location_type.id
LEFT JOIN
civicrm_group_contact ON contact_a.id = civicrm_group_contact.contact_id
LEFT JOIN
civicrm_group_contact_cache ON contact_a.id = civicrm_group_contact_cache.contact_id
LEFT JOIN
civicrm_entity_tag ON (civicrm_entity_tag.entity_table = 'civicrm_contact'
AND civicrm_entity_tag.entity_id = contact_a.id)
LEFT JOIN
civicrm_note ON (civicrm_note.entity_table = 'civicrm_contact'
AND contact_a.id = civicrm_note.entity_id)
LEFT JOIN
civicrm_worldregion ON civicrm_country.region_id = civicrm_worldregion.id
LEFT JOIN
civicrm_group_contact `civicrm_group_contact-5e9e3db3d6d9a` ON (contact_a.id = `civicrm_group_contact-5e9e3db3d6d9a`.contact_id
AND `civicrm_group_contact-5e9e3db3d6d9a`.status IN ('Added'))
LEFT JOIN
civicrm_tag ON civicrm_entity_tag.tag_id = civicrm_tag.id
LEFT JOIN
civicrm_website ON contact_a.id = civicrm_website.contact_id
WHERE
(((`civicrm_group_contact-5e9e3db3d6d9a`.group_id IN ('581'))))
AND (contact_a.is_deleted = 0)
AND contact_a.is_deleted != 1
AND contact_a.id IN (2,1016)
GROUP BY contact_a.id
LIMIT 0 , 100000;
```
</details>
<details>
<summary><b>Query2:</b> Slow query log and SQL</summary>
```sql
# Thread_id: 53 Schema: drupal QC_hit: No
# Query_time: 0.382351 Lock_time: 0.000376 Rows_sent: 2 Rows_examined: 138786
# Rows_affected: 0 Bytes_sent: 12525
# Tmp_tables: 8 Tmp_disk_tables: 5 Tmp_table_sizes: 30154752
# Full_scan: No Full_join: Yes Tmp_table: Yes Tmp_table_on_disk: Yes
# Filesort: Yes Filesort_on_disk: No Merge_passes: 0 Priority_queue: No
SELECT
contact_a.id AS contact_id,
contact_a.contact_type AS `contact_type`,
contact_a.contact_sub_type AS `contact_sub_type`,
contact_a.sort_name AS `sort_name`,
contact_a.display_name AS `display_name`,
contact_a.do_not_email AS `do_not_email`,
contact_a.do_not_phone AS `do_not_phone`,
contact_a.do_not_mail AS `do_not_mail`,
contact_a.do_not_sms AS `do_not_sms`,
contact_a.do_not_trade AS `do_not_trade`,
contact_a.is_opt_out AS `is_opt_out`,
contact_a.legal_identifier AS `legal_identifier`,
contact_a.external_identifier AS `external_identifier`,
contact_a.nick_name AS `nick_name`,
contact_a.legal_name AS `legal_name`,
contact_a.image_URL AS `image_URL`,
contact_a.preferred_communication_method AS `preferred_communication_method`,
contact_a.preferred_language AS `preferred_language`,
contact_a.preferred_mail_format AS `preferred_mail_format`,
contact_a.hash AS `hash`,
contact_a.source AS `contact_source`,
contact_a.first_name AS `first_name`,
contact_a.middle_name AS `middle_name`,
contact_a.last_name AS `last_name`,
contact_a.prefix_id AS `prefix_id`,
contact_a.suffix_id AS `suffix_id`,
contact_a.formal_title AS `formal_title`,
contact_a.communication_style_id AS `communication_style_id`,
contact_a.email_greeting_id AS email_greeting_id,
contact_a.postal_greeting_id AS postal_greeting_id,
contact_a.addressee_id AS addressee_id,
contact_a.job_title AS `job_title`,
contact_a.gender_id AS `gender_id`,
contact_a.birth_date AS `birth_date`,
contact_a.is_deceased AS `is_deceased`,
contact_a.deceased_date AS `deceased_date`,
contact_a.household_name AS `household_name`,
IF(contact_a.contact_type = 'Individual',
NULL,
contact_a.organization_name) AS organization_name,
contact_a.sic_code AS `sic_code`,
contact_a.user_unique_id AS `user_unique_id`,
contact_a.employer_id AS `current_employer_id`,
contact_a.is_deleted AS `contact_is_deleted`,
contact_a.created_date AS `created_date`,
contact_a.modified_date AS `modified_date`,
contact_a.addressee_display AS addressee_display,
contact_a.addressee_custom AS addressee_custom,
contact_a.email_greeting_display AS email_greeting_display,
contact_a.email_greeting_custom AS email_greeting_custom,
contact_a.postal_greeting_display AS postal_greeting_display,
contact_a.postal_greeting_custom AS postal_greeting_custom,
IF(contact_a.contact_type = 'Individual',
contact_a.organization_name,
NULL) AS current_employer,
civicrm_address.id AS address_id,
civicrm_location_type.id AS location_type_id,
civicrm_location_type.name AS `location_type`,
civicrm_address.street_address AS `street_address`,
civicrm_address.street_number AS `street_number`,
civicrm_address.street_number_suffix AS `street_number_suffix`,
civicrm_address.street_name AS `street_name`,
civicrm_address.street_unit AS `street_unit`,
civicrm_address.supplemental_address_1 AS `supplemental_address_1`,
civicrm_address.supplemental_address_2 AS `supplemental_address_2`,
civicrm_address.supplemental_address_3 AS `supplemental_address_3`,
civicrm_address.city AS `city`,
civicrm_address.postal_code_suffix AS `postal_code_suffix`,
civicrm_address.postal_code AS `postal_code`,
civicrm_address.geo_code_1 AS `geo_code_1`,
civicrm_address.geo_code_2 AS `geo_code_2`,
civicrm_address.manual_geo_code AS `manual_geo_code`,
civicrm_address.name AS `address_name`,
civicrm_address.master_id AS `master_id`,
civicrm_address.county_id AS county_id,
civicrm_address.state_province_id AS state_province_id,
civicrm_address.country_id AS country_id,
civicrm_phone.id AS phone_id,
civicrm_phone.phone_type_id AS `phone_type_id`,
civicrm_phone.phone AS `phone`,
civicrm_phone.phone_ext AS `phone_ext`,
civicrm_email.id AS email_id,
civicrm_email.email AS `email`,
civicrm_email.on_hold AS `on_hold`,
civicrm_email.is_bulkmail AS `is_bulkmail`,
civicrm_email.signature_text AS `signature_text`,
civicrm_email.signature_html AS `signature_html`,
civicrm_im.id AS im_id,
civicrm_im.provider_id AS `im_provider`,
civicrm_im.provider_id AS provider_id,
civicrm_im.name AS `im`,
civicrm_openid.id AS openid_id,
civicrm_openid.openid AS `openid`,
civicrm_worldregion.id AS worldregion_id,
civicrm_worldregion.name AS `world_region`,
civicrm_website.id AS website_id,
civicrm_website.url AS `url`,
/*
CONCAT_WS(',',
GROUP_CONCAT(DISTINCT IF(civicrm_group_contact.status = 'Added', civicrm_group_contact.group_id, '')),
GROUP_CONCAT(DISTINCT civicrm_group_contact_cache.group_id)) AS `groups`,
*/
CONCAT_WS(',', sel_group_list.group_list, sel_smartgroup_list.group_list) `groups`,
#GROUP_CONCAT(DISTINCT (civicrm_tag.name)) AS tags,
tags,
GROUP_CONCAT(DISTINCT (civicrm_note.note)) AS notes
FROM
civicrm_contact contact_a
LEFT JOIN
civicrm_address ON (contact_a.id = civicrm_address.contact_id)
LEFT JOIN
civicrm_country ON (civicrm_address.country_id = civicrm_country.id)
LEFT JOIN
civicrm_email ON (contact_a.id = civicrm_email.contact_id)
LEFT JOIN
civicrm_phone ON (contact_a.id = civicrm_phone.contact_id)
LEFT JOIN
civicrm_im ON (contact_a.id = civicrm_im.contact_id)
LEFT JOIN
civicrm_openid ON (civicrm_openid.contact_id = contact_a.id)
LEFT JOIN
civicrm_location_type ON civicrm_address.location_type_id = civicrm_location_type.id
LEFT JOIN
civicrm_group_contact ON contact_a.id = civicrm_group_contact.contact_id
#LEFT JOIN civicrm_group_contact_cache ON contact_a.id = civicrm_group_contact_cache.contact_id
#LEFT JOIN civicrm_entity_tag ON (civicrm_entity_tag.entity_table = 'civicrm_contact' AND civicrm_entity_tag.entity_id = contact_a.id)
LEFT JOIN civicrm_note ON (civicrm_note.entity_table = 'civicrm_contact' AND contact_a.id = civicrm_note.entity_id)
LEFT JOIN civicrm_worldregion ON civicrm_country.region_id = civicrm_worldregion.id
#LEFT JOIN civicrm_group_contact `civicrm_group_contact-5e9e3db3d6d9a` ON (contact_a.id = `civicrm_group_contact-5e9e3db3d6d9a`.contact_id AND `civicrm_group_contact-5e9e3db3d6d9a`.status IN ('Added'))
#LEFT JOIN civicrm_tag ON civicrm_entity_tag.tag_id = civicrm_tag.id
LEFT JOIN (
SELECT contact_id, CONCAT_WS(',', GROUP_CONCAT(cgc.group_id)) group_list FROM civicrm_group_contact cgc
LEFT JOIN civicrm_contact contact_a
ON contact_a.id = cgc.contact_id AND cgc.status = "ADDED"
WHERE contact_a.id in (2,1016)
GROUP BY contact_a.id
) AS sel_group_list ON contact_a.id = sel_group_list.contact_id
LEFT JOIN (
SELECT contact_id, CONCAT_WS(',', GROUP_CONCAT(cgcc.group_id)) group_list FROM civicrm_group_contact_cache cgcc
LEFT JOIN civicrm_contact contact_a
ON contact_a.id = cgcc.contact_id
WHERE contact_a.id in (2,1016)
GROUP BY contact_a.id
) AS sel_smartgroup_list ON contact_a.id = sel_smartgroup_list.contact_id
LEFT JOIN (
SELECT contact_a.id, cet.entity_id, GROUP_CONCAT(DISTINCT(ct.name)) tags FROM civicrm_contact contact_a
LEFT JOIN civicrm_entity_tag cet ON contact_a.id = cet.entity_id
LEFT JOIN civicrm_tag ct ON cet.tag_id = ct.id
WHERE contact_a.id in (2,1016)
GROUP BY contact_a.id
) AS sel_tag_list ON contact_a.id = sel_tag_list.entity_id
LEFT JOIN civicrm_website ON contact_a.id = civicrm_website.contact_id
WHERE
#(((`civicrm_group_contact-5e9e3db3d6d9a`.group_id IN ('581')))) AND
(contact_a.is_deleted = 0)
AND contact_a.is_deleted != 1
AND contact_a.id IN (2,1016)
GROUP BY contact_a.id
LIMIT 0 , 100000;
```
</details>https://lab.civicrm.org/dev/drupal/-/issues/118civicrm_views module no longer present causing updatedb error: "The module ci...2020-04-30T18:34:04Zjohnkcivicrm_views module no longer present causing updatedb error: "The module civicrm_views does not exist"CiviCRM 5.24.4 with Drupal 8.8.5. The last few version upgrades, now, I get this error from updatedb.
```
In ExtensionList.php line 265:
...CiviCRM 5.24.4 with Drupal 8.8.5. The last few version upgrades, now, I get this error from updatedb.
```
In ExtensionList.php line 265:
The module civicrm_views does not exist.
```
I thought I was able to fix it, before, with a `drush pm:uninstall civicrm_views`. (I don't need it.) But the error came back. This time I was only able to resolve it by copying civicrm_views from my vendor/civicrm/civicrm-drupal-8/modules/ into web/modules/contrib/civicrm/modules. I can then run the updatedb. Even though I now have this module disabled, I can't remove it from web/modules or I get this in my Apache log upon going to the status page:
```
Uncaught PHP Exception Drupal\\Core\\Extension\\Exception\\UnknownExtensionException: "The module civicrm_views does not exist or is not installed." at /var/www/acreresidency.org/web/core/lib/Drupal/Core/Extension/ExtensionList.php line 346
```
Was this recently taken out of the set of modules and stuff that gets installed by the composer scripts? I'm thinking back to when I used to have to do some of this install work manually, and then eventually composer took over for me.
If I had to take a guess at what this bug is: civicrm_views got removed from what is installed by default, but I already had it installed. And the module doesn't properly clean up after itself on uninstall?
I seem to recall I also, for one of the upgrades, had to remove a line saying "civicrm_views" from an extensions list that's in my config/sync. Just now I looked in the database too. I tried out some "LIKE" queries with "civicrm_view%", and I wasn't able to find anything in my drupal_config table that relates to this module.
How screwed up is my install now? I should have reported this sooner.https://lab.civicrm.org/dev/user-interface/-/issues/19Misplaced text in the Additional Details section on New Contribution2020-04-21T20:07:09ZDaveDMisplaced text in the Additional Details section on New ContributionAt the moment my guess is a field was removed but a malformed description row is still there. Will take a closer look.
![screenshot](/uploads/09a91f658435ea1047f4a877b9271d9e/screenshot.gif)At the moment my guess is a field was removed but a malformed description row is still there. Will take a closer look.
![screenshot](/uploads/09a91f658435ea1047f4a877b9271d9e/screenshot.gif)5.26.0https://lab.civicrm.org/dev/core/-/issues/1724Changes to CiviContribute Component Settings not saved2020-05-15T23:43:31ZkcristianoChanges to CiviContribute Component Settings not savedAs reported in [Mattermost](https://chat.civicrm.org/civicrm/pl/iy3p1aqambfadymorr5ean7e3w)
```
Anyone have any idea why the CiviContribute Component Settings page would not be updating values on save? Everything up to the “Enabled Defe...As reported in [Mattermost](https://chat.civicrm.org/civicrm/pl/iy3p1aqambfadymorr5ean7e3w)
```
Anyone have any idea why the CiviContribute Component Settings page would not be updating values on save? Everything up to the “Enabled Deferred Revenue” field saves fine, but nothing below it (although Civi reports “Your changes have been saved”. Civi v5.24.3 running on WordPress).
```
Reproduced on 5.25 RC and master in WP and Drupal 7.5.24.6https://lab.civicrm.org/dev/core/-/issues/1723Reciprocal relationship search with custom fields leads to an error on Advanc...2020-04-23T02:52:05ZjitendraReciprocal relationship search with custom fields leads to an error on Advanced searchTo replicate -
- Create a custom field extending relationships. Searchable = yes.
- Open Advanced search
- Select any reciprocal relationship eg spouse of, sibling, partner, etc
- Enter value for the custom field.
- Hit Search.
![image...To replicate -
- Create a custom field extending relationships. Searchable = yes.
- Open Advanced search
- Select any reciprocal relationship eg spouse of, sibling, partner, etc
- Enter value for the custom field.
- Hit Search.
![image](/uploads/57290e079190ceb39e47eebe5a3331e6/image.png)
possibly due to https://github.com/civicrm/civicrm-core/pull/15793. As it was merged ~5 months ago(included in 5.21), I think it should be labeled as regression?5.24.5https://lab.civicrm.org/dev/wordpress/-/issues/48Dashboard is empty2020-05-11T13:47:29ZcliffnhansenDashboard is emptyHello, I am brand new to CiviCRM and excited to use it, however my dashboard in Wordpress has suddenly gone blank. It says that there is a system alert, but when I click it, nothing happens. I've tried clicking the configure your dashboa...Hello, I am brand new to CiviCRM and excited to use it, however my dashboard in Wordpress has suddenly gone blank. It says that there is a system alert, but when I click it, nothing happens. I've tried clicking the configure your dashboard and reconfigure your dashboard options and nothing happens. I've also tried deactivating, and reinstalling the addon with the newest version, but still nothing happens. If you could help me out, I'd appreciate it!
![civicrdashboard](/uploads/929f6376069e33c9669c325368a5b929/civicrdashboard.JPG)https://lab.civicrm.org/dev/core/-/issues/1722advanced search: allow searching by first/last name; pass values from quickse...2023-03-18T05:03:32Zlcdwebadvanced search: allow searching by first/last name; pass values from quicksearchOverview
----------------------------------------
Currently, only the sort_name field is exposed to Advanced Search. As a result it's not possible to search by first or last name specifically. In addition, while you can choose first or l...Overview
----------------------------------------
Currently, only the sort_name field is exposed to Advanced Search. As a result it's not possible to search by first or last name specifically. In addition, while you can choose first or last name from the quick search option list, selecting one of those options and clicking Enter passes the value to the sort_name field -- where you lose that field-specific filter.
We should add the first/last name fields to Advanced Search and pass values from quick search (when selected) to the appropriate fields.
Example use-case
----------------------------------------
1. Create two contacts, one with "Jones" in the first name field, one with it in the last name field
2. From quick search, select first name, enter Jones, and click Enter
The search results will include contacts with Jones in either the first or last name if you have wildcard enabled. If you have wildcard disabled, only records with that value starting the last name will be returned (the opposite of what you were attempting to do via quick search).lcdweblcdwebhttps://lab.civicrm.org/dev/core/-/issues/1721case activity: follow-up activity assignee does not receive email2020-05-15T00:38:15Zlcdwebcase activity: follow-up activity assignee does not receive emailOverview
----------------------------------------
When creating a case activity follow-up activity with an assignee, the assignee is not sent an email.
Reproduction steps
----------------------------------------
1. Create a case
2. Crea...Overview
----------------------------------------
When creating a case activity follow-up activity with an assignee, the assignee is not sent an email.
Reproduction steps
----------------------------------------
1. Create a case
2. Create an activity within the case
3. Expand the "Schedule Follow-up" panel and complete the fields. Select a contact as the assignee.
Current behaviour
----------------------------------------
Follow-up assignee does not receive an email.
Expected behaviour
----------------------------------------
Assignee should receive an email
Comments
----------------------------------------
This appears to simply be a functionality gap.5.27.0lcdweblcdwebhttps://lab.civicrm.org/dev/core/-/issues/1720API error: '2' is not a valid option for field payment_instrument_id2023-04-28T05:03:19ZStoobAPI error: '2' is not a valid option for field payment_instrument_idUpon completion of upgrade to 5.24.x this message appears if the payment_instrument_id 2 does not exist in the Civi.
> API error: '2' is not a valid option for field payment_instrument_id
It is possible to run a CiviCRM without specif...Upon completion of upgrade to 5.24.x this message appears if the payment_instrument_id 2 does not exist in the Civi.
> API error: '2' is not a valid option for field payment_instrument_id
It is possible to run a CiviCRM without specific ID for payment instrument, IMHO the upgrader should not assume certain payment_instrument_id exists. Although the upgrade is completed 'just fine' this error can hopefully be removed or underlying issue resolved.https://lab.civicrm.org/dev/core/-/issues/1719Invalid and patchy XML encoding causes Case crash if status is not valid XML2020-04-19T00:22:23ZRichInvalid and patchy XML encoding causes Case crash if status is not valid XMLOverview
----------------------------------------
CiviCase will fail to fully create a CaseType if a Case Status' name is invalid in an XML context.
Reproduction steps
----------------------------------------
1. Create case status 'Si...Overview
----------------------------------------
CiviCase will fail to fully create a CaseType if a Case Status' name is invalid in an XML context.
Reproduction steps
----------------------------------------
1. Create case status 'Signed & Sealed'
1. Create case type
1. Got an error about XML encoding
Current behaviour
----------------------------------------
The form errors and won't save. Yet it has half created a Case - it's there but has no description. This is massively confusing because it looks like it might have worked, but creates havock if you actually try to use that case type.
Expected behaviour
----------------------------------------
It should create the case type without error.
Comments
----------------------------------------
This happens because of some incomplete and rather *lets-do-it-ourselves* XML encoding. There's even a comment in the code saying "Ugh!" about this!
The Statuses element was being written without being encoded, therefore `&` becomes an invalid entity.
I have a PR for this that replaces the homemade XML encoding function with PHP's built in XML Writer objects.5.26.0RichRichhttps://lab.civicrm.org/dev/core/-/issues/1718membership batch entry: fix join date population2020-04-22T00:44:07Zlcdwebmembership batch entry: fix join date populationWhen using the membership batch entry tool, if you select a contact with an existing membership it prepopulates with existing data. The join date is not prefilled because of an incorrect field name reference.When using the membership batch entry tool, if you select a contact with an existing membership it prepopulates with existing data. The join date is not prefilled because of an incorrect field name reference.5.26.0lcdweblcdwebhttps://lab.civicrm.org/dev/core/-/issues/1717SMTP temp failure causes "Call to undefined method CRM_Utils_Mail_FilteredPea...2020-04-25T19:32:57Zaydunsaidan.saunders@squiffle.ukSMTP temp failure causes "Call to undefined method CRM_Utils_Mail_FilteredPearMailer::disconnect()"Overview
----------------------------------------
SMTP temp fails can result in "Call to undefined method CRM_Utils_Mail_FilteredPearMailer::disconnect()"
This involves SMTP temp fails so is not easy to reproduce reliably.
The error se...Overview
----------------------------------------
SMTP temp fails can result in "Call to undefined method CRM_Utils_Mail_FilteredPearMailer::disconnect()"
This involves SMTP temp fails so is not easy to reproduce reliably.
The error seen in the log is:
`SMTP Socket Error or failed to set sender error. Message: Failed to set sender: b-6928-162335-5d12be6b881f1e84@example.com [SMTP: Invalid response code received from SMTP server while sending email. This is often caused by a misconfiguration in Outbound Email settings. Please verify the settings at Administer CiviCRM >> Global Settings >> Outbound Email (SMTP). (code: 421, response: 4.7.0 Try again later, closing connection. (MAIL) f22sm5685077lja.39 - gsmtp)], Code: 10004`
The mailing was initiated with `cv api Job.execute` producing:
`Call to undefined method CRM_Utils_Mail_FilteredPearMailer::disconnect()`
The problem seems to be here: https://github.com/civicrm/org.civicrm.flexmailer/blob/master/src/Listener/DefaultSender.php#L77
Possible fixes:
1) Change flexmailer not to call `$mailer->disconnect()`
2) Implement `CRM_Utils_Mail_FilteredPearMailer::disconnect()`
Also, the error message is misleading: a 421 or 4.7.0 is not an invalid response and was not caused but settings misconfiguration.
This seems to be a fairly minor problem in that it only occurs for temp fails and those are retried later.
Environment information
----------------------------------------
* __CiviCRM:__ 5.24.4
* __Flexmailer:__ 1.1.1
* __PHP:__ 7.2
* Mail configuration: SMTP to gmail.com5.24.5https://lab.civicrm.org/dev/core/-/issues/1716Create new customPre hook2020-06-12T16:52:05ZlcdwebCreate new customPre hookI'd like to create a new "customPre" hook that is fired prior to a custom field being stored in the DB. Currently we have a "custom" hook which is fired after a custom field is stored, but no way to impact the process prior to storing. S...I'd like to create a new "customPre" hook that is fired prior to a custom field being stored in the DB. Currently we have a "custom" hook which is fired after a custom field is stored, but no way to impact the process prior to storing. See also: https://civicrm.stackexchange.com/questions/35354/is-there-a-hook-to-impact-custom-data-before-its-saved/35364#35364
* why this is needed: there's currently no way to alter custom data before it is stored, or to do thing like check the existing value for the field prior to the update (in order to, for example, trigger a secondary action). Custom data currently does not hit the pre/post hooks.
* why not try to implement the existing "pre" hook in this context? we could potentially do that, but the custom data model is a little different from a standard object. For example, the custom group ID is useful to have available but is not part of the actual insert/update query for custom data. It seems more useful to follow the existing pattern with the "custom" hook. While we could pass that additional metadata as part of the pre hook's params, it would deviate from how that hook is implemented elsewhere.
The hook would be implemented in CRM_Core_BAO_CustomValueTable::create(), and would follow the pattern of how the "custom" hook is implemented there.
Looking for feedback and concept approval.5.28.0lcdweblcdwebhttps://lab.civicrm.org/dev/core/-/issues/1715CiviCase dashboard/find case: Additional columns and filters needed2022-05-11T14:39:19ZDetlev SieberCiviCase dashboard/find case: Additional columns and filters neededIn order to improve the workflow, the following should be added to the find case / dashboard functionality of CiviCase:
New column: "Case_ID" -> with sorting
New filter: "Next Sched. assignee"
* This should refer to the "Next Sched." ...In order to improve the workflow, the following should be added to the find case / dashboard functionality of CiviCase:
New column: "Case_ID" -> with sorting
New filter: "Next Sched. assignee"
* This should refer to the "Next Sched." actitivity from the list.
* The filter list should contain "myself" as the first entry, and then should display all other assignee contained in the activities displayed on the case list.
This functionality should be available on:
* case dashboard
* search results (case search)
* case dashletshttps://lab.civicrm.org/dev/core/-/issues/1714Search Builder - Activity Type search not working.2020-05-02T04:46:10ZBari PollardSearch Builder - Activity Type search not working.After upgrading to 5.24.3 some smart groups stopped working that were created using Search Builder and activity types.
Investigated on DMaster 5.26.alpha1.
Using Search Builder -
Choose Activity - Activity Type - = - Tell a Friend. ...After upgrading to 5.24.3 some smart groups stopped working that were created using Search Builder and activity types.
Investigated on DMaster 5.26.alpha1.
Using Search Builder -
Choose Activity - Activity Type - = - Tell a Friend.
Actual results - 0
Choose Activity - Activity Type ID - = - Tell a Friend. (Not listed as a ID just the label)
Actual results - 925.25.0https://lab.civicrm.org/dev/core/-/issues/1713After core upgrade, extension routes unavailable2020-08-04T07:44:04ZtottenAfter core upgrade, extension routes unavailableOverview
----------------------------------------
After upgrading 5.21.1 to 5.24.3, there appears to be a problem with rebuilding the menu/router listing. This has been specifically observed in Mosaico. I can't imagine why it would be u...Overview
----------------------------------------
After upgrading 5.21.1 to 5.24.3, there appears to be a problem with rebuilding the menu/router listing. This has been specifically observed in Mosaico. I can't imagine why it would be unique to Mosaico, but I have not tested with other extensions.
Reproduction steps
----------------------------------------
* Create `dmaster` site with Civi 5.21.1
* Install Mosaico 2.3
* Create a draft mailing. Add some content. Keep this page open (eg `civicrm/a/#/mailing/1`)
* Update code to Civi 5.24.3.
* Run the Civi upgrader. Use the web UI. (*The problem does not reproduce if you use cv upgrade.*) Wait for it finish.
* Reload the draft mailing page.
Current behaviour
----------------------------------------
The AJAX request for `/civicrm/mosaico/iframe?snippet=1` fails -- the response provides the fallback HTML page used by `/civicrm`. This content is rendered badly. The popup fails to open.
![Screen_Shot_2020-04-17_at_12.11.58_AM](/uploads/a74655b0fd5daf8dcf2eafee25cb6c1f/Screen_Shot_2020-04-17_at_12.11.58_AM.png)
If you explicitly flush the system cache, then the `civicrm/mosaico/iframe` route works as expected.
Expected behaviour
----------------------------------------
During the upgrade, when the system cache is flushed, it should activate the routes used by Mosaico.
The AJAX request for `/civicrm/mosaico/iframe?snippet=1` should work.
Environment information
----------------------------------------
<!-- Some of the items below may not be relevant for every bug - if in doubt please include more information than you think is neccessary. -->
* __Browser:__ _Firefox 74_
* __CiviCRM:__ _5.24.3_
* __PHP:__ _7.1_
* __CMS:__ _D7_
* __Database:__ _MySQL 5.7_
* __Web Server:__ _Apache 2_
Comments
----------------------------------------
NOTE: This issue was originally reported on Mattermost by Pete and Jitendra from Fuzion.5.26.0https://lab.civicrm.org/dev/core/-/issues/1712Can't edit Civicontribute component settings for tax and invoicing once set2022-10-04T13:21:48ZUpperholmeCan't edit Civicontribute component settings for tax and invoicing once setOverview
----------------------------------------
Having enabled the Tax and Invoicing feature and set the preferred options at /civicrm/admin/setting/preferences/contribute?reset=1, if you later want to alter one or more of these option...Overview
----------------------------------------
Having enabled the Tax and Invoicing feature and set the preferred options at /civicrm/admin/setting/preferences/contribute?reset=1, if you later want to alter one or more of these options, for example changing the number of days before the invoice falls due, or turning off the option to send a PDF invoice by default, edits to this page are not saved.
I've come across this on two installations, both running 5.24.x, and have replicated the issue on https://dmaster.demo.civicrm.org
Reproduction steps
----------------------------------------
1. Go to Administration > CiviContribute > CiviContribute Component Settings.
2. If Tax and Invoicing is already enabled, edit, for example, the number of days before an invoice falls due. Click 'Save'.
3. Return to Administration > CiviContribute > CiviContribute Component Settings
4. Note that your edit has not been saved and that the previous setting still stands.
Current behaviour
----------------------------------------
I've not tested every setting on this page, but for those that I have, edits are not saved and whatever the previous setting was remains in force.
Expected behaviour
----------------------------------------
The edit should be saved.
Environment information
----------------------------------------
* Browsers tested: Firefox 75/Safari 13.04
* CiviCRM: Master/5.24.1
* PHP: 7.3
* CMS: Drupal 7.69
* Database: MariaDB
* Web Server: Apache 2.45.25.0https://lab.civicrm.org/dev/drupal/-/issues/117Profiles used for editing user records in drupal 8 don't load civi javascript2021-06-14T15:53:02ZDaveDProfiles used for editing user records in drupal 8 don't load civi javascript1. Create a civi profile that has `View/Edit Drupal User Account` checked under "Used For".
1. Add the Individual birthdate field.
1. Clear drupal cache at admin/config/development/performance (otherwise the tab in the next step won't sh...1. Create a civi profile that has `View/Edit Drupal User Account` checked under "Used For".
1. Add the Individual birthdate field.
1. Clear drupal cache at admin/config/development/performance (otherwise the tab in the next step won't show up).
1. Edit any user record under /admin/people.
1. Click on the tab for the profile you made.
1. Birthdate field has no datepicker. If you turn off aggregation which is on by default you can see no civi javascript seems to be even listed (i.e. it's not a 404 or resource url issue).
Civi and civicrm-drupal-8 version is master.5.40.0https://lab.civicrm.org/dev/core/-/issues/1711Display SMS segment count when constructing a Mass SMS2021-08-23T19:49:34ZJohn TwymanDisplay SMS segment count when constructing a Mass SMSOverview
----------------------------------------
SMS messages over 160 characters in length are split into 153 character segments. If you use non-GSM characters (eg. ideograms, curly quotes, etc.) those messages are sent in UCS-2 encod...Overview
----------------------------------------
SMS messages over 160 characters in length are split into 153 character segments. If you use non-GSM characters (eg. ideograms, curly quotes, etc.) those messages are sent in UCS-2 encoding which limits segments to 67 characters.
SMS providers (Clickatel, Twilio, Nexmo, etc.) charge *per segment*.
I propose a modification to the New Mass SMS wizard to display the number of SMS segments required to send a Mass SMS on the *SMS Content* step of the New Mass SMS wizard. It will give users better visibility of the cost of their desired message.
Example use-case
----------------------------------------
1. Click on **Mailings -> New Mass SMS**.
2. Step through the first step of the wizard (Select Recipients).
3. On the SMS Content step, as the SMS message is typed in, the number of segments required is displayed and dynamically updated alongside the display of the number of characters entered.
Current behaviour
----------------------------------------
As a SMS message is entered Civi displays the number of characters used. There is no indication of the number of message segments.
Proposed behaviour
----------------------------------------
Modify [the maxCharInfoDisplay function in CRM/Contact/Form/Task/SMSCommon.tpl](https://github.com/civicrm/civicrm-core/blob/a3628d0b7a4d5d253fa38d87ddc7eb2525ef2775/templates/CRM/Contact/Form/Task/SMSCommon.tpl#L67) that dynamically displays the number of characters entered to also calculate and display how many segments are required.
![civisms-improvement-mockup](/uploads/c2bc774996a3b8a659c63e9e190abfd6/civisms-improvement-mockup.png)
The simplest approach might be to adopt this code ([thanks StackOverflow](https://stackoverflow.com/questions/12673120/how-to-detect-non-gsm-7-bit-alphabet-characters-in-input-field/12673229)):
`gsm = "@£$¥èéùìòÇØøÅåΔ_ΦΓΛΩΠΨΣΘΞ^{}\[~]|€ÆæßÉ!\"#¤%&'()*+,-./0123456789:;<=>?¡ABCDEFGHIJKLMNOPQRSTUVWXYZÄÖÑܧ¿abcdefghijklmnopqrstuvwxyzäöñüà";
var letter = 'a';
var letterInAlfabet = gsm.indexOf(letter) !== -1;`
We could also consider adding a help icon to trigger a modal that provides some basic information about SMS segments, "weird" characters, etc. too.
Comments
----------------------------------------
The mockup I've attached is a good example of how a message that looks like it fits within a single segment can produce unexpected outcomes. The use of curly quotes push 148 characters across three segments, tripling the cost per recipient.
This happens relatively frequently as people often copy/paste content from Outlook, Gmail, Word, etc., documents/messages.5.40.0