Development issueshttps://lab.civicrm.org/groups/dev/-/issues2024-02-12T05:03:28Zhttps://lab.civicrm.org/dev/core/-/issues/3543Provide warning for mailings that are being edited to prevent Mailing not sav...2024-02-12T05:03:28ZlarsssandergreenProvide warning for mailings that are being edited to prevent Mailing not saved errors.In our org, we have multiple users who might need to edit the same mailing. Unfortunately, this creates the possibility that someone leaves the mailing open and someone else opens it and begins editing, leading to the "Mailing not saved....In our org, we have multiple users who might need to edit the same mailing. Unfortunately, this creates the possibility that someone leaves the mailing open and someone else opens it and begins editing, leading to the "Mailing not saved. Content may be out of date" error (and tears and hair loss due to lost changes). In our case, this is mostly one user putting together a mailing and another user editing it, fixing formatting errors, etc. It's very hard to train users who are used to Google Docs not to leave the tab open, leading to the need for rigid hand off procedures. We've been trying to get people trained for years, but this continues to be an issue that we waste a lot of time on.
Would it be feasible to add some kind of warning that would warn another user who wants to open another mailing that is already open for editing? I'm thinking just setting a flag on the mailing with a 30 minute expiration, which is updated every time the mailing or content is saved. The flag would be unset when the user exits the mailing. Mailings with the flag would have a warning on the draft mailings screen that would say "Someone else may be editing this mailing" or a warning before opening the mailing. This won't be a perfect solution, but I think it would prevent 95% of problems.
We're using Mosaico, but I think this would be implemented in core and apply to both kinds of mailings.https://lab.civicrm.org/dev/core/-/issues/3542Move the cache for `CRM_Extension_Browser` out of the filesystem and use a `S...2022-06-16T02:21:42ZtiotsopMove the cache for `CRM_Extension_Browser` out of the filesystem and use a `SqlGroup` insteadThe current code in `CRM_Extension_Browser` is coded specifically for an adhoc, file-based caching logic. The recommendation standard is basically to change the backing/storage from a JSON file to the civicrm_cachetable (at least, for th...The current code in `CRM_Extension_Browser` is coded specifically for an adhoc, file-based caching logic. The recommendation standard is basically to change the backing/storage from a JSON file to the civicrm_cachetable (at least, for the typical usage).
Ref [#2](https://lab.civicrm.org/dev/cloud-native/issues/2#note_4790)5.52.0tiotsoptiotsophttps://lab.civicrm.org/dev/core/-/issues/3540Hardcoded filesystem paths in JS strings in cache2022-06-11T14:45:13ZherbdoolHardcoded filesystem paths in JS strings in cacheI've noticed hardcoded filesystem paths in civicrm_cache where group_name = js_strings. With dynamic containers on a host this could mean that the JS can't be found if it switches containers. Is there a way to make this relative?I've noticed hardcoded filesystem paths in civicrm_cache where group_name = js_strings. With dynamic containers on a host this could mean that the JS can't be found if it switches containers. Is there a way to make this relative?https://lab.civicrm.org/dev/core/-/issues/3539When communcating with external network services, detect transient failures a...2024-02-12T13:57:20ZmfbWhen communcating with external network services, detect transient failures and queue for retryFor improved resiliency, CiviCRM should detect transient failures when communicating with external network services and queue for retry. e.g.
* Transactional email: Queue for sending rather than sending during the request process?
* Bul...For improved resiliency, CiviCRM should detect transient failures when communicating with external network services and queue for retry. e.g.
* Transactional email: Queue for sending rather than sending during the request process?
* Bulk mail: Catch transient failures and ensure that they are retried. PRs: https://github.com/civicrm/civicrm-core/pull/11838 https://github.com/civicrm/civicrm-core/pull/11840 (and there are additional failure scenarios possible)
* ...https://lab.civicrm.org/dev/core/-/issues/3537Does CiviCRM handle UNIX filesystem functionality like symlinks?2024-02-10T05:03:27ZxurizaemonDoes CiviCRM handle UNIX filesystem functionality like symlinks?Some cloud hosting providers dynamically assign storage (disk) and may use symlinks to associate dynamic storage with predictable filesystem locations. CiviCRM seems to be confused by the presence of symlinks in the CiviCRM install path ...Some cloud hosting providers dynamically assign storage (disk) and may use symlinks to associate dynamic storage with predictable filesystem locations. CiviCRM seems to be confused by the presence of symlinks in the CiviCRM install path (and possibly resource paths).
This manifests as "weird" resource URLs or inability to resolve filesystem paths.
> This doesn't work in all configurations. For example, if your directory structure uses symlinks, it may not resolve correctly.
For examples, https://civicrm.stackexchange.com/search?q=symlink eg
* https://civicrm.stackexchange.com/questions/19495/set-civicrm-files/19498#19498
* https://civicrm.stackexchange.com/questions/20064/help-needed-in-setting-up-rest-api-in-civi-crm/20076#20076
* https://civicrm.stackexchange.com/questions/20265/mailchimp-plugin-creating-webhook-with-invalid-url
* https://civicrm.stackexchange.com/questions/21925/incorrect-resource-url-warning-but-menu-arrows-are-present
For WP specifically, https://github.com/civicrm/civicrm-core/pull/11086https://lab.civicrm.org/dev/core/-/issues/3536[META] Does CiviCRM make it easy for all connection-related configuration to ...2024-02-10T05:03:26Zherbdool[META] Does CiviCRM make it easy for all connection-related configuration to be provided via environment variables: database credentials, key-value database credentials, filesystem, and so on?It is possible by editing civicrm.settings.php to inject the environment variables.
I'm wondering if a more generic framework might be useful where CiviCRM provides some boilerplate for what strings are required from the environment va...It is possible by editing civicrm.settings.php to inject the environment variables.
I'm wondering if a more generic framework might be useful where CiviCRM provides some boilerplate for what strings are required from the environment variables.
This could help since there is so much repetition of variables that need to be assigned.
----
Issues towards this goal:
* https://lab.civicrm.org/dev/cloud-native/issues/12
* https://lab.civicrm.org/dev/cloud-native/issues/7
* https://lab.civicrm.org/dev/cloud-native/issues/8https://lab.civicrm.org/dev/core/-/issues/3535Does CiviCRM allow us to declare which directories are writable, allowing the...2022-06-11T14:45:00ZherbdoolDoes CiviCRM allow us to declare which directories are writable, allowing the rest to be locked-down?CiviCRM core seems to be consistent in only writing to a files directory. Until recently, however, it attempted to write to the extensions directory to store extension updates. It's unclear if CiviCRM enforces this for extensions. I've h...CiviCRM core seems to be consistent in only writing to a files directory. Until recently, however, it attempted to write to the extensions directory to store extension updates. It's unclear if CiviCRM enforces this for extensions. I've heard that some extensions will write to directories outside of the files (but I haven't been able to confirm this). If that's the case then it will conflict with PAAS that only allow certain directories to be writable.https://lab.civicrm.org/dev/core/-/issues/3532Determine CiviCRM host and port in Docker-friendly way - via an environment v...2024-02-25T05:03:20ZMichael McAndrewDetermine CiviCRM host and port in Docker-friendly way - via an environment variablehttps://lab.civicrm.org/dev/core/-/issues/3531Time zones should be in sync even if database server is set to different time...2024-02-09T05:03:19ZherbdoolTime zones should be in sync even if database server is set to different time zone from server with application (PHP)According to https://docs.civicrm.org/installation/en/latest/general/requirements/#mysql-time:
> PHP and MySQL may be running on different servers, and the clocks may be out-of-sync. Configuring automatic clock synchronization is the be...According to https://docs.civicrm.org/installation/en/latest/general/requirements/#mysql-time:
> PHP and MySQL may be running on different servers, and the clocks may be out-of-sync. Configuring automatic clock synchronization is the best solution.
It's not clear how necessary it is for CiviCRM to rely on the database server's time zone. This causes issues on Pantheon since there is no guarantee the database server is in the same location as the app server. The database server is set to use UTC. One can override it with `SET @@global.time_zone = '-04:00'` or whatever, but this is only temporary. Pantheon migrates databases between servers as upgrades are rolled out. The new server has the default settings. So we can't rely on setting that and Pantheon is currently not planning to allow changes to that configuration in a file.
In core (per CMS integration) the database time zone is set each database session. Such as with `CRM_Core_Config::singleton()->userSystem->setMySQLTimeZone();` in civicrm.module. But this doesn't help in the case with Pantheon. (I'm not entirely sure why).
But it's not really clear why it needs to be done this way. Could CiviCRM instead take the same approach as Drupal/Backdrop by saving all dates as UTC and then convert them to the CMS' timezone for display? Isn't that more robust?https://lab.civicrm.org/dev/core/-/issues/3528[META] Does CiviCRM make it easy to provide the absolute system path at runti...2024-02-08T05:03:20Zherbdool[META] Does CiviCRM make it easy to provide the absolute system path at runtime from the environment rather than hardcoded in a file?In CiviCRM 4.7 this seems to be mostly solved but issues keep cropping up now and then. The environment variables can be specified in the settings file. In CiviCRM 4.7.25 it finally got rid of the hardcoded absolute path in Config.IDS.in...In CiviCRM 4.7 this seems to be mostly solved but issues keep cropping up now and then. The environment variables can be specified in the settings file. In CiviCRM 4.7.25 it finally got rid of the hardcoded absolute path in Config.IDS.ini. Most recently I've noticed some hardcoded paths in civicrm_cache where group_name = js_strings. It's unclear if the cache is regenerated if the paths cannot be found. Even better would be to avoid saving the full path.
----
Issues towards this goal:
* https://lab.civicrm.org/dev/cloud-native/issues/12
* https://lab.civicrm.org/dev/cloud-native/issues/14
* https://lab.civicrm.org/dev/cloud-native/issues/15
* https://lab.civicrm.org/dev/cloud-native/issues/16https://lab.civicrm.org/dev/core/-/issues/3526Extensions Mapper caches full path which breaks with dynamic containers2022-06-11T14:42:45ZherbdoolExtensions Mapper caches full path which breaks with dynamic containersCRM_Extension_Mapper caches the full path to extensions which with dynamic containers (such as with Pantheon) means that the cached path might not match the active container's path. This causes a fatal error.
The only solution seems to ...CRM_Extension_Mapper caches the full path to extensions which with dynamic containers (such as with Pantheon) means that the cached path might not match the active container's path. This causes a fatal error.
The only solution seems to be to force CRM_Extension_Mapper to fetch the extensions paths from the database, or prevent it from caching the full path. Currently with the CiviCRM Starterkit we are forcing it to fetch the list from the database each time (https://www.drupal.org/node/2347897).
Relates to https://lab.civicrm.org/dev/cloud-native/issues/6https://lab.civicrm.org/dev/core/-/issues/3525Move example data to an optional/add-on step2024-02-07T05:03:27ZtottenMove example data to an optional/add-on stepDuring installation, the installer shows a checkbox which asks whether you want to load sample data. Then:
* If you don't want sample data, it loads `sql/civicrm_data.mysql`.
* If you do want sample data, it loads `sql/civicrm_generat...During installation, the installer shows a checkbox which asks whether you want to load sample data. Then:
* If you don't want sample data, it loads `sql/civicrm_data.mysql`.
* If you do want sample data, it loads `sql/civicrm_generated.mysql`.
This has some consequences:
* You must choose whether to load sample data during *installation*. This is presented in the installer UI.
* The `civicrm_generated.mysql` takes extra work to maintain (eg `bin/regen.sh`; the output of which is not merge-friendly).
The goal here would be to have one *installation* process -- which could be headless. *Afterwards*, you should have the *option* to enable create the sample data. This *option* may be specified as at same time as other install parameters (eg via arg) or perhaps via a separate command afterwards.https://lab.civicrm.org/dev/core/-/issues/3524Does CiviCRM make it possible to specify which directories are private and wh...2022-06-11T14:42:36ZherbdoolDoes CiviCRM make it possible to specify which directories are private and which are public-accessible?This is currently not possible in CiviCRM. A PAAS like Pantheon provides a specific folder for private file uploads and we can't change it to any other folder. However, CiviCRM assumes that the site is either running on Apache (and it ca...This is currently not possible in CiviCRM. A PAAS like Pantheon provides a specific folder for private file uploads and we can't change it to any other folder. However, CiviCRM assumes that the site is either running on Apache (and it can use .htaccess files) or that custom NGINX rules can be set. This is an unreasonable expectation.
CiviCRM requires some filepaths to be private and will complain if they're not configured properly: https://civicrm.org/advisory/civi-sa-2014-001-risk-information-disclosure
CiviCRM has hardcoded the filepaths of some things based on the path of CIVICRM_TEMPLATE_COMPILEDIR. The problem is that CIVICRM_TEMPLATE_COMPILEDIR needs to be private but many of the files that CiviCRM is trying to write based on that path need to be publicly available (e.g. dynamically written JS).
An audit of both baseFilePath() and CIVICRM_TEMPLATE_COMPILEDIR:
[audit-template_compiledir.txt](/uploads/9302db5820d0af0181edf9990fd47b90/audit-template_compiledir.txt).
Tim provided some technical guidance in another thread https://lab.civicrm.org/dev/cloud-native/issues/1#note_3124:
> - The references to `baseFilePath()` in `CRM_Utils_System_*` should become irrelevant if `civicrm.settings.php` has the `$civicrm_paths['civicrm.files']`
>
> - The references to `baseFilePath()` in `CRM_Utils_File::absoluteDirectory` and `::relativeDirectory` already appear to be irrelevant within `civicrm-core`. (The only usage I could find was one which explicitly set its own base.)
>
> - The reference to `baseFilePath()` in `CRM_Core_Config_Runtime` is more effort. I don't know if it'd work, but my first try would be (a) lookup a path-variable like `Civi::paths()->getVariable('civicrm.log', 'path')`, (b) declare the variable in `Civi\Core\Paths`, (c) change the relative boot order of `Civi\Core\Paths` and `CRM_Core_Config_Runtime`.
>
> - I think `CRM/Utils/Cache/SerializeCache.php` is unused.
>
> - The `CRM/Core/IDS.php` line feels silly. We should pick one folder! Pointing that at `templates_c` makes as much sense as `uploadDir`. (TBH, I'm not sure does anything now that `Config.IDS.ini` has been killed.)
>
> - `Civi/Core/Container.php` and `CRM/Extension/ClassLoader.php` are very similar to the Smarty use-case (i.e. writing out ephemeral PHP files to take advantage of opcode caching).https://lab.civicrm.org/dev/core/-/issues/3523Don't cache the full path of extensions so they don't break with dynamic paths2022-06-11T14:41:17ZherbdoolDon't cache the full path of extensions so they don't break with dynamic pathsExtension paths are cached as full paths, which are incompatible with cloud services that might have dynamic filepaths (such as Pantheon). Instead the full path can be determined by other means.
This is an example of what gets cached in...Extension paths are cached as full paths, which are incompatible with cloud services that might have dynamic filepaths (such as Pantheon). Instead the full path can be determined by other means.
This is an example of what gets cached in CRM_Extension_Mapper::getActiveModuleFiles:
```
a:6:{i:0;a:2:{s:6:"prefix";s:4:"iats";s:8:"filePath";s:101:"/var/www/vhosts/httpdocs/sites/all/civicrm_custom/extensions/com.iatspayments.civicrm/iats.php";}i:1;a:2:{s:6:"prefix";s:14:"extendedreport";s:8:"filePath";s:114:"/var/www/vhosts/httpdocs/sites/all/civicrm_custom/extensions/nz.co.fuzion.extendedreport/extendedreport.php";}i:2;a:2:{s:6:"prefix";s:12:"cividiscount";s:8:"filePath";s:116:"/var/www/vhosts/httpdocs/sites/all/civicrm_custom/extensions/org.civicrm.module.cividiscount/cividiscount.php";}i:3;a:2:{s:6:"prefix";s:9:"sumfields";s:8:"filePath";s:108:"/var/www/vhosts/httpdocs/sites/all/civicrm_custom/extensions/net.ourpowerbase.sumfields/sumfields.php";}i:4;a:2:{s:6:"prefix";s:4:"gdpr";s:8:"filePath";s:102:"/var/www/vhosts/httpdocs/sites/all/civicrm_custom/extensions/uk.co.vedaconsulting.gdpr/gdpr.php";}i:5;a:2:{s:6:"prefix";s:10:"l10nupdate";s:8:"filePath";s:107:"/var/www/vhosts/httpdocs/sites/all/civicrm_custom/extensions/com.cividesk.l10n.update/l10nupdate.php";}}
```
Previous solution was to use the patch in this issue https://www.drupal.org/node/2347897 but it meant there was a performance hit by always skipping cache.5.24.0https://lab.civicrm.org/dev/core/-/issues/3522Soften messages for read-only extensionsDir2022-06-11T14:41:07ZtiotsopSoften messages for read-only extensionsDirImprove messaging when someone has a different policy for managing `extensionsDir`.This a continuation of this [PR](https://github.com/civicrm/civicrm-core/pull/11895). Most of the messaging update has already been done. There is only on...Improve messaging when someone has a different policy for managing `extensionsDir`.This a continuation of this [PR](https://github.com/civicrm/civicrm-core/pull/11895). Most of the messaging update has already been done. There is only one message ("Read-Only Extensions"). It still encourages web-writable policy, but it lowers the severity and presents it a choice ("if you want X, do Y").
Probably, changing the `warning` into a `notice` is the only thing to update:- a `warning` implies something is wrong, while a `notice` says it's merely out of the ordinary.5.9tiotsoptiotsophttps://lab.civicrm.org/dev/core/-/issues/3519UI for tracking user jobs2023-03-20T10:07:38ZeileenUI for tracking user jobsThis is a place holder for the user being able to access a UI to see their user jobs and for sysadmins to be able to see all jobs.
I expect the UI would be a packaged search kit display & that should be a pretty easy add.
Wish list fea...This is a place holder for the user being able to access a UI to see their user jobs and for sysadmins to be able to see all jobs.
I expect the UI would be a packaged search kit display & that should be a pretty easy add.
Wish list features
- information from the queue (x jobs out of y completed)
- ability to see the related import table
- ability to update rows in the related import table & re-queue those rowshttps://lab.civicrm.org/dev/core/-/issues/3518Ajax error in import queue runner if too slow2022-09-05T22:47:42ZeileenAjax error in import queue runner if too slowI've been hitting this when it is slow because of debugging - in theory could be other reasons
![image](/uploads/9f8e318d8819d3b0a08cb6d8d510e17c/image.png)I've been hitting this when it is slow because of debugging - in theory could be other reasons
![image](/uploads/9f8e318d8819d3b0a08cb6d8d510e17c/image.png)https://lab.civicrm.org/dev/core/-/issues/3517Re-add a confirm step to Imports2023-01-31T04:41:48ZeileenRe-add a confirm step to ImportsThe really really ugly confirm warning got lost - but we probably need to re-add a `crmConfirm`The really really ugly confirm warning got lost - but we probably need to re-add a `crmConfirm`https://lab.civicrm.org/dev/core/-/issues/3516User Job & User import table cleanup strategy2023-03-20T10:08:27ZeileenUser Job & User import table cleanup strategyIdeally the user job and import would be dropped at the same time - either through a manual process or a cleanup job. As of writing, all import tables are dropped on upgrade or, if they are more than two days old, they are dropped on ca...Ideally the user job and import would be dropped at the same time - either through a manual process or a cleanup job. As of writing, all import tables are dropped on upgrade or, if they are more than two days old, they are dropped on cache clearing.https://lab.civicrm.org/dev/core/-/issues/3514Define interfaces for interacting with newly cleaned up import code2023-04-03T02:29:57ZeileenDefine interfaces for interacting with newly cleaned up import codeThis is a bit of a placeholder but I think it will be necessary for integrations to
- ~~add new user_job_types - currently this is a hard-coded list and the Parser class is hard-coded into it - we probably need to solve this to get the ...This is a bit of a placeholder but I think it will be necessary for integrations to
- ~~add new user_job_types - currently this is a hard-coded list and the Parser class is hard-coded into it - we probably need to solve this to get the csvimporter working~~ these are declared by any class that `implements UserJobInterface`
- alter the metadate - eg inject 'full_name' into the available fields to map to. UPDATE - am wondering if being able to add pseudofields for the api is enough - would work for full_name
- intervene once `getMappedRow` has run
- ideally we would extract out the `lookup` code where matching contacts are found or matching entities and add some interaction there - assuming the hook in `findDuplicates` is not enough. UPDATE - I think intervening at the end of `getMappedRow` will work
- be able to alter the validation of the mapping. I know a form rule could be added by hook but I think a specific hook is cleaner as the hook would also need to alter existing rules