Add support for loading ECMAScript Modules (ESM)

Introduction

There was a recent announcement about a milestone -- all major browsers now support ECMAScript Modules (ESM) with "import-maps". Import-maps make ESM viable for complex applications (like CiviCRM) without requiring NodeJS during deployment.

The aim of this issue is to fully support ESM for JS development in CiviCRM -- by leveraging browser support for import-maps. It is broken down into a few sections:

• Part 1: Basic ESM Support
• Part 2: Import Map Support

However, because this is relatively new functionality with some risk and compatibility issues, the approach described here is an incremental phase-in -- where specific files/pages/extensions may use ESM (without requiring it for existing screens).

Background

• As discussed in https://gist.github.com/totten/5c34e3885a4fe7002f990e09395b4294, one major factor in updating CiviCRM's Javascript tooling is support for ECMAScript Modules (ESM).
• ESM has increasingly become necessary for a wide of range Javascript tools and libraries.
• Originally, browsers did not support ESM. Instead, JS developers used NodeJS-based tools (such as Webpack or Rollup) to convert ESM's into basic Javascript. However, browser support for ESM has improved over the years.
• For a decent introduction to the concepts of ESM and its import-maps, see (e.g.) https://blog.logrocket.com/es-modules-in-browsers-with-import-maps/ (Reading this #4279 (closed) will be easier if you've already been introduced to the concepts.)
• This issue corresponds to approach 4 in the gist, except that (now) Firefox and Safari both have native support for import-maps.

Part 1. Basic ESM Support

CiviCRM currently allows you to add Javascript to a page. For example:

Civi::resources()->addScript('doStuff();');
Civi::resources()->addScriptFile('my_extension', 'js/foo.js');
Civi::resources()->addScriptUrl('https://example.com/bar.js');

generates output like:

<script type="text/javascript">
doStuff();
</script>

<script type="text/javascript" src="https://example.com/path/to/my_extension/js/foo.js"></script>

<script type="text/javascript" src="https://example.com/bar.js"></script>

ESM documents are slightly different from regular Javascript documents -- e.g. they support the import and export keywords. Browsers require a different declaration for ESM:

<script type="module">
import doStuff from './do-stuff.js';
doStuff();
</script>

<script type="module" src="https://example.com/path/to/my_extension/js/foo.js"></script>

<script type="module" src="https://example.com/bar.js"></script>

On the PHP side, we probably need a way to flag resources as ESMs. Perhaps:

Civi::resources()->addScript('import doStuff from \'./do-stuff.js\'; doStuff();', ['esm' => TRUE]);
Civi::resources()->addScriptFile('my_extension', 'js/foo.js', ['esm' => TRUE]);
Civi::resources()->addScriptUrl('https://example.com/bar.js', ['esm' => TRUE]);

or

Civi::resources()->addModule('import doStuff from \'./do-stuff.js\'; doStuff();');
Civi::resources()->addModuleFile('my_extension', 'js/foo.js');
Civi::resources()->addModuleUrl('https://example.com/bar.js');

Other considerations:

• Does it matter if or how ESM's appear in hook_civicrm_coreResourceList? Assuming we keep our current file-names (*.js), then there'd be no way to distinguish vanilla JS files from ESM files. OTOH, that hook was deprecated circa 5.31. The newer hook_civicrm_alterBundle should be more agreeable (e.g. allowing $bundle->addModuleFile(), $bundle->filter(), etc).
• Is it absolutely necessary to add more methods or flags to Civi::resources()? Hypothetically, we could do a global swap in all generated HTML (replace <script type="text/javascript"> with <script type="module">). However, there are differences of interpretation - e.g. wrt scoping and global variables. I did a quick hack to use type="module" globally... and the page didn't render. Nothing's impossible, but changing all-at-once feels like a heavier lift.

Part 2. Import Map Support

Outputting <script type="module"> is a prerequisite for using ESM's in Civi. However, it's not sufficient - a CiviCRM deployment is split across various folders (what with core, core-exts, downloaded-ext's, CMS plugins, and so on). With basic ESM's, it's tricky to reference all these folders.

By adding an import-map, we can load dependencies from all these different folders.

For example, on a site with civicrm, search_kit, and mosaico, you might generate an import-map that looks like this:

<script type="importmap">
{
  "imports": {
    "civicrm/": "https://example.com/modules/civicrm/js/",
    "search_kit/": "https://example.com/modules/civicrm/ext/search_kit/js/"
    "mosaico/": "https://example.com/files/civicrm/ext/mosaico/js"
  }
}
</script>

which would enable imports from those paths, as in

import searchFoo from 'search_kit/foo.js';
import mosaicoFoo from 'mosaico/foo.js';

There are a few important considerations here. In no particular order...

• Quote: "Only the first import map in the document with an inline definition is processed; any additional import maps and external import maps are ignored." -- This means that the importmap will (eventually) be subject to contention among applications like CiviCRM, Drupal, WordPress, etc. We'll (eventually) need a way to integrate the import-maps from different applications.
• AFAICS, the CMS's do not currently have a standard way to handle import-maps. But it's on their radar. (Ex: Drupal issue, WP/Gutenberg issue) We should expect that (eventually) some CMS's will define their own registry. Even when they don't, we should expect other CMS plugins will start adding import-maps.
• There is a logical namespace. We need some guideline for organizing in the logical-namespace.
  • Example: Map between CiviCRM extension-names and ESM folders (as in the search_kit and mosaico items above).
  • Given that every page is limited to one import-map, that means the logical namespace will be shared between Civi extensions and $open_ended_future_cms_stuff. We should probably use a prefix.
• Should we automatically map all extensions -- or require an opt-in (e.g. hook_civicrm_esm or hook_civicrm_importMaps)? How confident are we that CiviCRM extensions should be mapped 1-to-1 with ESM's folder namespace?
• "The src, async, ... attributes must not be specified." (source) -- The entire importmap must be outputted inline as part of the top-level HTML page; you can load an external map (<script type="importmap" src="url">). However, this should eventually change as WICG's specs define define a way to move the importmap to a src.

Given the constraints and expectation of future change, this approach comes to mind:

1. Define hook_civicrm_esmImportMap(array &$importMap) as a way for CiviCRM (core/exts) to modify the import-map.
2. Define mixin/esm@1.0.0. This is our first-best-guess at how to map between extensions and ESM folders, Maybe:

   return function ($mixInfo) {
     Civi::dispatcher()->addListner('hook_civicrm_esmImportMap', function (&$importMap) use ($mixInfo) {
       $logicalPath = 'civicrm/' . $mixInfo->shortName . '/';
       $physicalPath = Civi::resources()->getUrl($mixInfo->longName) . '/';
       $importMap[$logicalPath] = $physicalPath;
     });
   }

3. Provide an initial implementation which fires hook_esmImportMap and outputs the <script type="importmap">...</script> in the page-header.
   • However, we should expect that (in the future) UF-integrations will need to swap this -- they will instead ask CiviCRM for $importMap and pass that to a CMS API.
   • Additionally, if this mechanism is swappable, then that should improve the prognosis for adding the es-module-shims to extend compatibility to older browsers.