Skip to content

OAuth2 administration (email focus)

Background

CiviCRM integrates with various third-party services. When submitting read/write requests to contemporary web APIs, it is quite common to authenticate via OAuth - esp OAuth2 (https://tools.ietf.org/html/rfc6749). For this use-case, we consider CiviCRM (the web-app) acting a client to some service-provider. Two important workflows are:

  • Client registration: This is a manual, non-standardized process. The site admin logs into the Google/Microsoft/Facebook/Twitter/etc administrative system and generates a client_id and client_secret. These values must be copied into the web-app (CiviCRM) configuration.
  • Authorization Code Grant: This process obtains permissions for a specific user/resource/service. It involves (a) performing a semi-manual workflow to confirm permissions and (b) periodically requesting access tokens.

For CiviCRM integrations (core or contrib) that wish to speak to other systems, this creates a recurring administrative problem:

  • (Developer) You need to define several artifacts (docs, settings, routes) for managing client_id, client_secret, and each of the authorizations/grants.
  • (Administrator) You need to examine the separate artifacts (docs, settings, etc) required by each integration, and there will likely be arbitrary differences.

This proposal arises from mail#59 (moved) and parallel issues for Microsoft's email service. When we need examples to make this more concrete, we'll refer to IMAP and cloud email.

Goal

Define a general OAuth2 administration mechanism in CiviCRM.

To keep the scope limited, the initial concern is managing credentials for general system-level integrations (e.g. Administer => System Settings => Credentials). For example, one might use this to:

  • Connect CiviMail with a cloud mail service
  • Connect CiviContribute with a cloud accounting system

This does not address the per-user authorizations. For example, if you wished to allow each staff member or peer-to-peer fundraiser to link their own email account (for use with sending one-on-one emails), then that would be a different endeavor. (I believe it is a superset of the endeavor described here, and the "Add-ons/Strech-goals/Future-directions" touches on this.)

Model

We track information in three layers:

  1. Provider (Client-Type): Suppose, for example, that many CiviCRM deployments connect to services provided by Google and Microsoft. Anytime you connect to Google, it should go to a particular URL; similarly, anytime you connect to Microsoft, it goes a particular URL. You need store a list of these common URLs. For each provider, you'd expect a few key properties like:
    • authorize_url
    • token_url
    • default_scopes
    • provider_class (string; ex: in theleague/oauth2-client, it has \League\OAuth2\Client\Provider\GenericProvider for generic use; but alternate classes can provide extra options+data)
  2. Client: When a CiviCRM site (example.com) connects to a provider, it must identify itself. Key properties:
    • client_id (unique id; functions a username when contacting the OAuth2 service)
    • client_secret (secret credential; functions a password when contacting the OAuth2 service)
  3. Account Authorization: When a CiviCRM site is configured as a client, it may not have authorization to any particular resources/accounts/services. To acquire access, you need the user ("resource owner") to open the web-app and go through some redirects/authorization screens.

Ex: IMAP: UI Mockup

The current IMAP configuration screen accepts a username+password. Instead, one needs to use credentials that are stored in an OAuth2 subsystem. Compare:

The "Manage Credentials" screen gives the Civi administrator the ability to configure the client identity and the account authorizations. The layout might be:

Ex: IMAP: General tasks

A loose/general list of tasks:

  • Add OAuth data structures:
    • Client types: Define a data-structure (e.g. JSON file, hooks) for programmatically defining client types.
      • (Should this have a SQL storage option? IMHO, no, that's an extremely niche proposition. To require that the administrator to first create the "Client Type" in the web UI seems like an exercise in software brutalism.)
    • Clients: Define a data-structure for client_ids, client_secrets, etc. These details are necessarily different for each deployment, so it's necessary to store them somewhere in the admin's purview -- ie SQL and/or civicrm.settings.php.
    • Account Authorizations and Tokens: Define a data-structure for account authorizations and their tokens. These must be stored in SQL.
  • Update IMAP/POP data-structure:
    • Store a reference to OAuth
  • Implement UI: Like in the mockups above:
    • Implement the "Manage credentials" screen and "Add account" flow (incl thephpleague/oauth2-client)
    • Update the "New Mail Account" screen
  • Update IMAP/POP driver: The current IMAP/pop driver uses username/password. If a specific mailbox uses IMAP
  • Documentation:
    • Developer: How to define or edit a client-type
    • Developer: How to send a web-service request with stored credentials and Guzzle
    • Sysadmin: How to manage clients/connections

Add-ons / Stretch goals / Future directions

Here are few topics for potential exploration:

  • E2E Testing: Setup a test-case which makes real IMAP+OAuth2 connections
  • Reuse client-type/provider metadata: The list of client-types is, essentially, metadata. The oauthd project has a ton of metadata stored as JSON files. Consider using their metadata. https://github.com/oauth-io/oauthd/tree/master/providers
  • Aggregator support: This issue is oriented toward singular/dedicated deployments. However, if you have multi-tenant/multi-site deployments, then deploying OAuth2 is more onerous. (To wit: do you register every site with its own client_ids -- which is a lot of manual setup -- or do you copy/share the IDs -- which is less secure and may not be supported by all providers). An OAuth aggregator allows your various tenants/subsites to use the same client ids more securely. Generally, the goal would be to integrate the list of "Clients" in Civi with the aggregator.
    • (The oauthd project linked above is an aggregator. It seems it isn't being updated, but it does have a lot of forks. Curious.)
  • In-situ authorization or Wizards: The "Manage Credentials" screen is disjoint from the "New Mail Account" screen. With in-situ authorization or wizards, you could handle the authorization in-siteu (e.g. in "New Mail Account" or in CiviMail).
  • Authorization (Meta)Access Controls: This issue is oriented toward a global/shared service (like the CiviCase IMAP checker), but OAuth can also be used for per-user services. Example: Suppose staffer Alice wants to send a mail-merge document to Google Cloud Print. This probably requires OAuth authorization. Civi can store that credential and re-use it whenever Alice wants to print. But if Bob wants to print, then he needs to go through a different authorization (because he uses different printers).
Edited by totten
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information