MS Exchange - IMAP-XOAuth2 authentication fails

Overview

#2141 (closed) introduced support for (1) obtaining OAuth2 tokens and (2) using the OAuth2 tokens for IMAP-XOAuth2.

In my test setup, I am able to obtain a token from Microsoft which appears valid -- but their IMAP server is not accepting the token.

The "Critical Evaluation" below include a more thorough review of theories/data, but here's a high-level summary of debugging efforts:

Provider	OAuth2 initiation	Use token for IMAP/XOAuth2	Use token for HTTP/REST
Microsoft Exchange Online		fails, tho JWT has IMAP claims
Google Mail			(not tested)

Reproduction steps

Configure Civi's OAuth client as described in the draft documentation: documentation/docs/sysadmin!278 (merged)

Recap:

1. Enable oauth-client
2. Register the OAuth details in Civi and in Azure Portal.
3. In Civi's "Admin => CiviMail => Mail Accounts", add a mail account
4. For the newly created account, choose "Save & Test".

Current behavior

The IMAP server responds with NO AUTHENTICATE failed. and * BAD Command Error. 12

In the GUI, it appears as:

On a network level, the interaction is:

<< * OK The Microsoft Exchange IMAP4 service is ready. [{_LONG_RANDOM_CODE_}]
>> A0001 AUTHENTICATE XOAUTH2 {_THE_XOAUTH2_TOKEN_}
<< A0001 NO AUTHENTICATE failed.
>>
<< * BAD Command Error. 12
>> A0002 LOGOUT
<< * BAD Command Error. 12

(To see this, I applied a hack to log network I/O and re-ran the check via CLI.)

Expected behavior

The IMAP test should succeed.

Environment information

• Browser: Firefox
• CiviCRM: 5.32.beta1
• PHP: 7.1
• CMS: Drupal 7
• Database: MySQL 5.6
• Web Server: Apache 2.4

Critical evaluation

Here are a few theories for where the problem may reside.

1. The problem is in the OAuth2 client.
   1. The OAuth2 client is broken - it does not obtain a real token from Microsoft.
   2. The OAuth2 client obtains a real token - but the token does not have the proper scopes.
   3. The OAuth2 client obtains a valid token with proper scopes - but it doesn't speak IMAP-XOAuth2 correctly.
2. The problem is in the OAuth2 service provider (either the authorization-server or the resource-server/IMAP-server).
   1. There is a problem in the account flags/configuration.
   2. There is a bug in the OAuth2 service provider.

Let's consider each in turn to see if we can prove/disprove the theory:

(Theory 1.1) The OAuth2 client is broken - it does not obtain a real token from Microsoft.

This explanation doesn't hold -- because the same token works for other scenarios.

• Navigate to "Admin => System Settings => OAuth => ms-exchange"

  

• Click "Inspect"

• Find "Access Token: Raw". Copy the value.

• In CLI, make an HTTP request to MS with the same token:

  $ curl 'https://graph.microsoft.com/v1.0/me' \
    -H 'Accept: application/json' \
    -H 'Authorization: Bearer <TOKEN_VALUE>'
  
  {"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#users/$entity","businessPhones":["XXX-XXX-XXXX"],"displayName":"My Name","givenName":"My","jobTitle":null,"mail":"myname@wariomail.onmicrosoft.com","mobilePhone":null,"officeLocation":null,"preferredLanguage":"en-US","surname":"Name","userPrincipalName":"myname@wariomail.onmicrosoft.com","id":"XXXXX-XXXX-XXXXX"}

• The response is well-formed and has the correct information.

(Theory 1.2) The OAuth2 client obtains a real token - but the token does not have the proper scopes.

I can show that the client works according to spec.

• In the Microsoft documentation Authenticate an IMAP, POP or SMTP connection using OAuth, we can see the required scope is IMAP.AccessAsUser.All.

• In Azure Portal, navigate to "Azure Active Directory => App registrations => (my-app) => API permissions". Note the list of permissions includes IMAP:

• In CiviCRM, navigate to "Admin => System Settings => OAuth => ms-exchange"

• In the "Details" tab, we can see the list of scopes that requested during setup:

  

• In the "Client #X" tab, we can "Inspect" the token.

  • Observe: The "Token Record" shows that the raw response from the OAuth2 service. It indicates approval for the necessary scopes:
  • Observe: The "Access Token: JWT Payload" also reports these scopes:

      "scp": "IMAP.AccessAsUser.All POP.AccessAsUser.All SMTP.Send User.Read profile openid email",

So, it appears that every component (from Azure app registration, to the client's request configuration, to the received token data, to the decoded JWT token data) reports IMAP.AccessAsUser.All as an included scope.

There is a subtle discrepancy in how the scope is labeled - sometimes, it's presented as the shorter IMAP.AccessAsUser.All; other times, it's the longer https://outlook.office.com/IMAP.AccessAsUser.All. To be sure, I've requested tokens with both notations -- and in both cases, the outcome is the same. (To wit: the JWT token has IMAP.AccessAsUser.All - but it does not work for IMAP access.)

(Theory 1.3) The OAuth2 client obtains a valid token with proper scopes - but it doesn't speak IMAP-XOAuth2 correctly.

I've tested this theory a few ways:

1. Connect to another service (Google Mail) which uses the same protocol (IMAP-XOauth2). It works.
2. Hack the PHP IMAP library (imap_transport.php) to inspect the wire-communication. Manually decode the XOAuth2 statement to ensure that it follows the XOAuth2 formula.
3. Write a new script with a different PHP IMAP library to try the same token. The alternate library works with Google Mail but not Microsoft Exchange.

(Theory 2.1) There is a problem in the account flags/configuration.

As I mentioned above (theory 1.2), the "App registration" has all the scopes described in the docs

In the "Exchange Administration Center", the account has IMAP enabled:

In the Exchange webmail UI, there is a settings page with a search-box. Search for "IMAP", and it shows:

It's peculiar that searching for "IMAP" presents an inert radio-button for POP, but it does show connection details for IMAP.

TBH, Microsoft's backend here is labyrinthine, so it's hard to be sure that I've checked every possible permission. But I can't find anything else that would prevent IMAP access.

Perhaps someone with more experience in Microsoft's backend should try to run on a different account? Perhaps an account that has been demonstrated to work with IMAP?

(Theory 2.2) There is a bug in the OAuth2 service provider.

I can't really prove or disprove this theory. I don't really want to believe this is the problem (surely this stuff works for other people...). OTOH, maybe it's actually quite rare for people to use IMAP+XOAuth2 with Exchange Online...

moved from mail#79 (moved)

changed the description

I was going to mention I was having trouble with the microsoft one, but assumed it was on my end and was going to give it another go later. Haven't read all the above yet just wanted to say you're not the only one.

changed the description

I've made a variant of the workflow which runs as a simple/standalone script: https://github.com/totten/oauth2-imap-demo

Same problem

No success just adding that I did some tests that can maybe rule out some of the above:

1. Using openssl s_client and manually base64'ing and typing in protocol commands I get the same result.
2. It's not that IMAP isn't enabled. I can login and do IMAP stuff using regular IMAP logins.
3. POP and SMTP have the same problem.
4. It DOES work with thunderbird and using OAUTH2 for IMAP. I'm definitely seeing the list of folders that's on the server. So unless it's lying about using OAUTH2 and is falling back to regular AUTH LOGIN, the service itself is working. So it suggests either scope mismatches or maybe we're not using the right token?
   • For comparison these look like the same we have:
   • https://hg.mozilla.org/comm-central/file/df9a40274fe58403700e90a026d243718512249a/mailnews/base/src/OAuth2Providers.jsm#l36
   • https://hg.mozilla.org/comm-central/file/df9a40274fe58403700e90a026d243718512249a/mailnews/base/src/OAuth2Module.jsm#l174

Oh wait, the scopes aren't EXACTLY the same: We have "outlook.office.com" and they have "outlook.office365.com".

Hmm, unless I'm missing a spot it seems to make no difference.

Aha success! I think it is scope related.

I went through the flow manually and both when requesting a code and then a token I specified the full scope list as https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/POP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access.

Gonna take a break and then will write it up.

Ok so the problem is that you can't request scopes from different resources all in the same token. The User.Read scope is from graph.microsoft.com.

If you specify it bare at the top of the list, as it is now, it messes things up. Using its full scope https://graph.microsoft.com/User.Read doesn't help.

You can include a bare User.Read at the end of the list but then it converts it to the outlook.office365.com resource and so it doesn't recognize that permission, i.e. on the mail account page it doesn't guess your settings, but then if you manually fill them in and Save and Test then it works.

In other words, whatever resource is used for the first one in the list gets used for all of them. The offline_access one is special - technically it's graph, but it seems to be ok as-is.

So this is the patch:

--- a/ext/oauth-client/providers/ms-exchange.dist.json
+++ b/ext/oauth-client/providers/ms-exchange.dist.json
@@ -6,10 +6,9 @@
     "urlResourceOwnerDetails": "https://graph.microsoft.com/v1.0/me",
     "scopeSeparator": " ",
     "scopes": [
-      "User.Read",
-      "https://outlook.office.com/IMAP.AccessAsUser.All",
-      "https://outlook.office.com/POP.AccessAsUser.All",
-      "https://outlook.office.com/SMTP.Send",
+      "https://outlook.office365.com/IMAP.AccessAsUser.All",
+      "https://outlook.office365.com/POP.AccessAsUser.All",
+      "https://outlook.office365.com/SMTP.Send",
       "offline_access"
     ]
   },

So to have the system guess the user settings maybe you'd need 2 tokens. I haven't looked into it.

added comp:CiviMail git:civicrm-core triaged labels

@DaveD Wow, nice work. :) I can confirm that removing User.Read yields a working IMAP token.

I did some more testing, and there are a couple other ways to get the email address:

• If one decodes the main access_token (as JWT), then email is present as upn.
• If one adds the OpenID scopes, then the token-request returns an id_token... which is also JWT. The id_token is also more clearly documented as being JWT with specific claims.

So what we could do is treat the decoded id_token as the "resource owner" record.

I'm not certain how to fully validate the id_token (ie what key-pair does MS use?), but I think it doesn't matter since we receive the id_token as part of an otherwise-secure token request (https://login.microsoftonline.com/common/oauth2/v2.0/token), and the OpenID spec seems to affirm this:

If the ID Token is received via direct communication between the Client and the Token Endpoint (which it is in this flow), the TLS server validation MAY be used to validate the issuer in place of checking the token signature. The Client MUST validate the signature of all other ID Tokens according to JWS [JWS] using the algorithm specified in the JWT alg Header Parameter. The Client MUST use the keys provided by the Issuer.

Working on a patch...

Patch: https://github.com/civicrm/civicrm-core/pull/18951

Thanks! Will give it a spin.

mentioned in commit 47617d1e

mentioned in commit 4fdca40a

mentioned in commit d551e4c1

PR has been merged closing

closed

changed milestone to %5.32.0

mentioned in issue #2141 (closed)

mentioned in issue #2559 (closed)