A/B testing report should show unique opens, not total opens

@larssg are you able to provide a fix as well?

Should be fairly simple, so I should be able to handle it.

I've had a look at this and don't quite understand how things are working. If anyone is able to point me in the right direction, that would be much appreciated.

These stats are coming from here using crmApi, but I'm not quite clear on how crmApi pulls the stats from the API. The is_distinct: 0 at 510 sure looks suspicious! This might be as simple as replacing the 0 with a 1 (it looks like @scardinius added this).

After having used this further, I also see that clicks are also reporting total clicks, not unique clicks, which should also be fixed by the same change.

Unsubscribes is reporting unsubs + opt out, which should at least be noted (confusing because if you click through to the report or search, you get only unsubs). It might be better to report this separately as two lines, but will need more investigation.

I also note that the number of opens ignores your time zone, so if you're in GMT-7, the number of opens is 7 hours out of date, i.e. doesn't show any results until seven hours after they happened. Seems there are a few related issues here.

Digging further, I can see that using is_distinct: 1 here will improve the situation, giving the number of contacts who clicked any link, the number of contacts who opened and the number of contacts who unsubbed or opted out. But, the delivered and bounces values returned by the API are wrong: Delivered always returns 1 for any number of deliveries and bounces returns null for 0 bounces and 1 for one or more bounces.

I believe these should be fixed by removing the ifs here and here, but I want to make sure that makes sense before proceeding. Maybe there's some reason to want these to return those values? It makes sense to me that the distinct number of deliveries and the non-distinct number of deliveries would be the same (and the same for bounces).

I imagine @colemanw might now the answers to some of these questions.

The time issue is a little bit more obscure to me. I think what's happening is that the time of clicks, opens, etc are being recorded as UTC. However, the API is using the current time in the server time zone here for the query, with the result that if you are in GMT -6, for instance, the query won't return any results from the last six hours because those are in the future relative to server time. What's the solution here? My first thought is do we need to pass the time at all? Perhaps just leaving the time out would give us the full results, but I haven't tried this yet. This has been solved on the normal mailing report page, so I can look there, but that brings up a bigger question.

I wonder what the long term plan is with the respect to the two different mailing report pages. There's the normal mailing report and then there's this A/B mailing report that uses the API. It would make sense to use one of these only in the future. Does that mean fixing up the A/B version and using it for all mailings or vice versa? The API doesn't give intended recipients, forwards, replies, or opt out and unsub separately, while the normal mailing report doesn't give unique clicks. Does anyone have thoughts on the way forward here?

Here's a PR that fixes the unique opens and clicks counting issues and renames Unsubscribes to Unsubscribes & Opt-outs. I haven't dealt with the time zone issue or the bigger picture questions above, but this is starting point for discussion. I'm also not sure that distinct=1 in the API returning 1 for the number of deliveries might not be important elsewhere, though it seems unlikely.

@larssg agree with you that "Unique Opens" is a very important metric to add to A/B Test report page, but I'm not sure I would replace Total Opens by Unique Opens, instead of adding the Unique Open metric to the existing ones.

In our experience having both metrics (Totals and Unique, as they are in Traditional CiviMailing right now) can return useful comparison information to the Organization and to the developers (we had cases were weird Total open number led us to a bug in a recipient that had issues with its Mailing Client, opening several times a day the same email, and affecting the final stats).

So in my POV adding those stats is better than replacing.

@sluc23 Agree with you that we want both unique and total opens.

I think this goes back to the long-term question here. It doesn't really make sense to have two entirely separate ways of generating these stats (one for normal mailings, another for A/B mailings). So I think if there's a path forward to using the A/B mailing report stats, which come through the API, for all mailings, then we should definitely add total opens, unsubs and opt outs separately, and forwards and replies. See the PR for further discussion.

@larssg agree again that does not make much sense to have different tracking methods for A/B and regular CiviMailing.. I cannot add much about the implications of changing one or the other.

Just wanted to add that IF this PR moves forward would be good to somehow alert the users that this stats has changed.. if not the end users won't have any notice about that from CiviCRM x version the stats named as Opened was for Total Opens and after upgrading to x+ version now is Unique Opens.

If I'm not mistaken the label of the stats has not changed and it still will display label Opened (from your PR), so that could lead to confusion

@sluc23 Thanks for this, I updated the PR so the labels are clear that these are unique not total.

Although thinking about this more, I'm not convinced that we do want to show total and unique opens. Other than diagnosing weird issues as you mentioned, is there any use case for showing total opens? It's pretty much meaningless as a metric, I would think, as many mail clients won't re-fetch the tracking image (e.g. Gmail) on a second open. In other cases, you'll get one contact opening a message 8 times in a row, which isn't really meaningful.

mentioned in issue #2810 (closed)

mentioned in commit 508bfb89

closed

A/B testing report should show unique opens, not total opens

Child items ...

Activity