Docs falling off organic search results
I've noticed in the past several months that "docs" is appearing less and less on SERPs and, when it does, it's often not on page one. This is inconsistent with past experiences where "docs" often ranked within the top 3 positions on organic search.
A review of potential issues using Google Search Console indicates an increasing number of URLs marked as "noindex". A list of current URLs (365 as of today) flagged in this manner is here.
Originally, I thought this was specific to certain guides as I've seen a higher number of error URLs under "user" docs than under "dev", however it appears to be a bit more random than I initially thought.
I believe the cause is related to the canonical url declaration combined with a redirect. Google states:
Excluded: The page is not indexed, but we think that was your intention. (For example, you might have deliberately excluded it by a noindex directive, or it might be a duplicate of a canonical page that we've already indexed on your site.)
As an example:
-
https://docs.civicrm.org/dev/en/latest/api/v4/custom-data/
- The current "live" URL.
- Declared canonical: https://docs.civicrm.org/dev/en/stable/api/v4/custom-data/
- Google states that this page is not indexed and cites the "Alternate page with proper canonical tag".
-
https://docs.civicrm.org/dev/en/stable/api/v4/custom-data/
- Declared canonical URL is https://docs.civicrm.org/dev/en/stable/api/v4/custom-data/
- Redirects to https://docs.civicrm.org/dev/en/latest/api/v4/custom-data/
- Errors with: No: 'noindex' detected in 'X-Robots-Tag' http header (image below)
I'm not sure if this is correct, however it seems like for pages that are erroring, the issue is declaring a canonical URL that is flagged as "noindex" and that redirects back to the current page.
I believe the proper fix would be to mark the current URL as the user declared canonical (ensure that it is 'indexable', which it seems to be) and maintain the expired URLs as "noindex" as redirects to the current URL.