An Independent.ie SEO Case Study

Can you rank in Google’s Top stories without AMP?

The answer should have been no.

But in this column, you’ll learn what we found by testing non-AMP versus AMP on Top stories performance and the impact it had on traffic at Independent News and Media, where I lead SEO.

We’ll explore Google’s documentation on the topic, see what’s happening in the wild, and attempt to answer a few pressing questions in news SEO:

• Can non-AMP articles perform in Top stories?
• How does the organic search performance of non-AMP articles compare to AMP?
• How does mobile-first indexing impact Top stories performance?
• Will publishers drop AMP when the Page Experience update rolls out?
• Is AMP really worth it?

But first, let’s start with one important question.

Is AMP Required for Top Stories Eligibility?

According to Google’s documentation,

“To be considered for the carousel section of “Top stories” on mobile, content needs to be published in Accelerated Mobile Pages (AMP) format with article-specific structured data”

Google announced that they are removing the AMP requirement from Top stories eligibility on mobile when Page Experience ranking signals are introduced.

I’ll let you in on a secret, though; non-AMP pages are already appearing in Top stories and have been for quite some time.

Here’s an example from Belgian news publisher De Standaard, which has not adopted AMP.

Here. we see them ranking above an AMP news article:

We see the same result for one of the publishers in our group, Independent.ie, which also ranked in the Top stories on mobile with non-AMP in the example below.

How is this possible?

One word: relevance.

Google recently updated their Core Web Vitals FAQ resource to reaffirm the impact of the update on rankings:

“Intent of the search query is still a very strong signal, so a page with a subpar page experience may still rank highly if it has great, relevant content.”

And in December 2019, Google announced that:

“Publishers no longer need to submit their site to be eligible for the Google News app and website. Publishers are automatically considered for Top stories or the News tab of Search.”

Newzdash recently analyzed the percentage share of non-AMP stories in Top stories and compiled a report examining the impact of this announcement.

Their data (reflecting US news publishers) as shown below indicates that about 5-8% of all mobile Top stories on Google are non-AMP URLs.

Google generally tests most major rollouts extensively, which could explain why we are seeing a lot more non-AMP URLs in top stories in 2020 and 2021.

With all of that in mind, we decided to examine whether non-AMP could produce the same traffic as AMP.

What would the impact be if we decommissioned AMP?

Considerations for a/B Testing on a News Site

The challenge when testing AMP versus non-AMP for a news publisher is that news causes too many traffic spikes.

Unlike traditional keyword research, which predominantly uses 12 months of aggregated data, news keyword research is predominately based on trending topics or those that have not been searched before.

Take [COVID-19] pre-2020, for example.

Further, even though certain news topics repeatedly appear in the news agenda, these are hard to predict.

Take the popular celebrity name [britney spears], for example. In Ireland, Britney last made the headlines with news on her documentary.

Outside of that, there was little to no search interest.

See the query’s search performance on Independent.ie:

We see similar patterns for most news stories — a spike in traffic followed by little search interest and clicks.

This can also be observed for the rankings on a news site.

For example, when a celebrity or topic is in the news,  generally, news sites will rank well. But if that celebrity or news topic is not trending, the SERP will “reset,” causing news sites to lose these rankings.

This classic concept is known as QDF – Query Deserves Freshness and it affects news sites perhaps more than any others.

Google’s Freshness algorithm comes into effect when a topic sees a sudden rise in relevancy/mentions (news reports) and traffic (search volume).

When it comes to forecasting with news sites in terms of traffic or rankings, for the most part this concept makes the data less reliable for predictions or correlations.

Therefore, it is essential to find a more evergreen section on a news site for testing. Even so, I note that it is still not immune to the unpredictable nature of the news agenda.

How to Test AMP Versus Non-AMP on a News Site

Understanding how AMP indexing works is important when testing AMP versus non-AMP.

Following Google’s guidelines on removing AMP, you could consider the following steps.

Step 1. Remove the link tag with the relation “amphtml” from the canonical.

Googlebot discovers AMP articles by crawling the canonical and discovering a link tag with the relation “amphtml.” See the example below:

The first step in testing AMP versus non-AMP is to remove this tag.

If you were interested in new articles only, this is all you need to do. But as we had thousands of indexed AMPs, we needed two additional steps.

Step 2. 301 Redirect the AMP variant to the canonical.

Once Googlebot has indexed the AMP variants, these will remain in Google’s AMP cache and need to be redirected to the canonical.

Step 3. Resubmit XML sitemaps.

Although it’s possible to request that Google update its AMP cache, this is not scalable as it has to be done one URL at a time.

We found it was faster to resubmit the XML sitemaps. In the case of a news site, this also means resubmitting the News XML sitemaps.

How to Measure AMP Versus Non-AMP on a News Site

Measuring AMP versus non-AMP is different depending on your URL structure.

In this test on Independent.ie, we had a separate URL structure (www.independent.ie) for desktop, (m.independent.ie) for mobile web, and (amp.independent.ie) for AMP.

We decided to test the impact of AMP versus non-AMP on Google Search traffic by removing AMP from articles in one section of the site and comparing it to AMP in another.

We had also considered split testing articles at the ID level, or even testing every second published article using Cloudflare Workers.

But again, none of those scenarios would remove the unknown nature of the news agenda. One article may have a high search interest, whereas another may not.

The Life section on the site was chosen to test without AMP, as it was considered to be the most evergreen section. (We defined this section evergreen as it had topics driving clicks from search for more than 3 months).

It also had a similar average traffic level month-over-month.

As a control, we chose the Style section because it had similar average search traffic levels but was a bit more “spikey” due to celebrity news desked in that section.

For measurement, we set up a simple Google Analytics custom report filtered by hostname and site section.

User segments were then filtered by Google search traffic only:

We set conditions to include web organic traffic and exclude AMP traffic, as well as web traffic with AMP traffic and then AMP traffic only.

How Does the Organic Search Performance of Non-AMP Compare to AMP?

As mentioned in the custom report set up above, the AMP vs non-AMP isolated organic traffic to the AMP variant (amp.independent.ie/life/.*).

This data was compared to organic traffic excluding AMP (www.independent.ie/life/.*) and (m.independent.ie/.*), as well as organic traffic including AMP (amp.independent.ie|m.independent.ie|www.independent.ie).

Prior to testing, AMP accounted for approximately 50% of all search traffic to the site.

Our hypothesis was that total search traffic should decline with the Life section of Independent.ie having non-AMP only.

On February 1st, 2021, when AMP was removed from the Life section of the Independent.ie website, the orange line above (users excluding AMP), merged into the green line (all users) with the blue line (AMP) decreasing to null.

This finding suggests that traffic without the AMP article variant simply merged into the non-AMP.

Total search traffic in the Life section remained on par with the previous period.

When we examined the same graph on the Style section (the control), which had AMP, we observed that users from search remained constant on both non-AMP (m.independent.ie/style/.*|www.independent.ie/style/.*) as well as AMP (amp.independent.ie/style/.*).

Our conclusion is that having an AMP variant only split the traffic. It did not add any further benefit than our non-AMP news articles.

As mentioned above, the test was conducted on evergreen content. AMP is thought to play a bigger role in news sites due to the mobile Top stories carousel — not so much in regular organic search.

In March 2021, our company made the decision to decommission AMP across all publishers in the group.

If we run the same report for Google organic search traffic to the news section of Independent.ie, we see a similar graph to what we observed in the test above.

This finding now adds further confirmation to our assumptions that AMP did not offer more organic traffic than non-AMP.

It merely split the traffic between properties.

However, as mentioned above, measuring traffic alone does not paint the full picture.

We were also interested in measuring performance in Top stories and Google Discover as both of these referrers were believed to benefit from AMP only. The belief was that we needed AMP for these channels.

How to Measure Mobile Top Stories Performance on Google

To measure mobile Top stories performance in terms of traffic (clicks), we used the following Google Search Console performance report settings.

1. Select a device filter to “Mobile”

2. Select a search appearance as “rich results”

Note that the Google Search Console performance report filter ‘Rich Results‘ can give insights into the search performance of Top stories.

However, it is important to bear in mind that the data is sampled and averaged over a period of time.

Arguably, for a more accurate Top stories ranking visibility comparison you could look at specific Top stories rankings tools such as NewzDash or NewsDashboard.

If we were to run the test again, we would possibly explore such tools.

Here’s why.

Mobile-First Indexing Impact on Top Stories Performance

The visibility of news topics we cover has remained unchanged. However, with non-AMP only, our performance for “news” keywords has declined since decommissioning AMP.

See the screenshot below for our position over time on mobile (orange line) for “news” keywords that article detail pages are triggered on.

We have gone from ranking on average #2-3 on mobile to an average position of 4-5.

Let’s take a desktop result for the query [irish news] we see Independent.ie, for example.

If we inspect the page in incognito and switch user agent to mobile, Independent.ie is not found in the Top stories carousel for the exact same query.

As you can see, all publishers are now AMP only. Independent.ie, now fully non-AMP, has been swapped out with another AMP alternative.

For various other keyword variations checked outside of “news” the SERP is identical on desktop and mobile. This is because we are the only major publisher in Ireland to have decommissioned AMP.

This may suggest that Google’s eligibility may only apply to news-type queries in Top stories.

However, there is a caveat. Independent.ie has not switched to mobile-first indexing.

On mobile in Top stories, for fresh queries, Google is using our canonical and not our mobile alternate.

However, for older news (1-2 hours+), Google has no issue discovering and swapping to the mobile alternative.

What this may mean is that it is not so much an AMP issue as it may be a mobile-first index issue, with Google still crawling Independent.ie desktop first.

In the future, it would be interesting to test AMP versus non-AMP on a responsive website to see if similar results were found.

Can Non-AMP Articles Perform in Top Stories?

Above, you see clicks for Life section content on Independent.ie from November 2020 to February 2021.

Similar to what we observed with traffic, impressions and clicks are relative to seasonality. Again, it is difficult to measure cause and effect on a news site.

A most interesting finding is that the average position on Google’s mobile Top stories remained relatively the same at an average position of 2-3, despite the life section of Indepedent.ie only having non-AMP.

Also, the Life section received clicks from Top stories.

This finding suggests to us two things:

1. We observed no noticeable impact on our Top stories traffic from not using AMP.
2. Non-AMP content can rank on mobile Top stories on Google.

Can Non-AMP Perform in Google Discover?

When we plotted out Google Discover clicks over time, we saw the same sort of fluctuations in both November and January as we did with search traffic.

Despite the anecdotal assumption that AMP is required for Google Discover, we found that non-AMP in the testing period of February 2021 for Independent.ie received 6.9% additional clicks than in February 2020, when they were AMP.

In short, we saw no impact from removing AMP on Google Discover performance.

How does non-AMP compare to AMP with Core Web Vitals performance?

As documented in the Core Web Vitals FAQs, we can expect

“…a signal for ranking will apply only to mobile Search”.

Google Search Console groups page types from CrUX report data.

Therefore, we felt Google Search Console would be the best source to measure this.

We set up independent.ie/life/ as a new property in its Google Search Console, allowing us to isolate metrics to this section only.

As you can see below, there is a dramatic dropoff at that point of “good” URLs. This represents the period in which AMP was switched off in this section,  on the first day of February 2021.

What was most interesting about this test was that both the article detail page templates on m.independent.ie/life/.* and amp.independent.ie/life/.* used the exact same code.

The only difference was the AMP cache.

Although AMP code is considered a “stripped down” version of the page, the real benefit publishers get is the AMP cache.

See our Lighthouse tests for AMP html without the AMP cache:

And AMP html with the AMP cache (“https://www.google.com/amp/s/”):

The results of the above test indicate that Google may be using the version of the AMP cache in its Page Experience update.

Is it possible that although AMP is no longer a requirement for eligibility in Top stories, it scores perfectly with Core Web Vitals?

This would mean that all AMPs may have a ranking benefit with page experience by default.

See this AMP cache example for Search Engine Journal:

How Will Non-AMP Compete with the Speed of AMP?

AMP was developed with page experience in mind, but that is not why all AMP versions perfectly with Core Web Vitals.

The reason AMP has the advantage over the average mobile site is because of the AMP cache.

Best practices for Core Web Vitals such as image optimization, fast server-response times, and effective font-loading can’t be implemented client-side.

AMPs that appear on Google Search or Bing are served from the AMP cache. Google’s servers perform all of these optimizations for your site.

In short, non-AMP content will require “hand-tuning” for mobile loading speed, responsiveness, and content stability.

How Will the Page Experience Update Impact Top Stories?

According to the Core Web Vitals FAQs,

“…all web pages irrespective of their page experience status or Core Web Vitals score are eligible for Top Stories carousel.”

Compliance with Google News content policies will be the only requirement.

I believe that when Google rolled out its AMP Top Stories in 2015, it was very much a “carrot and a stick” for publishers.

To entice publishers in, they offered more exclusive mobile real estate for news on mobile search.

Back then, it was marketed as Google’s solution to “What could Google do for news?” and set out to help publishers address the change in user behavior to mobile.

It was meant to help them provide a better mobile web experience.

To give credit to Google where it is due, back then some news publishing mobile experiences took an average of 19 seconds to load.

From my perspective, what AMP really did was provide the solution to, “What could Google do for users?”

On the publisher side, there were a lot of development and functionality sacrifices, error handling, cost and ad revenue losses for many.

Outside of the eligibility documentation for Top stories, it’s important to note that Google has not come out publicly and stated that having AMP enabled on your site is a ranking signal.

They have always marketed it as a “framework site owners can use to build performant web pages to help improve page experience.”

But as we learned in the experiments above, it is now possible to rank in Top stories without it.

Is AMP Really Worth It Then?

Having AMP can be a shortcut to achieving speed and performance.

We observed this when AMP was implemented on Independent.ie, as overall traffic from mobile search increased by ~55%.

This was potentially due to the first-mover advantage of additional traffic from the AMP Top stories carousel, and also because at that time, we had a terribly slow mobile site.

But it’s not all positive AMP experiences within our group.

When we launched AMP on one of our regional title’s belfasttelegraph.co.uk in 2019, for example, we saw no measurable traffic impact but a disappointing 49% decline in mobile digital ad revenue.

This was partially due to AMP ads restrictions and errors.

Our titles are not in isolation. Research provided by Chartbeat together with the Daily Beast shows that only 1 in 3 publishers analyzed in that study could see clear statistical evidence of a traffic increase as a result of AMP.

Will Publishers Drop AMP When the Page Experience Update Rolls Out?

What it takes to succeed in news SEO with Top stories hasn’t changed — you still need a great mobile experience.

Publishers that integrate valid AMP can “ensure highly performant web pages.” And as we learned above, non-AMP news articles can also rank in Top stories. However, publishers that do not adopt AMP will be at a clear disadvantage if they ignore their poor mobile page experience.

Some publishers will need to invest heavily in “hand-tuning” their sites if they are to compete with AMP in Top stories in terms of page experience going forward.

The good news in terms of ranking in Top stories is that beginning in mid-June 2021, the Top stories carousel feature on Google Search will “officially” be updated to include all news content.

My prediction is that we will not see too much shift in Top stories, as most publishers have adopted AMP.

What we will see are more and more top publishers decommissioning AMP.

More Resources:

---

Image Credits

Featured image and all screenshots taken by author, April 2021