Dealing with Indexability issues?

Crawl your website with Sitebulb for 300+ tech SEO checks

Try for Free
Medium This Hint is worth investigating further, and may warrant further attention depending on the type and quantity of URLs affected. Potential Issue This Hint is unlikely to be affecting the site at the moment, but should be investigated as it could cause issues in the future.

Nofollow in HTML and HTTP header

This means that the URL in question has a nofollow directive in both the HTML and in the HTTP header.

Why is this important?

It is considered best practice to only specify robots directives once on any given URL. This is because doing it multiple times makes the configuration more open to human error.

In this scenario, you have a nofollow specified in the X-Robots-Tag in the HTTP header, and also a meta nofollow in the HTML <head>.

At this point, there would be nothing 'wrong', as all the robots directives are in agreement. However, if in the future you wanted to change the page from nofollow to follow, and went into your page template or plugin configuration to change the meta nofollow, there exists the potential that you may completely forget to also change the HTTP header.

The net result would be that the page would remain 'nofollow', even though you thought you'd changed it to not be. Google clearly states that if you have multiple directives which conflict, they will select the option that is most restrictive (and it is likely that the other search engines follow suit).

You can avoid such catastrophic futures by only specifying robots directives in one location.

What does the Hint check?

This Hint will trigger for any internal URL which contains nofollow directives in both the HTML and in the HTTP header.

Examples that trigger this Hint

The Hint would trigger for any URL that had both of the following;

Meta nofollow in the <head>,

<!doctype html>
<html lang="en">
<head>
<title>example</title>
<meta name="robots" content="noindex,nofollow">
...
</head>
<body>...</body>
</html>

AND in the HTTP header:

HTTP/... 200 OK
...
X-Robots-Tag: nofollow

Why is this Hint marked 'Potential Issue'?

This Hint is a 'Potential Issue', which means that it is unlikely to be affecting the site at the moment, but should be investigated as it could cause issues in the future.

Robots directives implemented multiple times is usually not deliberate, so this is flagged in Sitebulb so you can remove the potential for future damage. To do this you may need developer help, as you will need to adjust page templates, plugins or HTTP headers - removing the duplication so that robots directives are only defined once.

Sitebulb Desktop

Ideal for SEO professionals, consultants and digital marketing agencies.

Try our fully featured 14 day trial. No credit card required.

Try Sitebulb for Free

Sitebulb Cloud

Perfect for collaboration, remote teams and extreme scale.

Cloud crawling with no project limits and very generous crawl limits.

Explore Sitebulb Cloud