Crawling subdomains

By default, Sitebulb will not crawl subdomain URLs it encounters, and will effectively treat them as external URLs. You can control how Sitebulb treats subdomain URLs via the Subdomain options from the left hand menu of the audit setup.

Subdomain options

The default option is for the top box to be ticked, 'Check Subdomain Link Status' - this means that subdomain URLs will be treated as external URLs, and Sitebulb will run a HTTP status check on any it encounters.

Subdomain default option

This allows you to find any broken or redirected subdomain links. If you untick this box, Sitebulb will not run the status check on subdomains links.

How to crawl all subdomain URLs

The second option is a bit more involved, 'Treat Subdomains as Internal' - this means that any subdomain URLs that Sitebulb encounters will be treated as internal URLs. On other words, ticking this box is 'how to crawl all subdomain URLs on a website'. Sitebulb will run all the normal 'internal checks' on these URLs, including internal link calculations.

When you do tick the box, the top tickbox will grey out (because it is now redundant) and you will be faced with the option of explicitly adding the subdomains, one per line in the box underneath.

If you tick the box and don't add any subdomains into the box, ALL subdomain URLs will be crawled. However if you do add some subdomains to the box, ONLY those subdomains will be crawled.

Simply enter the subdomains, one per line. In the example below, Sitebulb would crawl https://blog.example.com and https://support.example.com and treat those subdomains as internal (but NOT https://community.example.com or any other subdomain).

Treat subdomains as internal

When you come to look at the completed audit results, you will see that URLs from subdomains will now appear in the 'Internal URLs' lists, and they will have URL Rank (UR) scores.

Subdomains in internal