Chrome Timed Out

Sitebulb is able to crawl and render JavaScript (like Google does), and you can select for Sitebulb to do this using the Chrome Crawler - which is run by a version of headless Chromium.

On certain websites, Sitebulb's default configuration for Chromium will not allow you to crawl the website, and you may see errors like this on the pre-audit:

This example reads 'Chrome Failed: Chrome Navigation Error' but you may encounter other instances of 'Chrome Failed' with different messages.

Following the steps on this article will help you find and solve Chrome crawling issues yourself - if you find yourself stuck however, please do not hesitate to contact our support team.

Test in the Single Page Analysis tool

You can diagnose the issue by checking how Chrome is running in Sitebulb in a different place - the Single Page Analysis tool. This will be a lot quicker than trying different project settings on new audits all the time.

The Single Page Analysis tool is located in the top navigation menu - head here and try the URL that is failing:

If it works, you'll see Sitebulb will have collected a bunch of data about the URL. If it does not work, you'll see a message like this:

An error at this stage is expected, since we already know that Sitebulb is having problems trying to crawl the website.

Open Advanced Settings

Click the little cog on the right hand side to open up the Advanced Settings. This is quite a long pane so there is a scroll bar on the right. Initially we will look at the top items:

• Device - switching from Mobile to Desktop (or vice versa) can yield different results, so this is worth testing.
• User Agent - this is probably the most common 'quick fix'; certain website servers reject particular user-agent strings, so simply changing it can allow a site to instantly become crawlable. It is worth trying a few different values for the User Agent.
• Render Timeout - it is very rare that the render timeout will impact whether or not a site can be crawled AT ALL, but if you want to rule this out you can increase the render timeout to something massive like 20 seconds (read more about the render timeout here)

These are the first things to try - literally change one of the elements, then press the green Check button at the top after each one you change.

If if does start working, you should have found your solution. So let's say that the User Agent needed changing to 'iPhone', you would the need to go back into your project settings and update the User Agent there.

More Advanced Settings

Assuming it is still not working, open up the Advanced Settings again, and this time scroll down a little on the right:

There are a number of checkbox options here, with each one, toggle the option on/off and then press the green Check button at the top after each one you change:

• Flatten Shadow DOM - By default Sitebulb will flatten the shadow DOM, so you can uncheck this option to disable Sitebulb flattening the DOM.
• Flatten Iframes - By default Sitebulb will flatten the shadow Iframes, so you can uncheck this option to disable Sitebulb flattening Iframes.
• Enable Cookies - Although most sites don't, certain sites require cookies to be saved in order to crawl them, so you can switch this setting on by checking the box.
• Incognito (Session Isolation) - Whether Incognito is selected should not impact whether or not Sitebulb is able to crawl a website, so you can probably ignore this option.
• Enable Service Workers - Although most sites don't, certain sites render content and links via service workers, so you can switch this setting on by checking the box.

In truth, it is very difficult to ascertain what might be the problem until you have experimented a little bit, so this process needs a bit of iterative work to step through each potential issue.

If if does start working, you should have found your solution. So let's say that you need to disable 'flatten shadow DOM' in order for the Single Page Analysis to yield results, you would the need to go back into your project settings and update the Chrome Crawler settings there.

If it is still not working, then please contact support and we will figure out how to resolve the situation.

Test in a project

Once you have worked out which settings need to be tweaked, you'll need to go and set up a new project in Sitebulb with your amended settings.

Depending on which setting(s) you needed to change in order to get the Single Page Analysis tool to work, you may need to adjust settings in a couple of different places.

The 'New Project' screen allows you to adjust the Device, User Agent and Cookies (Yes/No) - but to see these options you need to hit Advanced Settings in the bottom right

The remaining Chrome settings are available once Sitebulb has performed the pre-audit checks (which happens after you hit Save and Continue on the New Project page).

You first need to navigate to the Crawler Settings, from the left hand menu:

Then scroll down to find the Advanced Chrome Crawler Settings, which include all the toggles for Flatten Shadow DOM, Flatten Iframes and Enable Service Workers.

Again, you want to replicate the settings that worked for you on the Single Page Analysis tool.

Then, go ahead and see if Sitebulb will crawl the site (fingers crossed!).

If all else fails...

If you still can't get anywhere, it may be worth establishing if there is a broader problem with the Chrome Crawler on your machine (e.g. anti-virus blocking Chromium), please check our other troubleshooting guide for this.

But if you've tried everything listed above, but Sitebulb STILL will not crawl properly, it is probably something we have never seen before. In which we'll need to work with you to get to the bottom of the issue (which we will!).

Please email [email protected] and provide the following information:

• The website you are trying to audit
• Screenshots of the error(s) you are seeing
• Your Sitebulb log files (find out how to get these from here)

We'll look into it and figure out what we need to do to make it work!