Authentication for crawling staging sites

Some websites utilize authentication, meaning that you need to login using a username/password in order to access the page content. Most often you will encounter this when dealing with dev sites or staging servers.

There are two different ways which this can be implemented:

1. HTTP authentication (the server will 'challenge' a client request and prompt the client to provide authentication information, so you don't even get to access the page until you provide authentication).
2. Forms-based authentication (the page will respond with a 200 status and you can access the page, but will include an embedded form which requires authenticating before you can access the full page content).

Sitebulb allows authentication via either of these options, and both are found via the 'Crawler Settings' options on the left-hand menu in the audit setup:

How to crawl sites with HTTP authentication

If you are trying to crawl a site with HTTP authentication, you will actually be required to enter your login credentials at the stage where Sitebulb does the domain check, when setting up the Project.

The start URL will be marked 'Requires Auth', and you will need to 'Save and Continue' by hitting the green button:

You will then be challenged to enter the HTTP Authentication credentials by a pop-up. Enter the HTTP username and password, then hit 'Try Again'. Assuming you have entered them correctly, Sitebulb will progress forward to the normal audit setup page.

You can also check/add/adjust HTTP authentication credentials via the Crawler Settings.

Enable the HTTP Authentication checkbox, then enter your HTTP Username and HTTP Password. When you start your audit Sitebulb will use these details to log in to the website.

How to crawl sites with forms-based authentication

For forms-based authentication, you actually do not add the authentication credentials via the Advanced Settings. This is because the Start URL allows you to continue (e.g. a 200 status), so Sitebulb does not 'know' there is a form involved - you need to tell Sitebulb that this is the case.

Start a new project, and click the Show Advanced Settings button;

This will show some additional options on the left, and you want to select 'Yes' from the Forms Authentication dropdown:

This will open a small browser window taking you to the site you are going to be crawling.

Navigate to your login form, if you need to change the URL of where your login form is you can change the URL at the top of the window. Enter your site's login details. Once you have successfully logged in click the Add Authentication button at the bottom of the page.