I was recently asked the following question about Sitebulb on Twitter:
I tried and failed to write my answer in 140 characters, so I gave up and went all in with a full scale blog post.
TL;DR - it's actually not a 'crawler.' It's a website auditor.
First things first, Sitebulb is a desktop product (like Screaming Frog), and not a cloud/SaaS product (like DeepCrawl). This means that you install it on your own computer (Windows or Mac) and it runs using your own local resources (CPU/RAM).
This is massively beneficial. Since you don't need to wait 10 minutes for a cloud instance to spin up, Sitebulb can begin the crawl and start showing you data immediately.
Desktop also means that it does not need to be prohibitively expensive, like all cloud crawlers inevitably end up.
It might not be intuitively obvious why this happens, so let me explain...
We had a goal in mind when we built our crawler, we really wanted to help small agencies and consultants to do amazing technical SEO work, at scale, without it costing hundreds or thousands of dollars a month.
We actually spent 3 months building the product as a cloud crawler, before realising there would be no way to keep the product reasonably priced.
The reason cloud crawlers are expensive is because crawling websites produces a LOT of data. Sitebulb stores over 300 pieces of data for any one URL. Say you've crawled a site with 100,000 URLs... you're talking 30 million fields of data. And that's just for URL data, for a site with 100 links on each page, you could have another 10 million links, all of which needs storing - perhaps indefinitely - on your cloud architecture. That's a lot of bytes right there.
So any pricing package for a cloud crawler needs to cover both the potential bandwidth you will use crawling, but also the potential storage. Notice that this isn't actual storage, but potential storage, just in case you need to use it.
We realised that with a cloud setup we could not satisfy our goal of producing software that could work at scale, without costing a fortune.
So we had to slightly redefine our goal:
-> "To help small agencies and consultants do amazing technical SEO work, at scale, using desktop software."
And by scale, we mean this sort of thing:
Ok. It's 2020. Every SEO worth their salt knows what a crawler is. Every SEO worth their salt already pays for crawler software, often from multiple providers.
We knew this when we set out building Sitebulb, so we knew we couldn't build 'just another crawler'.
We knew we had to build more than just a crawler. We felt that, in order to compete in this market, we had to build something that would deliver answers rather than questions, would deliver opportunities rather than problems, and would deliver shortcuts to vital insights. And, importantly, would give you the ability to communicate all this to your clients or bosses.
We also know that there are lots of advanced technical SEOs who love to manipulate the raw data themselves. They like nothing more than a massive CSV export or SQL file. To be honest, if this is you, you're not really our target customer.
Why? Because Sitebulb will do a lot of this manipulation for you - saving you loads of time in the process. It then presents the data in a huge variety of ways; there are data tables and graphs galore, filtered URL Lists, pre-designed bulk exports, and context specific Hints to highlight potential issues.
When you crawl a website your purpose is not simply to end up with a pile of data, it is to end up with an understanding of the issues on the website so you are able to put forward a set of recommendations that will make a positive difference.
Sitebulb makes this job easy. Using the 'Hints' system, it shows you when you have critical issues that demand your attention:
It also shows you how much of the site is affected, and wheter these are indexable or not indexable URLs. All of this data provides the additional context you need in order to make quick decisions.
Every Hint comes with an importance status - Critical, High, Medium or Low - so you can instantly see where the issues lie and where to focus your time.
It is just as useful to understand which 'issues' are low impact, and can be ignored for the time being:
Every Hint also comes with a call to action button to 'Learn more', which takes you to a dedicated page on our website that explains exactly what the issue is and how to resolve it. You can either follow along yourself or pass it straight on to your developer to fix.
Every audit comes with a set of scores out of 100. Scores deliver instant meaning and provide a framework for further investigation - quickly spot areas of weakness to dive into, then scan through the prioritized Hints to understand what is holding the site back.
Scores are calculated for every section, meaning you can really zero in on areas for improvement. There is a score specifically for SEO, so you can easily demonstrate to clients or bosses that your SEO is on the right track by logging improvements in this score.
Sitebulb heaviliy utitilizes data visualization to help you quickly spot patterns, and they can be used in client reports to help express complex ideas with clarity.
And then there are our award-winning Crawl Maps. Crawl Maps tackle that big question that has been gnawing at SEOs for years: "how do we explain site architecture to clients (or other non-SEO muggles)?"
Crawl Maps provide a fantastic, visual, intuitive way to do this, which is one of the reasons they are so popular. The other, of course, is that they help the SEO themselves to better understand a website. One quick look at a Crawl Map can tell you information it would have taken hours to extract manually.
Understanding website issues is only half the battle. The other half involves communicating with your client/boss/developer and convincing them to take action to actually implement the changes you are recommending.
Sitebulb is designed with this task in mind:
You can either put this together using your own existing templates and methodology... or you can take advantage of Sitebulb's built-in PDF reporting.
At the click of a button, produce a completely white-label PDF report that can be customised for different use cases.
So what we've actually built is a product that doesn't simply crawl your website and spit out the data for you to analyse yourself. It will do a ton of the analysis for you. It would be much more accurate to call Sitebulb a 'website auditor' than a 'crawler.'
This point really sums up Sitebulb, as one of customers picked up on:
"This is so, so good. You’ve really nailed a significant problem that used to take me hours using a smash up of Screaming Frog, DeepCrawl, custom Ruby scripts, and a ton of manual hunt and peck."
If you love getting your hands dirty with the data and building all the reports yourself, then Sitebulb is probably not for you. But if you’re like Tony, and you want to save yourself a ton of time on unnecessary processes, then Sitebulb could be exactly the solution you are looking for.
Sitebulb offers the convenience and lower price point of desktop software, and marries this with enterprise level data visualisation and analysis. It can crawl websites of practically any size and scale, and utilizes a cutting-edge Chromium browser for JavaScript rendering - literally no website is out of reach.
Sitebulb has been built with SEO expertise baked in. This is perfect for less experienced SEOs as it helps steer them in the right direction, but it also means that more advanced SEOs can spend more time focusing on the really clever stuff. Instead of wasting hours in spreadsheets trying to identify issues, Sitebulb just tells you them straight off the bat.
If this sounds like it will save you time AND make you better at your job, that's because it will.
