You may want to control how fast Sitebulb is able to crawl, and you could only do this before with chrome Crawler by changing the instances of Chrome, but we have now added a way to do this more efficiently.
When setting up a new project or crawl navigate to the "Crawl Settings" Tab.
It should take you to this page, here can select "Limit URL Speed" and then Select "Max HTML URLs per Second" will significantly slow down or speed up the crawl speed depending on what settings you choose.
Limit URL Speed: yes
Instances of Chrome: 1
Max HTML URLs Per Second: 1
You can see with the above settings applied the Avg Speed is around 0.6 URLs per second. and in this 1 min clip, it managed to crawl 28 URLs. This is significantly slower than a normal everyday crawl and really helpful if you need to crawl a site at a slower rate for instance if you're getting a 429 Too Many Requests response code.
Limit URL Speed: yes
Instances of Chrome: 8
Max HTML URLs Per Second: 10
As you can see with the crawl settings set to a higher amount the crawl is so much quicker the speed of URLs per second is between 7-11. In this 1 min clip, it managed to crawl 194 URLs. this is significantly more URLs that the Results above.
