Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Very long crawl times

Jorik Heijstek Aug 19, 2019 09:05AM UTC

Hi,

Since Burp 2.0 is out of beta I started using it today for a project I'm working on. While crawling a website, I noticed it took a very long time to crawl the website.
So I decided to run a test on demo.testfire.net (IBM testsite for Appscan) with Burp 2.1.03 and 1.7.37. The old version performed a crawl under 2 minutes, while the new version with crawl limit - 60 minutes took 20 minutes. I also tested crawl strategy - fastest, which also took significantly more time than the old version.

While I understand the need for a new crawler, the increased crawl times make it very difficult to scan large websites, as these kind of projects often have a limited run time.
Is there any way this can be mitigated?

Kind regards


Liam Tai-Hogan Aug 19, 2019 02:27PM UTC Support Center agent

Thanks for your feedback Jorik. We’ve done some testing and we haven’t been able to replicate this crawl time. On the fastest setting, it took Burp 2 three minutes to perform the crawl. It crawled 160 locations and sent 687 requests.

Would it be possible to turn on the logging for the crawler and send us the results? (Crawl > Scan configuration > New > Crawl Optimization > Crawl strategy options > Enable logging).


Jorik Heijstek Aug 20, 2019 09:47AM UTC
Hi,

Will do that. I've already generated a log file for demo.testfire.net and I'm working an a log file for the project I'm working on. How can I send these log files to you?

Kind regards

Liam Tai-Hogan Aug 20, 2019 11:42AM UTC Support Center agent

You can send the files to support@portswigger.net. Thanks.


Post Your public answer

Your name
Your email address
Answer