Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Spidering - avoid getting all the products from store

Ori | Last updated: Jul 06, 2015 03:26PM UTC

Hi there, I've been trying to spider a site and adding the results to the scope. The problem I'm facing is if we want to spider a store with a catalog of, for example, 10k items, it will try to crawl all those items (the URLs are different and no params are specified). Is there any configuration I am missing to avoid getting all the items crawled? If Burp does not have it yet, is it planned to do it so? Cheers, Ori

PortSwigger Agent | Last updated: Jul 07, 2015 08:15AM UTC

This is a common problem with automated crawling of applications that use an "infinite" number of URLs with parameter data within the URL file path rather than conventional parameters. One possible solution would be to configure a lower maximum link depth at Spider / Options / Crawler settings. This may help limit the explosion of apparent content as Burp crawls the store. An alternative would be to mark the problematic paths of the app as out of scope, so that they aren't automatically crawled, and then perform some manual crawling via your browser to ensure that all relevant content and functionality has been covered, without visiting every distinct product.

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.