Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Utterly unclear on the purpose of spider

Spiderman's | Last updated: Apr 14, 2015 03:21PM UTC

My impression is that spider expands the sitemap as it crawls, aided by its form submission abilities, etc. But after I spider my entire host, I notice that manual active scanning the entire host does not make a difference in the number of URLs populating the scan queue, compared to just actively scanning the host alone without spidering first. So does spidering not increase the number of available URLs to scan? Or is spidering instead meant to potentially increase the number of vulnerabilities per URL being scanned? Definitely hoping someone can set me straight here. I'm just not sure when to spider, or what Burp functions that spidering is supposed to help complement.

PortSwigger Agent | Last updated: Apr 15, 2015 08:20AM UTC

Running the Spider will fetch all the in-scope items in the site map, identify any newly discovered content by analyzing the responses, add those items to the site map, and recurse. So typically, spidering will increase the amount of content in the site map. However, if you have already browsed around (part of) the application, Burp will have passively populated the site map with content discovered by analyzing responses. So, depending on the nature of the application and the amount of manual browsing you have done, the Spider might just request the unrequested items and not identify any/many new items from the resulting responses. In this situation, running the Spider will fill out the unrequested items in the site map without increasing the amount of items in it. When you send a branch of the site map to the Scanner, the Scanner will process all the selected items, including those that are not yet requested. Hence, in some situations, sending to the Scanner might result in the same number of scan queue items, regardless of whether you have previously run the Spider.

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.