Why does Burp Crawler make 4 requests to every endpoint?
I'm crawling a site and have noticed that the crawler will make 4 identical requests to every endpoint. What is the purpose of this? It's an authenticated crawl and it seems to only test one URL (4 times) before logging in again (4 times) and then following each redirect and testing the same pages over and over and over again. Looking through the logs, the crawler has the the exact same GET request to the exact same endpoint over 70 times during the crawl. How is that efficient? I have 2 sets of credentials, so the max amount I would guess that should be made to a particular endpoint (during a crawl) would be 3 (once for unauthenticated, and once each for each credential). I'm not sure if this is a bug or what.
The crawler employs multiple crawler “agents” to parallelize its work. Each agent represents a distinct user of the application navigating around with their own browser. You can read more about this crawling technique in our online documentation:
Have you tried using the “Crawl strategy – fastest” setting?