Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Crawl not finding items

Benjamin | Last updated: Jan 09, 2019 11:39PM UTC

I have a directory with a known file, e.g. https://website.com/folder/script.js. This appears as a grey item from the passive crawl in the Site Map under the Target tab, and the Response tab under Content is blank (because the resource hasn't actually been requested yet). If I directly browse to the file, it loads normally, appears in Proxy tab, however under Site Map it's still greyed out with no response. If I run a crawl scan set to "most complete" it also doesn't appear to request the file or update the Site Map.

Burp User | Last updated: Jan 09, 2019 11:40PM UTC

I also have the filter set to "show all".

PortSwigger Agent | Last updated: Jan 10, 2019 08:54AM UTC

The crawler doesn't load JavaScript; the current version is not able to process JavaScript files, although this is a feature that will be added in future. Script files will be loaded as part of JavaScript analysis in the audit phase, although this doesn't update the site map. I'm not sure why the file has remained greyed out after requesting it through Proxy. One thing to check: in the dashboard, is the "Live passive crawl from Proxy" task paused? If so, once you resume it, the request should get filled in correctly.

Burp User | Last updated: Sep 10, 2019 04:56PM UTC

I can't seem to get the burpsuite to crawl anything. It seems to just crawl the page I entered and that's it. It's pretty useless. I've tried every different way to configure it. it still refuses to visit the sitemap that it already has. Come on portswigger. This is a bug that badly needs to be fixed.

Mike, PortSwigger Agent | Last updated: Sep 11, 2019 09:48AM UTC

Hi Paul, can you provide more details about the application you are trying to crawl? like Paul Johnston previously mentioned we don't support JavaScript-based web applications so this isn't something that we can crawl at the moment but are working to support this in the future.

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.