Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Crawl not finding items

Benjamin Jan 09, 2019 11:39PM UTC

I have a directory with a known file, e.g. https://website.com/folder/script.js. This appears as a grey item from the passive crawl in the Site Map under the Target tab, and the Response tab under Content is blank (because the resource hasn't actually been requested yet).

If I directly browse to the file, it loads normally, appears in Proxy tab, however under Site Map it's still greyed out with no response.

If I run a crawl scan set to "most complete" it also doesn't appear to request the file or update the Site Map.


Benjamin Jan 09, 2019 11:40PM UTC
I also have the filter set to "show all".

Paul Johnston Jan 10, 2019 09:00AM UTC Support Center agent

The crawler doesn’t load JavaScript; the current version is not able to process JavaScript files, although this is a feature that will be added in future.

Script files will be loaded as part of JavaScript analysis in the audit phase, although this doesn’t update the site map.

I’m not sure why the file has remained greyed out after requesting it through Proxy. One thing to check: in the dashboard, is the “Live passive crawl from Proxy” task paused? If so, once you resume it, the request should get filled in correctly.


Post Your public answer

Your name
Your email address
Answer