Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Huge project files when scanning

Tiago | Last updated: Apr 16, 2018 04:28PM UTC

Hi, I'm using Burp Pro 1.7.33 and I noticed that the scanner is generating huge amounts of project data with no apparent reason. I'm talking about 1GB per minute more or less for a single scan which has the Burp Collaborator disabled. My scan generated around 3000 requests and the project file went from 30MB to 12GB. Is there a reason for the huge disk space increase? What's curious is that if I copy the project to another file, the new file takes up just a few dozen megabytes. Is this a bug? Thanks.

Liam, PortSwigger Agent | Last updated: Apr 17, 2018 06:51AM UTC

You can often reduce your project file size by using Burp > Save copy of project. Then close Burp and open the copy of the Project. This should remove any deleted history. Is this the method you are using to reduce the size f your project? Is Burp Suite performing static code analysis? Static analysis can consume large amounts of memory and processing, and so it may be desirable to restrict static analysis to key targets of interest. This static code analysis settings can be found at Scanner > Options > Static Code Analysis.

Burp User | Last updated: Apr 17, 2018 08:51AM UTC

Hi Liam, Yes, that's what I'm doing to reduce the size of the project. The problem is that I need to do that after each scan, otherwise the file just fills in all my disk space. The static code analysis is on, but I have no problems in terms of memory or CPU, it's really just the disk space that is the problem. Could this be an issue on this particular version of Burp? Because I've been using Burp for many years now and I've never seen this behaviour before. Thanks

Liam, PortSwigger Agent | Last updated: Apr 17, 2018 09:16AM UTC

Have any changed been made to the application. It's hard to say what is causing this issue without some further investigation. Have you tried using an older version of Burp on the same application with the same settings? You can download older versions of Burp Suite from your account page on our website: - https://portswigger.net/users

Burp User | Last updated: Apr 17, 2018 10:08AM UTC

Maybe this can be useful for fingerprinting the issue. I'm using a Pre-request macro that sends out a few request before my actual request. If I turn this off, the size of the project file increases at a normal rate (almost nothing at all). So it seems like Burp is saving the macro requests and responses in the project file somehow. Is this possible? However I think that even if Burp is logging all this data, the project file is still filling too fast. The biggest response size I'm having has around 0.2 MB, I currently have around 4000 request done. A rough estimation would make it 800MB but my project file has 14GB. I haven't tried with an older version but I'll give it a shot later.

Liam, PortSwigger Agent | Last updated: Apr 17, 2018 10:17AM UTC

Thanks for the additional information Tiago. Could you provide us more details of the pre-request macro? If there is any sensitive data, you can send the requests / responses / details to support@portswigger.net.

Burp User | Last updated: Apr 17, 2018 10:39AM UTC

Sorry, Liam. Unfortunately I can't share the details of my project due to NDAs. However I did just test the same scan with the same macro on version 1.7.25 and the project file is steady at around 65MB which is the normal behaviour. It seems to be a bug introduced in a recent version of Burp. I'll try to make a few tests on a non-sensitive website to see if I can reproduce the issue and I'll keep you updated with my results.

Liam, PortSwigger Agent | Last updated: Apr 17, 2018 12:10PM UTC

Thanks for checking that out for us. If you manage to reproduce the issue and can share the details, that would be helpful in our testing.

Burp User | Last updated: May 09, 2018 09:06AM UTC

Hi Liam, Sorry for the late reply on this but I tried to replicate this using DVWA and I couldn't get the same behaviour. I'll keep an eye on this in the future, if I run across the same issue again in a different scenario, I'll let you know. Thanks for the support.

Oleksii | Last updated: Jan 30, 2023 11:23AM UTC

Faced the same problem. During the crawl stage project file grows up to 85GB (it takes only 40-70 minutes to exhaust all available space). Unfortunately, it's not possible to share the webapp we're scanning. I've made a couple of experiments, e.g.: 1. Changed 'project_options/miscellaneous/disable_logging_to_history_and_site_map' to `true`; 2. Changed 'logger/capture_enabled' to `false`; 3. And disabled any 3d party extensions from BApp Store. and it didn't affect project file size in any manner. I ran a few scans with mock webapp, playing with size of the returned pages and it showed linear dependency between page size and project file size. So I assume it unconditionally stores at least all HTTP responses disregard the settings mentioned above. Is there any settings that I've missed? Thank you.

Liam, PortSwigger Agent | Last updated: Jan 31, 2023 10:02AM UTC

Thanks for this report, Oleksii. Could you try using this scan config - Audit checks - all except JavaScript analysis - https://portswigger.net/burp/documentation/desktop/scanning/built-in-configurations. Let us know if this helps.

Oleksii | Last updated: Feb 01, 2023 11:17AM UTC

Hey Liam, I've launched the scan with JavaScript analysis disabled. It consumed 31 GB for the first 20 minutes of crawl stage.

Oleksii | Last updated: Feb 01, 2023 11:24AM UTC

I've made another interesting observation. I intercepted every outgoing HTTP request and every corresponding HTTP response with Java API and summarized total data amount received from the webapp. After it had consumed ~84 GB of disk space for the project file, it turned out that it received only 527 MB of data from the webapp.

Liam, PortSwigger Agent | Last updated: Feb 01, 2023 02:51PM UTC

Thanks for following up, Oleksii. Could you send email us (support@portswigger.net) your diagnostics information? You can find this by going to "Help > Diagnostics" within Burp.

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.