Use Grep-Extract to Scrape Data in Intruder Responses

Introduction

In this lab, you will learn how to use one of Burp Intruder's powerful features: 'Grep - Extract'. This feature allows you to define rules to extract specific pieces of text from the responses received during an Intruder attack. It is incredibly useful for harvesting valuable information that might change with each request, such as CSRF tokens, session IDs, or other dynamically generated data.

By the end of this lab, you will be able to configure Burp Intruder to automatically find and display custom data from responses, making your security testing workflow more efficient. We will simulate a scenario where we need to extract a hidden token from a web page's response.

In Intruder > Options, Scroll to the 'Grep - Extract' Section

In this step, we will begin by launching Burp Suite and sending a sample request to the Intruder tool. This will set the stage for configuring our extraction rules.

First, you need to launch Burp Suite. You can typically find it in the application menu of your desktop environment.

1. Open the Terminal from your desktop.
2. Launch Burp Suite by running the command burpsuite. Select a temporary project and click "Next", then "Start Burp".
3. Go to the Proxy > Intercept tab and ensure "Intercept is on".
4. Open the built-in Chromium browser by going to Proxy > Intercept and clicking "Open Browser".
5. In the browser, navigate to any simple website, for example, http://portswigger.net.
6. The request will be intercepted in Burp Suite. Right-click anywhere in the request text area and select Send to Intruder.
7. Now, navigate to the Intruder tab. You will see the request you just sent.
8. Within the Intruder tab, click on the Options sub-tab.
9. Scroll down the page until you find the section titled Grep - Extract. This is where we will configure our data extraction rules.

You have now successfully located the 'Grep - Extract' section and are ready to add a rule.

Add a New Extraction Rule

In this step, you will add a new, empty rule to the 'Grep - Extract' section. This rule will later be configured to define what specific data we want to pull from the server's responses.

The 'Grep - Extract' feature works by creating a list of rules. Each rule defines a pattern to look for in the response. When a response matches a rule, the specified part of the response is extracted and displayed in a new column in the Intruder results table.

To add a new rule, simply follow this instruction:

1. In the Grep - Extract section you located in the previous step, click the Add button.

After clicking Add, a new entry will appear in the rule table. This new entry will have default settings, which we will configure in the next step. The dialog for defining the extraction will also pop up, ready for your input.

Define Start and End Delimiters to Extract Data (e.g., a hidden token)

In this step, you will configure the rule you just added. We will tell Burp what to extract by defining the text that comes immediately before and after our target data. This is done using start and end delimiters.

For this lab, let's imagine we are trying to extract a CSRF token from a response that contains the following HTML snippet:

<input type="hidden" name="csrf_token" value="a1b2c3d4e5f6g7h8i9j0" />

Our goal is to extract the value a1b2c3d4e5f6g7h8i9j0.

1. After clicking "Add" in the previous step, a "Define extract grep" dialog appears. If it's not open, click on the new rule you added to open it.
2. The tool automatically fetches the response from the original request you sent to Intruder. You can see it in the response view at the bottom.
3. To define the extraction, we will specify a start and end delimiter. The most reliable way is to use the "Define start and end" option.
4. Find a unique piece of text in the response that you want to extract. For our example, let's pretend the response contains the HTML snippet above.
5. In the response view, highlight the value you want to extract (e.g., a1b2c3d4e5f6g7h8i9j0).
6. The "Start" and "End" fields will be automatically populated.
   • The Start after expression field should contain the text immediately preceding your target data. For our example, this would be name="csrf_token" value=".
   • The End at delimiter field should contain the text immediately following your target data. For our example, this would be ".
7. Click OK to save the rule.

You have now configured a rule that will search every response for text between name="csrf_token" value=" and ", and extract it.

Run an Intruder Attack

In this step, you will configure a basic payload and launch the Intruder attack. As the attack runs, Burp will apply the Grep-Extract rule to every response it receives.

First, we need to set a payload position.

1. Navigate to the Intruder > Positions sub-tab.
2. The request template is shown here. By default, Burp may have automatically added payload markers (§...§) around parameter values. For this exercise, it doesn't matter where the payload is, as we are focused on the response. You can leave the default markers or clear them and add one anywhere. For example, you could add it to a cookie value or a parameter.
3. Ensure the attack type is set to Sniper.

Next, we configure a simple payload.

1. Navigate to the Intruder > Payloads sub-tab.
2. Under Payload Sets, keep the Payload set as 1.
3. Under Payload Options, select Numbers from the dropdown list.
4. Configure it to run from 1 to 5 with a step of 1. This will send 5 requests.

Finally, start the attack.

1. Click the Start attack button in the top-right corner of the Intruder tab.
2. A new "Intruder attack" window will open, showing the results of the attack in real-time.

The attack is now running, and Burp is busy extracting data based on the rule you created.

View the Extracted Data in the New Columns of the Results Table

In this final step, you will observe the results of your Grep-Extract configuration. The data you told Burp to find is now neatly organized in the attack results window.

1. Look at the Intruder attack window that opened in the previous step.
2. You will see a table with columns like "Request", "Position", "Payload", "Status", "Length", etc.
3. Scroll the table to the right. You should see a new column. By default, it will be named Extract 1. If you added more rules, they would be named Extract 2, and so on.
4. This new column contains the data that was extracted from each response using the rule you defined.

If the website you targeted doesn't actually contain the csrf_token string we used as an example, this column might be empty. However, the column's presence confirms that your rule was active. If you had targeted a real, dynamic token on a login page, this column would now be filled with the unique tokens from each response, ready for you to analyze or use in subsequent attacks.

You can right-click the table header to add or remove columns, or click on a column header to sort the results based on that data.

Summary

In this lab, you have successfully learned how to use the 'Grep - Extract' feature in Burp Intruder. You walked through the entire process, from sending a request to Intruder, adding and configuring an extraction rule with specific delimiters, running an attack, and finally, viewing the scraped data in the results window.

This skill is essential for automating the process of harvesting dynamic data from web applications, which is a common task in penetration testing and security analysis. You can now apply this knowledge to extract CSRF tokens, session identifiers, API keys, or any other specific information from HTTP responses during your security assessments.