Use Grep and cURL to Find Keywords in Web Responses

Introduction

In penetration testing and web application analysis, tools like Burp Suite Intruder are often used to send a large number of requests to a target. A key feature in this process is "Grep - Match," which allows you to automatically flag responses containing specific keywords (e.g., "Welcome", "Login successful", "admin_access_granted"). This helps you quickly sift through thousands of responses to find interesting ones.

In this lab, you will learn how to replicate this powerful "Grep - Match" functionality using standard Linux command-line tools. You will use curl to make HTTP requests, a shell loop to automate the process for a list of payloads, and grep to search for a success keyword in the responses. This is a fundamental skill for anyone interested in security automation and scripting.

In Intruder > Options, Scroll to the 'Grep - Match' Section

In a GUI tool like Burp Suite, this is the step where you would navigate to the options panel to configure your matching rules. In our command-line environment, the equivalent action is to prepare our necessary components: the list of payloads and the target URL.

The setup process has already created a file named payloads.txt in your ~/project directory. This file contains a list of potential values we want to test against the web application, serving the same purpose as an Intruder payload list.

Let's view the contents of this file.

cat ~/project/payloads.txt

You should see the following output:

orange
banana
apple
grape
mango

Our target application is a simple web server running on http://localhost:8000. We will test each payload against this target.

Clear the Default Rules

In Burp Suite, you would typically clear any pre-existing or default rules to ensure you are only matching what you explicitly define. For our simulation, this means establishing a baseline by making a single request with a known "failing" payload. This helps us understand what a normal, unsuccessful response looks like.

Let's use curl to make a request to the server with the payload orange, which we know is not the correct one.

curl "http://localhost:8000/?item=orange"

The command will fetch the web page and print its HTML source to the terminal. The output should look like this:

<!DOCTYPE html>
<html>
  <head>
    <title>Test Page</title>
  </head>
  <body>
    <script>
      const params = new URLSearchParams(window.location.search);
      if (params.get("item") === "apple") {
        document.body.innerHTML = "<h1>Welcome to the secret page!</h1>";
      } else {
        document.body.innerHTML = "<h1>Item not found.</h1>";
      }
    </script>
  </body>
</html>

Notice the JavaScript logic and the resulting "Item not found" text. This is our baseline "failure" response.

Add a New Rule to Match a Success Keyword (e.g., 'Welcome', 'Logged in')

This is the core of the "Grep - Match" technique. We need to define a unique string or pattern that will only appear in a "successful" response. Looking at the HTML from the previous step, we can see that a successful request (where item=apple) will result in the text "Welcome to the secret page!". The keyword "Welcome" is a perfect candidate for our rule.

We can use the grep command to filter the output of curl and check for the existence of our keyword. The -s flag for curl is used to run it in silent mode, which hides the progress meter.

First, let's test this rule against a failing payload. This command should produce no output, because the word "Welcome" is not in the response.

curl -s "http://localhost:8000/?item=orange" | grep "Welcome"

Now, let's test it against the known successful payload, apple.

curl -s "http://localhost:8000/?item=apple" | grep "Welcome"

This time, grep finds a match and prints the line containing the keyword:

            document.body.innerHTML = '<h1>Welcome to the secret page!</h1>';

The pipe (|) and grep "Welcome" command together form our "Grep - Match" rule. We have successfully created a way to distinguish a successful response from a failed one.

Run an Intruder Attack

An "Intruder attack" is simply the process of automating our request for every payload in our list. We can achieve this with a simple shell loop. The loop will read each line from payloads.txt, run our curl | grep command for it, and show us the results.

To make the output cleaner and more like a results table, we will create a small shell script. Use the nano editor to create a new file named attack.sh.

nano attack.sh

Now, copy and paste the following script into the nano editor:

#!/bin/bash

echo "Payload,Found"

for payload in $(cat ~/project/payloads.txt); do
  response=$(curl -s "http://localhost:8000/?item=$payload")
  if echo "$response" | grep -q "Welcome"; then
    echo "$payload,True"
  else
    echo "$payload,False"
  fi
done

This script first prints a header (Payload,Found). Then, it loops through each payload. Inside the loop, it uses grep -q (quiet mode) to check for the "Welcome" keyword. If found, it prints the payload followed by True; otherwise, it prints False.

Press Ctrl+X, then Y, and Enter to save the file and exit nano.

Next, make the script executable:

chmod +x attack.sh

Finally, run the attack:

./attack.sh

Observe the New Column in the Results Table for Your Match

After running the script in the previous step, you will see a clean, table-like output in your terminal.

Payload,Found
orange,False
banana,False
apple,True
grape,False
mango,False

This output is the command-line equivalent of the results table in Burp Suite Intruder. The first column lists the payload that was tested, and the second column, which we named "Found", acts as our "Grep - Match" indicator.

By simply scanning this second column for the value True, you can immediately identify the successful payload. As you can see, the line apple,True clearly stands out, telling us that the payload apple resulted in a response containing our target keyword. This demonstrates how effective the technique is for finding needles in a haystack of HTTP responses.

Summary

In this lab, you successfully simulated the "Grep - Match" feature, a cornerstone of many automated web security tools. You learned how to combine several fundamental Linux command-line tools to achieve a powerful result.

You practiced:

Using curl to make web requests from the terminal.
Using grep to search for specific keywords within the response data.
Writing a shell for loop to automate a task for a list of inputs.
Combining these tools into a single script to systematically test payloads and identify successful attempts.

This command-line approach is not only a great way to understand how sophisticated tools work under the hood but is also an extremely flexible and powerful skill for your own custom automation and security scripting tasks.

Use Grep-Match to Find Keywords in Intruder Responses