JSON Data Processing with jq

Introduction

Welcome to the Linux jq Programming Lab! In this lab, you'll learn how to use jq, a lightweight and flexible command-line JSON processor. jq is like sed for JSON data - you can use it to slice, filter, map, and transform structured data. This lab is designed to take you from basic to advanced usage of jq, with practical examples that you can apply in real-world scenarios, such as processing JSON data from APIs or configuration files.

Imagine you're a traveler in China, and you're using a travel app that provides information about various attractions, including their locations, opening hours, and reviews. The app's backend stores this data in JSON format, and you're tasked with extracting specific information to plan your trip efficiently. This lab will guide you through using jq to query and manipulate this JSON data, making it easier for you to find the perfect attractions to visit.

Skills Graph

Basic JSON Querying

Learn how to extract simple data from a JSON object.

Suppose you have the following JSON data named 'data.txt' in the 'home/labex/project/' representing a list of attractions:

[
  {
    "name": "The Great Wall of China",
    "location": "Shanxi Province",
    "opening_hours": "24 hours"
  },
  {
    "name": "Terracotta Warriors",
    "location": "XiAn",
    "opening_hours": "9:00 AM -  5:00 PM"
  }
]

Task: Extract the names of all attractions.

Solution:

cat ~/project/data.txt | jq '.[] | .name'

Example:

"The Great Wall of China"
"Terracotta Warriors"

Explanation:

• cat ~/project/data.txt reads the contents of the file.
• | pipes the output of the echo command to jq.
• jq '.[] | .name' tells jq to iterate over each object in the array (.[]) and extract the name field.

Filtering JSON Data

Learn how to filter JSON data based on specific criteria.

Task: Find attractions that are open 24 hours.

Solution:

cat ~/project/data.txt | jq '.[] | select(.opening_hours == "24 hours") | .name'

Example:

"The Great Wall of China"

Explanation:

• select(.opening_hours == "24 hours") filters the objects to only those where the opening_hours field is exactly "24 hours".
• The rest of the command is the same as in Step 1, extracting the name field of the filtered objects.

Transforming JSON Data

Learn how to transform JSON data into a different format.

Task: Convert the opening hours to a more readable format.

Solution:

cat ~/project/data.txt | jq '.[] | {name: .name, location: .location, opening_hours: (.opening_hours | if . == "24 hours" then "Open 24 hours" else "Open \(.)" end)}'

Example:

{
  "name": "The Great Wall of China",
  "location": "Shanxi Province",
  "opening_hours": "Open 24 hours"
}
{
  "name": "Terracotta Warriors",
  "location": "XiAn",
  "opening_hours": "Open 9:00 AM - 5:00 PM"
}

Explanation:

• {name: .name, location: .location, opening_hours: ...} constructs a new JSON object with the specified fields.
• if .opening_hours == "24 hours" then "Open 24 hours" else "Open " + .opening_hours end checks if the opening_hours field is "24 hours" and transforms it to "Open 24 hours"; otherwise, it prepends "Open " to the original opening_hours.

Summary

Congratulations! You've completed the Linux jq Programming Lab. You've learned how to query, filter, and transform JSON data using jq, a powerful tool for working with structured data in the command line. Whether you're processing data from APIs, configuration files, or any other JSON source, jq can help you extract, filter, and manipulate the data you need, making your tasks more efficient and your code cleaner.

Remember, practice is key to mastering jq and other command-line tools. Try applying these concepts to your own JSON data and experiment with different queries and transformations. Happy coding!