Import MongoDB Data

Introduction

In this lab, you will learn how to import data into MongoDB using both JSON and CSV formats. The first step covers importing JSON files, where you'll create a sample JSON file and use the mongoimport command to import it into a MongoDB database. The second step focuses on importing CSV data, guiding you through creating a CSV file and using the mongoimport command to load it into MongoDB. The lab also includes steps to check the import results, handle any errors, and verify the imported data, ensuring a comprehensive understanding of the MongoDB import process.

Skills Graph

Import JSON Files

In this step, you'll learn how to import JSON files into MongoDB using the mongoimport command. JSON (JavaScript Object Notation) is a popular data interchange format that MongoDB can easily import.

First, let's create a sample JSON file with some data. We'll use a simple dataset of books.

Open a terminal and navigate to the project directory:

cd ~/project

Create a file named books.json using nano:

nano books.json

Add the following JSON content:

[
  {
    "title": "MongoDB Basics",
    "author": "Jane Smith",
    "year": 2023
  },
  {
    "title": "Python Programming",
    "author": "John Doe",
    "year": 2022
  },
  {
    "title": "Data Science Handbook",
    "author": "Alice Johnson",
    "year": 2021
  }
]

Save the file by pressing Ctrl+X, then Y, and Enter.

Now, start the MongoDB shell:

mongosh

Create a new database and switch to it:

use library_database

Exit the MongoDB shell:

exit

Import the JSON file into the books collection:

mongoimport --db library_database --collection books --file ~/project/books.json --jsonArray

Example output:

[#######......] 2/3 documents
2023-xx-xx 00:00:00.000 [############] library_database.books 3/3 documents

Let's verify the import by using mongosh to count the documents:

mongosh library_database --eval "db.books.countDocuments()"

Example output:

3

Import CSV Data

In this step, you'll learn how to import CSV (Comma-Separated Values) files into MongoDB using the mongoimport command. CSV is a common format for storing tabular data.

First, let's create a sample CSV file with some data about library members.

Navigate to the project directory:

cd ~/project

Create a file named library_members.csv using nano:

nano library_members.csv

Add the following CSV content:

name,age,membership_type,join_date
John Doe,35,Premium,2023-01-15
Jane Smith,28,Standard,2022-11-20
Alice Johnson,42,Gold,2021-06-10
Bob Williams,55,Premium,2022-03-05

Save the file by pressing Ctrl+X, then Y, and Enter.

Start the MongoDB shell:

mongosh

Create a new database and switch to it:

use library_database

Exit the MongoDB shell:

exit

Import the CSV file into the members collection:

mongoimport --db library_database --collection members --type csv --file ~/project/library_members.csv --headerline

The --headerline flag tells MongoDB to use the first row of the CSV as field names.

Example output:

[#######......] 2/4 documents
2023-xx-xx 00:00:00.000 [############] library_database.members 4/4 documents

Let's verify the import by using mongosh to count the documents and view the first few records:

mongosh library_database --eval "db.members.countDocuments()"

Example output:

4

Now, let's view the imported documents:

mongosh library_database --eval "db.members.find().limit(2)"

Example output:

[
  {
    _id: ObjectId("..."),
    name: 'John Doe',
    age: 35,
    membership_type: 'Premium',
    join_date: '2023-01-15'
  },
  {
    _id: ObjectId("..."),
    name: 'Jane Smith',
    age: 28,
    membership_type: 'Standard',
    join_date: '2022-11-20'
  }
]

Check Import Results

In this step, you'll learn how to verify and explore the data you've imported into MongoDB. We'll use various MongoDB commands to check the imported collections and their contents.

Start the MongoDB shell:

mongosh library_database

First, let's list all collections in our database:

show collections

Example output:

books
members

Now, let's explore the data in more detail. We'll use different methods to check and query our imported collections.

Count the number of documents in each collection:

db.books.countDocuments()
db.members.countDocuments()

Example output:

3
4

Let's find specific documents using query filters:

// Find books published after 2022
db.books.find({ year: { $gt: 2022 } })

Example output:

[
  {
    _id: ObjectId("..."),
    title: 'MongoDB Basics',
    author: 'Jane Smith',
    year: 2023
  }
]

Now, let's query members with a specific membership type:

// Find Premium members
db.members.find({ membership_type: "Premium" })

Example output:

[
  {
    _id: ObjectId("..."),
    name: 'John Doe',
    age: 35,
    membership_type: 'Premium',
    join_date: '2023-01-15'
  },
  {
    _id: ObjectId("..."),
    name: 'Bob Williams',
    age: 55,
    membership_type: 'Premium',
    join_date: '2022-03-05'
  }
]

Export a collection to verify its contents:

exit

Use mongoexport to export the members collection to a JSON file:

mongoexport --db library_database --collection members --out ~/project/exported_members.json

View the exported file:

cat ~/project/exported_members.json

This allows you to verify the imported data outside of MongoDB.

Handle Import Errors

In this step, you'll learn how to handle common import errors when working with MongoDB and mongoimport. Understanding these errors will help you troubleshoot data import issues.

First, let's create a sample file with some import challenges:

cd ~/project

Create an intentionally problematic CSV file:

nano problematic_members.csv

Add the following content:

name,age,membership_type,join_date
John Doe,thirty-five,Premium,2023-01-15
Jane Smith,28,Standard,invalid-date
Alice Johnson,42,Gold,2021-06-10

Note the intentional errors:

• "thirty-five" is not a valid number
• "invalid-date" is not a proper date format

Try importing this file:

mongoimport --db library_database --collection problematic_members --type csv --file ~/project/problematic_members.csv --headerline

You'll likely see an error message like:

2024-xx-xx 00:00:00.000 [#######] Failed: document validation error

To handle such errors, use the --stopOnError and --jsonArray flags:

mongoimport --db library_database --collection problematic_members --type csv --file ~/project/problematic_members.csv --headerline --stopOnError

Create a clean version of the file:

nano clean_members.csv

Add the corrected content:

name,age,membership_type,join_date
John Doe,35,Premium,2023-01-15
Jane Smith,28,Standard,2022-11-20
Alice Johnson,42,Gold,2021-06-10

Now import with validation:

mongoimport --db library_database --collection clean_members --type csv --file ~/project/clean_members.csv --headerline

Verify the import:

mongosh library_database --eval "db.clean_members.find()"

Let's demonstrate another common error - incorrect JSON format:

nano invalid_books.json

Add an intentionally malformed JSON:

[
  {
    "title": "MongoDB Basics",
    "author": "Jane Smith",
    "year": 2023
  },
  {
    "title": "Incomplete Book"
    // Missing comma and closing brace
  }
]

Try importing this file:

mongoimport --db library_database --collection invalid_books --file ~/project/invalid_books.json --jsonArray

You'll see an error about JSON parsing.

Create a valid JSON file:

nano valid_books.json

Add the correct JSON:

[
  {
    "title": "MongoDB Basics",
    "author": "Jane Smith",
    "year": 2023
  },
  {
    "title": "Complete Book",
    "author": "John Doe",
    "year": 2022
  }
]

Import the valid JSON:

mongoimport --db library_database --collection valid_books --file ~/project/valid_books.json --jsonArray

Verify Imported Data

In this final step, you'll learn how to thoroughly verify and validate the data you've imported into MongoDB using various techniques and tools.

Start the MongoDB shell:

mongosh library_database

First, let's perform a comprehensive data validation:

// Check total number of collections
show collections

Example output:

books
members
clean_members
valid_books

Count documents in each collection:

db.books.countDocuments()
db.members.countDocuments()
db.clean_members.countDocuments()
db.valid_books.countDocuments()

Perform advanced queries to verify data integrity:

// Find books published after 2022
db.books.find({ year: { $gt: 2022 } })

// Find members older than 30
db.members.find({ age: { $gt: 30 } })

Exit the MongoDB shell:

exit

Use mongoexport to export collections for external verification:

## Export books collection
mongoexport --db library_database --collection books --out ~/project/exported_books.json

## Export members collection
mongoexport --db library_database --collection members --out ~/project/exported_members.json

Verify exported files:

## Check exported books
head -n 5 ~/project/exported_books.json

## Check exported members
head -n 5 ~/project/exported_members.json

Perform data consistency check:

## Compare collection sizes
wc -l ~/project/exported_books.json
wc -l ~/project/exported_members.json

Optional: Use jq for JSON validation:

## Install jq if not already installed
sudo apt-get update
sudo apt-get install -y jq

## Validate JSON structure
jq '.' ~/project/exported_books.json
jq '.' ~/project/exported_members.json

Let's create a simple verification script:

nano ~/project/verify_import.sh

Add the following content:

#!/bin/bash

## Verify MongoDB Import
echo "Checking MongoDB Collections:"
mongosh library_database --quiet --eval "
    print('Books count: ' + db.books.countDocuments());
    print('Members count: ' + db.members.countDocuments());
    print('Clean Members count: ' + db.clean_members.countDocuments());
    print('Valid Books count: ' + db.valid_books.countDocuments());
"

Make the script executable:

chmod +x ~/project/verify_import.sh

Run the verification script:

~/project/verify_import.sh

Summary

In this lab, you learned how to import JSON and CSV data into MongoDB using the mongoimport command. For the JSON files, you created a sample books.json file, imported it into a library_database database, and verified the import by counting the documents. For the CSV data, you created a library_members.csv file, imported it into the same database, and checked the import results. You also learned how to handle import errors and verify the imported data.