jq Command

jq is a lightweight and flexible command-line JSON processor. It allows you to slice, filter, map, and transform structured data with the same ease that sed, awk, grep provide for text processing.

Basic Concept

jq uses a simple functional language designed specifically for JSON. Every jq expression takes JSON input and produces JSON output. The language is built around the concept of filters that transform JSON data.

Syntax Overview

Basic Access

// Identity filter (returns input unchanged)
.

// Access property
.propertyName
.store.book

// Access array element
.[0]
.users[0]

// Access nested property
.users[0].address.city

Array Operations

// Get array length
. | length

// Access all array elements
.[]

// Array indexing
.[0]
.[1:3]    // Slice from index 1 to 2
.[-1]     // Last element

Sample JSON Data

{
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      },
      {
        "category": "fiction",
        "author": "Herman Melville", 
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  },
  "users": [
    {
      "id": 1,
      "name": "John Doe",
      "email": "[email protected]",
      "age": 30,
      "active": true,
      "address": {
        "street": "123 Main St",
        "city": "New York",
        "zipcode": "10001"
      }
    },
    {
      "id": 2,
      "name": "Jane Smith", 
      "email": "[email protected]",
      "age": 25,
      "active": false,
      "address": {
        "street": "456 Oak Ave",
        "city": "Los Angeles", 
        "zipcode": "90210"
      }
    }
  ]
}

Basic Queries

// Get all books
.store.book

// Get first book
.store.book[0]

// Get book title
.store.book[0].title

// Get bicycle color
.store.bicycle.color

// Get all users
.users

// Get user address
.users[0].address.city

Array Processing

Array Iteration

// Iterate over all books
.store.book[]

// Get all book titles
.store.book[].title

// Get all user names
.users[].name

// Get all user cities
.users[].address.city

// Get all authors
.store.book[].author

Array Indexing and Slicing

// First book
.store.book[0]

// Last book
.store.book[-1]

// First two books
.store.book[0:2]

// All books except first
.store.book[1:]

// Last two books
.store.book[-2:]

Filtering

Select Filter

// Books under $10
.store.book[] | select(.price < 10)

// Fiction books
.store.book[] | select(.category == "fiction")

// Books that have ISBN
.store.book[] | select(.isbn)

// Active users
.users[] | select(.active)

// Users over 25
.users[] | select(.age > 25)

// Users in New York
.users[] | select(.address.city == "New York")

Conditional Expressions

// Books with price condition
.store.book[] | select(.price < 10 and .category == "fiction")

// Users with multiple conditions
.users[] | select(.active and .age > 25)

// Books without ISBN
.store.book[] | select(.isbn | not)

Object Construction

Creating New Objects

// Select specific book fields
.store.book[0] | {title, author, price}

// Create custom object structure
.store.book[0] | {
  title: .title,
  author: .author,
  category: .category
}

// Transform all books
.store.book[] | {
  title: .title,
  author: .author,
  expensive: (.price > 10)
}

// User summary
.users[] | {
  name: .name,
  email: .email,
  city: .address.city,
  status: (if .active then "Active" else "Inactive" end)
}

Array Construction

// Create array of titles
[.store.book[].title]

// Create array of user names
[.users[].name]

// Mixed array
[.store.book[0].title, .store.book[0].author]

// Complex array construction
[.users[] | {name: .name, city: .address.city}]

Map and Transformation

Map Function

// Map over books
.store.book | map(.title)

// Map over users
.users | map(.name)

// Transform with map
.store.book | map({title, author, price})

// Complex mapping
.users | map({
  profile: {name, email},
  location: .address.city,
  active
})

Advanced Transformations

// Add computed fields
.store.book | map(. + {expensive: (.price > 10)})

// Transform structure
.users | map({
  user_info: {name, age},
  contact: {email, city: .address.city}
})

// Conditional transformation
.store.book | map(
  if .category == "fiction" 
  then . + {genre: "fiction"} 
  else . + {genre: "non-fiction"} 
  end
)

Aggregation and Calculations

Basic Aggregation

// Count books
.store.book | length

// Count users
.users | length

// Sum book prices
.store.book | map(.price) | add

// Average book price
(.store.book | map(.price) | add) / (.store.book | length)

// Maximum book price
.store.book | map(.price) | max

// Minimum book price
.store.book | map(.price) | min

Grouping

// Group books by category
.store.book | group_by(.category)

// Group users by activity status
.users | group_by(.active)

// Group and transform
.store.book | group_by(.category) | map({
  category: .[0].category,
  count: length,
  titles: map(.title)
})

Sorting

// Sort books by price
.store.book | sort_by(.price)

// Sort users by age (descending)
.users | sort_by(.age) | reverse

// Sort books by multiple criteria
.store.book | sort_by(.category, .price)

// Sort with custom logic
.users | sort_by(.address.city, -.age)

String Operations

String Functions

// Convert to string
.store.book[0].price | tostring

// String length
.users[0].name | length

// Uppercase/lowercase
.users[0].name | ascii_upcase
.users[0].name | ascii_downcase

// String splitting
.users[0].email | split("@")

// String joining
[.store.book[].author] | join(", ")

String Interpolation

// Create formatted strings
.users[] | "\(.name) lives in \(.address.city)"

// Book description
.store.book[] | "\(.title) by \(.author) - $\(.price)"

// Conditional strings
.users[] | "\(.name) is \(if .active then "active" else "inactive" end)"

Conditional Logic

if-then-else

// Simple condition
.users[] | if .active then .name else empty end

// Complex conditions
.store.book[] | if .price < 10 then "cheap" else "expensive" end

// Multiple conditions
.users[] | 
  if .age >= 30 then "senior"
  elif .age >= 18 then "adult"
  else "minor"
  end

// Conditional object construction
.store.book[] | {
  title,
  price_category: (if .price < 10 then "budget" else "premium" end)
}

try-catch

// Safe property access
.users[] | .profile.avatar // error if profile doesn't exist
.users[] | try .profile.avatar catch "no avatar"

// Multiple fallbacks
.store.book[] | try .isbn catch try .id catch "no identifier"

Advanced Features

Recursive Operations

// Recursive descent
.. | .price?

// Find all values recursively
.. | select(type == "number")

// Recursive transformation
walk(if type == "object" and has("price") then .price *= 1.1 else . end)

Path Operations

// Get paths to all prices
paths(.price)

// Get path and value
path(.store.book[0].title), .store.book[0].title

// Update at path
setpath(["store", "bicycle", "color"]; "blue")

Variables and Functions

// Using variables
.store.book as $books | $books | length

// Define and use variable
(.store.book | length) as $count | 
  "There are \($count) books"

// Complex variable usage
.users[] as $user | 
  .store.book[] | 
  select(.price < 10) | 
  {book: .title, recommended_for: $user.name}

Practical Examples

Data Analysis

// Book statistics
{
  total_books: (.store.book | length),
  categories: [.store.book[].category | unique[]],
  avg_price: ((.store.book | map(.price) | add) / (.store.book | length)),
  price_range: {
    min: (.store.book | map(.price) | min),
    max: (.store.book | map(.price) | max)
  }
}

// User demographics
{
  total_users: (.users | length),
  active_users: (.users | map(select(.active)) | length),
  cities: [.users[].address.city | unique[]],
  age_stats: {
    avg_age: ((.users | map(.age) | add) / (.users | length)),
    min_age: (.users | map(.age) | min),
    max_age: (.users | map(.age) | max)
  }
}

Data Transformation

// Create catalog
{
  store_info: {
    books: .store.book | map({title, author, price}),
    bicycle: .store.bicycle
  },
  customers: .users | map({
    id,
    name,
    contact: {email, city: .address.city},
    status: (if .active then "active" else "inactive" end)
  })
}

// Flatten structure
[
  (.store.book[] | {type: "book", item: ., location: "store"}),
  ({type: "bicycle", item: .store.bicycle, location: "store"}),
  (.users[] | {type: "user", item: ., location: "customers"})
]

Filtering and Search

// Complex search
{
  affordable_fiction: [.store.book[] | select(.category == "fiction" and .price < 10)],
  active_ny_users: [.users[] | select(.active and .address.city == "New York")],
  books_with_isbn: [.store.book[] | select(.isbn)]
}

// Multi-criteria filtering
.store.book[] | select(
  (.category == "fiction") and 
  (.price < 15) and 
  (.author | test("^[A-M]"; "i"))
)

Best Practices

1. Use pipes effectively: Chain operations with | for readability
2. Handle errors gracefully: Use try-catch for optional fields
3. Leverage built-in functions: Use map, select, group_by, etc.
4. Create reusable filters: Define complex logic as variables
5. Test incrementally: Build complex queries step by step
6. Use meaningful variable names: Make code self-documenting

Common Patterns

Safe Navigation

// Safe property access
.users[]? | .address?.city?

// Default values
.users[0].profile.avatar // "unknown"

// Multiple fallbacks  
(.users[0].profile.avatar // .users[0].profile.image // "no-image.png")

Data Validation

// Check required fields
.store.book[] | select((.title | length) > 0 and (.author | length) > 0)

// Validate structure
.users[] | select(has("id") and has("name") and has("email"))

// Type checking
.store.book[] | select(.price | type == "number")

jq provides a powerful and expressive language for JSON processing with its functional programming approach and rich set of built-in functions.