jq 1.6 实战：5个Shell脚本JSON处理场景与完整代码示例-洪萨配资

jq 1.6 实战：5个Shell脚本JSON处理场景与完整代码示例

JSON已经成为现代数据交换的事实标准，从API响应到配置文件，无处不在。对于开发者来说，能够高效处理JSON数据是一项必备技能。而jq作为命令行下的JSON处理神器，其强大功能和灵活性令人惊叹。本文将带你深入jq 1.6的实战应用，通过5个典型场景的完整Shell脚本示例，展示如何用jq解决实际问题。

1. API响应解析与数据提取

现代Web开发中，与API交互是家常便饭。假设我们有一个返回用户列表的API，响应如下：

{ "status": "success", "data": { "users": [ { "id": 101, "name": "张三", "email": "zhangsan@example.com", "active": true, "roles": ["admin", "editor"] }, { "id": 102, "name": "李四", "email": "lisi@example.com", "active": false, "roles": ["viewer"] } ], "total": 2 } }

需求：提取所有活跃用户的姓名和邮箱，并以CSV格式输出。

#!/bin/bash # 模拟API响应数据 api_response='{ "status": "success", "data": { "users": [ { "id": 101, "name": "张三", "email": "zhangsan@example.com", "active": true, "roles": ["admin", "editor"] }, { "id": 102, "name": "李四", "email": "lisi@example.com", "active": false, "roles": ["viewer"] } ], "total": 2 } }' # 使用jq处理并输出CSV echo "$api_response" | jq -r ' .data.users[] | select(.active == true) | [.name, .email] | @csv '

输出结果：

"张三","zhangsan@example.com"

关键jq技巧：

select()函数用于条件过滤
[]展开数组
@csv格式化输出为CSV
-r选项输出原始字符串（去掉JSON引号）

2. 配置文件动态修改

开发中经常需要修改JSON格式的配置文件。假设我们有一个应用配置文件：

{ "app": { "name": "MyApp", "version": "1.0.0", "settings": { "debug": false, "logLevel": "info", "maxConnections": 10 } } }

需求：编写脚本将debug模式开启，并将maxConnections增加到20。

#!/bin/bash # 原始配置文件 config_file='config.json' echo '{ "app": { "name": "MyApp", "version": "1.0.0", "settings": { "debug": false, "logLevel": "info", "maxConnections": 10 } } }' > "$config_file" # 使用jq修改配置 jq '.app.settings.debug = true | .app.settings.maxConnections = 20' "$config_file" > tmp.json && mv tmp.json "$config_file" # 显示修改后的配置 echo "修改后的配置文件内容：" cat "$config_file"

修改后的配置文件：

{ "app": { "name": "MyApp", "version": "1.0.0", "settings": { "debug": true, "logLevel": "info", "maxConnections": 20 } } }

关键jq技巧：

使用.操作符导航JSON结构
使用|管道组合多个修改操作
修改后输出到临时文件再替换原文件

3. 复杂JSON数据统计与分析

假设我们有一组销售数据：

[ { "date": "2023-01-01", "product": "A", "quantity": 10, "price": 99.99, "region": "North" }, { "date": "2023-01-02", "product": "B", "quantity": 5, "price": 199.99, "region": "South" }, { "date": "2023-01-03", "product": "A", "quantity": 8, "price": 99.99, "region": "North" } ]

需求：计算每种产品的总销售额和平均每单数量。

#!/bin/bash # 销售数据 sales_data='[ { "date": "2023-01-01", "product": "A", "quantity": 10, "price": 99.99, "region": "North" }, { "date": "2023-01-02", "product": "B", "quantity": 5, "price": 199.99, "region": "South" }, { "date": "2023-01-03", "product": "A", "quantity": 8, "price": 99.99, "region": "North" } ]' # 使用jq进行数据分析 echo "$sales_data" | jq ' group_by(.product) | map({ product: .[0].product, totalSales: (map(.quantity * .price) | add), averageQuantity: (map(.quantity) | add / length), transactionCount: length }) '

输出结果：

[ { "product": "A", "totalSales": 1799.82, "averageQuantity": 9, "transactionCount": 2 }, { "product": "B", "totalSales": 999.95, "averageQuantity": 5, "transactionCount": 1 } ]

关键jq技巧：

group_by()按字段分组
map()对数组元素进行转换
数学计算直接在jq中完成
add函数计算数组总和

4. 日志文件过滤与转换

假设我们有如下结构的日志条目：

{"timestamp": "2023-06-15T10:00:00Z", "level": "INFO", "message": "User logged in", "user": "user1"} {"timestamp": "2023-06-15T10:01:00Z", "level": "ERROR", "message": "Failed to connect to DB", "error": "Connection timeout"} {"timestamp": "2023-06-15T10:02:00Z", "level": "WARN", "message": "High memory usage", "memory": "85%"}

需求：提取所有ERROR级别的日志，并转换为更简洁的格式。

#!/bin/bash # 日志数据 log_entries=' {"timestamp": "2023-06-15T10:00:00Z", "level": "INFO", "message": "User logged in", "user": "user1"} {"timestamp": "2023-06-15T10:01:00Z", "level": "ERROR", "message": "Failed to connect to DB", "error": "Connection timeout"} {"timestamp": "2023-06-15T10:02:00Z", "level": "WARN", "message": "High memory usage", "memory": "85%"} ' # 使用jq处理日志 echo "$log_entries" | jq -c ' select(.level == "ERROR") | { time: .timestamp, msg: .message, detail: (.error // .memory // .user // "No additional details") } '

输出结果：

{"time":"2023-06-15T10:01:00Z","msg":"Failed to connect to DB","detail":"Connection timeout"}

关键jq技巧：

-c紧凑输出
//操作符提供默认值
条件选择特定日志级别
重构JSON结构

5. 多文件数据合并与处理

场景：我们有两个JSON文件，需要合并并根据某些条件处理。

users.json:

[ {"id": 1, "name": "Alice", "department": "Engineering"}, {"id": 2, "name": "Bob", "department": "Marketing"} ]

salaries.json:

[ {"userId": 1, "salary": 80000}, {"userId": 2, "salary": 75000} ]

需求：合并两个文件，计算每个部门的平均薪资。

#!/bin/bash # 创建示例文件 echo '[ {"id": 1, "name": "Alice", "department": "Engineering"}, {"id": 2, "name": "Bob", "department": "Marketing"} ]' > users.json echo '[ {"userId": 1, "salary": 80000}, {"userId": 2, "salary": 75000} ]' > salaries.json # 使用jq合并处理 jq -n ' # 加载第一个文件 input as $users | # 加载第二个文件 input as $salaries | # 合并数据 [ $users[] as $user | $salaries[] as $salary | select($user.id == $salary.userId) | { name: $user.name, department: $user.department, salary: $salary.salary } ] | # 按部门分组计算平均薪资 group_by(.department) | map({ department: .[0].department, avgSalary: (map(.salary) | add / length), employeeCount: length }) ' users.json salaries.json

输出结果：

[ { "department": "Engineering", "avgSalary": 80000, "employeeCount": 1 }, { "department": "Marketing", "avgSalary": 75000, "employeeCount": 1 } ]

关键jq技巧：

input加载多个文件
复杂的数据关联处理
多步骤数据处理管道
group_by和统计计算

jq高级技巧与最佳实践

通过以上5个实战场景，我们已经领略了jq的强大功能。下面总结一些高级技巧：

变量使用：使用--arg传递外部变量

jq --arg dept "Engineering" '.users[] | select(.department == $dept)' data.json

自定义函数：定义可复用的jq函数

def highlight($msg): ">>> \($msg) <<<"; .[] | highlight(.name)

错误处理：使用try/catch处理可能缺失的字段
```
try .someField catch "default value"
```
性能优化：对于大文件，使用--stream模式
```
jq --stream 'select(length == 2)' huge.json
```
格式化输出：利用@格式转换器
```
.[] | [.name, .age] | @tsv
```