如何使用 jq 对连续的 json 记录流调用操作

Question

所以我有一个每秒将 JSON 记录写入 stdout 的进程，我如何在写入一定数量 (=x) 的记录后使用 jq 触发一个动作并且每秒仍然有我的输出？

这个过程是，为了方便重放，用一个带睡眠的 bash shell 读取循环代替：

( read line ; while [[ -n "${line}" ]] ; do echo "${line}"; sleep 1 ; read line ; done ) < json.data

x条记录后的动作是统计记录的数量，忽略“null”值的行

当x为1时，对每一行进行处理，解决方案简单明了

( read line ; while [[ -n "${line}" ]] ; do echo "${line}"; sleep 1 ; read line ; done ) < json.data | \
jq -c '. , ([.] | del( .[] | select(to_entries[].value == null)) | length)'
{"date":230415072207,"kwL1Tl":0,"kwL1":0.022,"kwL2Tl":0,"kwL2":0.075,"kwL3Tl":0,"kwL3":0}
1
{"date":230415072311,"kwL1Tl":0,"kwL1":0.023,"kwL2Tl":0,"kwL2":0.08,"kwL3Tl":0,"kwL3":0}  
1
{"date":230415072312,"kwL1Tl":null,"kwL1":null,"kwL2Tl":null,"kwL2":null,"kwL3Tl":null,"kwL3":null}  
0
{"date":230415072415,"kwL1Tl":0,"kwL1":0.023,"kwL2Tl":0,"kwL2":0.08,"kwL3Tl":0,"kwL3":0}  
1
{"date":230415072416,"kwL1Tl":0,"kwL1":0.022,"kwL2Tl":0,"kwL2":0.08,"kwL3Tl":0,"kwL3":0}  
1
...

但是当我想对每 5 条或 100 条记录执行此操作时该怎么办？

x=5 的预期输出：

$ ( read line ; while [[ -n "${line}" ]] ; do echo "${line}"; sleep 1 ; read line ; done ) < json.data | \
alert
{"date":230415072207,"kwL1Tl":0,"kwL1":0.022,"kwL2Tl":0,"kwL2":0.075,"kwL3Tl":0,"kwL3":0}
{"date":230415072311,"kwL1Tl":0,"kwL1":0.023,"kwL2Tl":0,"kwL2":0.08,"kwL3Tl":0,"kwL3":0}
{"date":230415072312,"kwL1Tl":null,"kwL1":null,"kwL2Tl":null,"kwL2":null,"kwL3Tl":null,"kwL3":null}
{"date":230415072415,"kwL1Tl":0,"kwL1":0.023,"kwL2Tl":0,"kwL2":0.08,"kwL3Tl":0,"kwL3":0}
{"date":230415072416,"kwL1Tl":0,"kwL1":0.022,"kwL2Tl":0,"kwL2":0.08,"kwL3Tl":0,"kwL3":0}
4
{"date":230415072519,"kwL1Tl":0,"kwL1":0.021,"kwL2Tl":0,"kwL2":0.08,"kwL3Tl":0,"kwL3":0}
{"date":230415072520,"kwL1Tl":0,"kwL1":0.021,"kwL2Tl":0,"kwL2":0.08,"kwL3Tl":0,"kwL3":0}
{"date":230415072623,"kwL1Tl":0,"kwL1":0.022,"kwL2Tl":0,"kwL2":0.075,"kwL3Tl":0,"kwL3":0}
{"date":230415072624,"kwL1Tl":0,"kwL1":0.022,"kwL2Tl":0,"kwL2":0.075,"kwL3Tl":0,"kwL3":0}
{"date":230415072727,"kwL1Tl":0,"kwL1":0.022,"kwL2Tl":0,"kwL2":0.076,"kwL3Tl":0,"kwL3":0}
5
{"date":230415072728,"kwL1Tl":0,"kwL1":0.023,"kwL2Tl":0,"kwL2":0.076,"kwL3Tl":0,"kwL3":0}
{"date":230415072831,"kwL1Tl":0,"kwL1":0.023,"kwL2Tl":0,"kwL2":0.075,"kwL3Tl":0,"kwL3":0}
...

Bash 函数“alert”应该包含一个有效的 jq 命令。

Answer 1

下面的“警报”功能符合预期：

alert(){
    #
    # send alerts if needed
    jq -nc --argjson m "5" '
        def alerts(r): r | del( .[] | select(to_entries[].value == null)) | length ; 
        foreach inputs as $in (
             {"recs":[]};
             if (.recs|length) < $m then .recs += [$in] else .recs=[$in] end;
             .recs[-1], if (.recs|length) == $m then alerts(.recs) else empty end)
    '
}

解释一下：

• “foreach inputs”将流程输出读取为流
• 第一个“if”将输入的 JSON 记录添加到数组“recs”中，直到数组大小为 $m。 （在我们的例子中，“5”由 jq 程序的“-argjson m 5”选项指定）。如果它是 5 大，它会再次覆盖“recs”数组（使用新的第一个元素）。
• ".recs[-1]" 将收到的最后一条记录放在输出中。 （“$in”也可以）
• 当数组“recs”为 5 大时，第二个“if”执行内部函数“alerts”。
• jq 本地函数“alerts”在删除一个或多个字段值为“null”的记录后写出数组“recs”的大小。