使用脚本将cloud-init日志转换为json

Question

我正在尝试将cloud-init日志转换为json，以便filebeat可以将其拾取并将其发送到Kibana。我想通过使用shell脚本或python脚本来做到这一点。有没有将这些日志转换为json的脚本？

我的python脚本如下

import json
import subprocess

filename = "/home/umesh/Downloads/scripts/cloud-init.log"

def convert_to_json_log(line):
    """ convert each line to json format """
    log = {}
    log['msg'] = line
    log['logger-name'] = 'cloud-init'
    log['ServiceName'] = 'Contentprocessing'
    return json.dumps(log)


def log_as_json(filename):    
    f = subprocess.Popen(['cat','-F',filename],
                        stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    while True:
        line = f.stdout.readline()
        log = convert_to_json_log(line) 
        print log
        with open("/home/umesh/Downloads/outputs/cloud-init-json.log", 'a') as new:
            new.write(log + '\n')

log_as_json(filename)

脚本返回一个json格式的文件，但msg字段返回空字符串。我想将日志的每一行转换为消息字符串。

Answer 1

首先，尝试使用python内置函数读取原始日志文件，而不是使用子进程运行os命令，因为：

• 它将更加便携（适用于OS'）
• 更快，更不容易出错

重新编写你的log_as_json函数如下工作：

inputfile = "cloud-init.log"
outputfile = "cloud-init-json.log"

def log_as_json(filename):
    # Open cloud-init log file for reading
    with open(inputfile, 'r') as log:
        # Open the output file to append json entries
        with open(outputfile, 'a') as jsonlog:
            # Read line by line
            for line in log.readlines():
                # Convert to json and write to file
                jsonlog.write(convert_to_json(line)+"\n")

Answer 2

在花了一些时间准备自定义脚本后，我做了下面的脚本。它可能对许多其他人有帮助。

  import json

def convert_to_json_log(line):
    """ convert each line to json format """
    log = {}
    log['msg'] = json.dumps(line)
    log['logger-name'] = 'cloud-init'
    log['serviceName'] = 'content-processing'
    return json.dumps(log)

# Open the file with read only permit

f = open('/var/log/cloud-init.log', "r")

# use readlines to read all lines in the file

# The variable "lines" is a list containing all lines in the file

lines = f.readlines()

# close the file after reading the lines.
f.close()

jsonData = ''
for line in lines:
    jsonLine = convert_to_json_log(line)
    jsonData = jsonData + "\n" + jsonLine;


with open("/var/log/cloud-init/cloud-init-json.log", 'w') as new:
    new.write(jsonData)