我想在特定时期内将某些日志从ES Cluster提取到文件中(因为它们以百万计)
我已经尝试过https://github.com/taskrabbit/elasticsearch-dump及其-searchBody选项,如下所示:
elasticdump --input="some.endpoint" --output="somelocation/my_index.json" --ignoreType='mapping,settings,template' --limit 2000 -- --searchBody='{"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"stored_fields":["*"],"script_fields":{},"docvalue_fields":[{"field":"@timestamp","format":"date_time"}],"query":{"bool":{"must":[{"range":{"@timestamp":{"format":"strict_date_optional_time","gte":"2019-12-17T12:00:00.000Z","lte":"2019-12-17T12:00:15.000Z"}}}],"filter":[{"match_all":{}}],"should":[],"must_not":[]}}}'
它确实生成了带有日志的文件,尽管似乎它没有考虑-searchBody查询,因为日志从“ @ timestamp”:“” 2019-12-03T05:09:46.902+开始0000“并显示为infinity(必须手动停止该过程)
也尝试过--debug
并得到相同的结果参考
Thu, 19 Dec 2019 12:34:43 GMT | starting dump
Thu, 19 Dec 2019 12:34:43 GMT [debug] | discovered elasticsearch input major version: 7
Thu, 19 Dec 2019 12:34:43 GMT | got 100 objects from source elasticsearch (offset: 0)
Thu, 19 Dec 2019 12:34:43 GMT | sent 100 objects to destination file, wrote 100
Thu, 19 Dec 2019 12:34:43 GMT [debug] | scrollRequest: {"uri":"my.endpoint/_search/scroll","method":"POST","body":"{\"scroll\":\"10m\",\"scroll_id\":\"DnF1ZXJ5VGhlbkZldGNoVgAAAAAAAIaUFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACg_BZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpUWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJz-FmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACg-hZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpYWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJ0AFmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACg-xZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpcWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJz_FmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACdAhZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAhpgWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJ0BFmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACg_RZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpkWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIaaFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACg_hZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpsWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJ0DFmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACg_xZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQUWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKECFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChARZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQYWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEDFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChABZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQcWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEEFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChBRZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQQWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEGFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChBxZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQgWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEIFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChCRZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQkWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEKFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGnBZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQoWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKELFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGnhZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQsWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEMFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGnRZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQwWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKENFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGoRZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQ0WYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEOFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGoBZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQ4WYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEQFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChDxZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnRAWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKERFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACdDxZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAnREWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKESFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACdEhZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAnRMWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKETFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChFBZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnRQWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEVFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACdFRZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAnRYWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEWFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACdFxZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAnRgWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAIafFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGohZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAhqMWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIalFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGpBZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAhqYWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIanFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGqBZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAhqkWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIaqFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGqxZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAhqwWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIatFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGrhZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnRkWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAIavFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGsBZwYW4xcE1lZVJseXVOV01OWDRiTDBB\"}"}
Thu, 19 Dec 2019 12:34:43 GMT [debug] | body: "{\"_scroll_id\":\"DnF1ZXJ5VGhlbkZldGNoVgAAAAAAAIaUFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACg_BZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpUWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJz-FmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACg-hZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpYWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJ0AFmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACg-xZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpcWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJz_FmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACdAhZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAhpgWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJ0BFmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACg_RZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpkWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIaaFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACg_hZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAhpsWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAJ0DFmFiWGlwWFFOUjFpR01ZT3ZfT3JwancAAAAAAACg_xZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQUWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKECFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChARZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQYWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEDFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChABZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQcWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEEFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChBRZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQQWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEGFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChBxZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQgWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEIFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChCRZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnQkWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEKFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGnBZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQoWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKELFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGnhZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQsWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEMFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGnRZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQwWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKENFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGoRZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQ0WYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEOFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACGoBZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnQ4WYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEQFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChDxZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnRAWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKERFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACdDxZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAnREWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKESFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACdEhZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAnRMWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKETFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAAChFBZBOHVNTTN4R1NoT2RpRmpwODA0YUZRAAAAAAAAnRQWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEVFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACdFRZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAnRYWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAKEWFkE4dU1NM3hHU2hPZGlGanA4MDRhRlEAAAAAAACdFxZhYlhpcFhRTlIxaUdNWU92X09ycGp3AAAAAAAAnRgWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAIafFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGohZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAhqMWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIalFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGpBZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAhqYWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIanFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGqBZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAhqkWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIaqFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGqxZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAhqwWcGFuMXBNZWVSbHl1TldNTlg0YkwwQQAAAAAAAIatFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGrhZwYW4xcE1lZVJseXVOV01OWDRiTDBBAAAAAAAAnRkWYWJYaXBYUU5SMWlHTVlPdl9PcnBqdwAAAAAAAIavFnBhbjFwTWVlUmx5dU5XTU5YNGJMMEEAAAAAAACGsBZwYW4xcE1lZVJseXVOV01OWDRiTDBB\",\"took\":1,\"timed_out\":false,\"terminated_early\":true,\"_shards\":{\"total\":86,\"successful\":86,\"skipped\":0,\"failed\":0},\"hits\":{\"total\":{\"value\":1489826675,\"relation\":\"eq\"},\"max_score\":1.0,\"hits\":[{\"_index\":\"logs-2019-12-03\",\"_type\":\"log\",\"_id\":\"fEQ1ym4BchgN3v1WBpMd\",\"_score\":1.0,\"_source\":{\"@timestamp\":\"2019-12-03T05:21:42.642+0000\................"
但是,-searchBody中的同一查询正在kibana上运行,预期日志从{“ @ timestamp”:“ 2019-12-03T12:00:03.300 + 0000”]开始 ,我不确定这里会缺少什么。任何帮助,不胜感激。谢谢
我有一个类似的问题,即搜索主体无法正常工作,请问我有一个更新的方式解决这个问题的方法吗?