Amazon Cloudwatch Logs Insights with JSON字段

Question

我正在尝试使用Logs Insights和其中一个字段中包含JSON的数据，并解析JSON字段

当我将我的数据放入启动器代码的洞察中时，它看起来如下所示

fields @timestamp, @message
| sort @timestamp desc
| limit 25

如何在嵌套的JSON中轻松提取path变量以对其执行聚合？通过查看一些文档，我认为@message.path可以工作，但似乎并非如此。是否有人在Insights中成功地解释了JSON日志

​

​

编辑:我的数据样例

#
@timestamp
@message
1
2018-12-19 23:42:52.000
I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"user,tags,promotions,company_sector,similar_professionals.tags,similar_professionals.user","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@logStream  i-05d1d61ab853517a0
@message  I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@timestamp  1545262972000
2
2018-12-19 23:42:16.000
I, [2018-12-19T23:42:16.723472 #851] INFO -- : [ea712503-eb86-4a6e-ab38-ddbcd6c2b4d0] {"method":"GET","path":"/api/v1/heartbeats/new","format":"json","controller":"API::V1::Public::HeartbeatsController","action":"new","status":201,"duration":9.97,"view":3.2,"time":"2018-12-19T23:42:16.712+00:00","params":{"format":"json","compress":false},"@timestamp":"2018-12-19T23:42:16.722Z","@version":"1","message":"[201] GET /api/v1/heartbeats/new (API::V1::Public::HeartbeatsController#new)"}

Answer 1

CloudWatch洞察日志会自动发现以下日志类型的字段：

Lambda日志

JSON logs Insights会自动发现Lambda Logs、中的日志字段，但只针对每个日志事件(注意:我的日志)中的第一个嵌入的JSON片段。如果Lambda日志事件包含多个JSON片段，您可以使用parse命令解析和提取日志字段。有关详细信息，请参阅JSON日志中的字段。

CloudTrail日志

参见fields in JSON logs。

来源： Supported Logs and Discovered Fields

如果@message为I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}

然后，您可以选择并过滤字段，如下所示：

fields @timestamp, @message, method
| filter method = "GET"
| sort @timestamp desc

它也适用于嵌套字段，即params.format = "json"或results.0.firstName = "Paul"。

Answer 2

可以使用parse命令提取字段。

如果@message是

I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}

然后提取字段，如下所示：

fields @timestamp, @message
| parse "I, [*T*] INFO -- : {"method":"*"}" as @date, @time, @method
| filter method=GET
| sort @timestamp desc
| limit 20

就目前而言，文档相当简单。我可以通过将通配符*替换为正则表达式来获得结果，但随后解析失败。

Answer 3

在@pyb insights的基础上，我能够使用parse @message '"path":"*"' as path从@message中的任何位置提取路径。

由于这是@message上的第二次全局纯文本搜索，因此您可以通过管道传输另一个parse @message '"method":"*"' as method来获取方法，而无需考虑排序问题

如果您的@message是：

I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}

使用：

parse @message '"path":"*"' as path | parse @message '"method":"*"' as method

将产生以下字段：path = '/api/v1/professionals/ID'和method = 'GET'

注意，这仍然是简单的字符串解析，因此，它没有嵌套键的概念，就像params.format找不到json一样，然而，只要@message中没有另一个format字符串，只使用format就可以。

还要注意，这是针对Insights没有在消息中发现您的JSON的情况。我相信这就是@pyb在this answer中提到的情况。使用以下格式也无法发现我的日志

info - Request: {"method":"POST","path":"/auth/login/","body":{"login":{"email":"email@example.com","password":"********"}},"uuid":"36d76df2-aec4-4549-8b73-f237e8f14e23","ip":"*.*.*.*"}