我有一个rabbitmq,它成功地存储了消息,但是我的日志存储库读取队列时忽略了我的大部分消息。
RabbitMQ是OK,我有一个小python脚本来显示所有消息
import pika
i=0
def on_message(channel, method_frame, header_frame, body):
global i
print i
print("Message body", body)
channel.basic_ack(delivery_tag=method_frame.delivery_tag)
i+=1
credentials = pika.PlainCredentials('***', '***')
parameters = pika.ConnectionParameters('***',5672,'logstash', credentials=credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.exchange_declare(exchange="logstash", exchange_type="topic", passive=False, durable=True, auto_delete=False)
channel.queue_declare(queue="hbbtv", auto_delete=False, durable=True)
channel.queue_bind(queue="hbbtv", exchange="logstash", routing_key="hbbtv")
channel.basic_qos(prefetch_count=1)
channel.basic_consume(on_message, 'hbbtv')
try:
channel.start_consuming()
except KeyboardInterrupt:
channel.stop_consuming()
connection.close()我能看到我所有的留言
12 ('Message body','{"message":"212.95.70.118 - 25/Feb/2016:11:19:53 +0100 \"GET /services/web/index.php/OPA/categories/ARTEPLUS7/fr HTTP/1.1\“200 348 \"[DE&page=PLUS7&tv=false\](http://www.arte.tv/hbbtvv2/notv/cehtml/index.cehtml?lang=de_DE&page=PLUS7&tv=false%5C)”\"Opera/9.80 (Linux armv7l;HbbTV/1.1.1 (;Philips;;PhilipsTV);) CE-HTML/1.0 NETTV/4.3.1 PhilipsTV/2.1.1固件/003.015.000.001 (PhilipsTV,2.1.1,en) Presto/2.12.362版本/12.11 \“hbbtvdyn.arte.tv","@version":"1",”@time戳“:”2016-02-25T10:19:53.000Z“,”path“:”/data/log/access“,"host":"arte-hbbtvdyn-web1.sdv.fr","type":"apache-access","application":"hbbtv","clientip":"212.95.70.118","ident":"-",“auth”:--,“time戳”:“25/Feb/2016:11:19:53 +0100",”动词“:”GET“,"request":"/services/web/index.php/OPA/categories/ARTEPLUS7/fr","httpversion":"1.1","response":"200",“字节”:“348”,"referrer":"\"[DE&page=PLUS7&tv=false\](http://www.arte.tv/hbbtvv2/notv/cehtml/index.cehtml?lang=de_DE&page=PLUS7&tv=false%5C)"",“代理”:“\”Opera/9.80 (Linux armv7l;(;飞利浦;;;PhilipsTV;) CE-HTML/1.0 NETTV/4.3.1 PhilipsTV/2.1.1固件/003.015.000.001 (PhilipsTV,2.1.1,en) Presto/2.12.362版本/12.11\“,”hbbtvdyn.arte.tv“:”hbbtvdyn.arte.tv“,"geoip":{"ip":"212.95.70.118","country_code2":"FR","country_code3":"FRA",”country_name“:”法国“,"continent_code":"EU","region_name":"C1",“city_name”:“斯特拉斯堡”,“纬度”:48.60040000000001,“经度”:7.787399999999991,“时区”:“欧洲/巴黎”,"real_region_name":"Alsace","location":7.787399999999991,48.60040000000001}}') 13 (‘消息体’,{“信息”:“212.95.70.118- 25/Feb/2016:11:19:53 +0100 \"GET /services/web/index.php/OPA/videos/highlights/6/ARTEPLUS7/de/GE HTTP/1.1\”500 4519 \"[DE&page=PLUS7&tv=false\](http://www.arte.tv/hbbtvv2/notv/cehtml/index.cehtml?lang=de_DE&page=PLUS7&tv=false%5C)“\"Opera/9.80 (Linux armv7l;(;飞利浦;;;PhilipsTV;) CE-HTML/1.0 NETTV/4.3.1 PhilipsTV/2.1.1固件/003.015.000.001 (PhilipsTV,2.1.1,en) Presto/2.12.362版本/12.11 \“hbbtvdyn.arte.tv","@version":"1",”@time戳“:”2016-02-25T10:19:53.000Z“,”path“:”/data/log/access“,"host":"arte-hbbtvdyn-web1.sdv.fr","type":"apache-access","application":"hbbtv","clientip":"212.95.70.118","ident":"-",“auth”:--,“time戳”:“25/Feb/2016:11:19:53 +0100",”动词“:”GET“,"request":"/services/web/index.php/OPA/videos/highlights/6/ARTEPLUS7/de/GE","httpversion":"1.1","response":"500",“字节”:“4519”,"referrer":"\"[DE&page=PLUS7&tv=false\](http://www.arte.tv/hbbtvv2/notv/cehtml/index.cehtml?lang=de_DE&page=PLUS7&tv=false%5C)"",“代理”:“\”Opera/9.80 (Linux armv7l;(;飞利浦;;;PhilipsTV;) CE-HTML/1.0 NETTV/4.3.1 PhilipsTV/2.1.1固件/003.015.000.001 (PhilipsTV,2.1.1,en) Presto/2.12.362版本/12.11\“,”hbbtvdyn.arte.tv“:”hbbtvdyn.arte.tv“,"geoip":{"ip":"212.95.70.118","country_code2":"FR","country_code3":"FRA",”country_name“:”法国“,"continent_code":"EU","region_name":"C1",“city_name”:“斯特拉斯堡”,“纬度”:48.60040000000001,“经度”:7.787399999999991,“时区”:“欧洲/巴黎”,"real_region_name":"Alsace","location":7.787399999999991,48.60040000000001}}') 14 (‘消息体’,{“消息”:“212.95.70.119- 25/Feb/2016:11:19:53 +0100 \"GET /OPA/getOPAData.php?url=videoStreams%3Flanguage%3Dfr%26protocol%3DHTTP%26mediaType%3Dmp4%26quality%3DEQ%2CSQ%2CHQ%26profileAmm%3D%24nin%3AAMM-YTFR-HAB%2CAMM-YTFR%2CAMM-DT%26kind%3DSHOW%26availableScreens%3Dtv%26fields%3DprogramId%2Curl%2Cquality%2CaudioSlot%2CaudioCode%2CaudioLabel%2CaudioShortLabel%2Cchannel%26programId%3D048353-033-A%26platform%3DARTEPLUS7&filename=PLUS7_stream_048353-033-A_fr_FR.json HTTP/1.1\“2005508\”-\/5.0 (Linux;SmartHub;智能电视;SmartTV;U;( Maple2012) AppleWebKit/538.1+ (KHTML,类似于壁虎) TV Safari/538.1+ \“hbbtvdyn.arte.tv","@version":"1",”@time戳“:”2016-02-25T10:19:53.000Z“,”path“:”/data/log/access“,"host":"arte-hbbtvdyn-web1.sdv.fr","type":"apache-access","application":"hbbtv","clientip":"212.95.70.119","ident":"-“、"auth":"-”、“时间戳”:“25/Feb/2016:11:19:53 +0100”、“动词”:“GET”、"request":"/OPA/getOPAData.php?url=videoStreams%3Flanguage%3Dfr%26protocol%3DHTTP%26mediaType%3Dmp4%26quality%3DEQ%2CSQ%2CHQ%26profileAmm%3D%24nin%3AAMM-YTFR-HAB%2CAMM-YTFR%2CAMM-DT%26kind%3DSHOW%26availableScreens%3Dtv%26fields%3DprogramId%2Curl%2Cquality%2CaudioSlot%2CaudioCode%2CaudioLabel%2CaudioShortLabel%2Cchannel%26programId%3D048353-033-A%26platform%3DARTEPLUS7&filename=PLUS7_stream_048353-033-A_fr_FR.json","httpversion":"1.1",“响应”:“200”,“字节”:“5508”,“引用”:“\”-\“”,“代理”:“\”Mozilla/5.0(Linux);SmartHub;智能电视;SmartTV;U;( Maple2012) AppleWebKit/538.1+ (KHTML,类似壁虎) TV Safari/538.1+ \“,”hbbtvdyn.arte.tv“,"geoip":{"ip":"212.95.70.119","country_code2":"FR","country_code3":"FRA",”country_name“:”法国“,”continent_code“:”欧盟“,"region_name":"C1",”city_name“:”斯特拉斯堡“,“纬度”:48.60040000000001,“经度”:7.787399999999991,“时区”:“欧洲/巴黎”,“real_region_name”:“阿尔萨斯”,“real_region_name”
有了好的速率信息(每秒钟几次),我绝对不会觉得解析失败。
因此,当logstash读取消息时,问题就会发生。问题是

logstash的输入部分是
rabbitmq {
host=>"arte-elasticlog.sdv.fr"
user=>"***"
password=>"***"
queue=>"hbbtv"
vhost=>"logstash"
port=>5672
auto_delete=>false
durable=>true
type => "rabbit_hbbtv"
}发布于 2016-03-03 12:38:45
问题是由于我的logstash过滤器,我有两个apache访问权限,但是有不同的模式,所以当logstash试图查看一条消息时,有时他有一个好的模式--> ES,有时不是->没有消息。
现在,对于所有不同的日志,我添加( add )
application-->"my application name"在我的输入和我所有的grok过滤器取决于应用程序。
现在一切都好了,谢谢你的帮助。
发布于 2016-02-25 12:10:32
_grokparsefailure表示它无法解析消息。这意味着消息成功地从队列中读取,但是您的grok表达式有问题或无法应用于您的邮件内容。
还有一件事是,rabbitmq输入的默认编解码器是"json",如果您的rabbitmq消息内容不是json,您应该将输入的编解码器设置为例如:
编解码器=>平原{}
https://stackoverflow.com/questions/35622947
复制相似问题