blocks|key|1071370|text|Hadoop通常以离线方式使用。因此，我更愿意定期处理日志。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1071371|在我之前参与的一个项目中，我们让服务器生成每小时轮换一次的日志文件(每小时x:00)。我们有一个每小时运行一次的脚本(每小时x:30)将文件上传到HDFS+(那些还不在HDFS中的文件)。然后，您可以在Hadoop中随心所欲地运行作业来处理这些文件。|1071372|我相信也有更好的实时替代方案。|1071373|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|H|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|I|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|J|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|G|$]]

Hadoop is usually used in an offline manner. So I would rather process the logs periodically. 

In a project I was involved with previously, we made our servers produce log files that were rotated hourly (every hour at x:00). We had a script that ran hourly (every hour at x:30) uploaded the files into HDFS (those that weren't already there). Then you can run jobs as often as you like in Hadoop to process these files.

I am sure there are better real-time alternatives too.

blocks|key|1137772|text|Hadoop不用于实时处理。但它可以用于按小时处理日志，可能会落后一个小时，这几乎是实时的。我想知道在日志到来时处理它的需要是什么。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1137773|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Hadoop is not used for live real time processing. But it can be used to process logs on hourly basis may be one hour behind which is near real time. I wonder what is the need of processing logs as it comes.

blocks|key|4856849|text|您可以尝试使用Flume作为日志收集器，并将其存储在S3中以进行批处理：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4856850|http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/|offset|length|4856851|entityMap|0|LINK|mutability|MUTABLE|url^0|0|0|25|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|N|8|@]|9|@$D|O|E|P|1|Q]]|A|$]]|$1|F|3|-4|5|6|7|R|8|@]|9|@]|A|$]]]|G|$H|$5|I|J|K|A|$L|C]]]]

Something you can try is to use Flume as a log collector and store them in S3 for batch processing:

<a href="http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/" rel="nofollow">http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/</a>

blocks|key|1071443|text|如果您想要真正的实时处理，您可能想看看Twitter's+Storm，它是开源的并托管在GitHub上。Tutorial+here。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1071444|看起来像是大公司的used+in+production。|1071445|在这一点上，我自己并不使用Storm，实际上做了一些类似于问题和回答中提到的事情：|1071446|使用Apache的Hadoop日志事件(使用轮换日志每15/30更改一次日志文件每隔15/30将它们添加到minutes)|1071447|Upload+|unordered-list-item|1071448|
|1071449|将新步骤添加到现有|1071450|1071451|群集(在Amazon+EMR上)|1071452|1071453|使用Hadoop，您可以通过经常在集群上运行批处理并只添加一个新作业来接近实时，但不是真正的实时。为此，您需要Storm或类似的工具。|1071454|entityMap|0|LINK|mutability|MUTABLE|url|http://storm-project.net/|1|https://github.com/nathanmarz/storm/wiki/Tutorial|2|https://github.com/nathanmarz/storm/wiki/Powered-By^0|J|F|0|1G|D|1|0|9|I|2|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|18|8|@]|9|@$A|19|B|1A|1|1B]|$A|1C|B|1D|1|1E]]|C|$]]|$1|D|3|E|5|6|7|1F|8|@]|9|@$A|1G|B|1H|1|1I]]|C|$]]|$1|F|3|G|5|6|7|1J|8|@]|9|@]|C|$]]|$1|H|3|I|5|6|7|1K|8|@]|9|@]|C|$]]|$1|J|3|K|5|L|7|1L|8|@]|9|@]|C|$]]|$1|M|3|N|5|6|7|1M|8|@]|9|@]|C|$]]|$1|O|3|P|5|L|7|1N|8|@]|9|@]|C|$]]|$1|Q|3|N|5|6|7|1O|8|@]|9|@]|C|$]]|$1|R|3|S|5|L|7|1P|8|@]|9|@]|C|$]]|$1|T|3|-4|5|6|7|1Q|8|@]|9|@]|C|$]]|$1|U|3|V|5|6|7|1R|8|@]|9|@]|C|$]]|$1|W|3|-4|5|6|7|1S|8|@]|9|@]|C|$]]]|X|$Y|$5|Z|10|11|C|$12|13]]|14|$5|Z|10|11|C|$12|15]]|16|$5|Z|10|11|C|$12|17]]]]

If you want true real-time processing, you might want to look at <a href="http://storm-project.net/" rel="nofollow">Twitter's Storm</a>, which is open-source and hosted on GitHub. <a href="https://github.com/nathanmarz/storm/wiki/Tutorial" rel="nofollow">Tutorial here</a>.

It looks like it is being <a href="https://github.com/nathanmarz/storm/wiki/Powered-By" rel="nofollow">used in production</a> at large companies.

On that note, I don't use Storm myself, and actually do something similar to what has been in mentioned in the question and responses:

<ol>
<li>Log events using Apache (using rotatelogs for changing log files every 15/30 minutes)</li>
<li>Upload them every so often to S3</li>
<li>Add a new step to an existing Hadoop cluster (on Amazon EMR)</li>
</ol>

With Hadoop, you can get close to real-time by running the batch processing often on a cluster and just adding a new job, but not true real-time. For that you need Storm or something similar.

I want to process the logs from my web server as it comes in using Hadoop (Amazon Elastic mapreduce). I googled for help but nothing useful. I would like to know if this can be done or is there any alternative way to do this.

Processing live feed of logs from web server using Hadoop

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

EdgeOne AI 安全实战专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

我想使用Hadoop (Amazon Elastic mapreduce)处理来自我的web服务器的日志。我在谷歌上寻求帮助，但没有任何有用的东西。我想知道这是否可以做到，或者有没有其他方法可以做到这一点。

问使用Hadoop处理来自web服务器的日志的实时馈送
EN

回答 4

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问使用Hadoop处理来自web服务器的日志的实时馈送EN

回答 4

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问使用Hadoop处理来自web服务器的日志的实时馈送
EN