blocks|key|1624323|text|正如人们所说，“唯一的区别是你在库伯内特斯/集装箱”。现实是，这意味着在实际操作上有几件大事：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1624324|上面链接的舵图是一个玩具。|unordered-list-item|1624325|它建立香草hadoop+(即不是HDP或CDH)|1624326|它对HA没有影响|1624327|它不做kerberos|1624328|1624329|​|1624330|您必须管理您自己的卷|1624331|如果您在公共云上运行，这不是什么大问题，因为您可以动态地获得存储|1624332|1624333|1624334|因此，除非您只是想要一个超级轻量级的hdfs部署，或者您很乐意/愿意构建您自己部署的更复杂的k8s+hadoop部署，或者您愿意花钱购买一个具有hadoop支持的第三方kubernetes堆栈(例如robin.io)，我想说的是，总的来说，在k8s上运行它是不值得的。|1624335|注意，如果/当hadoop供应商制作自己的运算符时，这种情况可能会改变。|offset|length|1624336|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/operator-framework/operator-sdk^0|0|1|1|1|1|0|0|1|1|0|0|0|L|3|0|0^^$0|@$1|2|3|4|5|6|7|17|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|18|8|@]|9|@]|A|$]]|$1|E|3|F|5|D|7|19|8|@]|9|@]|A|$]]|$1|G|3|H|5|D|7|1A|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|1B|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|1C|8|@]|9|@]|A|$]]|$1|L|3|M|5|6|7|1D|8|@]|9|@]|A|$]]|$1|N|3|O|5|D|7|1E|8|@]|9|@]|A|$]]|$1|P|3|Q|5|D|7|1F|8|@]|9|@]|A|$]]|$1|R|3|-4|5|6|7|1G|8|@]|9|@]|A|$]]|$1|S|3|M|5|6|7|1H|8|@]|9|@]|A|$]]|$1|T|3|U|5|6|7|1I|8|@]|9|@]|A|$]]|$1|V|3|W|5|6|7|1J|8|@]|9|@$X|1K|Y|1L|1|1M]]|A|$]]|$1|Z|3|-4|5|6|7|1N|8|@]|9|@]|A|$]]]|10|$11|$5|12|13|14|A|$15|16]]]]

As people have said, "the only difference is you are in kubernetes/container". The reality is that means a couple of huge things in terms of actual operation:

<ul>
<li>The helm chart linked above is a toy.

<ul>
<li>It builds vanilla hadoop (i.e. not HDP or CDH)</li>
<li>It doesn't do HA namenodes</li>
<li>It doesn't do kerberos</li>
</ul></li>
<li>You have to manage your own volumes

<ul>
<li>If you are running on a public cloud this isn't a super big deal, as you can dynamically get storage</li>
</ul></li>
</ul>

So unless you just want a super lightweight hdfs deployment, or you are comfortable/willing to build out your own deployment of a more sophisticated k8s hadoop deployment, or you are willing to pay for a 3rd party kubernetes stack with hadoop support (e.g. robin.io), I would say that in general it is not worth running on k8s at this point. 

Note that if/when the hadoop vendors make their own <a href="https://github.com/operator-framework/operator-sdk" rel="noreferrer">operator</a>, this might change.

blocks|key|2467283|text|标准hadoop只是一个hadoop，它有map-+just、spark等，并得到HDFS的支持。|type|unordered-list-item|depth|inlineStyleRanges|entityRanges|data|2467284|在kubernetes上的Hadoop只是上面的标准Hadoop，但是运行在Kubernetes上。|2467285|在K8S上使用Hadoop的情况下，您可以得到kubernetes通常比传统基础设施提供的所有好处。|unstyled|2467286|还有一个头盔图：|2467287|https://github.com/helm/charts/tree/master/stable/hadoop|offset|length|2467288|entityMap|0|LINK|mutability|MUTABLE|url^0|0|0|0|0|0|1K|0|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|U|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|V|8|@]|9|@]|A|$]]|$1|G|3|H|5|F|7|W|8|@]|9|@]|A|$]]|$1|I|3|J|5|F|7|X|8|@]|9|@$K|Y|L|Z|1|10]]|A|$]]|$1|M|3|-4|5|F|7|11|8|@]|9|@]|A|$]]]|N|$O|$5|P|Q|R|A|$S|J]]]]

<ul>
<li>Standard hadoop is just hadoop with map-reduce , spark etc and backed by HDFS</li>
<li>Hadoop on kubernetes is just standard Hadoop as above , but running on Kubernetes </li>
</ul>

In case of Hadoop on K8S , you get all the benefits that kubernetes usually offers over traditional infrastructure.

There is a helm chart as well:

<a href="https://github.com/helm/charts/tree/master/stable/hadoop" rel="noreferrer">https://github.com/helm/charts/tree/master/stable/hadoop</a>

blocks|key|1624360|text|您可能需要考虑查看这图表集--简而言之，这是一组头盔图表，用于在K8s集群上拆分Hadoop服务。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1624361|举几个亮点：|1624362|支持HA+namenode|unordered-list-item|1624363|Kerberos的支持|1624364|支持k8s持久性田鼠|1624365|支持数据卷|1624366|等|1624367|希望这能有所帮助。干杯|1624368|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/apache-spark-on-k8s/kubernetes-HDFS/tree/master/charts^0|9|1|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@]|9|@$A|11|B|12|1|13]]|C|$]]|$1|D|3|E|5|6|7|14|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|15|8|@]|9|@]|C|$]]|$1|I|3|J|5|H|7|16|8|@]|9|@]|C|$]]|$1|K|3|L|5|H|7|17|8|@]|9|@]|C|$]]|$1|M|3|N|5|H|7|18|8|@]|9|@]|C|$]]|$1|O|3|P|5|H|7|19|8|@]|9|@]|C|$]]|$1|Q|3|R|5|6|7|1A|8|@]|9|@]|C|$]]|$1|S|3|-4|5|6|7|1B|8|@]|9|@]|C|$]]]|T|$U|$5|V|W|X|C|$Y|Z]]]]

you might want to consider looking at <a href="https://github.com/apache-spark-on-k8s/kubernetes-HDFS/tree/master/charts" rel="nofollow noreferrer">this</a> set of charts
In short, this is a collection of helm charts to spin up Hadoop services on K8s cluster.

to mention a few highlights:

<ul>
<li>support HA namenode</li>
<li>support of Kerberos</li>
<li>support k8s persistent vols</li>
<li>support of data volumes</li>
<li>etc</li>
</ul>

Hope this helps. Cheers

blocks|key|1624410|text|+Hadoop+:+Hadoop提供HDFS作为分布式文件系统，其中存储资源集群作为单个文件或文件系统呈现给应用程序堆栈。HDFS用于访问无法存储在单个硬盘上的大量数据。Hadoop通过复制实现数据可靠性管理，这样应用程序就不必担心存储堆栈语义。|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|1624411|总之，Hadoop将多个硬盘转换为单个卷。这样可以实现非常大的数据存储。Hadoop使用YARN+(另一个资源协商器)作为计算节点全局调度程序。Hadoop生态系统非常大，包括星火、动物园管理员、Hbase、Hive和许多其他大数据、分析和机器学习的解决方案。|1624412|+Kubernetes+:+Kubernetes是一个容器编排平台。容器取代VM，因为VM利用更聪明的和整个客户操作系统来隔离单个主机OS环境中的节点。这意味着VM创建了不必要的复制，并且通过在彼此之上运行多个操作系统而降低了效率。这种效率损失是明显的云环境运行100's的虚拟机。容器仅通过调用所需的功能来解决复制和模拟OS级别隔离的问题，而不需要每个容器的整个OS映像。这是通过linux功能实现的。Kubernetes提供了通过2级隔离来管理多个容器的方法。二级隔离是通过包含多个容器的吊舱提供的。Kubernetes还通过容器复制提供负载平衡和故障安全部署。|1624413|总之，容器提供操作系统级隔离，使单个操作系统看起来像许多OS。这使得资源的高效使用和应用程序的并行运行成为可能。Kubernetes能够管理许多容器。通常，Kubernetes和containers接口云存储平台(如S3+)，其中S3是一个对象存储，而不是Hadoop分布式文件系统。|1624414|Hadoop有或没有Kubernetes:虽然从根本上解决了Hadoop和Kubernetes不同的问题，但通过解决应用程序依赖和部署挑战，Kubernetes由于容器化的好处而获得了广泛的应用。Kubernetes和容器提供了大规模的并行性和可伸缩性。|1624415|Kubernetes是开源领域的最新进步，尽管Google已经使用了很多年，但是Hadoop是一个十年前的解决方案，缺乏一些现代化。|1624416|那么，我们需要使用Hadoop作为一个带有容器和Kubernetes的分布式文件系统吗？这取决于应用需求和价值主张需求。从技术上讲，使用Docker和Kubernetes运行Hadoop是可行的，但是整个生态系统缺乏顺畅的集成。最近的几个开源项目试图解决这个问题，但是如果Hadoop将是一个未来的解决方案，或者我们需要一个新的/不同的分布式文件系统平台，那么时间就会告诉我们。目前，我们有许多解决方案，如云存储平台、Kafka、Elastic/logstash等，它们在特定领域以自己的优势解决了存储可伸缩性问题，而Hadoop和整个Hadoop生态系统仍然是一个占主导地位的大数据平台。|1624417|entityMap^0|0|9|0|0|0|D|0|0|0|L|0|0|0^^$0|@$1|2|3|4|5|6|7|T|8|@$9|U|A|V|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|W|8|@]|D|@]|E|$]]|$1|H|3|I|5|6|7|X|8|@$9|Y|A|Z|B|C]]|D|@]|E|$]]|$1|J|3|K|5|6|7|10|8|@]|D|@]|E|$]]|$1|L|3|M|5|6|7|11|8|@$9|12|A|13|B|C]]|D|@]|E|$]]|$1|N|3|O|5|6|7|14|8|@]|D|@]|E|$]]|$1|P|3|Q|5|6|7|15|8|@]|D|@]|E|$]]|$1|R|3|-4|5|6|7|16|8|@]|D|@]|E|$]]]|S|$]]

Hadoop: Hadoop provides HDFS as a distributed file system, where cluster of storage resources are presented to application stack as a single file or file system. HDFS API's used to access large set of data that is not feasible to store on a single hard disk. Hadoop includes data reliability management through replication so that applications don't have to worry about storage stack semantics.

In summary - Hadoop converts multiple hard disks into single volume. This enables very large data storage. Hadoop uses YARN (Yet another resource negotiator) as a compute node global scheduler. Hadoop ecosystem is very large and includes Spark, Zookeeper, Hbase, Hive and many other solutions towards big data, analytics and machine learning.

Kubernetes: Kubernetes is a container orchestration platform. Container replaces VM's, because VM's utilize hyper-wiser's and entire guest OS to isolate compute nodes within a single host OS environment. That means VM's create unnecessary duplication and are less efficient by running multiple OS on top of each other. This efficiency loss is obvious in cloud environment running 100's of thousand VMs. Containers solves this problem of duplication and emulate OS level isolation by only invoking required functionality without requiring entire OS image for each container. This is achieved through linux etcd functionality. Kubernetes provides means to manage multiple containers through 2 level isolation. 2nd level isolation is provided through PODS that contain multiple containers. Kubernetes also provide load balancing and fail safe deployment through container replication.

In summary - Containers provide OS level isolation to make single OS look like many OS's. This enables efficient use of resources and parallel running of applications. Kubernetes enable management of many containers. Generally Kubernetes and containers interface cloud storage platforms like S3, where S3 is a object store and not like Hadoop distributed file system. 

Hadoop with or without Kubernetes: While fundamentally Hadoop and Kubernetes solves different problems, Kubernetes have gained popularity due to containerization benefits by solving application dependency and deployment challenges. Kubernetes and containers provides massive parallelism and scalability.

Kubernetes is fairly recent advancement in open source even-though Google have been using this for many years, on the other hand Hadoop is a decade old solution and lacks some of the modernization.

So do we need to use Hadoop as a distributed file system with Containers and Kubernetes? It really depends on application requirements and value proposition needs. Technically it's feasible to run Hadoop with Docker and Kubernetes, however the entire ecosystem lacks smooth integration. Recent couple of open source projects try to solve this problem however if Hadoop will be a going forward solution or we need a new/different distributed file system platform only time will tell. Currently we have many solutions like Cloud storage platforms, Kafka, Elastic-search/logstash solves the storage scalability problem with their own strengths in specific areas while Hadoop and entire Hadoop ecosystem continue to be a dominant big data platform.

What is the difference between Hadoop on Kubernetes and the standard Hadoop ?
and what is the benefit from deploying Hadoop on Kubernetes ?

Hadoop on Kubernetes vs Standard Hadoop

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云AI代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

功能1上新10个字符

功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符。

功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符。

功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符

功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符

功能4上新

文章&问答评论现已支持表情

全新交互，全新视觉，新增快捷键、悬浮工具栏、高亮块等功能并同时优化现有功能，全面提升创作效率和体验

社区富文本编辑器全新改版！诚邀体验～ 

精选全网热门MCP server，让你的AI更好用 🚀

💥开发者 MCP广场重磅上线！

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

在Kubernetes上Hadoop和标准Hadoop有什么区别？在Kubernetes上部署Hadoop有什么好处？

问Kubernetes对标准Hadoop的Hadoop
EN

回答 4

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Kubernetes对标准Hadoop的HadoopEN

回答 4

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Kubernetes对标准Hadoop的Hadoop
EN