blocks|key|1998202|text|结构化流是实时流处理引擎吗？|type|blockquote|depth|inlineStyleRanges|entityRanges|data|1998203|TL；博士No.或者是的。取决于“实时流处理引擎”的定义。|unstyled|offset|length|style|BOLD|1998204|到目前的2.3.0-SNAPSHOT+(当前的主控)，结构化流媒体使用的是微批，似乎没有任何迹象表明它在未来的版本中会有所不同。|1998205|深入研究结构化流的流查询引擎|1998206|StreamExecution (流查询的执行环境)+启动单独的执行线程。，用于检查新记录是否可用。|1998207|一旦启动，microBatchThread+(它是一个常规的java.lang.Thread对象)执行runBatches那个开始执行每个触发间隔。。|CODE|1998208|通过代码，可以发现所做的流查询的内部执行引擎，对每个触发器进行批处理。|1998209|我的理解是，就微配料而言，没有什么真正的改变。它在星火流中是这样的，在结构化流中也是如此。|1998210|无耻的插头:你可能想探索更多的细节，阅读我的关于结构化流媒体的手册，这正是我写的目的，以了解非常低层次的细节。欢迎评论。|1998211|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/apache/spark|1|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala|2|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L247|3|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L211|4|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L294|5|https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-StreamExecution.html^0|0|0|3|0|4|E|N|2|0|0|0|0|F|1|R|A|2|0|5|G|U|G|1F|A|3|1R|B|4|0|9|E|0|0|K|D|5|0^^$0|@$1|2|3|4|5|6|7|1F|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|1G|8|@$E|1H|F|1I|G|H]]|9|@]|A|$]]|$1|I|3|J|5|D|7|1J|8|@$E|1K|F|1L|G|H]]|9|@$E|1M|F|1N|1|1O]]|A|$]]|$1|K|3|L|5|D|7|1P|8|@]|9|@]|A|$]]|$1|M|3|N|5|D|7|1Q|8|@]|9|@$E|1R|F|1S|1|1T]|$E|1U|F|1V|1|1W]]|A|$]]|$1|O|3|P|5|D|7|1X|8|@$E|1Y|F|1Z|G|Q]|$E|20|F|21|G|Q]]|9|@$E|22|F|23|1|24]|$E|25|F|26|1|27]]|A|$]]|$1|R|3|S|5|D|7|28|8|@$E|29|F|2A|G|H]]|9|@]|A|$]]|$1|T|3|U|5|D|7|2B|8|@]|9|@]|A|$]]|$1|V|3|W|5|D|7|2C|8|@]|9|@$E|2D|F|2E|1|2F]]|A|$]]|$1|X|3|-4|5|D|7|2G|8|@]|9|@]|A|$]]]|Y|$Z|$5|10|11|12|A|$13|14]]|15|$5|10|11|12|A|$13|16]]|17|$5|10|11|12|A|$13|18]]|19|$5|10|11|12|A|$13|1A]]|1B|$5|10|11|12|A|$13|1C]]|1D|$5|10|11|12|A|$13|1E]]]]

<blockquote>
 Is Structured Streaming real-time stream processing engine?
</blockquote>

TL;DR No. Or yes. Depends on definition of "real-time stream processing engine".

Up to the 2.3.0-SNAPSHOT (current <a href="https://github.com/apache/spark" rel="noreferrer">master</a>), Structured Streaming uses micro-batches and nothing seems to suggest it's going to be different in the versions to come.

<h2>Deep Dive Into Structured Streaming's Streaming Query Engine</h2>

<a href="https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala" rel="noreferrer">StreamExecution</a> (the execution environment for a streaming query) <a href="https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L247" rel="noreferrer">starts a separate thread of execution</a> that checks whether new records are available.

Once started, the <code>microBatchThread</code> (which is a regular Java's <code>java.lang.Thread</code> object) executes <a href="https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L211" rel="noreferrer">runBatches</a> that <a href="https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L294" rel="noreferrer">starts execution every trigger interval</a>.

Going through the code reveals the internal execution engine for streaming queries that does batches every trigger.

My understanding is that nothing has really changed as far as micro-batching is concerned. It was like this in Spark Streaming and is used in Structured Streaming too.

<hr>

Shameless plug: You may want to explore the subject in more details reading <a href="https://jaceklaskowski.gitbooks.io/spark-structured-streaming/spark-sql-streaming-StreamExecution.html" rel="noreferrer">my gitbook about Structured Streaming</a> which I'm writing exactly for the purpose to understand the very low-level details. Comments welcome.

blocks|key|1956085|text|在上一次火花峰会简介+(SF+2017年6月)中，他们谈到了“连续管道”和一种新的执行模型，它没有微批，其延迟值小于1ms+(而不是今天可能的10-100+is+)，参见幻灯片35和火花-20928。|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|1956086|但目标版本是2.3.0.|1956087|entityMap|0|LINK|mutability|MUTABLE|url|https://spark-summit.org/2017/events/expanding-apache-spark-use-cases-in-22-and-beyond/|1|https://issues.apache.org/jira/browse/SPARK-20928^0|19|6|4|6|0|2J|8|1|0|6|6|0^^$0|@$1|2|3|4|5|6|7|R|8|@$9|S|A|T|B|C]]|D|@$9|U|A|V|1|W]|$9|X|A|Y|1|Z]]|E|$]]|$1|F|3|G|5|6|7|10|8|@$9|11|A|12|B|C]]|D|@]|E|$]]|$1|H|3|-4|5|6|7|13|8|@]|D|@]|E|$]]]|I|$J|$5|K|L|M|E|$N|O]]|P|$5|K|L|M|E|$N|Q]]]]

In the last <a href="https://spark-summit.org/2017/events/expanding-apache-spark-use-cases-in-22-and-beyond/" rel="nofollow noreferrer">Spark Summit's intro</a> (SF June 2017), they talked about "Continuous pipeline" and a new execution model without microbatches with checkpointing for latency &lt; 1ms (instead of 10-100ms that is possible today), see slide 35 and <a href="https://issues.apache.org/jira/browse/SPARK-20928" rel="nofollow noreferrer">Spark-20928</a>. 

But the targeted version is 
2.3.0.

We know Flink is a really real time streaming processing engine, which can deal with records just when they arrive, and we also know that spark streaming is a micro-batch streaming processing engine.

However we also know that spark released the structured-streaming, how about it? Is it also a really real time streaming processing engine just like Flink ,which can deal with the record immediately when it arrives instead of micro-batch or still using a micro-batch mode?

Is Structured Streaming real-time stream processing engine?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云AI代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

功能1上新10个字符

功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符。

功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符。

功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符

功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符

功能4上新

文章&问答评论现已支持表情

全新交互，全新视觉，新增快捷键、悬浮工具栏、高亮块等功能并同时优化现有功能，全面提升创作效率和体验

社区富文本编辑器全新改版！诚邀体验～ 

精选全网热门MCP server，让你的AI更好用 🚀

💥开发者 MCP广场重磅上线！

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

我们知道Flink是一个真正的实时流处理引擎，它可以在记录到达时处理记录，我们也知道火花流是一个微批量流处理引擎。然而，我们也知道星星之火发布了结构化的流媒体，它怎么样？它是否也是一个真正的实时流处理引擎，就像Flink一样，它可以在记录到达时立即处理记录，而不是微批处理，或者仍然使用微批处理模式？

问结构化流是实时流处理引擎吗？
EN

回答 2

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问结构化流是实时流处理引擎吗？EN

回答 2

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问结构化流是实时流处理引擎吗？
EN