blocks|key|1525975|text|我在reddit上发布了这个问题，得到了很多有趣的答案。任何对这个问题感兴趣的人都可以在这里看一看：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1525976|https://www.reddit.com/r/MachineLearning/comments/mfca0p/d_什么_这个_最简单_最多_轻量级_但是/|offset|length|1525977|无论如何，新的答案将是非常感谢的。由于这是一个非常固执己见的主题，不同的人是有意义的，可以互相补充。|entityMap|0|LINK|mutability|MUTABLE|url|https://www.reddit.com/r/MachineLearning/comments/mfca0p/d_whats_the_simplest_most_lightweight_but/^0|0|0|27|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|P|8|@]|9|@$D|Q|E|R|1|S]]|A|$]]|$1|F|3|G|5|6|7|T|8|@]|9|@]|A|$]]]|H|$I|$5|J|K|L|A|$M|N]]]]

I posted this question on reddit and got lots of very interesting answers. Anyone interested in this subject can take a look here:
<a href="https://www.reddit.com/r/MachineLearning/comments/mfca0p/d_whats_the_simplest_most_lightweight_but/" rel="nofollow noreferrer">https://www.reddit.com/r/MachineLearning/comments/mfca0p/d_whats_the_simplest_most_lightweight_but/</a>
Anyway, new answers will be much appreciated. As this is a very opinionated subject, different anwers make sense and may complement each other.

I know this has been asked many times and in many different ways. And there are tons of blog posts, articles, videos and courses addressing this and comparing hundreds of tools, libraries, frameworks… And that’s part of my problem: I am facing so many options that I feel like Buridan’s ass, dying of starvation for not knowing what to do.
Although I don’t want to write too much, I need to speak a little about our situation, in order to put the question in our context.
Our Team
Our Team is small. We have only four people, which could be qualified as beginner data scientists. One of us has a profile that is a little bit more “engineer”, so data engineer could be more suitable for him. Anyway, we don’t have much experience, neither in Python Projects nor in Machine Learning. What we have is passion and love for ML!
For a couple of years, we have been functioning with SAS, but now we plan to change to the Python landscape, as it is much more vivid and exciting. In the last year, we have made two projects in Python, but without using any good practices at all. Every step was made by hand and prone to error, models were neither monitored nor even deployed (they only were used for making some batch predictions), projects were not properly structured, documentation was painful…
So we know that we need to change it before it becomes unmanageable.
We don’t expect the size of the team to grow fast. Let’s say in a couple of years we can expect 10-12 people working with us (the organization knows the importance of Machine Learning, but economic issues can be an obstacle).
Our Projects
For the moment, we have only made “classical” Machine Learning. I mean: no Deep Learning. We have used Pandas and Scikit-Learn, XGBoost, etc. And only in Batch mode. But we expect it to change in less than a year, because we will need to train an image classifier to detect anomalies in customs packages, so it will need to be:
Trained using a deep learning convolutional network.
Integrated with other applications (that are coded in Java) and fast (real-time).
Other change we expect is to need more distributed computing, as we will need to manage some huge databases that simply do not fit in a pandas dataframe. This are the most important challenges we face.
Our Company
We work for a big company, which also imposes some restrictions to us. Mainly:
We do not have budget to spend in MLOps solutions, so everything has to be open and free.
We won’t hire data scientist / data engineers for the moment.
There are some tools, uses by other teams, that we should use as part of the MLOps stack, although they are not the best in the class.
Regarding the last item, a short list of this set of restrictions is the following:
<ul>
<li>We have a Cloudera Express installation. It’s the most basic and cheaper Cloudera option, so it does not come with any tool for Machine Learning management. It only gives to us HDFS, Impala, Spark and a set of nodes to run Python scripts on them.
</li>
<li>We have Control-M as the orchestrator and workflow manager tool.
</li>
<li>We have DataStage as the ETL tool.
</li>
<li>We use SVN as the code version system (yes, no git).
</li>
<li>We deploy our projects using a very simplified and self-made version of Docker. It’s a little bit awkward and I think that, if we push a little bit, we could convince the organization to let us use Docker. But if Docker is reachable, Kubernetes is out of our capabilities.
</li>
<li>We have Jenkins for CI.
</li>
<li>We have Visual Studio Code professional licenses.
</li>
</ul>
Toolset
With this premises, I have two different and opposed concerns or even fears.
<ul>
<li>Fear of not using enough tools and good practices and arriving in a couple of years to a state where we cannot manage our own code, project and models.
</li>
<li>Fear of using so many tools that they impose a burden our small team cannot bear.
</li>
</ul>
It’s clear that we need some MLOps, but how much, I don’t know. I will review some things I have been reading, and I hope you can help me choosing the right tools.
Python Programming
It looks like we will program using Visual Studio. We will use a remote interpreter, because we will run things on the Cloudera Nodes, although we will program locally and integrate the code with a SVN repository.
Do we need tools for standardizing our code, like PyLint, Flake8, MyPy or Black? Would you recommend any of those?
CI and Deployment
We will use Jenkins. For deployment of our code, is Docker a no brainer, a minimum standard? I tend to think so from what I read, but I’d like to be sure and to have good arguments.
Do we need more tools?
Project Scaffolding
I have been reading about PyScaffold, CookieCutter and, best of all (from my point of view), Kedro. I think we will stick to Kedro template, because it offers much more functionality, and I like to think of each project as a set of pipelines to be run. What do you think of Kedro?
Documentation
Would you recommend having separate documents, or generating the documentation from the projects, using Sphinx or another similar tool? I tend to prefer the second option, because the first one very likely tend to generate obsolete docs. But I don’t know if the “burden” of the second is too big, and if the generated docs can suffice for a typical ML project.
Project registry
Is there any tool that could be used as a “project registry”, like a simple web app where we could navigate through our projects, read the docs and thinks like that? I don’t know. If not, the registry will be the SVN repo with all our projects as folders, and that’s all.
Data Exploration and Preparation
I think that Matplotlib, Seaborn and Pandas should suffice, and when things go big, we should use PySpark, Scala or even plain SQL in Impala. However, I know Dask exists, and newer tools like Koalas or Vaex. What do you think?
For creating data transformation pipelines, we will use Kedro, although there are lots of tools that look interesting, like Dagster.
When we enter the “deep learning” realm, can we keep using the same tools? Should we use another framework like TFX? I’d prefer not, cause learning one framework is hard, and two is worse. If a solution is valid for all our projects it’s better. Or TFX is valid for “classical” ML and Deep Learning?
Tests
I think unit testing can be too much burden for us. But I have come to Great Expectations library and think it’s well suited for ML projects. Would you recommend it as an important part of our MLOps stack?
By the way, there is a Kedro-Great Expectations plugin, so we could benefit from that.
Feature Store
Is it really needed, especially considering our team size and experience? If so, I have read about Feast and Snorkel.
Data Versioning
Is it really needed, especially considering our team size and experience? If so, I have read about DVC.
Experimenting
I think it’s an important piece, although I wonder if we really need a tool or we could use our own standard of reports and artifacts to follow what we have tried. But the risk that it goes unmanageable is high.
Kedro has a journal, I don’t know if it can suffice. Also it has a Kedro-MLFlow plugin, so that we could benefit from using MLFlow as the experiment tool.
I have also read about Guild, that seems really lightweigh and easy. I don’t know much more.
Training
I developed my own library for doing nested cross validation and, with the same function:
Optimizing hyperparameters (of model and pipeline).
Generating a report of the training to assess the quality of the model.
It’s build on top of Skopt. I did it pip installable, it’s here:
<a href="https://github.com/JaimeArboleda/nestedcvtraining" rel="nofollow noreferrer">https://github.com/JaimeArboleda/nestedcvtraining</a>
So we plan to use it with the typical models like XGBoost, LightGBM and all Scikit-Learn. And when we need other frameworks like TensorFlow or Keras, we will see.
Model Registry
I think it’s an important piece, although I don’t know if we even could build our own with an standard database. If not, MLFlow seems a mature option.
Model serving
I am not sure if it’s included in the previous point or not. Anyway, I have read about Streamlint and FastAPI. Would you recommend any of those?
Is Apache Kafka needed for real time predictions?
Visualization
With this I mean sharing with the organization basic web apps with customizable plots, explainable predictions and things like that. I have read about panel, which has the ability of transform a Jupyter Notebook into a simple web app. It might be interesting.
Model monitoring
Is there a good free tool for monitoring the models and detecting loss of accuracy, data drift and things like that? Or we should better generate our own script of monitoring to be run periodically?
BigData
As I said before, we plan to use mainly Spark when need.
I know it’s a lot of info. Maybe I have overcomplicated myself and I should use only 20% of what I think I should. Or maybe not. I have no idea. Any help will be GREATLY appreciated. Thanks in advance.

What’s the simplest, most lightweight but complete and 100% open source MLOps toolkit?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云AI代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

功能1上新10个字符

功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符。

功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符。

功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符

功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符

功能4上新

文章&问答评论现已支持表情

全新交互，全新视觉，新增快捷键、悬浮工具栏、高亮块等功能并同时优化现有功能，全面提升创作效率和体验

社区富文本编辑器全新改版！诚邀体验～ 

精选全网热门MCP server，让你的AI更好用 🚀

💥开发者 MCP广场重磅上线！

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

腾讯云开发者社区有与腾讯云相关的官方技术问答，也引入了来自Stack Overflow的优质外文问答。找寻与 什么是最简单、最轻量级但完整且100%的开源MLOps工具包？ 相关的问答，快来腾讯云开发者社区看看吧！

问什么是最简单、最轻量级但完整且100%的开源MLOps工具包？
EN

回答 1

Software Recommendation用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问什么是最简单、最轻量级但完整且100%的开源MLOps工具包？EN