Skip to content

Fix broken link in NovoGrad docstring #2096

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 16, 2020
Merged

Fix broken link in NovoGrad docstring #2096

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3bd380e
Fix broken link in NovoGrad docstring
seungjaeryanlee Aug 15, 2020
6f4b7f0
Fix typo in NovoGrad paper title
seungjaeryanlee Aug 15, 2020
360d8cc
Add missing parenthesis
seungjaeryanlee Aug 15, 2020
3fbd1ee
Remove unneeded wordy phrase
seungjaeryanlee Aug 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions tensorflow_addons/optimizers/novograd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,16 @@

@tf.keras.utils.register_keras_serializable(package="Addons")
class NovoGrad(tf.keras.optimizers.Optimizer):
"""The NovoGrad Optimizer was first proposed in [Stochastic Gradient
Methods with Layerwise Adaptvie Moments for training of Deep
Networks](https://arxiv.org/pdf/1905.11286.pdf)

NovoGrad is a first-order SGD-based algorithm, which computes second
moments per layer instead of per weight as in Adam. Compared to Adam,
NovoGrad takes less memory, and has been found to be more numerically
stable. More specifically we compute (for more information on the
computation please refer to this
[link](https://nvidia.github.io/OpenSeq2Seq/html/optimizers.html):
"""Optimizer that implements NovoGrad.

The NovoGrad Optimizer was first proposed in [Stochastic Gradient
Methods with Layerwise Adaptive Moments for training of Deep
Networks](https://arxiv.org/pdf/1905.11286.pdf) NovoGrad is a
first-order SGD-based algorithm, which computes second moments per
layer instead of per weight as in Adam. Compared to Adam, NovoGrad
takes less memory, and has been found to be more numerically stable.
(For more information on the computation please refer to this
[link](https://nvidia.github.io/OpenSeq2Seq/html/optimizers.html))

Second order moment = exponential moving average of Layer-wise square
of grads:
Expand Down