Skip to content

Decoupled weight decay #164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 30, 2019
Merged

Decoupled weight decay #164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
93450a2
Add decoupled weight decay optimizers and helper class for optimizer
PhilJd Apr 9, 2019
821ac69
Adapt README.md. Fix broken link to keras_utils.
PhilJd Apr 11, 2019
ec55b29
Remove TF private API calls.
PhilJd Apr 11, 2019
aa3d7b0
Add imports to __init__.py
PhilJd Apr 17, 2019
9ebc02b
Fix indentation of comments, remove call to tf.test.main from optimiz…
PhilJd Apr 18, 2019
98a42c6
Move optimizer_test_base into weight_decay_test for now. In the optim…
PhilJd Apr 25, 2019
077924c
Delete optimizer_test_base.py Remove keras object registration in the
PhilJd Apr 29, 2019
f8b6bdf
Merge branch 'master' into philjd/decaoupled_weight_decay
PhilJd Apr 30, 2019
d6ccae6
Fix code formatting via patch file.
PhilJd Apr 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions tensorflow_addons/optimizers/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ py_library(
"__init__.py",
"lazy_adam.py",
"moving_average.py",
"weight_decay_optimizers.py",
],
srcs_version = "PY2AND3",
deps = [
Expand Down Expand Up @@ -40,3 +41,16 @@ py_test(
":optimizers",
],
)

py_test(
name = "weight_decay_optimizers_test",
size = "small",
srcs = [
"weight_decay_optimizers_test.py",
],
main = "weight_decay_optimizers_test.py",
srcs_version = "PY2AND3",
deps = [
":optimizers",
],
)
7 changes: 5 additions & 2 deletions tensorflow_addons/optimizers/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,23 @@
|:---------- |:------------- |:--------------|
| lazy_adam | SIG-Addons | [email protected] |
| moving_average | Dheeraj R. Reddy | [email protected] |
| weight_decay_optimizers | Phil Jund | [email protected] |


## Components
| Submodule | Optimizer | Reference |
|:----------------------- |:---------------------- |:---------|
|:--------- |:---------- |:---------|
| lazy_adam | LazyAdam | https://arxiv.org/abs/1412.6980 |
| moving_average | MovingAverage | |
| weight_decay_optimizers | SGDW, AdamW, extend_with_decoupled_weight_decay | https://arxiv.org/pdf/1711.05101.pdf |


## Contribution Guidelines
#### Standard API
In order to conform with the current API standard, all optimizers
must:
* Inherit from either `keras.optimizer_v2.OptimizerV2` or its subclasses.
* [Register as a keras global object](https://github.com/tensorflow/addons/blob/master/tensorflow_addons/utils/python/keras_utils.py)
* [Register as a keras global object](https://github.com/tensorflow/addons/blob/master/tensorflow_addons/utils/keras_utils.py)
so it can be serialized properly.
* Add the addon to the `py_library` in this sub-package's BUILD file.

Expand Down
4 changes: 4 additions & 0 deletions tensorflow_addons/optimizers/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,8 @@
from __future__ import print_function

from tensorflow_addons.optimizers.lazy_adam import LazyAdam
from tensorflow_addons.optimizers.weight_decay_optimizers import AdamW
from tensorflow_addons.optimizers.weight_decay_optimizers import SGDW
from tensorflow_addons.optimizers.weight_decay_optimizers import (
extend_with_decoupled_weight_decay)
from tensorflow_addons.optimizers.moving_average import MovingAverage
Loading