Decoupled weight decay

[PhilJd]

This PR ports the decoupled weight decay optimizers (SGDW, AdamW) to tensorflow 2.0, minus all v1 tests, as tensorflow_addons depends on tf-2.0 anyway.
Note that I factored out the testing code that is duplicated in most optimizer tests (also in base tensorflow) into optimizer_test_base. After this PR has been merged, I'd adapt the LazyAdam optimizer to inherit from OptimizerTestBase.

Sorry for the long delay!
Cheers,
Phil :)

Closes #24.

[facaiy]

@PhilJd Welcome, thanks for the PR, and your patience. I'll take a look later this week :-)

[mfojtak]

You probably need to add this

from tensorflow_addons.optimizers.weight_decay_optimizers import AdamWOptimizer

into tensorflow_addons/optimizers/__init__.py in order to expose those classes

[PhilJd]

Thanks! I've added the respective classes and functions to __init__.py :)

[facaiy]

Thanks for the PR, I'll take another look at weekend :-)

[facaiy]

Do you need it?

[PhilJd]

Thanks for the comments @facaiy, I've updated the PR!

[facaiy]

Nice work! Leave some questions for tf 2.0

[facaiy]

@PhilJd Phil, don't forget to run make code-format :-)

[facaiy]

@PhilJd Hi, Phil. Could you address all comments? Thank you for the high quality PR, can't wait to merge it :-)

[PhilJd]

@facaiy Thanks a lot for the comments! :)
I've addressed them all, except for the tests. I think someone put testing on the agenda for the next SIG call.
On top of that:

• I found additional documentation bugs, code samples were not compatible with tf-2.0.
• The last commit also fixes extend_with_weight_decay function doesn't exist?  tensorflow#26360.
• I've renamed AdamWOptimizer/SGDWOptimizer to AdamW/SGDW, which follows the keras naming scheme instead of tf-1.0 naming scheme
• AdamW and SGDW now have a note that the decay also needs to be scheduled, originally only the extend function and the base class had this note.
• I've updated the paper title to the title of the published version

By the way, I ran make code-format before, but it seems this doesn't touch comments ;)
Thanks again for taking a detailed look! I also hope we can merge soon :)

[facaiy]

very close :-)

[PhilJd]

Oops, I forgot to commit optimizer_test_base ;)
Thanks for the comments, they should be resolved with the latest commit!

[facaiy]

Looks great, thank you, PhilJ! Could you resolve the merge conflict with master branch?

By the way, you can put your name in the contact info list and code owner file if you'd like to maintain the module contributed by yourself.

[facaiy]

@seanpmorgan @WindQAQ Sean, Tzu-Wei, do you have any concerns about this change?

[seanpmorgan]

@seanpmorgan @WindQAQ Sean, Tzu-Wei, do you have any concerns about this change?

No, looks like a very nice PR.. Just needs to resolve conflicts and test IMO

[PhilJd]

Conflicts with the master are resolved and I've put my name as maintainer into the README ;)

[PhilJd]

Interesting, running make code-format locally doesn't complain.

[facaiy]

@PhilJd Phil, could you run make code-format again? And if you'd like, please add the line below

addons/.github/CODEOWNERS

Line 32 in 4118a11

/tensorflow_addons/losses/sparsemax*.py @AndreasMadsen

/tensorflow_addons/optimizers/weight_decay_optimizers*.py  @PhilJd 

[facaiy]

Please ping me when you get it done, and I'll merge it. Thanks for your patience :-)

[PhilJd]

I've applied the patch from the build server. I wasn't able to find out why the code-format doesn't complain locally. I've tried the clang format version the build task uses, wget https://llvm.org/svn/llvm-project/cfe/trunk/tools/clang-format/git-clang-format, but still no changes...

[facaiy]

Thanks!