Clean up

Author: seanpmorgan

tensorflow_addons/examples/optimizers_lazyadam.ipynb

@@ -76,11 +76,32 @@
       },
       "cell_type": "markdown",
       "source": [
-        "## Overview\n",
+        "# Overview\n",
         "\n",
         "This notebook will demonstrate how to use the lazy adam optimizer from the Addons package.\n"
       ]
     },
+    {
+      "metadata": {
+        "id": "bQwBbFVAyHJ_",
+        "colab_type": "text"
+      },
+      "cell_type": "markdown",
+      "source": [
+        "# LazyAdam\n",
+        "\n",
+        "> LazyAdam is a variant of the Adam optimizer that handles sparse updates moreefficiently.\n",
+        "    The original Adam algorithm maintains two moving-average accumulators for\n",
+        "    each trainable variable; the accumulators are updated at every step.\n",
+        "    This class provides lazier handling of gradient updates for sparse\n",
+        "    variables.  It only updates moving-average accumulators for sparse variable\n",
+        "    indices that appear in the current batch, rather than updating the\n",
+        "    accumulators for all indices. Compared with the original Adam optimizer,\n",
+        "    it can provide large improvements in model training throughput for some\n",
+        "    applications. However, it provides slightly different semantics than the\n",
+        "    original Adam algorithm, and may lead to different empirical results."
+      ]
+    },
     {
       "metadata": {
         "colab_type": "text",
@@ -145,10 +166,11 @@
       },
       "cell_type": "code",
       "source": [
-        "model = tf.keras.Sequential()\n",
-        "model.add(tf.keras.layers.Dense(64, input_shape=(784,), activation='relu', name='dense_1'))\n",
-        "model.add(tf.keras.layers.Dense(64, activation='relu', name='dense_2'))\n",
-        "model.add(tf.keras.layers.Dense(10, activation='softmax', name='predictions'))"
+        "model = tf.keras.Sequential([\n",
+        "    tf.keras.layers.Dense(64, input_shape=(784,), activation='relu', name='dense_1'),\n",
+        "    tf.keras.layers.Dense(64, activation='relu', name='dense_2'),\n",
+        "    tf.keras.layers.Dense(10, activation='softmax', name='predictions'),\n",
+        "])"
       ],
       "execution_count": 0,
       "outputs": []
@@ -223,11 +245,11 @@
       "metadata": {
         "id": "1Y--0tK69SXf",
         "colab_type": "code",
+        "outputId": "163a7751-e35b-4d9f-cc07-1f8580bdf6bf",
         "colab": {
           "base_uri": "https://localhost:8080/",
-          "height": 67
-        },
-        "outputId": "4a33a1c1-da98-4da9-b226-ee9af7b903d3"
+          "height": 68
+        }
       },
       "cell_type": "code",
       "source": [
@@ -236,14 +258,14 @@
         "results = model.evaluate(x_test, y_test, batch_size=128)\n",
         "print('Test loss = {0}, Test acc: {1}'.format(results[0], results[1]))"
       ],
-      "execution_count": 19,
+      "execution_count": 9,
       "outputs": [
         {
           "output_type": "stream",
           "text": [
             "Evaluate on test data:\n",
-            "10000/10000 [==============================] - 0s 28us/sample - loss: 0.1149 - accuracy: 0.9762\n",
-            "Test loss = 0.11493133022264228, Test acc: 0.9761999845504761\n"
+            "10000/10000 [==============================] - 0s 21us/sample - loss: 0.0884 - accuracy: 0.9752\n",
+            "Test loss = 0.08840992146739736, Test acc: 0.9751999974250793\n"
           ],
           "name": "stdout"
         }