diff --git a/docs/api/paddle/matrix_power_cn.rst b/docs/api/paddle/linalg/matrix_power_cn.rst
similarity index 86%
rename from docs/api/paddle/matrix_power_cn.rst
rename to docs/api/paddle/linalg/matrix_power_cn.rst
index 210b41e61c9..c1f771a92f0 100644
--- a/docs/api/paddle/matrix_power_cn.rst
+++ b/docs/api/paddle/linalg/matrix_power_cn.rst
@@ -3,7 +3,7 @@
matrix_power
-------------------------------
-.. py:function:: paddle.matrix_power(x, n, name=None)
+.. py:function:: paddle.linalg.matrix_power(x, n, name=None)
计算一个或一批方阵的 ``n`` 次幂。
@@ -41,17 +41,17 @@ matrix_power
x = paddle.to_tensor([[1, 2, 3],
[1, 4, 9],
[1, 8, 27]], dtype='float64')
- print(paddle.matrix_power(x, 2))
+ print(paddle.linalg.matrix_power(x, 2))
# [[6. , 34. , 102.],
# [14. , 90. , 282.],
# [36. , 250., 804.]]
- print(paddle.matrix_power(x, 0))
+ print(paddle.linalg.matrix_power(x, 0))
# [[1., 0., 0.],
# [0., 1., 0.],
# [0., 0., 1.]]
- print(paddle.matrix_power(x, -2))
+ print(paddle.linalg.matrix_power(x, -2))
# [[ 12.91666667, -12.75000000, 2.83333333 ],
# [-7.66666667 , 8. , -1.83333333 ],
- # [ 1.80555556 , -1.91666667 , 0.44444444 ]]
\ No newline at end of file
+ # [ 1.80555556 , -1.91666667 , 0.44444444 ]]
diff --git a/docs/api/paddle/multi_dot_cn.rst b/docs/api/paddle/linalg/multi_dot_cn.rst
similarity index 97%
rename from docs/api/paddle/multi_dot_cn.rst
rename to docs/api/paddle/linalg/multi_dot_cn.rst
index 8dc63f4a419..e6200eecbdd 100755
--- a/docs/api/paddle/multi_dot_cn.rst
+++ b/docs/api/paddle/linalg/multi_dot_cn.rst
@@ -3,7 +3,7 @@
multi_dot
-------------------------------
-.. py:function:: paddle.multi_dot(x, name=None)
+.. py:function:: paddle.linalg.multi_dot(x, name=None)
Multi_dot是一个计算多个矩阵乘法的算子。
diff --git a/docs/guides/01_paddle2.0_introduction/basic_concept/amp_cn.ipynb b/docs/guides/01_paddle2.0_introduction/basic_concept/amp_cn.ipynb
new file mode 100644
index 00000000000..e5a5b2106b8
--- /dev/null
+++ b/docs/guides/01_paddle2.0_introduction/basic_concept/amp_cn.ipynb
@@ -0,0 +1,463 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "# 自动混合精度训练\n",
+ "\n",
+ "一般情况下，训练深度学习模型时使用的数据类型为单精度（FP32）。2018年，百度与NVIDIA联合发表论文：[MIXED PRECISION TRAINING](https://arxiv.org/pdf/1710.03740.pdf)，提出了混合精度训练的方法。混合精度训练是指在训练过程中，同时使用单精度（FP32）和半精度（FP16），其目的是相较于使用单精度（FP32）训练模型，在保持精度持平的条件下，能够加速训练。本文将介绍如何使用飞桨框架，实现自动混合精度训练。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 一、半精度浮点类型 FP16\n",
+ "\n",
+ "首先介绍半精度（FP16）。如图1所示，半精度（FP16）是一种相对较新的浮点类型，在计算机中使用2字节（16位）存储。在IEEE 754-2008标准中，它亦被称作binary16。与计算中常用的单精度（FP32）和双精度（FP64）类型相比，FP16更适于在精度要求不高的场景中使用。\n",
+ "\n",
+ "\n",
+ " ![missing](\"https://paddleweb-static.bj.bcebos.com/images/fp16.png\")
\n",
+ " 图 1. 半精度和单精度数据示意图\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 二、NVIDIA GPU的FP16算力\n",
+ "在使用相同的超参数下，混合精度训练使用半精度浮点（FP16）和单精度（FP32）浮点即可达到与使用纯单精度训练相同的准确率，并可加速模型的训练速度。这主要得益于英伟达推出的Volta及Turing架构GPU在使用FP16计算时具有如下特点：\n",
+ "- FP16可降低一半的内存带宽和存储需求，这使得在相同的硬件条件下研究人员可使用更大更复杂的模型以及更大的batch size大小。\n",
+ "- FP16可以充分利用英伟达Volta及Turing架构GPU提供的Tensor Cores技术。在相同的GPU硬件上，Tensor Cores的FP16计算吞吐量是FP32的8倍。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 三、使用飞桨框架实现自动混合精度\n",
+ "使用飞桨框架提供的API，``paddle.amp.auto_cast`` 和 ``paddle.amp.decorate`` 和 ``paddle.amp.GradScaler`` 能够实现自动混合精度训练（Automatic Mixed Precision，AMP），即在相关OP的计算中，根据一定的规则，自动选择FP16或FP32计算。飞桨的AMP为用户提供了两种模式：\n",
+ "- level=’O1‘：采用黑名名单策略的混合精度训练，使用FP16与FP32进行计算的OP列表可见该[文档](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/amp/Overview_cn.html)。\n",
+ "- level=’O2‘：纯FP16训练，除用户自定义黑名单中指定的OP和不支持FP16计算的OP之外，全部使用FP16计算。\n",
+ "\n",
+ "下面来看一个具体的例子，来了解如果使用飞桨框架实现混合精度训练。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "### 3.1 辅助函数\n",
+ "首先定义辅助函数，用来计算训练时间。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "\n",
+ "# 开始时间\n",
+ "start_time = None\n",
+ "\n",
+ "def start_timer():\n",
+ " # 获取开始时间\n",
+ " global start_time\n",
+ " start_time = time.time()\n",
+ "\n",
+ "def end_timer_and_print(msg):\n",
+ " # 打印信息并输出训练时间\n",
+ " end_time = time.time()\n",
+ " print(\"\\n\" + msg)\n",
+ " print(\"共计耗时 = {:.3f} sec\".format(end_time - start_time))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "### 3.2 构建一个简单的网络\n",
+ "\n",
+ "构建一个简单的网络，用于对比使用普通方法进行训练与使用混合精度训练的训练速度。该网络由三层 ``Linear`` 组成，其中前两层 ``Linear`` 后接 ``ReLU`` 激活函数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import paddle\n",
+ "import paddle.nn as nn\n",
+ "\n",
+ "class SimpleNet(nn.Layer):\n",
+ "\n",
+ " def __init__(self, input_size, output_size):\n",
+ " \n",
+ " super(SimpleNet, self).__init__()\n",
+ " self.linear1 = nn.Linear(input_size, output_size)\n",
+ " self.relu1 = nn.ReLU()\n",
+ " self.linear2 = nn.Linear(input_size, output_size)\n",
+ " self.relu2 = nn.ReLU()\n",
+ " self.linear3 = nn.Linear(input_size, output_size)\n",
+ "\n",
+ " def forward(self, x):\n",
+ "\n",
+ " x = self.linear1(x)\n",
+ " x = self.relu1(x)\n",
+ " x = self.linear2(x)\n",
+ " x = self.relu2(x)\n",
+ " x = self.linear3(x)\n",
+ "\n",
+ " return x"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "设置训练的相关参数，这里为了能有效的看出混合精度训练对于训练速度的提升，将 ``input_size`` 与 ``output_size`` 的值设为较大的值，为了使用GPU 提供的``Tensor Core`` 性能，还需将 ``batch_size`` 设置为 8 的倍数。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "W1110 18:42:02.362493 104 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1\n",
+ "W1110 18:42:02.367755 104 device_context.cc:465] device: 0, cuDNN Version: 7.6.\n"
+ ]
+ }
+ ],
+ "source": [
+ "epochs = 5\n",
+ "input_size = 4096 # 设为较大的值\n",
+ "output_size = 4096 # 设为较大的值\n",
+ "batch_size = 512 # batch_size 为8的倍数\n",
+ "nums_batch = 50\n",
+ "\n",
+ "train_data = [paddle.randn((batch_size, input_size)) for _ in range(nums_batch)]\n",
+ "labels = [paddle.randn((batch_size, output_size)) for _ in range(nums_batch)]\n",
+ "\n",
+ "mse = paddle.nn.MSELoss()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "### 3.3 使用默认的训练方式进行训练"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,\n",
+ " [1.24519622])\n",
+ "\n",
+ "默认耗时:\n",
+ "共计耗时 = 2.926 sec\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = SimpleNet(input_size, output_size) # 定义模型\n",
+ "\n",
+ "optimizer = paddle.optimizer.SGD(learning_rate=0.0001, parameters=model.parameters()) # 定义优化器\n",
+ "\n",
+ "start_timer() # 获取训练开始时间\n",
+ "\n",
+ "for epoch in range(epochs):\n",
+ " datas = zip(train_data, labels)\n",
+ " for i, (data, label) in enumerate(datas):\n",
+ "\n",
+ " output = model(data)\n",
+ " loss = mse(output, label)\n",
+ "\n",
+ " # 反向传播\n",
+ " loss.backward()\n",
+ "\n",
+ " # 训练模型\n",
+ " optimizer.step()\n",
+ " optimizer.clear_grad()\n",
+ "\n",
+ "print(loss)\n",
+ "end_timer_and_print(\"默认耗时:\") # 获取结束时间并打印相关信息"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "### 3.4 使用AMP训练模型\n",
+ "\n",
+ "在飞桨框架中，使用自动混合精度训练，需要进行四个步骤：\n",
+ "\n",
+ "- Step1： 定义 ``GradScaler`` ，用于缩放 ``loss`` 比例，避免浮点数下溢\n",
+ "- Step2： 使用 ``decorate`` 在level=’O1‘模式下不做任何处理，无需调用该api，在level=’O2‘模式下，将网络参数从FP32转换为FP16\n",
+ "- Step3： 使用 ``auto_cast`` 用于创建AMP上下文环境，该上下文中自动会确定每个OP的输入数据类型（FP16或FP32）\n",
+ "- Step4： 使用 Step1中定义的 ``GradScaler`` 完成 ``loss`` 的缩放，用缩放后的 ``loss`` 进行反向传播，完成训练\n",
+ "\n",
+ "\n",
+ "采用level=’O1‘模式训练："
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,\n",
+ " [1.24815702])\n",
+ "\n",
+ "使用AMP-O1模式耗时:\n",
+ "共计耗时 = 1.294 sec\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = SimpleNet(input_size, output_size) # 定义模型\n",
+ "\n",
+ "optimizer = paddle.optimizer.SGD(learning_rate=0.0001, parameters=model.parameters()) # 定义优化器\n",
+ "\n",
+ "# Step1：定义 GradScaler，用于缩放loss比例，避免浮点数溢出\n",
+ "scaler = paddle.amp.GradScaler(init_loss_scaling=1024)\n",
+ "\n",
+ "start_timer() # 获取训练开始时间\n",
+ "\n",
+ "for epoch in range(epochs):\n",
+ " datas = zip(train_data, labels)\n",
+ " for i, (data, label) in enumerate(datas):\n",
+ "\n",
+ " # Step2：创建AMP上下文环境，开启自动混合精度训练\n",
+ " with paddle.amp.auto_cast():\n",
+ " output = model(data)\n",
+ " loss = mse(output, label)\n",
+ "\n",
+ " # Step3：使用 Step1中定义的 GradScaler 完成 loss 的缩放，用缩放后的 loss 进行反向传播\n",
+ " scaled = scaler.scale(loss)\n",
+ " scaled.backward()\n",
+ "\n",
+ " # 训练模型\n",
+ " scaler.minimize(optimizer, scaled)\n",
+ " optimizer.clear_grad()\n",
+ "\n",
+ "print(loss)\n",
+ "end_timer_and_print(\"使用AMP-O1模式耗时:\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "采用level=’O2‘模式训练："
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,\n",
+ " [1.25423336])\n",
+ "\n",
+ "使用AMP-O2模式耗时:\n",
+ "共计耗时 = 0.890 sec\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = SimpleNet(input_size, output_size) # 定义模型\n",
+ "\n",
+ "optimizer = paddle.optimizer.SGD(learning_rate=0.0001, parameters=model.parameters()) # 定义优化器\n",
+ "\n",
+ "# Step1：定义 GradScaler，用于缩放loss比例，避免浮点数溢出\n",
+ "scaler = paddle.amp.GradScaler(init_loss_scaling=1024)\n",
+ "\n",
+ "# Step2：在level=’O2‘模式下，将网络参数从FP32转换为FP16\n",
+ "model, optimizer = paddle.amp.decorate(models=model, optimizers=optimizer, level='O2', master_weight=None, save_dtype=None)\n",
+ "\n",
+ "start_timer() # 获取训练开始时间\n",
+ "\n",
+ "for epoch in range(epochs):\n",
+ " datas = zip(train_data, labels)\n",
+ " for i, (data, label) in enumerate(datas):\n",
+ "\n",
+ " # Step3：创建AMP上下文环境，开启自动混合精度训练\n",
+ " with paddle.amp.auto_cast(enable=True, custom_white_list=None, custom_black_list=None, level='O2'):\n",
+ " output = model(data)\n",
+ " loss = mse(output, label)\n",
+ "\n",
+ " # Step4：使用 Step1中定义的 GradScaler 完成 loss 的缩放，用缩放后的 loss 进行反向传播\n",
+ " scaled = scaler.scale(loss)\n",
+ " scaled.backward()\n",
+ "\n",
+ " # 训练模型\n",
+ " scaler.minimize(optimizer, scaled)\n",
+ " optimizer.clear_grad()\n",
+ "\n",
+ "print(loss)\n",
+ "end_timer_and_print(\"使用AMP-O2模式耗时:\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 四、进阶用法\n",
+ "### 4.1 使用梯度累加\n",
+ "梯度累加是指在模型训练过程中，训练一个batch的数据得到梯度后，不立即用该梯度更新模型参数，而是继续下一个batch数据的训练，得到梯度后继续循环，多次循环后梯度不断累加，直至达到一定次数后，用累加的梯度更新参数，这样可以起到变相扩大 batch_size 的作用。\n",
+ "\n",
+ "在自动混合精度训练中，也支持梯度累加，使用方式如下："
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,\n",
+ " [1.25602019])\n",
+ "\n",
+ "使用AMP模式耗时:\n",
+ "共计耗时 = 1.026 sec\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = SimpleNet(input_size, output_size) # 定义模型\n",
+ "\n",
+ "optimizer = paddle.optimizer.SGD(learning_rate=0.0001, parameters=model.parameters()) # 定义优化器\n",
+ "\n",
+ "accumulate_batchs_num = 10 # 梯度累加中 batch 的数量\n",
+ "\n",
+ "# 定义 GradScaler\n",
+ "scaler = paddle.amp.GradScaler(init_loss_scaling=1024)\n",
+ "\n",
+ "start_timer() # 获取训练开始时间\n",
+ "\n",
+ "for epoch in range(epochs):\n",
+ " datas = zip(train_data, labels)\n",
+ " for i, (data, label) in enumerate(datas):\n",
+ "\n",
+ " # 创建AMP上下文环境，开启自动混合精度训练\n",
+ " with paddle.amp.auto_cast():\n",
+ " output = model(data)\n",
+ " loss = mse(output, label)\n",
+ "\n",
+ " # 使用 GradScaler 完成 loss 的缩放，用缩放后的 loss 进行反向传播\n",
+ " scaled = scaler.scale(loss)\n",
+ " scaled.backward()\n",
+ "\n",
+ " # 当累计的 batch 为 accumulate_batchs_num 时，更新模型参数\n",
+ " if (i + 1) % accumulate_batchs_num == 0:\n",
+ "\n",
+ " # 训练模型\n",
+ " scaler.minimize(optimizer, scaled)\n",
+ " optimizer.clear_grad()\n",
+ "\n",
+ "print(loss)\n",
+ "end_timer_and_print(\"使用AMP模式耗时:\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 五、总结\n",
+ "从上面的示例中可以看出，使用自动混合精度训练，O1模式共计耗时约 1.294s，O2模式共计耗时约 0.890s，而普通的训练方式则耗时 2.926s，O1模式训练速度提升约为 2.1倍，O2模式训练速度提升约为 3.0倍。如需更多使用混合精度训练的示例，请参考飞桨模型库： [paddlepaddle/models](https://github.com/PaddlePaddle/models)。"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "py35-paddle1.2.0"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
diff --git a/docs/guides/01_paddle2.0_introduction/basic_concept/amp_cn.md b/docs/guides/01_paddle2.0_introduction/basic_concept/amp_cn.md
index bc96b6736a4..646e01ecd37 100644
--- a/docs/guides/01_paddle2.0_introduction/basic_concept/amp_cn.md
+++ b/docs/guides/01_paddle2.0_introduction/basic_concept/amp_cn.md
@@ -1,6 +1,6 @@
# 自动混合精度训练
-一般情况下，训练深度学习模型时使用的数据类型为单精度（FP32）。2018年，百度与NVIDIA联合发表论文：[MIXED PRECISION TRAINING](https://arxiv.org/pdf/1710.03740.pdf)，提出了混合精度训练的方法。混合精度训练是指在训练过程中，同时使用单精度（FP32）和半精度（FP16），其目的是相较于使用单精度（FP32）训练模型，在保持精度持平的条件下，能够加速训练。本文将介绍如何使用飞桨框架，实现自动混合精度训练。
+一般情况下，训练深度学习模型时使用的数据类型为单精度（FP32）。2018年，百度与NVIDIA联合发表论文：[MIXED PRECISION TRAINING](https://arxiv.org/pdf/1710.03740.pdf)，提出了混合精度训练的方法。混合精度训练是指在训练过程中，同时使用单精度（FP32）和半精度（FP16），其目的是相较于使用单精度（FP32）训练模型，在保持精度持平的条件下，能够加速训练。本文将介绍如何使用飞桨框架，实现自动混合精度训练。
## 一、半精度浮点类型 FP16
@@ -57,6 +57,7 @@ import paddle.nn as nn
class SimpleNet(nn.Layer):
def __init__(self, input_size, output_size):
+
super(SimpleNet, self).__init__()
self.linear1 = nn.Linear(input_size, output_size)
self.relu1 = nn.ReLU()
@@ -91,6 +92,10 @@ labels = [paddle.randn((batch_size, output_size)) for _ in range(nums_batch)]
mse = paddle.nn.MSELoss()
```
+ W1110 18:42:02.362493 104 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
+ W1110 18:42:02.367755 104 device_context.cc:465] device: 0, cuDNN Version: 7.6.
+
+
### 3.3 使用默认的训练方式进行训练
@@ -120,10 +125,10 @@ end_timer_and_print("默认耗时:") # 获取结束时间并打印相关信息
```
Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
- [1.24609220])
-
+ [1.24519622])
+
默认耗时:
- 共计耗时 = 2.819 sec
+ 共计耗时 = 2.926 sec
### 3.4 使用AMP训练模型
@@ -138,6 +143,7 @@ end_timer_and_print("默认耗时:") # 获取结束时间并打印相关信息
采用level=’O1‘模式训练：
+
```python
model = SimpleNet(input_size, output_size) # 定义模型
@@ -170,14 +176,15 @@ end_timer_and_print("使用AMP-O1模式耗时:")
```
Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
- [1.24609900])
-
+ [1.24815702])
+
使用AMP-O1模式耗时:
- 共计耗时 = 1.324 sec
+ 共计耗时 = 1.294 sec
采用level=’O2‘模式训练：
+
```python
model = SimpleNet(input_size, output_size) # 定义模型
@@ -212,11 +219,17 @@ print(loss)
end_timer_and_print("使用AMP-O2模式耗时:")
```
+ in ParamBase copy_to func
+ in ParamBase copy_to func
+ in ParamBase copy_to func
+ in ParamBase copy_to func
+ in ParamBase copy_to func
+ in ParamBase copy_to func
Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
- [1.24997652])
-
+ [1.25423336])
+
使用AMP-O2模式耗时:
- 共计耗时 = 0.933 sec
+ 共计耗时 = 0.890 sec
## 四、进阶用法
@@ -263,10 +276,11 @@ end_timer_and_print("使用AMP模式耗时:")
```
Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
- [1.24623466])
-
+ [1.25602019])
+
使用AMP模式耗时:
- 共计耗时 = 1.020 sec
+ 共计耗时 = 1.026 sec
+
## 五、总结
-从上面的示例中可以看出，使用自动混合精度训练，O1模式共计耗时约 1.324s，O2模式共计耗时约 0.933s，而普通的训练方式则耗时 2.819s，O1模式训练速度提升约为 2.1倍，O2模式训练速度提升约为 3.0倍。如需更多使用混合精度训练的示例，请参考飞桨模型库： [paddlepaddle/models](https://github.com/PaddlePaddle/models)。
+从上面的示例中可以看出，使用自动混合精度训练，O1模式共计耗时约 1.294s，O2模式共计耗时约 0.890s，而普通的训练方式则耗时 2.926s，O1模式训练速度提升约为 2.1倍，O2模式训练速度提升约为 3.0倍。如需更多使用混合精度训练的示例，请参考飞桨模型库： [paddlepaddle/models](https://github.com/PaddlePaddle/models)。
diff --git a/docs/guides/01_paddle2.0_introduction/basic_concept/amp_en.ipynb b/docs/guides/01_paddle2.0_introduction/basic_concept/amp_en.ipynb
new file mode 100644
index 00000000000..22c12fcfed1
--- /dev/null
+++ b/docs/guides/01_paddle2.0_introduction/basic_concept/amp_en.ipynb
@@ -0,0 +1,453 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "# Automatic Mixed Precision Training\n",
+ "\n",
+ "In general, the datatype of training deep learning models is single-precision floating-point format(also called FP32). In 2018, Baidu and NVIDIA jointly published the paper: [MIXED PRECISION TRAINING](https://arxiv.org/pdf/1710.03740.pdf), which proposed mixed precision training. During the process of training, some operators use FP32 and other operators use half precision(also called FP16) in the same time. Its purpose is to speed up training, while compared with the FP32 training model, the same accuracy is maintained. This tutorial will introduce how to use automatic mixed precision training with PaddlePaddle."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 1. Half Precision (FP16)\n",
+ "\n",
+ "First introduce FP16. As shown in Figure 1, FP16 occupies 16 bits (two bytes in modern computers) of computer memory. In the IEEE 754-2008 standard, it is also named binary16. Compared with FP32 and double precision (also called FP64) commonly used, FP16 is more suitable for the usage in scenarios with low precision requirements.\n",
+ "\n",
+ "\n",
+ " \n",
+ " Figure 1. Half precision(FP16) and single precision(FP32)\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 2. FP16 Computing Power of NVIDIA GPU\n",
+ "\n",
+ "When the same hyperparameters are used, mixed precision training using FP16 and FP32 can achieve the same accuracy as that of pure single precision used, and can accelerate the training speed. It mainly attributes to the features that NVIDIA Volta and NVIDIA Turing use FP16 to calculate:\n",
+ "- FP16 can reduce memory bandwidth and storage requirements by half, which allows researchers to use more complex models and larger batch sizes under the same hardware conditions.\n",
+ "- FP16 can make full use of Tensor Cores technology provided by NVIDIA Volta and NVIDIA Turing. On the same GPU hardware, the computing throughput of Tensor Cores' FP16 is 8 times bigger than that of FP32."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 3. Automatic Mixed Precision Training with PaddlePaddle\n",
+ "\n",
+ "Using PaddlePaddle's API ``paddle.amp.auto_cast`` and ``paddle.amp.GradScaler`` can realize automatic mixed precision training (AMP), which can automatically choose FP16 or FP32 for different operators' calculation. After the AMP mode is turned on, the operator list calculated by FP16 and FP32 can be found in this [document](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/amp/Overview_cn.html). This is a specific example to understand how to use PaddlePaddle to achieve mixed precision training."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "### 3.1 Auxiliary Function\n",
+ "First define the auxiliary function to calculate the training time."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "\n",
+ "# start time\n",
+ "start_time = None\n",
+ "\n",
+ "def start_timer():\n",
+ " # get start time\n",
+ " global start_time\n",
+ " start_time = time.time()\n",
+ "\n",
+ "def end_timer_and_print(msg):\n",
+ " # print message and total training time\n",
+ " end_time = time.time()\n",
+ " print(\"\\n\" + msg)\n",
+ " print(\"total time = {:.3f} sec\".format(end_time - start_time))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "### 3.2 A Simple Network\n",
+ "\n",
+ "Define a simple network to compare the training speed of common methods and mixed precision. The network is composed of three layers of ``Linear``. The first two layers of ``Linear`` are followed by the ``ReLU`` activation function."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import paddle\n",
+ "import paddle.nn as nn\n",
+ "\n",
+ "class SimpleNet(nn.Layer):\n",
+ "\n",
+ " def __init__(self, input_size, output_size):\n",
+ " \n",
+ " super(SimpleNet, self).__init__()\n",
+ " self.linear1 = nn.Linear(input_size, output_size)\n",
+ " self.relu1 = nn.ReLU()\n",
+ " self.linear2 = nn.Linear(input_size, output_size)\n",
+ " self.relu2 = nn.ReLU()\n",
+ " self.linear3 = nn.Linear(input_size, output_size)\n",
+ "\n",
+ " def forward(self, x):\n",
+ "\n",
+ " x = self.linear1(x)\n",
+ " x = self.relu1(x)\n",
+ " x = self.linear2(x)\n",
+ " x = self.relu2(x)\n",
+ " x = self.linear3(x)\n",
+ "\n",
+ " return x"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "Set the parameters of training. In order to effectively show the improvement of training speed by mixed precision training, please set the larger values of ``input_size`` and ``output_size``. And in order to use the ``Tensor Core`` provided by GPU, ``batch_size`` needs to be set as a multiple of 8."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "epochs = 5\n",
+ "input_size = 4096 # set to a larger value\n",
+ "output_size = 4096 # set to a larger value\n",
+ "batch_size = 512 # batch_size is a multiple of 8\n",
+ "nums_batch = 50\n",
+ "\n",
+ "train_data = [paddle.randn((batch_size, input_size)) for _ in range(nums_batch)]\n",
+ "labels = [paddle.randn((batch_size, output_size)) for _ in range(nums_batch)]\n",
+ "\n",
+ "mse = paddle.nn.MSELoss()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "### 3.3 Training with Default Method"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,\n",
+ " [1.24072289])\n",
+ "\n",
+ "Default time:\n",
+ "total time = 2.935 sec\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = SimpleNet(input_size, output_size) # define model\n",
+ "\n",
+ "optimizer = paddle.optimizer.SGD(learning_rate=0.0001, parameters=model.parameters()) # define optimizer\n",
+ "\n",
+ "start_timer() # get the start time of training\n",
+ "\n",
+ "for epoch in range(epochs):\n",
+ " datas = zip(train_data, labels)\n",
+ " for i, (data, label) in enumerate(datas):\n",
+ "\n",
+ " output = model(data)\n",
+ " loss = mse(output, label)\n",
+ "\n",
+ " # backpropagation\n",
+ " loss.backward()\n",
+ "\n",
+ " # update parameters\n",
+ " optimizer.step()\n",
+ " optimizer.clear_grad()\n",
+ "\n",
+ "print(loss)\n",
+ "end_timer_and_print(\"Default time:\") # print massage and total time"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "### 3.4 Training with AMP\n",
+ "\n",
+ "Using automatic mixed precision training with PaddlePaddle requires four steps:\n",
+ "\n",
+ "- Step1: Define ``GradScaler``, which is used to scale the ``loss`` to avoid underflow\n",
+ "- Step2: Use ``decorate``, to do nothing in level='O1' mode without using this api, and in level='O2' mode to convert network parameters from FP32 to FP16\n",
+ "- Step3: Use ``auto_cast`` to create an AMP context, in which the input datatype(FP16 or FP32) of each oprator will be automatically determined\n",
+ "- Step4: Use ``GradScaler`` defined in Step1 to complete the scaling of ``loss``, and use the scaled ``loss`` for backpropagation to complete the training\n",
+ "\n",
+ "In level=’O1‘ mode："
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,\n",
+ " [1.24848151])\n",
+ "\n",
+ "AMP time in O1 mode:\n",
+ "total time = 1.299 sec\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = SimpleNet(input_size, output_size) # define model\n",
+ "\n",
+ "optimizer = paddle.optimizer.SGD(learning_rate=0.0001, parameters=model.parameters()) # define optimizer\n",
+ "\n",
+ "# Step1：define GradScaler\n",
+ "scaler = paddle.amp.GradScaler(init_loss_scaling=1024)\n",
+ "\n",
+ "start_timer() # get start time\n",
+ "\n",
+ "for epoch in range(epochs):\n",
+ " datas = zip(train_data, labels)\n",
+ " for i, (data, label) in enumerate(datas):\n",
+ "\n",
+ " # Step2：create AMP context environment\n",
+ " with paddle.amp.auto_cast():\n",
+ " output = model(data)\n",
+ " loss = mse(output, label)\n",
+ "\n",
+ " # Step3：use GradScaler complete the loss scaling\n",
+ " scaled = scaler.scale(loss)\n",
+ " scaled.backward()\n",
+ "\n",
+ " # update parameters\n",
+ " scaler.minimize(optimizer, scaled)\n",
+ " optimizer.clear_grad()\n",
+ "\n",
+ "print(loss)\n",
+ "end_timer_and_print(\"AMP time in O1 mode:\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "In level='O2' mode："
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 30,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "in ParamBase copy_to func\n",
+ "Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,\n",
+ " [1.25075114])\n",
+ "\n",
+ "AMP time in O2 mode:\n",
+ "total time = 0.888 sec\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = SimpleNet(input_size, output_size) # define model\n",
+ "\n",
+ "optimizer = paddle.optimizer.SGD(learning_rate=0.0001, parameters=model.parameters()) # define optimizer\n",
+ "\n",
+ "# Step1：define GradScaler\n",
+ "scaler = paddle.amp.GradScaler(init_loss_scaling=1024)\n",
+ "\n",
+ "# Step2：in level='O2' mode, convert network parameters from FP32 to FP16\n",
+ "model, optimizer = paddle.amp.decorate(models=model, optimizers=optimizer, level='O2', master_weight=None, save_dtype=None)\n",
+ "\n",
+ "start_timer() # get start time\n",
+ "\n",
+ "for epoch in range(epochs):\n",
+ " datas = zip(train_data, labels)\n",
+ " for i, (data, label) in enumerate(datas):\n",
+ "\n",
+ " # Step3：create AMP context environment\n",
+ " with paddle.amp.auto_cast(enable=True, custom_white_list=None, custom_black_list=None, level='O2'):\n",
+ " output = model(data)\n",
+ " loss = mse(output, label)\n",
+ "\n",
+ " # Step4：use GradScaler complete the loss scaling\n",
+ " scaled = scaler.scale(loss)\n",
+ " scaled.backward()\n",
+ "\n",
+ " # update parameters\n",
+ " scaler.minimize(optimizer, scaled)\n",
+ " optimizer.clear_grad()\n",
+ "\n",
+ "print(loss)\n",
+ "end_timer_and_print(\"AMP time in O2 mode:\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 4. Advanced Usage\n",
+ "### 4.1 Gradient Accumulation\n",
+ "\n",
+ "Gradient accumulation means running a configured number of steps without updating the model variables. Until certain steps, use the accumulated gradients to update the variables.\n",
+ "\n",
+ "In automatic mixed precision training, gradient accumulation is also supported, and the usage is as follows:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,\n",
+ " [1.25853443])\n",
+ "\n",
+ "AMP time:\n",
+ "total time = 1.034 sec\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = SimpleNet(input_size, output_size) # define model\n",
+ "\n",
+ "optimizer = paddle.optimizer.SGD(learning_rate=0.0001, parameters=model.parameters()) # define optimizer\n",
+ "\n",
+ "accumulate_batchs_num = 10 # the batch numbers of gradients accumulation\n",
+ "\n",
+ "# define GradScaler\n",
+ "scaler = paddle.amp.GradScaler(init_loss_scaling=1024)\n",
+ "\n",
+ "start_timer() # get start time\n",
+ "\n",
+ "for epoch in range(epochs):\n",
+ " datas = zip(train_data, labels)\n",
+ " for i, (data, label) in enumerate(datas):\n",
+ "\n",
+ " # create AMP context environment\n",
+ " with paddle.amp.auto_cast():\n",
+ " output = model(data)\n",
+ " loss = mse(output, label)\n",
+ "\n",
+ " # use GradScaler complete the loss scaling\n",
+ " scaled = scaler.scale(loss)\n",
+ " scaled.backward()\n",
+ "\n",
+ " # when the accumulated batch is accumulate_batchs_num, update the model parameters\n",
+ " if (i + 1) % accumulate_batchs_num == 0:\n",
+ "\n",
+ " # update parameters\n",
+ " scaler.minimize(optimizer, scaled)\n",
+ " optimizer.clear_grad()\n",
+ "\n",
+ "print(loss)\n",
+ "end_timer_and_print(\"AMP time:\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 5. Conclusion\n",
+ "\n",
+ "As can be seen from the above example, using the automatic mixed precision training, in O1 mode the total time is about 1.299s, in O2 mode the total time is about 0.888s, while the ordinary training method takes 2.935s, and the training speed is increased by about 2.4 times in O1 mode and 2.4 times in O2 mode. For more examples of using mixed precision training, please refer to paddlepaddle's models: [paddlepaddle/models](https://github.com/PaddlePaddle/models)."
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "py35-paddle1.2.0"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.7.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
diff --git a/docs/guides/01_paddle2.0_introduction/basic_concept/amp_en.md b/docs/guides/01_paddle2.0_introduction/basic_concept/amp_en.md
index ee31dc70ba1..6c5f15edfae 100644
--- a/docs/guides/01_paddle2.0_introduction/basic_concept/amp_en.md
+++ b/docs/guides/01_paddle2.0_introduction/basic_concept/amp_en.md
@@ -1,6 +1,6 @@
# Automatic Mixed Precision Training
-In general, the datatype of training deep learning models is single-precision floating-point format(also called FP32). In 2018, Baidu and NVIDIA jointly published the paper: [MIXED PRECISION TRAINING](https://arxiv.org/pdf/1710.03740.pdf), which proposed mixed precision training. During the process of training, some operators use FP32 and other operators use half precision(also called FP16) in the same time. Its purpose is to speed up training, while compared with the FP32 training model, the same accuracy is maintained. This tutorial will introduce how to use automatic mixed precision training with PaddlePaddle.
+In general, the datatype of training deep learning models is single-precision floating-point format(also called FP32). In 2018, Baidu and NVIDIA jointly published the paper: [MIXED PRECISION TRAINING](https://arxiv.org/pdf/1710.03740.pdf), which proposed mixed precision training. During the process of training, some operators use FP32 and other operators use half precision(also called FP16) in the same time. Its purpose is to speed up training, while compared with the FP32 training model, the same accuracy is maintained. This tutorial will introduce how to use automatic mixed precision training with PaddlePaddle.
## 1. Half Precision (FP16)
@@ -55,6 +55,7 @@ import paddle.nn as nn
class SimpleNet(nn.Layer):
def __init__(self, input_size, output_size):
+
super(SimpleNet, self).__init__()
self.linear1 = nn.Linear(input_size, output_size)
self.relu1 = nn.ReLU()
@@ -118,19 +119,22 @@ end_timer_and_print("Default time:") # print massage and total time
```
Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
- [1.25010288])
-
+ [1.24072289])
+
Default time:
- total time = 2.943 sec
+ total time = 2.935 sec
### 3.4 Training with AMP
-Using automatic mixed precision training with PaddlePaddle requires three steps:
+Using automatic mixed precision training with PaddlePaddle requires four steps:
+
+- Step1: Define ``GradScaler``, which is used to scale the ``loss`` to avoid underflow
+- Step2: Use ``decorate``, to do nothing in level='O1' mode without using this api, and in level='O2' mode to convert network parameters from FP32 to FP16
+- Step3: Use ``auto_cast`` to create an AMP context, in which the input datatype(FP16 or FP32) of each oprator will be automatically determined
+- Step4: Use ``GradScaler`` defined in Step1 to complete the scaling of ``loss``, and use the scaled ``loss`` for backpropagation to complete the training
-- Step1: Define ``GradScaler``, which is used to scale the ``loss`` and ``gradients``to avoid underflow
-- Step2: Use ``auto_cast`` to create an AMP context, in which the input datatype(FP16 or FP32) of each oprator will be automatically determined
-- Step3: Use ``GradScaler`` defined in Step1 to complete the scaling of ``loss``, and use the scaled ``loss`` for backpropagation to complete the training
+In level=’O1‘ mode：
```python
@@ -161,14 +165,64 @@ for epoch in range(epochs):
optimizer.clear_grad()
print(loss)
-end_timer_and_print("AMP time:")
+end_timer_and_print("AMP time in O1 mode:")
```
Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
- [1.23644269])
+ [1.24848151])
+
+ AMP time in O1 mode:
+ total time = 1.299 sec
- AMP time:
- total time = 1.222 sec
+
+In level='O2' mode：
+
+
+```python
+model = SimpleNet(input_size, output_size) # define model
+
+optimizer = paddle.optimizer.SGD(learning_rate=0.0001, parameters=model.parameters()) # define optimizer
+
+# Step1：define GradScaler
+scaler = paddle.amp.GradScaler(init_loss_scaling=1024)
+
+# Step2：in level='O2' mode, convert network parameters from FP32 to FP16
+model, optimizer = paddle.amp.decorate(models=model, optimizers=optimizer, level='O2', master_weight=None, save_dtype=None)
+
+start_timer() # get start time
+
+for epoch in range(epochs):
+ datas = zip(train_data, labels)
+ for i, (data, label) in enumerate(datas):
+
+ # Step3：create AMP context environment
+ with paddle.amp.auto_cast(enable=True, custom_white_list=None, custom_black_list=None, level='O2'):
+ output = model(data)
+ loss = mse(output, label)
+
+ # Step4：use GradScaler complete the loss scaling
+ scaled = scaler.scale(loss)
+ scaled.backward()
+
+ # update parameters
+ scaler.minimize(optimizer, scaled)
+ optimizer.clear_grad()
+
+print(loss)
+end_timer_and_print("AMP time in O2 mode:")
+```
+
+ in ParamBase copy_to func
+ in ParamBase copy_to func
+ in ParamBase copy_to func
+ in ParamBase copy_to func
+ in ParamBase copy_to func
+ in ParamBase copy_to func
+ Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
+ [1.25075114])
+
+ AMP time in O2 mode:
+ total time = 0.888 sec
## 4. Advanced Usage
@@ -204,7 +258,7 @@ for epoch in range(epochs):
scaled = scaler.scale(loss)
scaled.backward()
- # when the accumulated batch is accumulate_batchs_num, update the model parameters
+ # when the accumulated batch is accumulate_batchs_num, update the model parameters
if (i + 1) % accumulate_batchs_num == 0:
# update parameters
@@ -216,12 +270,12 @@ end_timer_and_print("AMP time:")
```
Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=False,
- [1.25127280])
-
+ [1.25853443])
+
AMP time:
- total time = 1.006 sec
+ total time = 1.034 sec
## 5. Conclusion
-As can be seen from the above example, using the automatic mixed precision training, the total time is about 1.222s, while the ordinary training method takes 2.943s, and the training speed is increased by about 2.4 times. For more examples of using mixed precision training, please refer to paddlepaddle's models: [paddlepaddle/models](https://github.com/PaddlePaddle/models).
+As can be seen from the above example, using the automatic mixed precision training, in O1 mode the total time is about 1.299s, in O2 mode the total time is about 0.888s, while the ordinary training method takes 2.935s, and the training speed is increased by about 2.4 times in O1 mode and 2.4 times in O2 mode. For more examples of using mixed precision training, please refer to paddlepaddle's models: [paddlepaddle/models](https://github.com/PaddlePaddle/models).
diff --git a/docs/guides/01_paddle2.0_introduction/basic_concept/autograd_cn.rst b/docs/guides/01_paddle2.0_introduction/basic_concept/autograd_cn.rst
index 3951f03c09d..fcf36e1d774 100644
--- a/docs/guides/01_paddle2.0_introduction/basic_concept/autograd_cn.rst
+++ b/docs/guides/01_paddle2.0_introduction/basic_concept/autograd_cn.rst
@@ -35,7 +35,7 @@ PaddlePaddle的神经网络核心是自动微分，本篇文章主要为你介
.. parsed-literal::
- 2.1.1
+ 2.2.0
本案例首先定义网络。因为本示例着重展示如何使用飞桨进行自动微分，故组网部分不过多展开，直接使用高层API中封装好的模型\ ``vgg11``\ 。
@@ -291,4 +291,4 @@ PaddlePaddle的神经网络核心是自动微分，本篇文章主要为你介
五、总结
------------------------
-本文章主要介绍了如何使用飞桨的自动微分，以及飞桨的自动微分机制。
+本文章主要介绍了如何使用飞桨的自动微分，以及飞桨的自动微分机制。
\ No newline at end of file
diff --git a/docs/guides/01_paddle2.0_introduction/basic_concept/gradient_clip_cn.rst b/docs/guides/01_paddle2.0_introduction/basic_concept/gradient_clip_cn.rst
index 5f32441212d..7d5cd89b959 100644
--- a/docs/guides/01_paddle2.0_introduction/basic_concept/gradient_clip_cn.rst
+++ b/docs/guides/01_paddle2.0_introduction/basic_concept/gradient_clip_cn.rst
@@ -20,6 +20,8 @@ Paddle提供了三种梯度裁剪方式：
.. code:: ipython3
+ import paddle
+
linear = paddle.nn.Linear(10, 10)
clip = paddle.nn.ClipGradByValue(min=-1, max=1)
sdg = paddle.optimizer.SGD(learning_rate=0.1, parameters=linear.parameters(), grad_clip=clip)
diff --git a/docs/guides/01_paddle2.0_introduction/basic_concept/gradient_clip_en.rst b/docs/guides/01_paddle2.0_introduction/basic_concept/gradient_clip_en.rst
index b6d58570b4f..31fd73f8b11 100644
--- a/docs/guides/01_paddle2.0_introduction/basic_concept/gradient_clip_en.rst
+++ b/docs/guides/01_paddle2.0_introduction/basic_concept/gradient_clip_en.rst
@@ -20,6 +20,8 @@ By default, Gradients of all parameters in SGD optimizer will be clipped:
.. code:: ipython3
+ import paddle
+
linear = paddle.nn.Linear(10, 10)
clip = paddle.nn.ClipGradByValue(min=-1, max=1)
sdg = paddle.optimizer.SGD(learning_rate=0.1, parameters=linear.parameters(), grad_clip=clip)
diff --git a/docs/guides/01_paddle2.0_introduction/basic_concept/tensor_introduction_cn.md b/docs/guides/01_paddle2.0_introduction/basic_concept/tensor_introduction_cn.md
index 3eb03db37b8..00efa373a39 100644
--- a/docs/guides/01_paddle2.0_introduction/basic_concept/tensor_introduction_cn.md
+++ b/docs/guides/01_paddle2.0_introduction/basic_concept/tensor_introduction_cn.md
@@ -81,8 +81,8 @@ array([[1., 2., 3.],
**Tensor**不仅支持 floats、ints 类型数据，也支持 complex numbers数据，如果输入为复数数据，则**Tensor**的dtype为 ``complex64`` 或 ``complex128`` ，其每个元素均为1个复数：
```python
-ndim_2_tensor = paddle.to_tensor([[1.0, 2.0, 3.0],
- [4.0, 5.0, 6.0]])
+ndim_2_tensor = paddle.to_tensor([[(1+1j), (2+2j)],
+ [(3+3j), (4+4j)]])
print(ndim_2_tensor)
```
@@ -473,7 +473,6 @@ x.logical_not(y) #对两个bool型tensor逐元素进行逻辑非操
### 线性代数相关
```python
-x.cholesky() #矩阵的cholesky分解
x.t() #矩阵转置
x.transpose([1, 0]) #交换axis 0 与axis 1的顺序
x.norm('fro') #矩阵的Frobenius 范数
diff --git a/docs/guides/01_paddle2.0_introduction/basic_concept/tensor_introduction_en.md b/docs/guides/01_paddle2.0_introduction/basic_concept/tensor_introduction_en.md
index 9e44ad029a7..f9dfcde4c58 100644
--- a/docs/guides/01_paddle2.0_introduction/basic_concept/tensor_introduction_en.md
+++ b/docs/guides/01_paddle2.0_introduction/basic_concept/tensor_introduction_en.md
@@ -80,8 +80,8 @@ array([[1., 2., 3.],
**Tensor** supports not only floats and ints but also complex numbers data, If input complex number data, the dtype of **Tensor** is ``complex64`` or ``complex128`` :
```python
-ndim_2_tensor = paddle.to_tensor([[1.0, 2.0, 3.0],
- [4.0, 5.0, 6.0]])
+ndim_2_tensor = paddle.to_tensor([[(1+1j), (2+2j)],
+ [(3+3j), (4+4j)]])
print(ndim_2_tensor)
```
@@ -482,7 +482,6 @@ x.logical_not(y) #logic not operation for two bool tensor
### linear algebra operators
```python
-x.cholesky() #cholesky decomposition of a matrix
x.t() #matrix transpose
x.transpose([1, 0]) #swap axis 0 with axis 1
x.norm('fro') #Frobenius Norm of matrix
diff --git a/docs/guides/01_paddle2.0_introduction/load_old_format_model.rst b/docs/guides/01_paddle2.0_introduction/load_old_format_model_cn.rst
similarity index 100%
rename from docs/guides/01_paddle2.0_introduction/load_old_format_model.rst
rename to docs/guides/01_paddle2.0_introduction/load_old_format_model_cn.rst
diff --git a/docs/guides/01_paddle2.0_introduction/migration_cn.rst b/docs/guides/01_paddle2.0_introduction/migration_cn.rst
index f04a2ee8835..94f9e2ee60d 100644
--- a/docs/guides/01_paddle2.0_introduction/migration_cn.rst
+++ b/docs/guides/01_paddle2.0_introduction/migration_cn.rst
@@ -66,7 +66,7 @@ paddle_upgrade_tool 可以使用下面的方式，快速使用:
开始
^^^^
-在使用paddle_upgrade_tool前，需要确保已经安装了Paddle 2.0.0版本。
+在使用paddle_upgrade_tool前，需要确保已经安装了Paddle 2.0.0+版本。
.. code:: ipython3
diff --git a/docs/guides/01_paddle2.0_introduction/update_cn.md b/docs/guides/01_paddle2.0_introduction/update_cn.md
index 2e1c44ab4ac..7f367547d13 100644
--- a/docs/guides/01_paddle2.0_introduction/update_cn.md
+++ b/docs/guides/01_paddle2.0_introduction/update_cn.md
@@ -558,5 +558,5 @@ https://github.com/PaddlePaddle/paddle_upgrade_tool
### 2.0文档教程
以下提供了2.0版本的一些示例教程：
-你可以在官网[应用实践](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/tutorial/index_cn.html)栏目内进行在线浏览，也可以下载在这里提供的源代码:
-https://github.com/PaddlePaddle/book/tree/develop/paddle2.0_docs
+你可以在官网[应用实践](https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/practices/index_cn.html)栏目内进行在线浏览，也可以下载在这里提供的源代码:
+https://github.com/PaddlePaddle/docs/tree/develop/docs/practices
diff --git a/docs/guides/02_paddle2.0_develop/05_train_eval_predict_cn.rst b/docs/guides/02_paddle2.0_develop/05_train_eval_predict_cn.rst
index 789be2a9394..3c2182c9b33 100644
--- a/docs/guides/02_paddle2.0_develop/05_train_eval_predict_cn.rst
+++ b/docs/guides/02_paddle2.0_develop/05_train_eval_predict_cn.rst
@@ -7,7 +7,7 @@
.. note::
- 高层API实现的模型训练与预测如\ ``Model.fit()、Model.evaluate()、Model.predict()``\ 都可以通过基础API实现，本文先介绍高层API的训练方式，然后会将高层API拆解为基础API的方式，方便对比学习。最后会补充介绍如何使用paddle inference进行预测。
+ 高层API实现的模型训练与预测如\ ``Model.fit()、Model.evaluate()、Model.predict()``\ 都可以通过基础API实现，本文先介绍高层API的训练方式，然后会将高层API拆解为基础API的方式，方便对比学习。
一、训练前准备
---------------------
@@ -137,11 +137,6 @@ numpy_ndarray_n是对应原始数据经过模型计算后得到的预测数据
除了通过第一部分的高层API实现模型的训练与预测，飞桨框架也同样支持通过基础API对模型进行训练与预测。简单来说，\ ``Model.prepare()、Model.fit()、Model.evaluate()、Model.predict()``\ 都是由基础API封装而来。下面通过拆解高层API到基础API的方式，来了解如何用基础API完成模型的训练与预测。
-
-.. note::
-
- 对于网络模型的创建你依旧可以选择Sequential组网方式，也可以采用SubClass组网方式，为方便后续使用paddle inference进行预测，我们使用SubClass组网方式创建网络，若后续使用paddle inference预测，需通过paddle.jit.save保存适用于预测部署的模型，并在forward函数前加@paddle.jit.to_static装饰器，将函数内的动态图API转化为静态图API。
-
.. code:: ipython3
# 定义网络结构( 采用SubClass 组网 )
@@ -153,9 +148,7 @@ numpy_ndarray_n是对应原始数据经过模型计算后得到的预测数据
self.linear_2 = paddle.nn.Linear(512, 10)
self.relu = paddle.nn.ReLU()
self.dropout = paddle.nn.Dropout(0.2)
-
- #后续若不使用paddle inferece，可对 @paddle.jit.to_static 进行注释
- @paddle.jit.to_static
+
def forward(self, inputs):
y = self.flatten(inputs)
y = self.linear_1(y)
@@ -214,9 +207,6 @@ numpy_ndarray_n是对应原始数据经过模型计算后得到的预测数据
# 梯度清零
optim.clear_grad()
- ##保存模型，会生成*.pdmodel、*.pdiparams、*.pdiparams.info三个模型文件
- path='./mnist/inference_model'
- paddle.jit.save(layer=mnist,path=path)
.. parsed-literal::
@@ -284,101 +274,3 @@ numpy_ndarray_n是对应原始数据经过模型计算后得到的预测数据
.. parsed-literal::
predict finished
-
-
-部署预测模型
-=====================
-其中预测方法除以上两种外，还可采用原生推理库paddle inference 进行推理部署，该方法支持TeansorRT加速，支持第三方框架模型，支持量化、裁剪后的模型，适合于工业部署或对推理性能、通用性有要求的用户。
-
-
-四、通过paddle inference实现预测
------------------------------------------
-
-paddle inference与model.predict()以及基础API的预测相比，可使用MKLDNN、CUDNN、TensorRT进行预测加速，同时支持用 X2Paddle 工具从第三方框架（TensorFlow、Pytorh 、 Caffe 等）产出的模型，可联动PaddleSlim，支持加载量化、裁剪和蒸馏后的模型部署。针对不同平台不同的应用场景进行了深度的适配优化，保证模型在服务器端即训即用，快速部署。在这里，我们只简单的展示如何用paddle inference实现该模型的部署预测。
-
-4.1 准备预测部署模型
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-要使用paddle inference预测需得到paddle预测格式的模型，所以你需要在训练过程中通过 paddle.jit.save(layer=mnist,path=path) 来保存模型，注意在训练时在forward函数前加@paddle.jit.to_static装饰器，将函数内的动态图API转化为静态图API。在第三章节基础API模型的训练中已加入相关配置。
-
-.. code:: ipython3
-
- #模型目录如下：
- mnist/
- ├── inference.pdmodel
- ├── inference.pdiparams.info
- └── inference.pdiparams
-4.2 准备预测部署程序
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-将以下代码保存为python_demo.py文件：
-
-.. code:: ipython3
-
- import argparse
- import numpy as np
- from skimage import transform,data
-
- # 引用 paddle inference 预测库
- import paddle.inference as paddle_infer
- from PIL import Image
-
- def main():
- args = parse_args()
-
- # 创建 config
- config = paddle_infer.Config(args.model_file, args.params_file)
-
- # 根据 config 创建 predictor
- predictor = paddle_infer.create_predictor(config)
-
- # 获取输入的名称
- input_names = predictor.get_input_names()
- input_handle = predictor.get_input_handle(input_names[0])
-
- # 设置输入，自定义一张输入照片，图片大小为28*28
- im=Image.open('./img3.png').convert('L')
- im=np.array(im).reshape(1,1,28,28).astype(np.float32)
- input_handle.copy_from_cpu(im)
-
- # 运行predictor
- predictor.run()
-
- # 获取输出
- output_names = predictor.get_output_names()
- output_handle = predictor.get_output_handle(output_names[0])
- output_data = output_handle.copy_to_cpu() # numpy.ndarray类型，是10个分类的概率
- print(output_data)
- print("Output data size is {}".format(output_data.size))
- print("Output data shape is {}".format(output_data.shape))
- pred=np.argmax(output_data) #选出概率最大的一个
- print("The predicted data is ： {}".format(pred.item()))
-
- def parse_args():
- parser = argparse.ArgumentParser()
- parser.add_argument("--model_file", type=str, help="model filename")
- parser.add_argument("--params_file", type=str, help="parameter filename")
- parser.add_argument("--batch_size", type=int, default=1, help="batch size")
- return parser.parse_args()
-
- if __name__ == "__main__":
- main()
-
-
-4.3 执行预测程序
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. code:: ipython3
-
- python python_demo.py --model_file ./mnist/inference_model.pdmodel --params_file ./mnist/inference_model.pdiparams --batch_size 2
-
-.. parsed-literal::
-
- #输出如下
-
- [[-1347.5923 -1156.918 -774.73865 3387.0623 -1553.3696 107.96879
- -2631.2185 -701.50323 -1094.3896 206.71666]]
- Output data size is 10
- Output data shape is (1, 10)
- The predicted data is ： 3
-
-详细教程可参照paddle inference文档：https://paddle-inference.readthedocs.io/en/latest/quick_start/python_demo.html
-
diff --git a/docs/guides/performance_improving/index_cn.rst b/docs/guides/performance_improving/index_cn.rst
index 241893eca6b..64faa2caf93 100644
--- a/docs/guides/performance_improving/index_cn.rst
+++ b/docs/guides/performance_improving/index_cn.rst
@@ -2,6 +2,11 @@
性能调优
########
+你可以通过以下内容，了解飞桨框架性能调优相关的内容：
+
+- `模型量化 <./quantization.html>`_ : 使用飞桨框架进行模型量化。
+
.. toctree::
- :maxdepth: 1
+ :hidden:
+ quantization.md
\ No newline at end of file
diff --git a/docs/install/docker/fromdocker.rst b/docs/install/docker/fromdocker.rst
index aa25d82d3d7..62905f664d7 100644
--- a/docs/install/docker/fromdocker.rst
+++ b/docs/install/docker/fromdocker.rst
@@ -5,5 +5,4 @@
.. toctree::
:maxdepth: 1
- linux-docker.md
macos-docker.md
diff --git a/docs/install/docker/fromdocker_en.rst b/docs/install/docker/fromdocker_en.rst
index c0b2b487411..af6a1a7fafe 100644
--- a/docs/install/docker/fromdocker_en.rst
+++ b/docs/install/docker/fromdocker_en.rst
@@ -5,5 +5,4 @@
.. toctree::
- linux-docker_en.md
macos-docker_en.md
diff --git a/docs/practices/cv/image_ocr.ipynb b/docs/practices/cv/image_ocr.ipynb
new file mode 100644
index 00000000000..d3b9c516c16
--- /dev/null
+++ b/docs/practices/cv/image_ocr.ipynb
@@ -0,0 +1,722 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "# 通过OCR实现验证码识别\n",
+ "\n",
+ "**作者:** [GT_老张](https://github.com/GT-ZhangAcer) \n",
+ "\n",
+ "**时间:** 2021.11\n",
+ "\n",
+ "**摘要:** 本篇将介绍如何通过飞桨实现简单的CRNN+CTC自定义数据集OCR识别模型，数据集采用[CaptchaDataset](https://github.com/GT-ZhangAcer/CaptchaDataset)中OCR部分的9453张图像，其中前8453张图像在本案例中作为训练集，后1000张则作为测试集。 \n",
+ "在更复杂的场景中推荐使用[PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)产出工业级模型，模型轻量且精度大幅提升。 \n",
+ "同样也可以在[PaddleHub](https://www.paddlepaddle.org.cn/hubdetail?name=chinese_ocr_db_crnn_mobile&en_category=TextRecognition)中快速使用PaddleOCR。"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 一、环境配置\n",
+ "\n",
+ "本教程基于Paddle 2.2.0 编写，如果你的环境不是本版本，请先参考官网[安装](https://www.paddlepaddle.org.cn/install/quick) PaddlePaddle 2.2 。"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "2.2.0\n"
+ ]
+ }
+ ],
+ "source": [
+ "import paddle\n",
+ "print(paddle.__version__)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "## 二、自定义数据集读取器\n",
+ "\n",
+ "常见的开发任务中，我们并不一定会拿到标准的数据格式，好在我们可以通过自定义Reader的形式来随心所欲读取自己想要数据。 \n",
+ "\n",
+ "设计合理的Reader往往可以带来更好的性能，我们可以将读取标签文件列表、制作图像文件列表等必要操作在`__init__`特殊方法中实现。这样就可以在实例化`Reader`时装入内存，避免使用时频繁读取导致增加额外开销。同样我们可以在`__getitem__`特殊方法中实现如图像增强、归一化等个性操作，完成数据读取后即可释放该部分内存。 \n",
+ "需要我们注意的是，如果不能保证自己数据十分纯净，可以通过`try`和`expect`来捕获异常并指出该数据的位置。当然也可以制定一个策略，使其在发生数据读取异常后依旧可以正常进行训练。 "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": false
+ },
+ "source": [
+ "### 2.1 数据展示\n",
+ "\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
- "
\n",
- "
\n",
- "
\n",
- "
\n",
- "
\n",
- "