Add FQA for optimizer #4081
Merged
Add FQA for optimizer #4081
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ligoml
reviewed
Nov 12, 2021
docs/faq/train_cn.md
Outdated
|
|
||
| ---------- | ||
|
|
||
| ##### 问题：如何在同一个优化器中定义不同参数的优化策略,比如bias的参数weight_decay的值为0.0，非bias的参数weight_decay的值为0.01？ |
Collaborator
There was a problem hiding this comment.
注意标点符号统一：
另外序号格式有些问题，需要改下 md
Ligoml
reviewed
Nov 12, 2021
docs/faq/train_cn.md
Outdated
| ##### 问题：如何在同一个优化器中定义不同参数的优化策略,比如bias的参数weight_decay的值为0.0，非bias的参数weight_decay的值为0.01？ | ||
|
|
||
| + 答复： | ||
| - 1. `AdamW`的参数`apply_decay_param_fun`可以用来选择哪些参数使用decay_weight策略。可以参考AdamW优化器的[API文档](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/optimizer/AdamW_cn.html#adamw) |
Collaborator
There was a problem hiding this comment.
api 文档的链接方式通常采用直接在出现位置加超链接的形式，建议统一下：
AdamW的参数apply_decay_param_fun可以用来选择哪些参数使用decay_weight策略。
docs/faq/train_cn.md
Outdated
|
|
||
| ---------- | ||
|
|
||
| #### 问题：paddle fluid如何自定义优化器，自定义更新模型参数的规则？ |
Collaborator
There was a problem hiding this comment.
官网的解析可能和本地 markdown 不太一致，还是以官网的预览为准吧
docs/faq/train_cn.md
Outdated
| #### 问题：paddle fluid如何自定义优化器，自定义更新模型参数的规则？ | ||
| + 答复： | ||
| - 1. 要定义全新优化器，自定义优化器中参数的更新规则，可以通过继承fluid.Optimizer，重写_append_optimize_op方法实现。不同优化器实现原理各不相同，一般流程是先获取learning_rate，gradients参数，可训练参数，以及该优化器自身特别需要的参数，然后实现更新参数的代码，最后返回更新后的参数。代码样例可参考[paddle源码](https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/fluid/optimizer.py)中AdamOptimizer等优化器的实现。 | ||
| - 2. 使用现有的常用优化器，可以在创建`Param`的时候，可以通过设置`ParamAttr`的属性来控制参数的属性，可以通过设置`regularize`，`learning_rate`等参数简单设置参数的更新规则。可以参考[ParamAttr文档](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/ParamAttr_cn.html#paramattr)。 |
Collaborator
There was a problem hiding this comment.
内容设置方面，建议从动态图和静态图两个方面分别来讲下如何自定义优化器，让这条 FAQ 能帮到更多的用户~
Contributor
Author
There was a problem hiding this comment.
用户开发的话，动态图和静态图代码可以是统一的
除非自己重写C++代码，开发底层的算子，重新编译paddle，但是这对用户来说会比较繁琐
sljlp
force-pushed
the
add_faq_for_opt
branch
from
9e5f3d8 to
028e2bc
Compare
Ligoml
approved these changes
Nov 12, 2021
sljlp
force-pushed
the
add_faq_for_opt
branch
from
028e2bc to
7807552
Compare
sljlp
added a commit
to sljlp/docs
that referenced
this pull request
Nov 12, 2021
Ligoml
added a commit
that referenced
this pull request
Nov 16, 2021
Co-authored-by: Ligoml <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add FQA for Optimizer