Switch from sin(x) to exp(x) as the target function in polynomial_autograd (pytorch#3597)

Fixes pytorch#3596

This PR branch uses 'exp(x)' instead of 'sin(x)' as the 'y' (target)
variable. The reason for this is faster and clearer convergence, since
exp(x) is well approximated by a Taylor expansion around the origin.
Also, all the derivatives are exp(0)=1, so the polynomial coefficients
are trivially '1/n! '(for the 'n-th' term).

I am also improving the information output to the user inside the
optimization loop.

Finally, I am making the top doc string a raw string, to avoid a python3
warning about it.


cc @albanD @jbschlosser

---------

Co-authored-by: Svetlana Karslioglu <[email protected]>

Author: 2 people

beginner_source/examples_autograd/polynomial_autograd.py

@@ -1,4 +1,4 @@
-"""
+r"""
 PyTorch: Tensors and autograd
 -------------------------------
 
@@ -27,8 +27,8 @@
 # Create Tensors to hold input and outputs.
 # By default, requires_grad=False, which indicates that we do not need to
 # compute gradients with respect to these Tensors during the backward pass.
-x = torch.linspace(-math.pi, math.pi, 2000, dtype=dtype)
-y = torch.sin(x)
+x = torch.linspace(-1, 1, 2000, dtype=dtype)
+y = torch.exp(x) # A Taylor expansion would be 1 + x + (1/2) x**2 + (1/3!) x**3 + ...
 
 # Create random Tensors for weights. For a third order polynomial, we need
 # 4 weights: y = a + b x + c x^2 + d x^3
@@ -39,17 +39,23 @@
 c = torch.randn((), dtype=dtype, requires_grad=True)
 d = torch.randn((), dtype=dtype, requires_grad=True)
 
-learning_rate = 1e-6
-for t in range(2000):
+initial_loss = 1.
+learning_rate = 1e-5
+for t in range(5000):
     # Forward pass: compute predicted y using operations on Tensors.
     y_pred = a + b * x + c * x ** 2 + d * x ** 3
 
     # Compute and print loss using operations on Tensors.
     # Now loss is a Tensor of shape (1,)
     # loss.item() gets the scalar value held in the loss.
     loss = (y_pred - y).pow(2).sum()
+
+    # Calculare initial loss, so we can report loss relative to it
+    if t==0:
+        initial_loss=loss.item()
+
     if t % 100 == 99:
-        print(t, loss.item())
+        print(f'Iteration t = {t:4d}  loss(t)/loss(0) = {round(loss.item()/initial_loss, 6):10.6f}  a = {a.item():10.6f}  b = {b.item():10.6f}  c = {c.item():10.6f}  d = {d.item():10.6f}')
 
     # Use autograd to compute the backward pass. This call will compute the
     # gradient of loss with respect to all Tensors with requires_grad=True.