进阶 tensorflow.org 2026-10-22 02:47:47 · 22 阅读
第6章 自动微分
# 梯度与自动微分入门
## 自动微分与梯度
自动微分对于实现反向传播等训练神经网络的机器学习算法非常有用。
本指南将介绍在 TensorFlow 中计算梯度的各种方法,尤其是在 eager execution 模式下的用法。
## 环境准备
```python
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
```
2024-08-15 01:30:07.003169: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-08-15 01:30:07.023862: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-08-15 01:30:07.029954: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered## 计算梯度 要实现自动微分,TensorFlow 需要在前向传播过程中记住各操作的执行顺序,然后在反向传播时按相反的顺序遍历这些操作,从而计算梯度。 ## 梯度带 TensorFlow 提供 `tf.GradientTape` API 来实现自动微分,也就是计算某个计算相对于其输入(通常是 `tf.Variables`)的梯度。TensorFlow 会把在 `tf.GradientTape` 上下文中执行的相关操作"记录"到一条"带"(tape)上,然后利用反向模式微分,根据这条带来计算"被记录"计算的梯度。 下面是一个简单的例子: ```python x = tf.Variable(3.0) with tf.GradientTape() as tape: y = x2 ```
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1723685409.408818 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.408818 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.408818 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685409.408818 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1723685410.685265 20970 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355记录了一些操作之后,就可以用 `GradientTape.gradient(target, sources)` 来计算某个目标(通常是损失值)相对于某个源(通常是模型的变量)的梯度: ```python # dy = 2x * dx dy_dx = tape.gradient(y, x) dy_dx.numpy() ```
6.0上面的例子用的是标量,但 `tf.GradientTape` 对任意张量同样适用: ```python w = tf.Variable(tf.random.normal((3, 2)), name='w') b = tf.Variable(tf.zeros(2, dtype=tf.float32), name='b') x = [[1., 2., 3.]] with tf.GradientTape(persistent=True) as tape: y = x @ w + b loss = tf.reduce_mean(y2) ``` 要计算 `loss` 相对于两个变量的梯度,可以把它们一起作为源传给 `gradient` 方法。这个方法对源的传入形式很灵活,接受列表或字典的任意嵌套组合,并以相同的结构返回梯度(参见 `tf.nest`)。 ```python [dl_dw, dl_db] = tape.gradient(loss, [w, b]) ``` 相对于每个源的梯度,其形状与该源相同: ```python print(w.shape) print(dl_dw.shape) ```
(3, 2) (3, 2)下面再次计算梯度,这次传入的是变量字典: ```python my_vars = { 'w': w, 'b': b } grad = tape.gradient(loss, my_vars) grad['b'] ```
<tf.Tensor: shape=(2,), dtype=float32, numpy=array([7.5046124, 2.0025406], dtype=float32)>## 相对于模型的梯度 为了做检查点保存和导出,通常会把 `tf.Variables` 收集到 `tf.Module` 或其子类(`layers.Layer`、`keras.Model`)中。 在大多数情况下,你需要计算的是相对于模型可训练变量的梯度。由于 `tf.Module` 的所有子类都会把变量聚合到 `Module.trainable_variables` 属性中,只需几行代码即可完成计算: ```python layer = tf.keras.layers.Dense(2, activation='relu') x = tf.constant([[1., 2., 3.]]) with tf.GradientTape() as tape: # 前向传播 y = layer(x) loss = tf.reduce_mean(y2) # 计算相对于每个可训练变量的梯度 grad = tape.gradient(loss, layer.trainable_variables) for var, g in zip(layer.trainable_variables, grad): print(f'{var.name}, shape: {g.shape}') ```
kernel, shape: (3, 2) bias, shape: (2,)## 控制带所监视的内容 默认行为是:在访问可训练的 `tf.Variable` 之后,记录所有相关操作。这样做的原因是: - 带需要知道前向传播中要记录哪些操作,才能在反向传播时计算梯度。 - 带会持有中间输出的引用,所以没必要记录多余的操作。 - 最常见的场景就是计算损失相对于模型所有可训练变量的梯度。 例如,下面的代码无法计算梯度,因为默认情况下 `tf.Tensor` 不会被"监视",而 `tf.Variable` 也不可训练: ```python # 可训练的变量 x0 = tf.Variable(3.0, name='x0') # 不可训练 x1 = tf.Variable(3.0, name='x1', trainable=False) # 不是 Variable:变量 + 张量的结果是张量 x2 = tf.Variable(2.0, name='x2') + 1.0 # 不是变量 x3 = tf.constant(3.0, name='x3') with tf.GradientTape() as tape: y = (x02) + (x12) + (x22) grad = tape.gradient(y, [x0, x1, x2, x3]) for g in grad: print(g) ```
tf.Tensor(6.0, shape=(), dtype=float32) None None None可以用 `GradientTape.watched_variables` 方法列出带正在监视的变量: ```python [var.name for var in tape.watched_variables()] ```
['x0:0']`tf.GradientTape` 提供了一些钩子,让用户可以控制监视的对象。 要记录相对于某个 `tf.Tensor` 的梯度,需要调用 `GradientTape.watch(x)`: ```python x = tf.constant(3.0) with tf.GradientTape() as tape: tape.watch(x) y = x2 # dy = 2x * dx dy_dx = tape.gradient(y, x) print(dy_dx.numpy()) ```
6.0反过来,如果想禁用监视所有 `tf.Variables` 的默认行为,可以在创建梯度带时设置 `watch_accessed_variables=False`。下面的计算用了两个变量,但只计算其中一个变量的梯度: ```python x0 = tf.Variable(0.0) x1 = tf.Variable(10.0) with tf.GradientTape(watch_accessed_variables=False) as tape: tape.watch(x1) y0 = tf.math.sin(x0) y1 = tf.nn.softplus(x1) y = y0 + y1 ys = tf.reduce_sum(y) ``` 由于没有对 `x0` 调用 `GradientTape.watch`,因此不会计算相对于它的梯度: ```python # dys/dx1 = exp(x1) / (1 + exp(x1)) = sigmoid(x1) grad = tape.gradient(ys, {'x0': x0, 'x1': x1}) print('dy/dx0:', grad['x0']) print('dy/dx1:', grad['x1'].numpy()) ```
dy/dx0: None dy/dx1: 0.9999546## 中间结果 你也可以请求输出相对于 `tf.GradientTape` 上下文中计算出的中间值的梯度。 ```python x = tf.constant(3.0) with tf.GradientTape() as tape: tape.watch(x) y = x * x z = y * y # 用带计算 z 相对于中间值 y 的梯度。 # dz_dy = 2 * y,而 y = x 2 = 9 print(tape.gradient(z, y).numpy()) ```
18.0默认情况下,只要调用 `GradientTape.gradient` 方法,`GradientTape` 持有的资源就会被释放。要在同一个计算上计算多个梯度,需要创建 `persistent=True` 的梯度带。这样就可以多次调用 `gradient` 方法,资源会在带对象被垃圾回收时才释放。例如: ```python x = tf.constant([1, 3.0]) with tf.GradientTape(persistent=True) as tape: tape.watch(x) y = x * x z = y * y print(tape.gradient(z, x).numpy()) # [4.0, 108.0](即 x = [1.0, 3.0] 处的 4 * x3) print(tape.gradient(y, x).numpy()) # [2.0, 6.0](即 x = [1.0, 3.0] 处的 2 * x) ```
[ 4. 108.] [2. 6.]```python del tape # 释放对带的引用 ``` ### 性能注意事项 - 在梯度带上下文中执行操作会带来很小的开销。对于大多数 eager execution 来说这个成本并不明显,但还是应该只在必要的地方使用带上下文。 - 梯度带会把中间结果(包括输入和输出)保存在内存中,供反向传播使用。 为了效率,有些运算(如 ReLU)不需要保留中间结果,在前向传播时就会被裁剪掉。但如果给带设置了 `persistent=True`,则什么都不丢弃,峰值内存占用会更高。 ## 非标量目标的梯度 梯度本质上是对标量进行的运算。 ```python x = tf.Variable(2.0) with tf.GradientTape(persistent=True) as tape: y0 = x2 y1 = 1 / x print(tape.gradient(y0, x).numpy()) print(tape.gradient(y1, x).numpy()) ```
4.0 -0.25因此,如果请求多个目标的梯度,每个源得到的结果是: - 各目标之和的梯度,等价于 - 各目标的梯度之和。 ```python x = tf.Variable(2.0) with tf.GradientTape() as tape: y0 = x2 y1 = 1 / x print(tape.gradient({'y0': y0, 'y1': y1}, x).numpy()) ```
3.75类似地,如果目标不是标量,计算的就是其总和的梯度: ```python x = tf.Variable(2.) with tf.GradientTape() as tape: y = x * [3., 4.] print(tape.gradient(y, x).numpy()) ```
7.0这样处理一组损失之和的梯度,或逐元素损失计算之和的梯度就非常简单。 如果需要对每一项分别求梯度,请参考 Jacobians 一节。 某些情况下可以省去 Jacobian。对于逐元素计算,总和的梯度就给出了每个元素相对于其输入元素的导数,因为各元素相互独立: ```python x = tf.linspace(-10.0, 10.0, 200+1) with tf.GradientTape() as tape: tape.watch(x) y = tf.nn.sigmoid(x) dy_dx = tape.gradient(y, x) plt.plot(x, y, label='y') plt.plot(x, dy_dx, label='dy/dx') plt.legend() _ = plt.xlabel('x') ```  ## 控制流 由于梯度带是在操作执行时记录它们的,所以 Python 控制流(例如 `if` 和 `while` 语句)可以被自然地处理。 下面在 `if` 的每个分支中使用了不同的变量,梯度只会连接到实际被使用的那一个: ```python x = tf.constant(1.0) v0 = tf.Variable(2.0) v1 = tf.Variable(2.0) with tf.GradientTape(persistent=True) as tape: tape.watch(x) if x > 0.0: result = v0 else: result = v12 dv0, dv1 = tape.gradient(result, [v0, v1]) print(dv0) print(dv1) ```
tf.Tensor(1.0, shape=(), dtype=float32) None注意,控制语句本身是不可微分的,因此对基于梯度的优化器来说是不可见的。 在上面例子中,根据 `x` 的取值,带要么记录 `result = v0`,要么记录 `result = v12`。相对于 `x` 的梯度始终为 `None`: ```python dx = tape.gradient(result, x) print(dx) ```
None## gradient 返回 None 的情况 当目标与源之间没有连接时,`gradient` 会返回 `None`。 ```python x = tf.Variable(2.) y = tf.Variable(3.) with tf.GradientTape() as tape: z = y * y print(tape.gradient(z, x)) ```
None这里 `z` 显然与 `x` 无关,但还有一些不那么明显的情况也会导致梯度断开。 ### 1. 用张量替换了变量 在"控制带所监视的内容"一节中提到,带会自动监视 `tf.Variable`,但不会监视 `tf.Tensor`。 一个常见的错误是无意中用 `tf.Tensor` 替换了 `tf.Variable`,而不是用 `Variable.assign` 来更新 `tf.Variable`。例如: ```python x = tf.Variable(2.0) for epoch in range(2): with tf.GradientTape() as tape: y = x+1 print(type(x).__name__, ":", tape.gradient(y, x)) x = x + 1 # 这里应该用 `x.assign_add(1)` ```
ResourceVariable : tf.Tensor(1.0, shape=(), dtype=float32) EagerTensor : None### 2. 在 TensorFlow 之外进行了计算 如果计算跳出了 TensorFlow,带就无法记录梯度路径。例如: ```python x = tf.Variable([[1.0, 2.0], [3.0, 4.0]], dtype=tf.float32) with tf.GradientTape() as tape: x2 = x2 # 这一步用 NumPy 计算 y = np.mean(x2, axis=0) # 与大多数运算一样,reduce_mean 会通过 `tf.convert_to_tensor` # 把 NumPy 数组转换为常量张量 y = tf.reduce_mean(y, axis=0) print(tape.gradient(y, x)) ```
None### 3. 对整数或字符串求梯度 整数和字符串是不可微分的。如果计算路径使用了这些数据类型,就不会有梯度。 没有人会期待字符串可微分,但如果没指定 dtype,很容易不小心创建出 int 类型的常量或变量。 ```python x = tf.constant(10) with tf.GradientTape() as g: g.watch(x) y = x * x print(g.gradient(y, x)) ```
WARNING:tensorflow:The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int32 NoneTensorFlow 不会自动做类型转换,所以实际使用中通常会直接报类型错误,而不是悄悄丢失梯度。 ### 4. 穿过了有状态的对象 状态会阻断梯度。读取有状态对象时,带只能看到当前状态,而看不到产生该状态的历史。 `tf.Tensor` 是不可变的,创建之后无法更改。它有一个值,但没有状态。到目前为止讨论的所有运算也都是无状态的:`tf.matmul` 的输出只取决于输入。 `tf.Variable` 有内部状态——它的值。使用变量时会读取这个状态。对变量求梯度很正常,但变量的状态会阻断梯度继续向更早的计算回溯。例如: ```python x0 = tf.Variable(3.0) x1 = tf.Variable(0.0) with tf.GradientTape() as tape: # 更新 x1 = x1 + x0 x1.assign_add(x0) # 带从 x1 开始记录 y = x12 # y = (x1 + x0)2 # 这样行不通 print(tape.gradient(y, x0)) #dy/dx0 = 2*(x1 + x0) ```
None类似地,`tf.data.Dataset` 迭代器和 `tf.queues` 也是有状态的,会阻断经过它们的所有张量的梯度。 ### 未注册梯度 有些 `tf.Operations` 被注册为不可微分,会返回 `None`;另一些则根本没有注册梯度。 tf.raw_ops 页面列出了哪些低层运算注册了梯度。 如果你对一个没有注册梯度的浮点运算求梯度,带会直接抛出错误,而不是悄悄返回 `None`,这样你就能知道出了问题。 例如,`tf.image.adjust_contrast` 函数封装了 raw_ops.AdjustContrastv2,它本可以有梯度,但梯度并未实现: ```python image = tf.Variable([[[0.5, 0.0, 0.0]]]) delta = tf.Variable(0.1) with tf.GradientTape() as tape: new_image = tf.image.adjust_contrast(image, delta) try: print(tape.gradient(new_image, [image, delta])) assert False # 正常情况下不应执行到这里 except LookupError as e: print(f'{type(e).__name__}: {e}') ```
LookupError: gradient registry has no entry for: AdjustContrastv2如果确实需要对这个运算求微分,要么实现并注册它的梯度(使用 `tf.RegisterGradient`),要么用其他运算重新实现该函数。 ### 用零代替 None 有时用 `0` 代替 `None` 来表示未连接的梯度会更方便。你可以通过 `unconnected_gradients` 参数来决定未连接梯度时返回什么: ```python x = tf.Variable([2., 2.]) y = tf.Variable(3.) with tf.GradientTape() as tape: z = y2 print(tape.gradient(z, x, unconnected_gradients=tf.UnconnectedGradients.ZERO)) ```
tf.Tensor([0. 0.], shape=(2,), dtype=float32)