A 2D case: We write $f_{\{q,k\}}$ in a multiplication matrix:
$f_{\{q,k\}}(x_m, m) = \begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix} \begin{pmatrix} W_{\{q,k\}}^{(11)} & W_{\{q,k\}}^{(12)} \\ W_{\{q,k\}}^{(21)} & W_{\{q,k\}}^{(22)} \end{pmatrix} \begin{pmatrix} x_m^{(1)} \\ x_m^{(2)} \end{pmatrix}$
General form
-
MatMul
-
$f_{\{q,k\}}(x_m, m) = R_{\Theta,m}^d W_{\{q,k\}} x_m$
-
where $R_{\Theta,m}^d$ denotes the rotation matrix for $m-th$ token with dimension $d$
$R_{\Theta,m}^d = \begin{pmatrix} \cos m\theta_1 & -\sin m\theta_1 & 0 & 0 & \cdots & 0 & 0 \\ \sin m\theta_1 & \cos m\theta_1 & 0 & 0 & \cdots & 0 & 0 \\ 0 & 0 & \cos m\theta_2 & -\sin m\theta_2 & \cdots & 0 & 0 \\ 0 & 0 & \sin m\theta_2 & \cos m\theta_2 & \cdots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & 0 & 0 & \cdots & \cos m\theta_{d/2} & -\sin m\theta_{d/2} \\ 0 & 0 & 0 & 0 & \cdots & \sin m\theta_{d/2} & \cos m\theta_{d/2} \end{pmatrix}$
-
$\Theta = \{\theta_i = 10000^{-2(i-1)/d}, i \in [1, 2, ..., d/2]\}.$
- One can prove that this setting provides a long-term decay property, which means the inner-product will decay when the relative position increase.
-
Notes
- Dimension of $R_{\Theta,m}^d$ is actually $(d \times d_k)$, the above matrix shows the case where $d_k = d$
- Multiplying with $R_{\Theta,m}^d$ means that every pair of two numbers in the vector will be rotated by $m\theta$
- It is also important to remember that $R_{\Theta,m}^d$ is rotating $Q$ or $K$, i.e., the multiplication result from $W_{\{q,k\}} x_m$, but not $W_{\{q,k\}}$, although it makes perfect sense according to math expression.
-
-
Visualization
