←July 27, 2026

Inference APIAPI

You Could Have Come Up With Kimi Delta Attention

Math notation

A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣q⟩\lvert q\rangle∣q⟩ is a column vector, ⟨k∣\langle k\rvert⟨k∣ is a row vector, ⟨k∣q⟩\langle k\rvert q\rangle⟨k∣q⟩ is a number, and ∣v⟩⟨k∣\lvert v\rangle\langle k\rvert∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space.

Modern linear attention variants are complex, and a upon first glance it is not so easy to see what they are designed to achieve. For reference here is the state update equation for Kimi Delta Attention (KDA):

S~t=St−1Diag⁡(αt)\widetilde S_t

S_{t-1}\operatorname{Diag}(\alpha_t)St​=St−1​Diag(αt​) ∣v^t⟩=S~t∣kt⟩\lvert\widehat v_t\rangle = \widetilde S_t\lvert k_t\rangle∣vt​⟩=St​∣kt​⟩ ∣et⟩=βt(∣vt⟩−∣v^t⟩)\lvert e_t\rangle = \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right)∣et​⟩=βt​(∣vt​⟩−∣vt​⟩) St=S~t+∣et⟩⟨kt∣S_t = \widetilde S_t+\lvert e_t\rangle\langle k_t\rvertSt​=St​+∣et​⟩⟨kt​∣ ∣ot⟩=St(dk−1/2∣qt⟩)\lvert o_t\rangle = S_t\left(d_k^{-1/2}\lvert q_t\rangle\right)∣ot​⟩=St​(dk−1/2​∣qt​⟩)

The reason they are so difficult to understand is that this is the latest in a family of linear attention variants that have been developed over the last few years and the complexity of them has inevitably ballooned such that from the outside the latest variants appear inaccessible.

In this post we are going to walk through the DeltaNet family of linear attention variants, two of which are used by the latest Qwen and Kimi model families, and show how you might have arrived at the same equations by asserting simple things about your hidden state.

That is the route we will take:

softmax attention  →  linear attention  →  DeltaNet  →  Gated DeltaNet  →  KDA

Only after deriving KDA will we turn to the recurrent and chunkwise Triton programs that execute it.

1. Begin with quadratic attention

For a query at token ttt, ordinary causal softmax attention is

ati=exp⁡ ⁣(s⟨ki∣qt⟩)∑j≤texp⁡ ⁣(s⟨kj∣qt⟩),s=dk−1/2,∣ot⟩=∑i≤tati∣vi⟩.\begin{aligned} a_{ti} &= \frac{ \exp!\left(s\langle k_i\rvert q_t\rangle\right) }{ \sum_{j\leq t} \exp!\left(s\langle k_j\rvert q_t\rangle\right) }, \qquad s=d_k^{-1/2},\ \lvert o_t\rangle &= \sum_{i\leq t}a_{ti}\lvert v_i\rangle. \end{aligned}ati​∣ot​⟩​=∑j≤t​exp(s⟨kj​∣qt​⟩)exp(s⟨ki​∣qt​⟩)​,s=dk−1/2​,=i≤t∑​ati​∣vi​⟩.​

Every attention weight is a scalar. It measures the similarity between one key and one query, then softmax turns all of the scores for that query into a distribution. The output is a weighted sum of value vectors.

Over a sequence of length TTT, there are T2T^2T2 key-query pairs. During autoregressive inference we can cache the keys and values instead of recomputing them, but the cache still grows with the sequence and every new query still has to inspect the entire history.

The obstacle to rearranging this computation is the softmax. Its denominator depends jointly on the current query and every earlier key. So, for the moment, remove it.

1.1 Remove the softmax

For clarity, absorb the constant scale sss into the query. The deliberately bare version of attention is then

∣ot⟩=∑i≤t⟨ki∣qt⟩∣vi⟩.\lvert o_t\rangle

\sum_{i\leq t} \langle k_i\rvert q_t\rangle \lvert v_i\rangle.∣ot​⟩=i≤t∑​⟨ki​∣qt​⟩∣vi​⟩.

The scalar inner product can move to the right:

∣ot⟩=∑i≤t∣vi⟩⟨ki∣qt⟩=(∑i≤t∣vi⟩⟨ki∣)∣qt⟩.\begin{aligned} \lvert o_t\rangle &= \sum_{i\leq t} \lvert v_i\rangle \langle k_i\rvert q_t\rangle\ &= \left( \sum_{i\leq t} \lvert v_i\rangle\langle k_i\rvert \right) \lvert q_t\rangle. \end{aligned}∣ot​⟩​=i≤t∑​∣vi​⟩⟨ki​∣qt​⟩=(i≤t∑​∣vi​⟩⟨ki​∣)∣qt​⟩.​

Everything that depends on the past can now be collected into one matrix of a fixed size V×KV \times KV×K:

St=∑i≤t∣vi⟩⟨ki∣\boxed{ S_t = \sum_{i\leq t} \lvert v_i\rangle\langle k_i\rvert }St​=i≤t∑​∣vi​⟩⟨ki​∣​

and attention becomes a recurrent write followed by a read:

St=St−1+∣vt⟩⟨kt∣,∣ot⟩=St∣qt⟩.\boxed{ \begin{aligned} S_t &= S_{t-1} + \lvert v_t\rangle\langle k_t\rvert,\ \lvert o_t\rangle &= S_t\lvert q_t\rangle. \end{aligned} }St​∣ot​⟩​=St−1​+∣vt​⟩⟨kt​∣,=St​∣qt​⟩.​​

The identity

(∣v⟩⟨k∣)∣q⟩=⟨k∣q⟩∣v⟩\left(\lvert v\rangle\langle k\rvert\right)\lvert q\rangle

\langle k\rvert q\rangle\lvert v\rangle(∣v⟩⟨k∣)∣q⟩=⟨k∣q⟩∣v⟩

is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state StS_tSt​.

This is linear in sequence length rather than quadratic: scan the tokens once, updating the same dv×dkd_v\times d_kdv​×dk​ state at every step. We have paid for that efficiency by discarding softmax’s normalization and selectivity. More sophisticated linear-attention methods use feature maps and normalizers, but this unadorned form exposes the memory problem that motivates DeltaNet.

1.2 Addition is not assignment

Suppose we write a pair ∣vt⟩⟨kt∣\lvert v_t\rangle\langle k_t\rvert∣vt​⟩⟨kt​∣ and immediately query the new state with that same key:

St∣kt⟩=(St−1+∣vt⟩⟨kt∣)∣kt⟩=St−1∣kt⟩+∣vt⟩⟨kt∣kt⟩⏟1=St−1∣kt⟩+∣vt⟩.\begin{aligned} S_t\lvert k_t\rangle &= \left( S_{t-1} + \lvert v_t\rangle\langle k_t\rvert \right) \lvert k_t\rangle\ &= S_{t-1}\lvert k_t\rangle + \lvert v_t\rangle \underbrace{\langle k_t\rvert k_t\rangle}{1}\ &= S{t-1}\lvert k_t\rangle+\lvert v_t\rangle. \end{aligned}St​∣kt​⟩​=(St−1​+∣vt​⟩⟨kt​∣)∣kt​⟩=St−1​∣kt​⟩+∣vt​⟩1⟨kt​∣kt​⟩​​=St−1​∣kt​⟩+∣vt​⟩.​

The write does not make the memory return ∣vt⟩\lvert v_t\rangle∣vt​⟩. It adds ∣vt⟩\lvert v_t\rangle∣vt​⟩ to whatever the memory already returned.

If the old state already produced the correct value, the additive write makes the new state produce twice that value. More generally, keys are not mutually orthogonal, so every write can interfere with previous writes. Linear attention has given us a compact associative memory, but its update behaves like += when what we want is closer to =.

2. DeltaNet: write the error, not the value

DeltaNet replaces the unconditional linear-attention write with a delta-rule correction. There are two useful ways to derive it.

2.1 Derivation one: demand that the write can be read back

Before writing token ttt, ask the memory what it currently associates with the new key:

∣v^t⟩=St−1∣kt⟩.\lvert\widehat v_t\rangle

S_{t-1}\lvert k_t\rangle.∣vt​⟩=St−1​∣kt​⟩.

If we want the memory to return ∣vt⟩\lvert v_t\rangle∣vt​⟩, we should not add the whole value. We should add only the difference:

∣vt⟩−∣v^t⟩.\lvert v_t\rangle-\lvert\widehat v_t\rangle.∣vt​⟩−∣vt​⟩.

Introduce a learned write strength βt∈[0,1]\beta_t\in[0,1]βt​∈[0,1] and define

∣et⟩=βt(∣vt⟩−St−1∣kt⟩).\lvert e_t\rangle

\beta_t \left( \lvert v_t\rangle - S_{t-1}\lvert k_t\rangle \right).∣et​⟩=βt​(∣vt​⟩−St−1​∣kt​⟩).

Then write this error at the current key:

St=St−1+∣et⟩⟨kt∣.\boxed{ S_t = S_{t-1} + \lvert e_t\rangle\langle k_t\rvert. }St​=St−1​+∣et​⟩⟨kt​∣.​

Now immediately read the same key:

St∣kt⟩=St−1∣kt⟩+∣et⟩⟨kt∣kt⟩=(1−βt)St−1∣kt⟩+βt∣vt⟩.\begin{aligned} S_t\lvert k_t\rangle &= S_{t-1}\lvert k_t\rangle + \lvert e_t\rangle \langle k_t\rvert k_t\rangle\ &= (1-\beta_t)S_{t-1}\lvert k_t\rangle + \beta_t\lvert v_t\rangle. \end{aligned}St​∣kt​⟩​=St−1​∣kt​⟩+∣et​⟩⟨kt​∣kt​⟩=(1−βt​)St−1​∣kt​⟩+βt​∣vt​⟩.​

When βt=1\beta_t=1βt​=1, the result is exactly ∣vt⟩\lvert v_t\rangle∣vt​⟩. Smaller βt\beta_tβt​ moves the old prediction partway towards the target.

The correction is also local in key space. For any query ∣x⟩\lvert x\rangle∣x⟩ orthogonal to the current key,

⟨kt∣x⟩=0⟹(St−St−1)∣x⟩=∣et⟩⟨kt∣x⟩⏟0=0.\langle k_t\rvert x\rangle=0 \quad\Longrightarrow\quad (S_t-S_{t-1})\lvert x\rangle = \lvert e_t\rangle \underbrace{\langle k_t\rvert x\rangle}_{0} =0.⟨kt​∣x⟩=0⟹(St​−St−1​)∣x⟩=∣et​⟩0⟨kt​∣x⟩​​=0.

So the rank-one write changes the response in the selected key direction while leaving every orthogonal direction alone.

2.2 Derivation two: take one step on reconstruction loss

The same update falls out of an online learning objective. Treat the current key-value pair as one training example for the linear map SSS:

Lt(S)=12∥S∣kt⟩−∣vt⟩∥22.\mathcal L_t(S)

\frac12 \left| S\lvert k_t\rangle-\lvert v_t\rangle \right|_2^2.Lt​(S)=21​∥S∣kt​⟩−∣vt​⟩∥22​.

Its gradient with respect to the state is

∇SLt(S)=(S∣kt⟩−∣vt⟩)⟨kt∣.\nabla_S\mathcal L_t(S)

\left( S\lvert k_t\rangle-\lvert v_t\rangle \right) \langle k_t\rvert.∇S​Lt​(S)=(S∣kt​⟩−∣vt​⟩)⟨kt​∣.

This is visibly an outer product: a value-space prediction error times the key bra at which that error was observed. Take one gradient-descent step of size βt\beta_tβt​ from St−1S_{t-1}St−1​:

St=St−1−βt∇SLt(St−1)=St−1−βt(St−1∣kt⟩−∣vt⟩)⟨kt∣=St−1+βt(∣vt⟩−St−1∣kt⟩)⟨kt∣.\begin{aligned} S_t &= S_{t-1} - \beta_t\nabla_S\mathcal L_t(S_{t-1})\ &= S_{t-1} - \beta_t \left( S_{t-1}\lvert k_t\rangle-\lvert v_t\rangle \right) \langle k_t\rvert\ &= S_{t-1} + \beta_t \left( \lvert v_t\rangle-S_{t-1}\lvert k_t\rangle \right) \langle k_t\rvert. \end{aligned}St​​=St−1​−βt​∇S​Lt​(St−1​)=St−1​−βt​(St−1​∣kt​⟩−∣vt​⟩)⟨kt​∣=St−1​+βt​(∣vt​⟩−St−1​∣kt​⟩)⟨kt​∣.​

This is exactly the update we got by requiring immediate reconstruction. The two interpretations are the same:

• as a memory operation, βt\beta_tβt​ controls how strongly to replace the old association;
• as online learning, βt\beta_tβt​ is the step size;
• as linear algebra, the change is a rank-one outer product.

2.3 The DeltaNet state transition

Expanding the error exposes DeltaNet as a structured state transition plus a new input:

St=St−1+βt(∣vt⟩−St−1∣kt⟩)⟨kt∣=St−1(I−βt∣kt⟩⟨kt∣)+βt∣vt⟩⟨kt∣.\begin{aligned} S_t &= S_{t-1} + \beta_t \left( \lvert v_t\rangle-S_{t-1}\lvert k_t\rangle \right) \langle k_t\rvert\ &= S_{t-1} \left( I-\beta_t\lvert k_t\rangle\langle k_t\rvert \right) + \beta_t\lvert v_t\rangle\langle k_t\rvert. \end{aligned}St​​=St−1​+βt​(∣vt​⟩−St−1​∣kt​⟩)⟨kt​∣=St−1​(I−βt​∣kt​⟩⟨kt​∣)+βt​∣vt​⟩⟨kt​∣.​

For a unit key, I−βt∣kt⟩⟨kt∣I-\beta_t\lvert k_t\rangle\langle k_t\rvertI−βt​∣kt​⟩⟨kt​∣ has eigenvalue 1−βt1-\beta_t1−βt​ in the current key direction and eigenvalue 111 in every orthogonal direction. It removes the old association along the current key before adding the new one.

DeltaNet fixes the write. It does not yet fix the lifetime of the state.

3. Gated DeltaNet: sometimes old information should disappear

The linear state compresses the whole history into one matrix. A read

St∣q⟩=∑i≤t⟨ki∣q⟩∣vi⟩S_t\lvert q\rangle

\sum_{i\leq t} \langle k_i\rvert q\rangle\lvert v_i\rangleSt​∣q⟩=i≤t∑​⟨ki​∣q⟩∣vi​⟩

cannot choose to skip an individual old token after that token has been folded into StS_tSt​. Every stored direction that overlaps the query contributes. The delta rule can correct the state around the current key, but stale information in other directions remains available and can distort future reads.

We therefore need a way to forget the old state before using it. Let αt∈[0,1]\alpha_t\in[0,1]αt​∈[0,1] be a learned scalar retention gate:

S~t=αtSt−1.\widetilde S_t

\alpha_t S_{t-1}.St​=αt​St−1​.

Run the same delta rule against this gated state:

S~t=αtSt−1,forget,∣v^t⟩=S~t∣kt⟩,predict,∣et⟩=βt(∣vt⟩−∣v^t⟩),correct,St=S~t+∣et⟩⟨kt∣,write.\boxed{ \begin{aligned} \widetilde S_t &= \alpha_tS_{t-1}, &&\text{forget},\ \lvert\widehat v_t\rangle &= \widetilde S_t\lvert k_t\rangle, &&\text{predict},\ \lvert e_t\rangle &= \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right), &&\text{correct},\ S_t &= \widetilde S_t+\lvert e_t\rangle\langle k_t\rvert, &&\text{write}. \end{aligned} }St​∣vt​⟩∣et​⟩St​​=αt​St−1​,=St​∣kt​⟩,=βt​(∣vt​⟩−∣vt​⟩),=St​+∣et​⟩⟨kt​∣,​​forget,predict,correct,write.​​

This is Gated DeltaNet. The order matters: forget first, predict from the retained state, then correct that prediction. If we predicted before forgetting, the error would describe a different memory from the one we update.

Expanding the recurrence gives

St=αtSt−1(I−βt∣kt⟩⟨kt∣)+βt∣vt⟩⟨kt∣.S_t

\alpha_tS_{t-1} \left( I-\beta_t\lvert k_t\rangle\langle k_t\rvert \right) + \beta_t\lvert v_t\rangle\langle k_t\rvert.St​=αt​St−1​(I−βt​∣kt​⟩⟨kt​∣)+βt​∣vt​⟩⟨kt​∣.

The delta rule gives targeted replacement; the scalar gate gives global erasure. They solve different problems and are complementary.

But αt\alpha_tαt​ still makes one decision for the entire matrix. The model must retain or forget every key channel at the same rate.

4. Kimi Delta Attention: forget each channel independently

Kimi Delta Attention replaces Gated DeltaNet’s scalar retention with a vector αt∈[0,1]dk\alpha_t\in[0,1]^{d_k}αt​∈[0,1]dk​. Put the vector on the diagonal:

Dt=Diag⁡(αt)∈Rdk×dk.D_t

\operatorname{Diag}(\alpha_t) \in\mathbb R^{d_k\times d_k}.Dt​=Diag(αt​)∈Rdk​×dk​.

Our state maps keys to values, so the key channels are the columns of SSS. Right-multiplication applies a different retention factor to every one:

S~t=St−1Dt.\widetilde S_t

S_{t-1}D_t.St​=St−1​Dt​.

Everything else is the delta rule we have already derived:

S~t=St−1Dt,forget each key channel,∣v^t⟩=S~t∣kt⟩,predict,∣et⟩=βt(∣vt⟩−∣v^t⟩),correct,St=S~t+∣et⟩⟨kt∣,write,∣ot⟩=St(s∣qt⟩),s=dk−1/2,read.\boxed{ \begin{aligned} \widetilde S_t &= S_{t-1}D_t, &&\text{forget each key channel},\ \lvert\widehat v_t\rangle &= \widetilde S_t\lvert k_t\rangle, &&\text{predict},\ \lvert e_t\rangle &= \beta_t \left( \lvert v_t\rangle-\lvert\widehat v_t\rangle \right), &&\text{correct},\ S_t &= \widetilde S_t+\lvert e_t\rangle\langle k_t\rvert, &&\text{write},\ \lvert o_t\rangle &= S_t(s\lvert q_t\rangle), \qquad s=d_k^{-1/2}, &&\text{read}. \end{aligned} }St​∣vt​⟩∣et​⟩St​∣ot​⟩​=St−1​Dt​,=St​∣kt​⟩,=βt​(∣vt​⟩−∣vt​⟩),=St​+∣et​⟩⟨kt​∣,=St​(s∣qt​⟩),s=dk−1/2​,​​forget each key channel,predict,correct,write,read.​​

That is KDA. Compared with Gated DeltaNet, the conceptual change is only the promotion

αt⟶Dt=Diag⁡(αt).\alpha_t \quad\longrightarrow\quad D_t=\operatorname{Diag}(\alpha_t).αt​⟶Dt​=Diag(αt​).

The effect is substantial: one channel can be cleared while another is retained.

4.1 Why the transition is diagonal-plus-low-rank

Expand the KDA correction:

St=St−1Dt+βt(∣vt⟩−St−1Dt∣kt⟩)⟨kt∣=St−1Dt(I−βt∣kt⟩⟨kt∣)⏟At+βt∣vt⟩⟨kt∣.\begin{aligned} S_t &= S_{t-1}D_t + \beta_t \left( \lvert v_t\rangle - S_{t-1}D_t\lvert k_t\rangle \right) \langle k_t\rvert\ &= S_{t-1} \underbrace{ D_t \left( I-\beta_t\lvert k_t\rangle\langle k_t\rvert \right) }_{A_t} + \beta_t\lvert v_t\rangle\langle k_t\rvert. \end{aligned}St​​=St−1​Dt​+βt​(∣vt​⟩−St−1​Dt​∣kt​⟩)⟨kt​∣=St−1​At​Dt​(I−βt​∣kt​⟩⟨kt​∣)​​+βt​∣vt​⟩⟨kt​∣.​

The key-space transition is

At=Dt−βtDt∣kt⟩⟨kt∣=Dt−∣bt⟩⟨at∣,\begin{aligned} A_t &= D_t-\beta_tD_t\lvert k_t\rangle\langle k_t\rvert\ &= D_t-\lvert b_t\rangle\langle a_t\rvert, \end{aligned}At​​=Dt​−βt​Dt​∣kt​⟩⟨kt​∣=Dt​−∣bt​⟩⟨at​∣,​

where

∣bt⟩=Dt∣kt⟩,⟨at∣=βt⟨kt∣.\lvert b_t\rangle=D_t\lvert k_t\rangle, \qquad \langle a_t\rvert=\beta_t\langle k_t\rvert.∣bt​⟩=Dt​∣kt​⟩,⟨at​∣=βt​⟨kt​∣.

So AtA_tAt​ is a diagonal matrix minus a rank-one matrix: a diagonal-plus-low-rank, or DPLR, transition. “DPLR” describes the dk×dkd_k\times d_kdk​×dk​ transition acting on key space. The memory state itself is still the dv×dkd_v\times d_kdv​×dk​ matrix StS_tSt​.

The full journey can now be summarized compactly:

Mechanism	State update	What it adds
Linear attention	S+∣v⟩⟨k∣S+\lvert v\rangle\langle k\rvertS+∣v⟩⟨k∣	Fixed-size recurrent memory
DeltaNet	S+β(∣v⟩−S∣k⟩)⟨k∣S+\beta(\lvert v\rangle-S\lvert k\rangle)\langle k\rvertS+β(∣v⟩−S∣k⟩)⟨k∣	Targeted replacement
Gated DeltaNet	Apply αS\alpha SαS, then the delta update	Whole-state forgetting
KDA	Apply SDSDSD, then the delta update	Per-key-channel forgetting

The implementation usually stores gt=log⁡αtg_t=\log\alpha_tgt​=logαt​ with gt≤0g_t\leq0gt​≤0, then obtains the retention factors as exp⁡(gt)\exp(g_t)exp(gt​). In the transposed dk×dvd_k\times d_vdk​×dv​ layout used by the reference code, the recurrence is only five lines:

state = state * g_t.exp().unsqueeze(-1)
prediction = einsum("bhkv,bhk->bhv", state, k_t)
residual = beta_t.unsqueeze(-1) * (v_t - prediction)
state = state + einsum("bhk,bhv->bhkv", k_t, residual)
output = einsum("bhk,bhkv->bhv", q_t * scale, state)

See the official naive_recurrent_kda reference.

5. The fused recurrent Triton kernel

The recurrence above is the natural implementation for autoregressive decode. KDA has two principal execution regimes:

Regime	Best use	Parallel unit
Fused recurrent	Decode, short sequences, stateful serving	One sequence, value head, and value tile
Chunkwise	Training and long prefill	Chunks, token subchunks, and key/value tiles

The recurrent Triton launch uses one program per sequence, value head, and 32-wide value tile:

BK = triton.next_power_of_2(K)
BV = 32
grid = (triton.cdiv(V, BV) * N * HV,)

See the fused_recurrent_kda_fwd launch code.

BK covers the key dimension in the normal supported configuration. Each program owns a [BK, BV] tile of the implementation’s transposed state and loops over tokens in order. Different value tiles, heads, and sequences run independently.

The kernel is almost a literal transcription of the recurrence:

state *= tl.exp(g_t[:, None])
prediction = tl.sum(state * k_t[:, None], axis=0)
residual = beta_t * (v_t - prediction)
state += k_t[:, None] * residual[None, :]
out_t = tl.sum(state * (q_t * SCALE)[:, None], axis=0)

The prediction and read are reductions; the write is an outer product. This is excellent for decode, where only one new token is available at a time. It is less attractive for training and long prefill because these vector operations do not become the large matrix multiplications on which tensor cores are most efficient.

That motivates a second view of exactly the same recurrence.

6. Chunkwise KDA

Chunkwise KDA processes CCC tokens together. It must produce exactly the same states and outputs as the token-by-token recurrence, but it reorganizes the work into matrix products.

For each chunk ccc, we need two results:

1. the state Sc+1S_{c+1}Sc+1​ after the entire chunk, given the incoming state ScS_cSc​;
2. every causal token output inside the chunk.

The only difficulty is that token iii‘s delta error depends on writes made by earlier tokens in the same chunk. A four-token example makes those dependencies explicit.

6.1 Decay notation for a four-token chunk

Take tokens 0,1,2,30,1,2,30,1,2,3 and define

Di=Diag⁡(αi).D_i=\operatorname{Diag}(\alpha_i).Di​=Diag(αi​).

The cumulative decay between the chunk boundary and token iii is

D0:i=D0D1⋯Di.D_{0:i}=D_0D_1\cdots D_i.D0:i​=D0​D1​⋯Di​.

The decay carrying a write at token jjj forward to token iii is

Dj+1:i=Dj+1Dj+2⋯Di,j<i,D_{j+1:i}=D_{j+1}D_{j+2}\cdots D_i, \qquad j<i,Dj+1:i​=Dj+1​Dj+2​⋯Di​,j<i,

with Di+1:i=ID_{i+1:i}=IDi+1:i​=I when no intervening decay exists. All of these matrices are diagonal, so they commute with one another.

6.2 Start with provisional errors

First pretend that each token can see the appropriately decayed incoming state but none of the other writes inside its chunk:

∣eˉi⟩=βi(∣vi⟩−ScD0:i∣ki⟩).\boxed{ \lvert\bar e_i\rangle = \beta_i \left( \lvert v_i\rangle - S_cD_{0:i}\lvert k_i\rangle \right). }∣eˉi​⟩=βi​(∣vi​⟩−Sc​D0:i​∣ki​⟩).​

For four tokens this gives four provisional value-space error kets:

∣eˉ0⟩,∣eˉ1⟩,∣eˉ2⟩,∣eˉ3⟩.\lvert\bar e_0\rangle,\quad \lvert\bar e_1\rangle,\quad \lvert\bar e_2\rangle,\quad \lvert\bar e_3\rangle.∣eˉ0​⟩,∣eˉ1​⟩,∣eˉ2​⟩,∣eˉ3​⟩.

They are easy to compute in parallel, but all except the first are wrong: earlier writes in the same chunk also contribute to their predictions.

6.3 Restore the causal dependencies

Token 000 has no earlier in-chunk write, so

∣e0⟩=∣eˉ0⟩.\lvert e_0\rangle=\lvert\bar e_0\rangle.∣e0​⟩=∣eˉ0​⟩.

Token 111 sees token 000‘s write after it has passed through D1D_1D1​:

∣e1⟩=∣eˉ1⟩−β1⟨k0∣D1∣k1⟩∣e0⟩.\lvert e_1\rangle

\lvert\bar e_1\rangle

\beta_1 \langle k_0\rvert D_1\lvert k_1\rangle \lvert e_0\rangle.∣e1​⟩=∣eˉ1​⟩−β1​⟨k0​∣D1​∣k1​⟩∣e0​⟩.

Token 222 sees both preceding writes:

∣e2⟩=∣eˉ2⟩−β2⟨k0∣D1D2∣k2⟩∣e0⟩−β2⟨k1∣D2∣k2⟩∣e1⟩.\begin{aligned} \lvert e_2\rangle &= \lvert\bar e_2\rangle\ &\quad- \beta_2 \langle k_0\rvert D_1D_2\lvert k_2\rangle \lvert e_0\rangle\ &\quad- \beta_2 \langle k_1\rvert D_2\lvert k_2\rangle \lvert e_1\rangle. \end{aligned}∣e2​⟩​=∣eˉ2​⟩−β2​⟨k0​∣D1​D2​∣k2​⟩∣e0​⟩−β2​⟨k1​∣D2​∣k2​⟩∣e1​⟩.​

Token 333 sees all three:

∣e3⟩=∣eˉ3⟩−β3⟨k0∣D1D2D3∣k3⟩∣e0⟩−β3⟨k1∣D2D3∣k3⟩∣e1⟩−β3⟨k2∣D3∣k3⟩∣e2⟩.\begin{aligned} \lvert e_3\rangle &= \lvert\bar e_3\rangle\ &\quad- \beta_3 \langle k_0\rvert D_1D_2D_3\lvert k_3\rangle \lvert e_0\rangle\ &\quad- \beta_3 \langle k_1\rvert D_2D_3\lvert k_3\rangle \lvert e_1\rangle\ &\quad- \beta_3 \langle k_2\rvert D_3\lvert k_3\rangle \lvert e_2\rangle. \end{aligned}∣e3​⟩​=∣eˉ3​⟩−β3​⟨k0​∣D1​D2​D3​∣k3​⟩∣e0​⟩−β3​⟨k1​∣D2​D3​∣k3​⟩∣e1​⟩−β3​⟨k2​∣D3​∣k3​⟩∣e2​⟩.​

Every bracket ⟨kj∣Dj+1:i∣ki⟩\langle k_j\rvert D_{j+1:i}\lvert k_i\rangle⟨kj​∣Dj+1:i​∣ki​⟩ is a scalar. Define the causal key-key coefficient

ρij=βi⟨kj∣Dj+1:i∣ki⟩,j<i.\boxed{ \rho_{ij} = \beta_i \langle k_j\rvert D_{j+1:i}\lvert k_i\rangle, \qquad j<i. }ρij​=βi​⟨kj​∣Dj+1:i​∣ki​⟩,j<i.​

Then all four equations have the compact form

∣e0⟩=∣eˉ0⟩,∣e1⟩=∣eˉ1⟩−ρ10∣e0⟩,∣e2⟩=∣eˉ2⟩−ρ20∣e0⟩−ρ21∣e1⟩,∣e3⟩=∣eˉ3⟩−ρ30∣e0⟩−ρ31∣e1⟩−ρ32∣e2⟩.\begin{aligned} \lvert e_0\rangle &=\lvert\bar e_0\rangle,\ \lvert e_1\rangle &=\lvert\bar e_1\rangle-\rho_{10}\lvert e_0\rangle,\ \lvert e_2\rangle &=\lvert\bar e_2\rangle-\rho_{20}\lvert e_0\rangle -\rho_{21}\lvert e_1\rangle,\ \lvert e_3\rangle &=\lvert\bar e_3\rangle-\rho_{30}\lvert e_0\rangle -\rho_{31}\lvert e_1\rangle-\rho_{32}\lvert e_2\rangle. \end{aligned}∣e0​⟩∣e1​⟩∣e2​⟩∣e3​⟩​=∣eˉ0​⟩,=∣eˉ1​⟩−ρ10​∣e0​⟩,=∣eˉ2​⟩−ρ20​∣e0​⟩−ρ21​∣e1​⟩,=∣eˉ3​⟩−ρ30​∣e0​⟩−ρ31​∣e1​⟩−ρ32​∣e2​⟩.​

Collect the coefficients into a strictly lower-triangular matrix:

Rc=[0000ρ10000ρ20ρ2100ρ30ρ31ρ320],Ackk=(I+Rc)−1.R_c

\begin{bmatrix} 0&0&0&0\ \rho_{10}&0&0&0\ \rho_{20}&\rho_{21}&0&0\ \rho_{30}&\rho_{31}&\rho_{32}&0 \end{bmatrix}, \qquad A^{kk}_c=(I+R_c)^{-1}.Rc​=​0ρ10​ρ20​ρ30​​00ρ21​ρ31​​000ρ32​​0000​​,Ackk​=(I+Rc​)−1.

Stack the error kets as columns:

Eˉc=[∣eˉ0⟩∣eˉ1⟩∣eˉ2⟩∣eˉ3⟩],\bar E_c

\begin{bmatrix} \lvert\bar e_0\rangle& \lvert\bar e_1\rangle& \lvert\bar e_2\rangle& \lvert\bar e_3\rangle \end{bmatrix},Eˉc​=[∣eˉ0​⟩​∣eˉ1​⟩​∣eˉ2​⟩​∣eˉ3​⟩​],

and likewise for EcE_cEc​. The causal substitutions are then

Ec=Eˉc(Ackk)T.\boxed{ E_c = \bar E_c\left(A^{kk}_c\right)^\mathsf T. }Ec​=Eˉc​(Ackk​)T.​

The implementation does not need to form a general dense inverse. Because I+RcI+R_cI+Rc​ is triangular with ones on its diagonal, the operation is a causal triangular solve, applied independently to every value channel.

6.4 Fast-forward the state

At the end of the chunk, the incoming state has passed through all four decays. Each in-chunk write has passed through only the decays after it:

Sc+1=ScD0D1D2D3+∣e0⟩⟨k0∣D1D2D3+∣e1⟩⟨k1∣D2D3+∣e2⟩⟨k2∣D3+∣e3⟩⟨k3∣.\begin{aligned} S_{c+1} &= S_cD_0D_1D_2D_3\ &\quad+ \lvert e_0\rangle\langle k_0\rvert D_1D_2D_3\ &\quad+ \lvert e_1\rangle\langle k_1\rvert D_2D_3\ &\quad+ \lvert e_2\rangle\langle k_2\rvert D_3\ &\quad+ \lvert e_3\rangle\langle k_3\rvert. \end{aligned}Sc+1​​=Sc​D0​D1​D2​D3​+∣e0​⟩⟨k0​∣D1​D2​D3​+∣e1​⟩⟨k1​∣D2​D3​+∣e2​⟩⟨k2​∣D3​+∣e3​⟩⟨k3​∣.​

Define the matrix whose rows are the keys as they arrive at the end boundary:

Kcend=[⟨k0∣D1D2D3⟨k1∣D2D3⟨k2∣D3⟨k3∣].K_c^{\mathrm{end}}

\begin{bmatrix} \langle k_0\rvert D_1D_2D_3\ \langle k_1\rvert D_2D_3\ \langle k_2\rvert D_3\ \langle k_3\rvert \end{bmatrix}.Kcend​=​⟨k0​∣D1​D2​D3​⟨k1​∣D2​D3​⟨k2​∣D3​⟨k3​∣​​.

Because EcE_cEc​ stacks the error kets as columns, all four outer-product writes become one matrix multiplication:

Sc+1=ScD0:3+EcKcend.\boxed{ S_{c+1} = S_cD_{0:3} + E_cK_c^{\mathrm{end}}. }Sc+1​=Sc​D0:3​+Ec​Kcend​.​

This is the first required chunk result: advance the recurrent state by four tokens at once.

6.5 Compute every causal output

KDA reads after writing. If S[i+1]S^{[i+1]}S[i+1] is the local state after token iii, then

∣oi⟩=sS[i+1]∣qi⟩.\lvert o_i\rangle

sS^{[i+1]}\lvert q_i\rangle.∣oi​⟩=sS[i+1]∣qi​⟩.

Expand the four outputs:

∣o0⟩=sScD0∣q0⟩+s⟨k0∣q0⟩∣e0⟩,∣o1⟩=sScD0D1∣q1⟩+s⟨k0∣D1∣q1⟩∣e0⟩+s⟨k1∣q1⟩∣e1⟩,∣o2⟩=sScD0D1D2∣q2⟩+s⟨k0∣D1D2∣q2⟩∣e0⟩+s⟨k1∣D2∣q2⟩∣e1⟩+s⟨k2∣q2⟩∣e2⟩,∣o3⟩=sScD0D1D2D3∣q3⟩+s⟨k0∣D1D2D3∣q3⟩∣e0⟩+s⟨k1∣D2D3∣q3⟩∣e1⟩+s⟨k2∣D3∣q3⟩∣e2⟩+s⟨k3∣q3⟩∣e3⟩.\begin{aligned} \lvert o_0\rangle &= sS_cD_0\lvert q_0\rangle + s\langle k_0\rvert q_0\rangle\lvert e_0\rangle,\ \lvert o_1\rangle &= sS_cD_0D_1\lvert q_1\rangle\ &\quad+ s\langle k_0\rvert D_1\lvert q_1\rangle\lvert e_0\rangle + s\langle k_1\rvert q_1\rangle\lvert e_1\rangle,\ \lvert o_2\rangle &= sS_cD_0D_1D_2\lvert q_2\rangle\ &\quad+ s\langle k_0\rvert D_1D_2\lvert q_2\rangle\lvert e_0\rangle\ &\quad+ s\langle k_1\rvert D_2\lvert q_2\rangle\lvert e_1\rangle + s\langle k_2\rvert q_2\rangle\lvert e_2\rangle,\ \lvert o_3\rangle &= sS_cD_0D_1D_2D_3\lvert q_3\rangle\ &\quad+ s\langle k_0\rvert D_1D_2D_3\lvert q_3\rangle\lvert e_0\rangle\ &\quad+ s\langle k_1\rvert D_2D_3\lvert q_3\rangle\lvert e_1\rangle\ &\quad+ s\langle k_2\rvert D_3\lvert q_3\rangle\lvert e_2\rangle + s\langle k_3\rvert q_3\rangle\lvert e_3\rangle. \end{aligned}∣o0​⟩∣o1​⟩∣o2​⟩∣o3​⟩​=sSc​D0​∣q0​⟩+s⟨k0​∣q0​⟩∣e0​⟩,=sSc​D0​D1​∣q1​⟩+s⟨k0​∣D1​∣q1​⟩∣e0​⟩+s⟨k1​∣q1​⟩∣e1​⟩,=sSc​D0​D1​D2​∣q2​⟩+s⟨k0​∣D1​D2​∣q2​⟩∣e0​⟩+s⟨k1​∣D2​∣q2​⟩∣e1​⟩+s⟨k2​∣q2​⟩∣e2​⟩,=sSc​D0​D1​D2​D3​∣q3​⟩+s⟨k0​∣D1​D2​D3​∣q3​⟩∣e0​⟩+s⟨k1​∣D2​D3​∣q3​⟩∣e1​⟩+s⟨k2​∣D3​∣q3​⟩∣e2​⟩+s⟨k3​∣q3​⟩∣e3​⟩.​

Define the causal query-key coefficient

χij=s⟨kj∣Dj+1:i∣qi⟩,j≤i,\boxed{ \chi_{ij} = s\langle k_j\rvert D_{j+1:i}\lvert q_i\rangle, \qquad j\leq i, }χij​=s⟨kj​∣Dj+1:i​∣qi​⟩,j≤i,​

and place the coefficients in a lower-triangular read matrix:

Acqk=[χ00000χ10χ1100χ20χ21χ220χ30χ31χ32χ33].A^{qk}_c

\begin{bmatrix} \chi_{00}&0&0&0\ \chi_{10}&\chi_{11}&0&0\ \chi_{20}&\chi_{21}&\chi_{22}&0\ \chi_{30}&\chi_{31}&\chi_{32}&\chi_{33} \end{bmatrix}.Acqk​=​χ00​χ10​χ20​χ30​​0χ11​χ21​χ31​​00χ22​χ32​​000χ33​​​.

The zeros enforce causality. The diagonal is included because token iii reads after making its own write.

Now stack the boundary-decayed query kets as columns,

Qcboundary=[D0∣q0⟩D0D1∣q1⟩D0D1D2∣q2⟩D0D1D2D3∣q3⟩],Q_c^{\mathrm{boundary}}

\begin{bmatrix} D_0\lvert q_0\rangle& D_0D_1\lvert q_1\rangle& D_0D_1D_2\lvert q_2\rangle& D_0D_1D_2D_3\lvert q_3\rangle \end{bmatrix},Qcboundary​=[D0​∣q0​⟩​D0​D1​∣q1​⟩​D0​D1​D2​∣q2​⟩​D0​D1​D2​D3​∣q3​⟩​],

and stack the output kets the same way. All four outputs are

Oc=sScQcboundary+Ec(Acqk)T.\boxed{ O_c = sS_cQ_c^{\mathrm{boundary}} + E_c\left(A^{qk}_c\right)^\mathsf T. }Oc​=sSc​Qcboundary​+Ec​(Acqk​)T.​

The first matrix product reads the appropriately decayed incoming state. The second adds the causal contribution of writes made inside the chunk. This is the second required chunk result.

6.6 How the Triton pipeline is organized

The chunkwise implementation turns the equations above into a pipeline of kernel launches rather than one monolithic kernel.

It first computes chunk-local cumulative log decays. Differences between two prefix sums encode Dj+1:iD_{j+1:i}Dj+1:i​ without explicitly multiplying a long chain of retention vectors. It then constructs the causal AqkA^{qk}Aqk and AkkA^{kk}Akk interaction matrices. AkkA^{kk}Akk is used to form a WY-style representation of the chunk’s corrected writes.

A state kernel performs the only inter-chunk scan, producing the state entering each chunk and resolving its delta errors. Once those incoming states are known, an output kernel can calculate the tokens in different chunks and tiles in parallel.

The real source contains fused and tiled variants of these stages. In particular, it computes 16-token diagonal interaction blocks before a fused off-diagonal and triangular-solve kernel. The forward dataflow is approximately:

def chunkwise_kda(q, k, v, log_decay, beta, initial_state, scale):
    # Within-chunk prefix sums. G[i] - G[j] encodes the decay
    # carrying a state or write from token j to token i.
    G = chunk_local_cumsum(log_decay)

    # Build causal query-key interactions and the triangular system
    # that resolves dependencies between delta errors.
    A_qk_diag, A_kk_diag = intra_token_parallel(
        q, k, G, beta, scale
    )
    A_qk, A_kk = inter_and_triangular_solve(
        q, k, G, beta, A_qk_diag, A_kk_diag, scale
    )

    # Convert the chunk into its WY pseudo-key/pseudo-value form.
    W, U, K_to_end = build_wy_factors(k, v, G, beta, A_kk)

    # The only recurrence left is over chunk boundaries.
    H, E, final_state = scan_chunk_states(
        K_to_end, W, U, G, initial_state
    )

    # Combine reads from each incoming chunk state with causal
    # contributions from writes made inside that chunk.
    output = calculate_outputs(q, E, G, A_qk, H, scale)
    return output, final_state

In the reference source these stages are orchestrated by chunk_kda_fwd. Its principal implementation entry points are chunk_kda_fwd_intra, chunk_gated_delta_rule_fwd_h, and chunk_gla_fwd_o_gk. Names such as v_new, h, and kg in the code correspond to the resolved errors, incoming chunk states, and keys decayed to the end of a chunk.

The recurrent and chunkwise programs are therefore not two different attention mechanisms. They are two schedules for the same KDA recurrence: serial vector operations for low-latency decode, and chunked matrix operations for tensor-core-heavy training and prefill.

Cite this post

@misc{doubleword-you-could-have-come-up-with-kimi-delta-attention,
  title        = {You Could Have Come Up With Kimi Delta Attention},
  author       = {Jamie Dborin},
  year         = {2026},
  howpublished = {Doubleword Blog},
  url          = {https://blog.doubleword.ai/you-could-have-come-up-with-kimi-delta-attention},
}