Answers · How models work

Does it matter which layers I attach adapters to?

In shortYes. For the same number of trainable parameters, adapters spread over more weight matrices, including the feed-forward ones, usually score higher than adapters concentrated on a few.

Yes. Two runs with the same number of trainable parameters can end at different scores if the corrections sit on different weight matrices, because a correction can only change the matrix it is attached to. The familiar default of correcting only the query and value projections is a habit from the first demonstration, not an optimum.

P_{\text{train}} = \sum_i r_i (d_{\text{in},i} + d_{\text{out},i})

For a projection with input width $d_{\text{in}}$ and output width $d_{\text{out}}$ , a bottleneck of width $r$ trains $r(d_{\text{in}}+d_{\text{out}})$ numbers. Take a block where each corrected projection has $d_{\text{in}}+d_{\text{out}}=8192$ . A width of 64 on two matrices costs $64 \times 2 \times 8192 = 1,048,576$ trainable numbers. The same budget buys width 16 on eight matrices: $16 \times 8 \times 8192 = 1,048,576$ . It also buys width 8 on sixteen: $8 \times 16 \times 8192 = 1,048,576$ . The product $r \times N$ is 128 in each case, so the budget is fixed and only the placement changes.

FIG 1

Bottleneck width r	Weight matrices corrected	r times matrices	Trainable parameters	Score
64	2	128	1,048,576	0.88
16	8	128	1,048,576	0.90
8	16	128	1,048,576	0.91
4	32	128	1,048,576	0.91

The natural guess fails because it treats placement as a labelling choice rather than a reachability choice. A correction is a low-rank change added to one weight matrix, so it can only move the function through that matrix. If the key projection or an output projection or a feed-forward matrix is left untouched, the adapted model has no route to whatever that matrix would have needed to do. Spending more parameters on a matrix that is already covered cannot compensate: a very wide correction on the query projection is still only a change to the query projection.

Given a fixed budget, breadth beats width. In one controlled comparison, every row spent 1,048,576 trainable numbers. Width 64 on 2 matrices scored 0.88; width 16 on 8 scored 0.90; width 8 on 16 scored 0.91; width 4 on 32 scored 0.91. The arithmetic for the budget is the same in every row: $64\times2=128$ , $16\times8=128$ , $8\times16=128$ , $4\times32=128$ , and $128\times8192=1,048,576$ . What changes is how many places the correction can act. The jump from 2 to 8 places is worth 0.02; from 8 to 16 is worth 0.01; from 16 to 32 is worth 0.00. That is why placement should be decided before width is tuned: widening a bottleneck that is already wide enough buys less than covering another matrix.

A detail that makes a placement mistake less catastrophic: one of the two thin matrices is initialised with small random numbers and the other with zeros, so their product is exactly zero. The adapted model therefore begins identical to the original, whether you attach to two matrices or sixteen. There is no shock at the start of training, and a run that goes badly degrades towards the pretrained model rather than away from it into something arbitrary.

The rule stops paying when breadth is already saturated. In the trace, 16 matrices at width 8 and 32 matrices at width 4 both scored 0.91, so sixteen places were already enough for that task and the extra coverage did nothing. If your budget is large enough to give every matrix a generous bottleneck, the choice between breadth and width stops being a trade. There is also a scale interaction: widening the bottleneck produces a larger correction for the same trained values, so raising the width without dividing the scale by that width quietly raises the effective training rate. A team that widens and sees a different run should check the scale before blaming the placement.

The order that follows is: cover all four attention projections at minimum, add the feed-forward matrices when the budget allows, make each bottleneck as narrow as that coverage requires, and only then tune how loudly the correction speaks. The worst case is still bounded by the pretrained model because the correction starts at zero.

Worth keeping

At a fixed parameter budget, spreading adapters over more weight matrices usually beats making a few bottlenecks wider.
A correction can only affect the matrix it is attached to, so untouched matrices are outside the adaptation's reach.
Adapter corrections start at zero, so adding them does not change the pretrained model until training moves them.

Where this goes next

This question is one step of Adapting a Model You Did Not Train, a course that works through its subject from the beginning. The page that teaches this part of it is Which Weights, How Wide, How Loud, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course

Does it matter which layers I attach adapters to?

Worth keeping

Where this goes next

Read alongside