Basis Lab

Answers · Where the data lives

Why do new rows show up when I run the same query twice inside one transaction?

In shortThe transaction is not holding one fixed set of rows: a later run of the same range query can see rows another transaction inserted and committed in between.

The extra rows are a phantom read: the second execution of the same range query sees rows that another transaction inserted and committed between the two executions. A transaction does not have a fixed row set for its whole life unless its isolation level gives it a snapshot, and the weaker levels do not.

The predicate is re-evaluated

Suppose the predicate is amount>100amount > 100. Transaction T1 first runs the query and gets the three rows 120, 240 and 410. Transaction T2 then inserts a row with amount 310 and commits. T1 runs the identical query again and gets 120, 240, 310 and 410. The row 310 was not there when T1 first looked, and no row that T1 read has changed. The first result is a subset of the second, and the difference is exactly one row, 4−3=14-3=1. The database has not remembered the first answer; it has evaluated the condition again against the committed data visible at that moment.

first run: {120,240,410},∣{120,240,410}∣=3,second run: {120,240,310,410},∣{120,240,310,410}∣=4,difference: {310},4−3=1.\begin{aligned} \text{first run: } &\{120,240,410\}, && |\{120,240,410\}| = 3,\\ \text{second run: } &\{120,240,310,410\}, && |\{120,240,310,410\}| = 4,\\ \text{difference: } &\{310\}, && 4-3 = 1. \end{aligned}
FIG 1
executionrows matching countrow 310 visible
first120, 240, 4103no
second120, 240, 310, 4104yes

Nothing in the table is an error report. Both executions succeed, and both return rows that were committed when they were read. The transaction has simply been given two different answers to one question, because the question is a predicate rather than a row identity. That is why the anomaly can pass every individual assertion in the code and still break a report or a check-then-act decision.

It is tempting to say that locking the rows the first query returned will stop this. It will not. Locks on 120, 240 and 410 protect those row identities, but the row that causes the trouble is 310, which did not exist when the locks were taken and therefore could not be locked. T2's insert into 310 does not conflict with any lock on 120, 240 or 410 because it does not touch those rows. The lock has to cover the condition amount>100amount > 100, not just the rows that happen to satisfy it now. That is a predicate lock or a range lock, or the coarser mechanism a serialisable isolation level uses; it is more expensive because it constrains inserts that are not yet in the table.

This is different from reading the same row twice and getting two values. There the row identity is fixed and a row lock is enough. Here the row identity 310 is new, so a row lock on anything already read cannot help. The enabling condition is precise: the transaction must read a range or a non-unique condition twice, and another transaction must commit an insert that satisfies that condition between the two reads. It does not need a read of uncommitted data, and it does not need the transaction to write anything back. The anomaly is about the range, not about the value of any individual row.

Where the guarantee changes

Under read committed, each statement gets a fresh view, so the second run can see the committed insert; that is the usual way this appears. Under repeatable read the answer depends on the implementation: some databases give every plain SELECT in the transaction one snapshot and the extra row will not appear, while others still permit phantoms or only protect rows already read. Under serialisable this anomaly is eliminated, at the cost of making the condition itself stable and potentially blocking or aborting the inserting transaction. If the insert is not committed, or if it is rolled back, it is not visible at read committed and above, so it cannot cause the second run to grow.

In a check-then-act pattern, count the bookings for a room and insert one if the count is below a limit. The count can be 3 on the first run and 4 on the second, and a decision based on the first count is already stale. The remedy is to run that pattern at a level that fixes the predicate, or to take an explicit lock on the condition, not merely on the rows the count returned.

Worth keeping

  • A transaction sees a fixed set of rows only if its isolation level gives it a stable snapshot for the whole transaction.
  • A new row appearing between two runs of a range query is a phantom read, not a non-repeatable read of an existing row.
  • Locking rows already returned cannot stop a phantom, because the offending row did not exist when those locks were taken.

Where this goes next

This question is one step of What a Transaction Promises, a course that works through its subject from the beginning. The page that teaches this part of it is The Four Things That Go Wrong, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that pageThe whole course

Other questions people ask about this

  1. Why is the date off by one day after I save it and read it back?You saved a calendar date as if it were a precise moment, so showing it in another time zone can move it to the day before or after.
  2. Is UTF-8 the same thing as Unicode, or what's the difference?No: Unicode assigns a number to each character, while UTF-8 is a way of writing those numbers as bytes, using one to four bytes per character.
  3. Why does my CSV file fall apart when one of the fields has a comma in it?Because the same comma both separates fields and sits inside the value, so a reader splitting on commas finds one field too many and every later column shifts.
  4. Why do programmers write numbers in hexadecimal instead of just normal numbers?Programmers use hexadecimal because each digit stands for exactly four bits, so one byte is always written as two digits and the columns line up.
  5. Why does my count come out doubled after the job restarts?Your count doubled because the restart replayed events the saved counter already held: the state and input position were not saved as one snapshot.
  6. How do I make a step safe to run again when I'm not sure whether it already ran?Change the operation so that sending it twice leaves the same stored data as sending it once, instead of relying on remembering not to send it again.

Read alongside