Answers · Where the data lives

Is UTF-8 the same thing as Unicode, or what's the difference?

In shortNo: Unicode assigns a number to each character, while UTF-8 is a way of writing those numbers as bytes, using one to four bytes per character.

No. Unicode is the catalogue that gives every character a number. UTF-8 is a layout that writes those numbers down as bytes, using one byte for the first hundred and twenty eight characters and two, three or four bytes for the rest. The two words answer different questions: which character is this, and how is it stored. A file is written in an encoding; UTF-8 is one encoding, and Unicode is the list behind it.

Two facts per character

Take four characters and write down both facts. The capital A has catalogue number 65 and takes one byte in UTF-8. The e with an acute accent has number 233 and takes two bytes. A character in a Chinese script has number 27700 and takes three bytes. An emoji has number 128512 and takes four bytes. The first number is the character's identity; the second is its width on disk. Under an old one byte scheme the first two characters each took one byte, while the Chinese character and the emoji had no representation at all.

FIG 1

character	Unicode number	UTF-8 bytes	old one-byte scheme bytes
capital A	65	1	1
e with acute accent	233	2	1
character in Chinese script	27700	3	0
emoji	128512	4	0

The arithmetic of length follows from those widths. For a string that uses only one byte and two byte characters, if B is the byte length and a is the number of one byte characters, then the number m of two byte characters is fixed by the bytes left over.

B = a + 2m \quad\Rightarrow\quad m = \frac{B-a}{2}

A name of twenty three bytes with twenty one plain characters has one two byte character: $23-21=2$ leftover bytes, and $2/2=1$ character. A program that counts bytes reports 23. A person who counts characters reports 22. Neither is wrong, because they are counting different things. The same arithmetic explains why a length limit expressed in bytes is not a length limit expressed in characters.

Why the natural guess fails

If UTF-8 were just another name for Unicode, a field limited to 255 bytes would hold 255 characters for every user. It does not. Plain English characters take one byte each, so $255 \times 1 = 255$ fit. Characters from a two byte script fit only 127 whole characters, because $127 \times 2 = 254$ and the next character would need 256 bytes. Characters from a three byte script fit 85, because $85 \times 3 = 255$ . The same field is three different limits, and the difference is not in which characters exist but in how many bytes each one occupies.

This is why using the words interchangeably hides the diagnosis. A rejected form may fail because the target scheme has no number for a character, which is a catalogue problem. Or it may fail because the bytes overflow a fixed size, which is a layout problem. The fixes are different. Re-encoding cannot invent a character the target scheme lacks; a bigger byte limit cannot restore a character that was already replaced during conversion.

What broken text tells you

Broken text has two useful shapes. A pair or triple of strange characters where one accented character should be means the bytes are intact and were read one at a time under an old agreement. The original is recoverable: read the same bytes again under the right agreement. A question mark, a small box, or a single replacement character means something tried to write the character into a scheme that had no number for it and substituted a stand in. The original character is gone from that file; re-reading cannot bring it back.

Where the distinction disappears: if every character in the string is in the first 128, UTF-8 and the old one byte scheme agree byte for byte, so byte count and character count are equal. That is why plain English hides the issue. It also means the simple two width formula stops applying as soon as a three or four byte character appears. For a general string the byte count is the sum of the widths:

B = n_1 + 2n_2 + 3n_3 + 4n_4

with $n_1$ one byte characters, $n_2$ two byte characters, $n_3$ three byte characters and $n_4$ four byte characters. The character count is $n_1+n_2+n_3+n_4$ , which is smaller than B whenever any $n_2$ , $n_3$ or $n_4$ is nonzero.

So the useful question is not whether a file is 'Unicode' but which numbers were assigned and which bytes were written. If the text can be repaired by re-reading, the bytes survived and the agreement was wrong. If a stand in has replaced the character, the agreement had no number for it and the information is absent from that copy. The byte count tells you about storage; the catalogue tells you about identity.

Worth keeping

A character's Unicode number and its UTF-8 byte sequence are different things, and the byte sequence can take one to four bytes.
A pair of strange characters usually means intact bytes were read under the wrong agreement; a question mark usually means the character was already lost in conversion.
A byte limit is not a character limit once any character takes more than one byte.

Where this goes next

This question is one step of Bytes, Text and File Formats, a course that works through its subject from the beginning. The page that teaches this part of it is Letters as Numbers, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course