Letters as Numbers

Last timeCounting in Twos and Sixteens

A letter is a number, and the agreement that assigns the numbers has changed twice. A name arriving with a question mark where the accent was is that history showing through.

From a keystroke to bytes

Between pressing a key and the disk there are two decisions, and keeping them

apart makes the whole subject easier.

The first is which number this character gets. That is settled by a catalogue

which assigns a number to every character in every writing system: the Latin

capital A is sixty five, the e with an acute accent is two hundred and thirty

three, and a given emoji is a number well past a hundred thousand. The

catalogue is one agreed list and there is no controversy left in it.

The second decision is how that number is written down as bytes. A number up to

a hundred thousand plainly does not fit in one byte, so a layout is needed, and

the layout the world settled on writes the first hundred and twenty eight

characters in one byte each and everything else in two, three or four.

FIG 1Four characters through both decisions

	its number in the catalo	bytes it takes in the mo	bytes under an old one b
capital A	65	1	1
e with an acute accent	233	2	1
a character in a Chinese	27700	3	0
an emoji	128512	4	0

Coded 0 in the last column where the character cannot be represented at all. The first row is the useful property of the modern layout: plain English text is byte for byte identical to what the old scheme produced, which is why the transition was possible at all. The second row is where the trouble is, because the same character was one byte then and is two bytes now, and anything that still assumes the old width reads those two bytes as two characters.

Why one byte was not enough

One byte distinguishes two hundred and fifty six things. The first hundred and

twenty eight were agreed early and cover the English alphabet, the digits,

punctuation and some control codes. That left a hundred and twenty eight for

everyone else, which is not enough for the accented letters of Western Europe

alone, let alone Greek, Cyrillic, Arabic, Hebrew and the scripts of Asia.

What happened instead was a few dozen incompatible schemes, each claiming the

upper half of the byte for a different set of characters. A file written under

one and read under another produced plausible looking nonsense, and because

nothing in the file said which scheme it was, the disagreement was invisible

until somebody read the text.

FIG 2How many characters in a string took two bytes

: the length of the string in bytes
: how many characters are plain one byte ones
: how many took two bytes each

A name of twenty three bytes containing twenty one plain characters has one two byte character in it. The arithmetic matters because the two counts drift apart silently: a program that measures length in bytes and a person who counts characters will disagree, and neither is wrong. The relation here assumes nothing takes three or four bytes, which holds for European text and fails for anything else, where the same idea needs a term for each width.

Characters are not bytes

Once characters have different widths, any limit stated in bytes becomes a

different limit for every language.

A field allowed two hundred and fifty five bytes holds two hundred and fifty

five English letters. The same field holds about a hundred and twenty seven

characters of Greek or Cyrillic, and eighty five of a Chinese script. A form

that says a field may hold two hundred and fifty five characters is therefore

telling some users the truth and others something false, and the failure

arrives as a rejected form or, worse, as text silently cut off.

The lesson stops here

3 more paragraphs to go

You have read the opening. The rest of the argument, the problems that check whether it landed, and the lines worth keeping at the end all come with a plan.

The first lesson of every course in the library reads the whole way through, free, so you can see exactly what the rest of them are.

See the plan The contents

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents