Everything Is a Run of Bytes

Every stored thing is a sequence of bytes plus an agreement about how to read them. The bytes are the fact and the agreement is the convention, and only one of them is in the file.

Bytes and an agreement

A disk stores numbers. Each one is between zero and two hundred and fifty five,

and the usual name for one of them is a byte. A file is a run of those numbers

with a length. That is the whole of what is stored.

Everything else, that this file is a photograph and that one is a spreadsheet,

that these eight bytes are a price and those four are a date, is an agreement

held somewhere other than in the bytes. The agreement is real and necessary and

it is not in the file.

FIG 1Five bytes, read under five agreements

step	the agreement	what the five bytes 72 101 108 108 111 become	is it valid
1	letters in the common Western scheme	the word hello	yes
2	five separate small whole numbers	72, 101, 108, 108, 111	yes
3	four of them as one whole number, then one spare	1214606444, and 111 left over	yes
4	pixel brightnesses in a row	five grey dots, mid to light	yes
5	a musical instrument instruction	a note on, of middling pitch	yes

5 steps

Every row is a correct reading of the same five numbers, and nothing in those numbers prefers one row over another. The last column is the uncomfortable part of the figure: there is no wrong answer here, only a reading that does not match what the writer intended. That mismatch, rather than any damage to the bytes themselves, is what people are describing when they say a file is corrupted.

Where the agreement actually lives is worth listing, because each place it can

live is a place it can be lost. In the name of the file, as an extension, which

anybody can change. In the first few bytes of the file, as a marker that

particular formats agree to begin with. In a column definition in a database.

In a header sent alongside the file over a network. And, most often, in the

assumption of whoever wrote the program that opens it.

What one byte holds

A byte distinguishes two hundred and fifty six things, because it is eight

switches and each switch doubles the count. That number, and the way it grows,

decides a great deal further on in this course.

FIG 2How many distinct values fit in a given number of bytes

values a run of n bytes distinguishes

The curve is drawn in millions so that the shape is visible, and it climbs out of the top of the picture almost immediately: three bytes is sixteen million, four is over four thousand million. The practical readings are that one byte is enough for a letter in one alphabet and not for a letter in all of them, that two bytes is enough for most counts and not for a population, and that four is enough for most things until it is suddenly not, which is the subject of the fourth lesson.

FIG 3Widths and what they are enough for

	bytes	distinct values	enough for
a flag	1	256	1
a letter in one alphabet	1	256	1
a colour	3	16777216	1
a count of people in a c	2	65536	0
an instant to the second	4	4294967296	0

The last column is coded 1 for yes and 0 for no. The fourth row is a real and common bug: two bytes counts to sixty five thousand, which is plenty for a town and nothing like enough for a country, and the failure when it arrives is not an error message but a wrong number. The last row is the one with a date on it, since four bytes of seconds runs out in 2038, and the whole of the fourth lesson is about what happens at the end of a range.

A file is a count of bytes

When every record in a file is the same width, the size of the file stops being

metadata and becomes arithmetic.

FIG 4Records in a fixed width file

: the size of the file in bytes
: the size of the header, if there is one
: the width of one record
: how many records the file holds

A file of 48,240 bytes with a 240 byte header and 48 byte records holds exactly a thousand records, and that can be known without reading any of them. The more valuable use is the other one: if the division does not come out whole, the file is damaged or truncated, and you know this before parsing a single field. That check costs nothing and catches the most common form of real corruption, which is a transfer that stopped early.

This is also the reason fixed width records survive in places where they look

archaic. Finding the ten thousandth record is a multiplication rather than a

search, which is a property no format with variable length rows can offer, and

the sixth and seventh lessons both come back to it.

The same bytes, read two ways

The single idea to carry out of this lesson is that a run of bytes has no

intrinsic meaning, so the question to ask of a confusing file is never what it

contains but what agreement it was written under.

FIG 5One run of bytes, two readings, in a few lines

python

raw = bytes([0xC3, 0xA9, 0x74, 0xC3, 0xA9])   # five bytes, nothing more

print(raw.decode("utf-8"))      # the modern agreement: three letters
print(raw.decode("latin-1"))    # an older one: five letters, two of them odd
print(list(raw))                # no letters at all: 195, 169, 116, 195, 169

All three lines succeed and all three are correct. The first reads the bytes under the agreement they were written with. The second reads them under an older agreement where every byte is one letter, which produces the familiar pair of strange characters where an accented letter should be. The third declines to read them as letters at all. The third lesson of this course is entirely about which of these happens and why, and the exercise worth doing is running these three lines on a file that has gone wrong at work.

One practical habit follows. When something arrives broken, look at the bytes

before theorising, because the bytes are the only part of the situation that is

not somebody's assumption. The next lesson is about how to read them.

Recap

A file is a run of numbers from zero to two hundred and fifty five, and nothing in that run says what the numbers mean.
The same bytes read under two different agreements give two different and equally valid answers, which is the single idea the rest of this course is built on.
The agreement usually lives outside the file, in a name, a header, a column type or somebody's assumption, and most corruption is the agreement being lost rather than the bytes being damaged.

This is the reading half

Starting the course gives you your own copy of it. Every idea on every page has problems standing under it, marked with a reason rather than a tick, and any sentence you do not believe can be opened and argued with. None of that can happen on a page nobody owns.

The contents

NextCounting in Twos and Sixteens →