Turn a record into bytes

Our lookup program keeps records with named fields in memory. A file stores a sequence of bytes. To save a record, the writer must turn its values into bytes; a reader must know how to turn them back.

A byte is a group of eight bits and can represent an integer from 0 through 255. Larger integers and text require agreed rules for interpreting one or more bytes. Those rules are an encoding.

Design one small record

For this lesson, store only a product number and a name. Use this deliberately limited format:

UTF-8 is a standard way to encode text. For the four letters in “Tote,” the byte values are 84, 111, 116, and 101. Other characters may use more than one byte, so the length must count encoded bytes rather than displayed letters.

Converting a record into this representation is serialization. Decoding it reconstructs the values. This is our own teaching format, not PostgreSQL’s on-disk representation.

Why store the length?

Place another record immediately after the first. The reader consumes the first two bytes, learns that four name bytes follow, and therefore knows the next record begins at offset 6.

Without a length or another boundary rule, the reader could not tell whether a byte belonged to this name or the next product number. Text that looks obvious to a person still needs an unambiguous representation.

The writer and reader must agree on every rule. If one interprets the second byte as a character count while the other interprets it as a byte count, non-ASCII names can break the boundary calculation.

Reject incomplete data

Suppose the stored header says the name has four bytes, but only two remain in the file. The reader must report an incomplete record. Guessing at the missing bytes would turn a storage failure into invented data.

Likewise, our writer must reject product number 300 because it cannot fit in the chosen one-byte field. A later format can use a wider field, but the reader must know which format it is reading. A file header containing a format version makes that distinction explicit.

Predict the next boundary

A record starts at offset 6 and says its name occupies three bytes. At which offset does the next record begin?

We can now describe exactly what should be written. Next we distinguish saving those bytes during an ordinary run from surviving an interruption halfway through the save.

Further reading

Python’s Unicode HOWTO explains text encodings. CMU’s storage notes provide the later bridge from byte representations to database storage layouts.