base32

stdlib/extra/base32.hk: RFC 4648 and Crockford base32 for bytes.

The middle of the bytes-to-text family: hex spends two characters per byte, base64 spends four per three, and base32 spends eight per five. It is the widest of the three and the only one meant to be read aloud, typed off a screen, or printed on a card. That is why it exists at all, and why its two alphabets differ in kind and not in two characters the way base64's do.

The two variants do not share a decoder, which is the one structural difference from base64. There, no character means one thing in one alphabet and something else in the other, and one permissive decode reads both. Here B is 1 under RFC 4648 and 11 under Crockford; every letter collides. Each variant therefore gets its own decode, and a caller has to know which it has, which it always does: the two come from different worlds.

Pure and whole-input, like the rest of the family: every entry point is @encapsulated, and the builder it writes into is allocated and finished inside the call and nothing mutable escapes. Callable from meta and from ordinary pure code, on both tiers. There is no streaming form.

Scoped out of this version by decision and no oversight: Crockford's optional trailing check symbol (the value modulo 37, written from a 37-symbol alphabet that adds *, ~, $, = and U to the 32; U returns there, which is what makes it a check symbol and no data one), and its rule that hyphens may be inserted for readability and ignored on decode. Both are real parts of that specification; neither is needed to read a ULID or a key, which is what the family was missing.

_rfc

_rfc: string = "ABCDEFGHIJKLMNOPQRSTUVWXYZ234567"

RFC 4648 §6: the twenty-six letters, then the six digits 2-7. The digits 0, 1, 8 and 9 are absent so that no character resembles another.

_crockford

_crockford: string = "0123456789ABCDEFGHJKMNPQRSTVWXYZ"

Crockford's alphabet: all ten digits, then the letters minus I, L, O and U. The first three are dropped because they resemble digits, U because dropping it leaves accidental obscenities out of generated identifiers.

Base32Error

type Base32Error
  BadCharacter(int)
  BadLength(int)
  BadPadding(int)
end

Why base32 text could not be decoded, with the byte offset at which the problem was found (the Base64Error idiom).

encode

def encode(input: bytes) -> string

Encode input in the RFC 4648 alphabet with = padding: the form a TOTP secret, an S/MIME body or anything the RFC calls base32 without qualification takes.

encode("".to_bytes())      => ""
encode("f".to_bytes())     => "MY======"
encode("fo".to_bytes())    => "MZXQ===="
encode("foo".to_bytes())   => "MZXW6==="
encode("foob".to_bytes())  => "MZXW6YQ="
encode("fooba".to_bytes()) => "MZXW6YTB"

encode_crockford

def encode_crockford(input: bytes) -> string

Encode input in Crockford's alphabet, unpadded: the form for an identifier a person will read or retype. Crockford specifies no padding at all, and there is therefore no padded twin.

encode_crockford("".to_bytes())      => ""
encode_crockford("f".to_bytes())     => "CR"
encode_crockford("fo".to_bytes())    => "CSQG"
encode_crockford("foo".to_bytes())   => "CSQPY"
encode_crockford("foob".to_bytes())  => "CSQPYRG"
encode_crockford("fooba".to_bytes()) => "CSQPYRK1"

decode

def decode(text: string) -> Result<bytes, Base32Error>

Decode RFC 4648 base32. Padding is optional, a TOTP secret commonly arriving without it, and a = that appears must end the input and leave a group carrying whole bytes. Lower case is accepted, since the RFC leaves that to the decoder and every producer of a hand-typed secret assumes it. Whitespace and a newline are skipped no more than a = is: a codec that drops characters unannounced cannot tell a wrapped body from a corrupted one.

decode("MZXW6YTB").unwrap_or("".to_bytes()).to_string().unwrap_or("") => "fooba"
decode("MZXW6===").unwrap_or("".to_bytes()).to_string().unwrap_or("") => "foo"
decode("MZXW6").unwrap_or("".to_bytes()).to_string().unwrap_or("")    => "foo"
decode("mzxw6").unwrap_or("".to_bytes()).to_string().unwrap_or("")    => "foo"
decode("M").unwrap_or("rejected".to_bytes()).length                   => 8
decode("M8").unwrap_or("rejected".to_bytes()).length                  => 8
decode(encode("hi".to_bytes())).unwrap_or("".to_bytes()).to_string().unwrap_or("") => "hi"

decode_crockford

def decode_crockford(text: string) -> Result<bytes, Base32Error>

Decode Crockford base32, with the forgiveness its specification requires and no more. That specification states the rule in one sentence, "upper and lower case letters are accepted, and i and l will be treated as 1 and o will be treated as 0", and those three are the whole confusion set.

S is no alias for 5, however alike the two look, and the alphabet is the reason and not the taste: ten digits plus the twenty-two letters left after excluding I, L, O and U is thirty-two, with no slack. An alias slot exists only for a character the encoding does not already use, which is what excluding I, L and O leaves. S is 25, and forgiving it would mean dropping a fifth letter, leaving a result that is no longer base32.

U is the other half of that rule: excluded too, but for accidental obscenity and not for confusion, and it therefore aliases to nothing and is rejected here.

decode_crockford("CSQPYRK1").unwrap_or("".to_bytes()).to_string().unwrap_or("") => "fooba"
decode_crockford("csqpyrk1").unwrap_or("".to_bytes()).to_string().unwrap_or("") => "fooba"
decode_crockford("CSQPYRKI").unwrap_or("".to_bytes()).to_string().unwrap_or("") => "fooba"
decode_crockford("CSQPYRKL").unwrap_or("".to_bytes()).to_string().unwrap_or("") => "fooba"
decode_crockford("0").unwrap_or("rejected".to_bytes()).length                   => 8
decode_crockford("CU").unwrap_or("rejected".to_bytes()).length                  => 8
decode_crockford("OR").unwrap_or("".to_bytes()) == decode_crockford("0R").unwrap_or("x".to_bytes()) => true
decode_crockford(encode_crockford("hi".to_bytes())).unwrap_or("".to_bytes()).to_string().unwrap_or("") => "hi"

_encode

def _encode(input: bytes, alphabet: string, pad: bool) -> string

The eight characters of one 40-bit group, or fewer at the end. pad fills the group out to eight with =.

_at

def _at(input: bytes, i: int) -> u64

Byte i of input widened, or 0 past the end: the zero fill RFC 4648 specifies for a short final group. u64 throughout, one group being 40 bits that no narrower width fits; the narrowing back to u8 is the mask, which leaves nothing needing a range check and nothing able to crash.

charsfor

def _chars_for(left: int) -> int

How many characters left remaining bytes produce: every whole 5 bits of the 8, 16, 24, 32 or 40 they carry.

bytesfor

def _bytes_for(left: int) -> int

The inverse: how many whole bytes left remaining characters carry. The counts 1, 3 and 6 are no whole byte and never reach here: _data_length has already rejected them.

_char

def _char(v: u64, alphabet: string) -> string

The alphabet character for the low 5 bits of v.

_upper

def _upper(ch: u8) -> u8

ch upper-cased, if it is an ASCII lower-case letter.

_value

def _value(ch: u8, crockford: bool) -> Option<u64>

The 5-bit value of one character, or None. Arithmetic and no scan, which spares either alphabet a lookup per character.

valuecrockford

def _value_crockford(u: u8) -> Option<u64>

Crockford's mapping over an already upper-cased character: the digits, the three confusion characters, then the letters.

crockfordletter

def _crockford_letter(u: u8) -> Option<u64>

A letter's value, counting past the gaps its alphabet leaves. I, L and O never arrive (the caller maps them first); U is in no alphabet and is the one letter rejected here.

_decode

def _decode(text: string, crockford: bool) -> Result<bytes, Base32Error>

datalength

def _data_length(raw: bytes) -> Result<int, Base32Error>

How much of raw is data and not padding, or why the padding is wrong. At most six trailing =, nothing after them, the data left over must not end a group with 1, 3 or 6 characters, none of which is a whole byte and each of which means the input was cut short, and any padding present must be the amount RFC 4648 §6 writes for that group. Padding remains optional, as it is in base64: the family's value is that the codecs behave alike.

padsfor

def _pads_for(rest: int) -> int

How many = RFC 4648 §6 writes after a data run of this length. Residues of 1, 3 and 6 never reach here - _data_length has already rejected them as BadLength, none of them being a whole byte.

decodedata

def _decode_data(raw: bytes, stop: int, crockford: bool) -> Result<bytes, Base32Error>

The bytes of raw's first stop characters, eight at a time.

_offset

def _offset(e: Base32Error) -> int