url

stdlib/extra/url.hk: URLs taken apart and put back. A Url value, a parser with positioned errors, a renderer that round-trips, percent-encoding and -decoding for the pieces, and query strings as a name -> values map.

Every program that builds a query string needs the encoding, and a set that is almost right produces URLs that almost work: a stray & or = left unencoded splits one parameter into two with no word about it, and the reader of the resulting request has no way to tell that from what the caller meant. Every program that reads one needs the inverse, and the same care: + is a space in a form body and a plus in a path, and a %XX that is not two hex digits is an error to report and no byte to guess at.

parse follows RFC 3986 §3's generic syntax: scheme, then an authority (// userinfo @ host : port) when the // is present, then path, ? query, # fragment. Every part comes back verbatim: no case folding, no dot-segment removal, no percent-normalising. It decodes nothing, because which decoding applies is a property of the part (a form query is not a path), and the caller therefore applies decode_component / decode_form to the piece it reads. What it refuses is what no URL may carry: a missing or malformed scheme, a byte that is whitespace or a control character or one of "<>\^\{|}, an unclosed [IPv6 literal], a port that is not a number in 0..65535`. Each error names the byte offset it was found at.

There are two encodings and no single one: they look interchangeable and are not, and picking the wrong one fails with no word about it.

encodecomponent RFC 3986 §2.3. A space is %20. What a path segment, a header value, and a signed request (OAuth, AWS SigV4) all specify. encodeform application/x-www-form-urlencoded, the WHATWG URL serializer. A space is +. What an HTML form body and most query strings carry.

The two safe sets differ as well: the form encoding leaves * alone and encodes ~, where RFC 3986 does the reverse. That is what browsers, curl and java.net.URLEncoder emit, and output here is byte-identical to theirs.

Both are total. Every input has an encoding, and there is no error type. Both are defined over the UTF-8 bytes of the text, which is what makes a non-ASCII component encode the same way everywhere.

Decoding mirrors the pair: decode_component reads %XX only and decode_form reads + as a space as well, and both refuse a malformed escape or a result that is not UTF-8, with the offset.

Pure and whole-input: every entry point that builds is @encapsulated, so the builder each writes into is allocated and finished inside the call and nothing mutable escapes. That leaves them callable from meta and from ordinary pure code, on both tiers. http.parse_url is the narrower, http-flavoured reading of the same syntax; this is the general one.

Url

struct Url
  scheme: string
  userinfo: Option<string>
  host: Option<string>
  port: Option<u16>
  path: string
  query: Option<string>
  fragment: Option<string>
end

A URL taken apart. host is None when the URL has no authority at all (mailto:a@b.example) and Some("") when it has an empty one (file:///etc/hosts), which lets a render put the // back where it was. An IPv6 literal host retains its brackets ([::1]). path is whatever followed the authority, "" included; query and fragment are what followed their ? / #, without the marker, and absent when the marker was. Every part is verbatim: decode a piece with decode_component (or decode_form for a form-encoded query) when you read it.

impl Display<Url>

to_string

def to_string(self) -> string

The URL spelled back out - render.

parse("http://h:8080/p?q#f").map(|u| u.to_string()).unwrap_or("") => "http://h:8080/p?q#f"

impl Eq<Url>

eq?

def eq?(self, other: Self) -> bool

Equal part by part.

parse("http://h/p").unwrap_or(_empty_url()).eq?(parse("http://h/p").unwrap_or(_empty_url())) => true

UrlError

type UrlError
  BadScheme(int)
  BadCharacter(int)
  BadHost(int)
  BadPort(int)
  BadEscape(int)
  BadUtf8(int)
end

Why a URL could not be parsed, or a piece of one decoded, with the byte offset at which the problem was found (the json.JsonError idiom).

impl Display<UrlError>

to_string

def to_string(self) -> string

Renders as <what> at offset <at>.

BadPort(9).to_string() => "invalid port at offset 9"

impl Eq<UrlError>

eq?

def eq?(self, other: Self) -> bool

Equal when the kind and the offset match.

BadPort(9).eq?(BadPort(9)) => true
BadPort(9).eq?(BadHost(9)) => false

emptyurl

def _empty_url() -> Url

The placeholder a doctest falls back to where a Result<Url, _> has to be unwrapped; never produced by parse, whose scheme is never empty.

parse

def parse(text: string) -> Result<Url, UrlError>

Take text apart (RFC 3986 §3). Every part comes back verbatim, Url naming what each one is, and an error names the byte it was found at.

parse("https://u@h:8080/a/b?q=1#top").map(|u| u.scheme).unwrap_or("")                     => "https"
parse("https://u@h:8080/a/b?q=1#top").map(|u| u.userinfo.unwrap_or("-")).unwrap_or("")    => "u"
parse("https://u@h:8080/a/b?q=1#top").map(|u| u.host.unwrap_or("-")).unwrap_or("")        => "h"
parse("https://u@h:8080/a/b?q=1#top").map(|u| u.port.unwrap_or(0u16)).unwrap_or(1u16)     => 8080u16
parse("https://u@h:8080/a/b?q=1#top").map(|u| u.path).unwrap_or("")                       => "/a/b"
parse("https://u@h:8080/a/b?q=1#top").map(|u| u.query.unwrap_or("-")).unwrap_or("")       => "q=1"
parse("https://u@h:8080/a/b?q=1#top").map(|u| u.fragment.unwrap_or("-")).unwrap_or("")    => "top"
parse("mailto:a@b.example").map(|u| u.host.unwrap_or("(none)")).unwrap_or("")             => "(none)"
parse("file:///etc/hosts").map(|u| u.host.unwrap_or("(none)")).unwrap_or("x")             => ""
parse("//no-scheme/p")                                                                    => Err(BadScheme(0))
parse("http://h:99999/")                                                                  => Err(BadPort(9))
parse("http://h/a b")                                                                     => Err(BadCharacter(10))

parseafter_scheme

def _parse_after_scheme(text: string, scheme: string, from: int) -> Result<Url, UrlError>

The parts after scheme:, starting at byte from.

_Authority

struct _Authority
  userinfo: Option<string>
  host: string
  port: Option<u16>
end

The three parts of an authority.

parseauthority

def _parse_authority(text: string, from: int, stop: int) -> Result<_Authority, UrlError>

text[from, stop) as [userinfo @] host [: port]: the userinfo is what precedes the last @ (a password may bear an encoded one, and the spec says the last), a bracketed host runs to its ], and otherwise the port is what follows the last :.

parseport

def _parse_port(text: string, from: int, stop: int) -> Result<Option<u16>, UrlError>

text[from, stop) as a port: empty is no port (RFC 3986 allows host:), otherwise digits that fit a u16.

schemeend

def _scheme_end(raw: bytes) -> Result<int, UrlError>

Where the scheme's : is, once letter (letter|digit|+|-|.)* runs up to it; the error names the offset of the first byte that broke the rule.

schemescan

def _scheme_scan(raw: bytes, i: int) -> Result<int, UrlError>

_alpha?

def _alpha?(b: u8) -> bool

firstbad_character

def _first_bad_character(raw: bytes) -> Option<int>

The first byte no URL may carry unencoded: anything at or below a space, DEL, and the RFC 3986 §2 excluded set "<>\^\{|} (a [/] is structure, %` an escape, and a byte past ASCII is left to the caller's decoding).

_excluded?

def _excluded?(b: u8) -> bool

findfrom

def _find_from(text: string, needle: string, from: int) -> Option<int>

The first needle at or after byte from.

findbefore

def _find_before(text: string, needle: string, from: int, stop: int) -> Option<int>

The first needle in [from, stop).

rfindbetween

def _rfind_between(text: string, needle: string, from: int, stop: int) -> Option<int>

The last needle in [from, stop).

rfindstep

def _rfind_step(text: string, needle: string, from: int, stop: int, last: Option<int>) -> Option<int>

render

def render(u: Url) -> string

Spell u back out: scheme: then, when host is present, // with the userinfo, host and port the way they were; then the path, ?query, #fragment. parse(render(u)) gives back u for any u parse produced.

render(Url(scheme="http", userinfo=None, host=Some("h"), port=Some(80u16), path="/p", query=Some("a=1"), fragment=None)) => "http://h:80/p?a=1"
render(Url(scheme="mailto", userinfo=None, host=None, port=None, path="a@b", query=None, fragment=None))                 => "mailto:a@b"

decode_component

def decode_component(text: string) -> Result<string, UrlError>

Undo encode_component: every %XX becomes the byte, everything else is left alone (a + is a plus), and the bytes must read back as UTF-8.

decode_component("hello%20world") => Ok("hello world")
decode_component("a%2Bb")         => Ok("a+b")
decode_component("%C3%A4i")       => Ok("äi")
decode_component("100%")          => Err(BadEscape(3))
decode_component("%ZZ")           => Err(BadEscape(0))

decode_form

def decode_form(text: string) -> Result<string, UrlError>

Undo encode_form: as decode_component, and a + is a space.

decode_form("hello+world") => Ok("hello world")
decode_form("a%2Bb")       => Ok("a+b")

_decode

def _decode(text: string, plus_is_space: bool) -> Result<string, UrlError>

escapebyte

def _escape_byte(raw: bytes, i: int) -> Option<u8>

The byte a %XX at i spells, or None when the two hex digits are not there.

hexvalue

def _hex_value(b: u8) -> Option<u8>

parse_query

def parse_query(query: string) -> Result<Map<string, List<string>>, UrlError>

A query string (a=1&b=2&a=3) as a name -> values map, in the http.Headers arrangement: a repeated name has every value, in order; a pair with no = has the value ""; an empty segment (a&&b) is skipped. Names and values are form-decoded (+ is a space). An error's offset is into the query as given.

parse_query("a=1&b=2&a=3").map(|m| m.get("a").unwrap_or([])).unwrap_or([])     => ["1", "3"]
parse_query("q=hello+world").map(|m| m.get("q").unwrap_or([])).unwrap_or([])   => ["hello world"]
parse_query("flag&x=").map(|m| m.get("flag").unwrap_or([])).unwrap_or([])      => [""]
parse_query("a=%ZZ")                                                            => Err(BadEscape(2))

querysegments

def _query_segments(query: string, from: int, acc: Result<Map<string, List<string>>, UrlError>) -> Result<Map<string, List<string>>, UrlError>

querypair

def _query_pair(query: string, from: int, stop: int, m: Map<string, List<string>>) -> Result<Map<string, List<string>>, UrlError>

_shift

def _shift(e: UrlError, by: int) -> UrlError

An error found inside a slice, re-offset into the whole.

render_query

def render_query(pairs: List<Pair<string, string>>) -> string

Name/value pairs as a query string, each side form-encoded, joined by & in the order given: the inverse of parse_query up to that order.

render_query([Pair(first="q", second="hello world"), Pair(first="page", second="2")]) => "q=hello+world&page=2"
render_query([])                                                                        => ""

unreservedextra

_unreserved_extra: string = "-._~"

The characters each set leaves alone beyond the alphanumerics.

formextra

_form_extra: string = "*-._"

_printable

_printable: string = "*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~"

ASCII 0x2A through 0x7E, indexed by byte - 42. Every character either set leaves verbatim sits in that span, and one table answers both.

hexdigits

_hex_digits: string = "0123456789ABCDEF"

Uppercase, per RFC 3986 §2.1: a decoder must accept either case, but two encoders that disagree produce different bytes for the same input, and a request signature is computed over those bytes.

encode_component

def encode_component(text: string) -> string

Percent-encode text for one component of a URL: a path segment, a query parameter name or value, or anything else that must not be read as structure. RFC 3986's unreserved set (A-Z a-z 0-9 - . _ ~) survives and every other byte becomes %XX, a space included.

encode_component("hello world") => "hello%20world"
encode_component("a+b&c=d")     => "a%2Bb%26c%3Dd"
encode_component("-._~")        => "-._~"
encode_component("äi")          => "%C3%A4i"

encode_form

def encode_form(text: string) -> string

Percent-encode text for an application/x-www-form-urlencoded body or query string, where a space is + and never %20. A-Z a-z 0-9 * - . _ survive; everything else, ~ included, becomes %XX.

A + in the input is itself encoded, which leaves the two spellings of a space distinguishable to whatever reads the result back.

encode_form("hello world") => "hello+world"
encode_form("a+b&c=d")     => "a%2Bb%26c%3Dd"
encode_form("*-._")        => "*-._"
encode_form("~")           => "%7E"

_encode

def _encode(text: string, extra: string, space: string) -> string

One pass over the UTF-8 bytes: a byte in the safe set is copied, a space becomes space, and anything else becomes %XX. A multi-byte scalar is never special-cased: each of its bytes falls through to the percent form, which is what RFC 3986 §2.5 specifies.

_safe?

def _safe?(b: u8, extra: string) -> bool

Whether b is kept verbatim. The range test comes first because _char is only defined over it, and because it rules out every byte of a multi-byte scalar in one comparison.

_alphanumeric?

def _alphanumeric?(b: u8) -> bool

_char

def _char(b: u8) -> string

The ASCII character for b, which callers ask for only after _safe?'s range test has held.

_hex

def _hex(nibble: u8) -> string