206 Partial Content · pfh-autosuggest-simple · September 2025

Byte Range

Autosuggest over a file nobody downloads

I wanted a search box over the 44,331 plant names in a 54,207-row table, on a host that runs nothing, and that table is 10,830,850 bytes, which is far more than a search box should ever ask anyone to download. So I built the index in two tiers: an SSTable of 44,333 normalised-name entries, each carrying the byte range of its own row in the CSV, and above it a 6,355-node prefix index that says which slice of the SSTable to read. Typing Symphytum costs one 300-byte read of the SSTable and one 1,438-byte read of the CSV, so 1,738 bytes in all, which is 0.013% of the three files those offsets address. The box below is that client rebuilt around the same files, and the byte counts it prints are ones your browser actually reported.

Addressed: 13,171,478bytes
Shipped here: 3,288,094bytes
Index nodes: 6,355
SSTable entries: 44,333
Median block: 161bytes
Refusal guard: 10,240bytes

Type something ↓

i The box

Type a plant name, and from the third character the page resolves your query against the prefix index it already holds, sends one HTTP Range request for the block that prefix points at, filters the lines that come back and then works out which rows of the CSV those lines address. The three bars are the three files drawn at their true lengths and the lit segment on each is what got asked for. The status line and the Content-Range header printed beside each bar are read off the response your browser received, and the byte count comes off a Resource Timing entry that did not exist before the request went out, so if the server ignores the Range header and hands over the whole file the page will say that instead of quietly printing the number I wanted.

query

loading the prefix index…

received this session0bytes of 13,171,478 addressed

the first four are the project's own test fixtures, with the reasons it gives for them

1taxon-scientificname-terms-index.txt147,886 Bwhole file, once

0147,886

no query yet

2taxon-scientificname-terms.bin2,192,742 BRange per query

02,192,742

no query yet

3taxon-scientificname.csv10,830,850 Bcomputed, then read locally

010,830,850

no query yet

the same window, magnifiednothing fetched yet

Bar two is honest and unreadable at the same time, so the same file is drawn three times over, each strip magnifying the slice the one above it has bracketed, until the block is wide enough to have a name written in it. Every hairline is a real block boundary out of the prefix index sitting in memory, and each strip prints its own scale.

the blocks have not been checked yet

the bytes the server sends will be printed here exactly as they arrived, newlines and all

the block that came back

Nothing has been fetched yet. The block is printed here exactly as it sits in the file, one term[START:END] per line, with the lines that survive startsWith(query) lit. Click any line, kept or dropped, to open the row it addresses.

the rows those lines address

The CSV columns are taxonID,scientificName,scientificNameAuthorship,source,taxonStatus.

every request, in order

#	query	key	how	Range	status	Content-Range	wire	ms

wire = transferSize from performance.getEntriesByName, taken only from an entry that did not exist before the request went out, so no request is ever credited with an earlier one's bytes. 0 means the browser answered out of its own cache and nothing crossed the network, and a dot means no timing entry arrived at all, which the page prints rather than guessing.

The CSV is 10,830,850 bytes and I could not put that in this repository, so it travels here as the source project's own gzip of it, 922,769 bytes, which the browser unpacks once and keeps. Gzip and Range do not mix, and the dev server in the original project says so outright by answeringAccept-Ranges: none on any path ending.gz, so the third bar prints the request it would have sent, the merged runs and the byte total, then reads the record text out of the bytes sitting at exactly those offsets. It is labelled computed every time and never 206.

The file behind bar two is taxon-scientificname-terms.txt byte for byte, SHA-256 60ec210c4d6b5993…, carrying a .bin extension only so that a CDN serves it asapplication/octet-stream and applies no compression transform on the way through. A transformed body and a byte offset cannot both be right.

ii The format

Both files are the same shape and there is nothing else in them: no header, no magic bytes, no separator, no length field, just a name, a bracket, two integers and a newline.indexer.py writes them with two f-strings,f"{term}[{start}:{end}]\n" on line 144 andf"{prefix}[{start}:{end}]\n" on line 165, and the client reads them back with one regex, /^(.+?)\[(\d+):(\d+)\]$/. Everything else is arithmetic on the two integers.

tier 1 · prefix index · 147,886 Bberberis v[271062:271247]a prefix, and the byte range in the SSTable that holds every name under it

tier 2 · SSTable · 2,192,742 Bberberis vulgaris l.[1808612:1808689]a normalised name, and the byte range of its own row in the CSV

tier 3 · the row · 10,830,850 Burn:lsid:ipni.org:names:328527-2,Berberis vulgaris L.,,MPNS Import,Accepted77 bytes, CRLF-terminated, exactly where the SSTable said it would be

the three lines above are read out of the shipped files at those offsets · the end integer is exclusive, so the header the client builds isbytes=${start}-${end - 1} (search-client.js:321), which is why 271247 on the page becomes 271246 on the wire

The whole tuning surface

Four values in a six-line dataclass decide the shape of every index the project builds, and I never needed a fifth.

max_terms_per_leaf30
min_prefix_length1
max_prefix_depth10
unicode_normalisationNFKD

indexer.py lines 24–29, IndexConfig

Normalising a name

Two statements, and they are the whole of it:unicodedata.normalize(self.config.unicode_normalisation, term), where that setting defaults to NFKD, thentext.lower().strip(). Internal spaces and punctuation survive, so panax quinquefolius l. stays one atomic key rather than three tokens, which is why a prefix walk over it behaves like a walk over a sentence.

Terms then sort by UTF-8 byte order rather than by string comparison,sorted(term_groups.keys(), key=lambda x: x.encode('utf-8')), and that is why the last key in the index is the hybrid sign×[2192559:2192742], whose end offset is exactly the length of the SSTable.

indexer.py lines 122–125 and 134

Where the depth comes from

A node stops splitting when it holds 30 terms or fewer, or when it has descended 10 characters, and its range is then the covering span of everything beneath it:min(r[0] for r in all_ranges) tomax(r[1] for r in all_ranges). Nothing about that rule mentions bytes, which is how you end up with a 14,007-byte leaf sitting next to a31-byte one.

indexer.py lines 178–187

iii The shape of the index

The tree only goes deep where the names crowd. Of the 6,355 nodes exactly one is a single character, 79 are two characters and 965 have run all the way to the depth cap of 10, where splitting stops whether or not the names have thinned out. A quiet corner of the alphabet is finished in two characters, while euphorbia and digitaria hit the cap still crowded, which is how they end up as leaves of 14,007 and 12,241 bytes, the two largest in the file.

279

3994

41,245

5547

6524

7567

8777

9656

10965

prefix length in characters × node count · counted over the shipped index file

What a block weighs

smallest31 B
median161 B
mean345 B
p90894 B
p992,421 B
largest14,007 B

percentiles by nearest rank, over all 6,355 blocks

The two that break the guard

The client refuses any aggregated span over 10,240 bytes. Only two leaves in this whole index are bigger than that on their own, and both are genera with a lot of grass or a lot of spurge behind them.

euphorbia [857653:871660]14,007 B1.37× the guard
digitaria [678028:690269]12,241 B1.20× the guard

the same index built over all six datasets

dataset	indexed column	prefix index	SSTable	CSV	index ÷ CSV
taxon-taxonid	taxonID	340,132	2,711,744	10,830,850	3.14%
taxon-scientificname	scientificName	147,886	2,192,742	10,830,850	1.37%
vernacular-taxonid	taxonID	51,739	9,258,673	31,849,147	0.16%
vernacular-vernacularname	vernacularName	368,270	5,551,093	31,849,147	1.16%
measurementorfact-taxonid	taxonID	136,688	16,312,518	144,107,220	0.09%
measurementorfact-measurementvalue	measurementValue	74,881	14,768,431	144,107,220	0.05%

file sizes measured in the source project's data/ directory · the row this page runs on is lit

iv What a query costs

To find out what the index actually costs I took every distinct three-character opening of the 44,333 entries in the SSTable, ran the realfindBestPrefix over each one and recorded the size of the block it resolved to. There are 1,554 of those openings. Half of them cost 497 bytes or less, the mean is dragged up to 1,201 by the crowded genera and 30 of them are refused outright for straying over the guard. Add a fourth character and the refusals drop to 10; by the eighth there are 4 left in 11,606 queries.

chars	queries	exact key	aggregated	refused	min	median	mean	p90	p99	max
3	1,554	1,145	409	30	36	497	1,201	3,352	8,191	10,240
4	3,546	3,158	388	10	33	432	761	1,604	5,225	10,110
6	6,133	5,849	284	5	31	446	648	1,354	3,581	10,179
8	11,606	11,407	199	4	31	455	600	1,275	2,058	10,179

That needs every name in the corpus, so it costs the whole SSTable, 2,192,742 bytes, in a single request. It is 4,412 times the median three-character query, and avoiding exactly this download is the entire reason the index exists. The button is here anyway, because a number you cannot check is a number you should not believe.

v Where it snags

There are three places where the mechanism does something a person typing would not expect, and all three fall straight out of the leaf rule and the guard rather than out of anything going wrong, so they are behaviour rather than bugs, which is not the same as saying they are fine.

1 · four keystrokes that fetch nothing

digit is not a key in the index. Only two keys start with it, digital[677728:678028] and digitaria [678028:690269], so the client falls to its second branch and aggregates: lowest start, highest end, one span covering both. That span is over the guard, so nothing is requested at all.

digaggregate677526–69026912,743 Bover 10,240, no request sent
digiaggregate677728–69026912,541 Bover 10,240, no request sent
digitaggregate677728–69026912,541 Bover 10,240, no request sent
digitaaggregate677728–69026912,541 Bover 10,240, no request sent
digitalexact677728–678028300 B7 names back for 850 bytes of CSV
digitaliexact677728–678028300 B7 names back for 850 bytes of CSV
digitalisexact677728–678028300 B7 names back for 850 bytes of CSV

The seventh character turns it into an exact hit and the cost collapses from the 12,541 bytes the sixth would have needed to 300 bytes actually fetched, which brings back all 7 foxgloves at once. The 4 queries before it show the visitor nothing at all, so press walk digitalis in the box above and watch it happen.

2 · a two-letter plant name that swallows a query

The greedy walk keeps the longest index key that is a prefix of the query, and it only aggregates when it has found none at all. In the vernacular-name index am is a real key, because four records are literally named am, so its block is four lines long and holds nothing else:

am[1038847:1039026]
am[1039026:1039183]
am[1039183:1039340]
am[1039340:1039497]

The first of the four rows those lines point at reads:

urn:lsid:ipni.org:names:69913-1,am,en,,"SEPASAL database. (RBG Kew, 1999)",http://creativecommons.org/licenses/by-nc-sa/3.0,,urn:lsid:plantsforhealth.kew.org:vernacularName:3349

Neither ame nor amer nor any prefix up to american is a key, so every query from three characters to nine resolves back to am's 80-byte block, reads its four lines, matches none of them and returns nothing. Typing american ginseng stays empty for seven keystrokes and opens on the tenth character, where american g is a key and 15 terms come back.

chars	query so far	key it resolves to	block	lines back	matched
2	am	am	80 B	4	4
3	ame	am	80 B	4	0
4	amer	am	80 B	4	0
5	ameri	am	80 B	4	0
6	americ	am	80 B	4	0
7	america	am	80 B	4	0
8	american	am	80 B	4	0
9	american	am	80 B	4	0
10	american g	american g	523 B	15	15
11	american gi	american g	523 B	15	9
12	american gin	american g	523 B	15	9
13	american gins	american g	523 B	15	9
14	american ginse	american g	523 B	15	9
15	american ginsen	american g	523 B	15	9
16	american ginseng	american g	523 B	15	9

replayed over data/vernacular-vernacularname-terms-index.txt and -terms.txt · those two files are 368,270 and 5,551,093 bytes and are not shipped here, so this table is quoted rather than run live

3 · the same name twice, and the rows nothing points at

The SSTable holds 44,333 entries for 44,331 distinct names, so a key can repeat. It happens exactly twice in this dataset, and both times it is one name arriving from two sources with two different rows:

aristolochia fangchi y.c.wu ex l.d.chow & s.m.hwang
[1386415:1386529]urn:lsid:ipni.org:names:77221314-1,Aristolochia fangchi Y.C.Wu ex L.D.Chow & S.M.Hwang,USDA,GRIN (USDA),Accepted
[1386529:1386911]urn:lsid:ipni.org:names:92924-1,Aristolochia fangchi Y.C.Wu ex L.D.Chow & S.M.Hwang,"Diazgranados, M., Allkin, B., Black, N., Cámara-Leret, R., Canteiro, C., Carretero, J., Eastwood, R., Hargreaves, S., Hudson, A., Milliken, W., Nesbitt, M., Ondo, I., Patmore, K., Pironon, S., Turner, R. & Ulian, T.","World Checklist of Useful Pl. Spp. (Diazgranados et al., 2020)",Illegitimate
cissus rotundifolia lam.
[2702557:2702692]urn:lsid:ipni.org:names:50981090-1,Cissus rotundifolia Lam.,"Royal Botanic Gardens, Kew","SEPASAL database. (RBG Kew, 1999)",Accepted
[2702692:2703043]urn:lsid:ipni.org:names:67922-1,Cissus rotundifolia Lam.,"Diazgranados, M., Allkin, B., Black, N., Cámara-Leret, R., Canteiro, C., Carretero, J., Eastwood, R., Hargreaves, S., Hudson, A., Milliken, W., Nesbitt, M., Ondo, I., Patmore, K., Pironon, S., Turner, R. & Ulian, T.","World Checklist of Useful Pl. Spp. (Diazgranados et al., 2020)",Accepted

At the other end, 9,874 of the 54,207 rows have an empty scientificName and are simply never indexed, which is where 54,207 − 9,874 = 44,333 comes from. The sorter puts them at the front, so the first 484,618 bytes of the CSV cannot be reached through this index by any query: the very first line of the SSTable is aaronsohnia factorovskyi warb. & eig[484618:484981]. There is also a smaller asymmetry I never fixed, which is that the indexer NFKD-normalises every term while the client only lower-cases and trims the query, so a name typed with a composed accent will not find its decomposed key.

vi What is shipped here

file	from the source project	bytes here	how the page uses it
taxon-scientificname-terms-index.txt	taxon-scientificname-terms-index.txt	147,886	fetched whole on page load, exactly as _doInitialise() does, then parsed into a Map of 6,355 keys
taxon-scientificname-terms.bin	taxon-scientificname-terms.txt	2,192,742	never fetched whole for a query, only with a Range header, one request per keystroke that resolves; the recompute button in section iv is the one exception and says so in the log
taxon-scientificname.csv.gz	compressed/taxon-scientificname.csv.gz	922,769	the 10,830,850-byte CSV, gzipped, unpacked once in the browser and read at true offsets
meta.json	derived	24,697	every static figure on this page, written by scripts/build-byte-range-data.mjs
total		3,288,094	against 13,171,478 bytes of index, SSTable and CSV that these offsets address

the three data files are byte-for-byte copies · SHA-256 d4b8b861c4f3…, 60ec210c4d6b… and 9f1fa928497c… for the gzip, which unpacks to 8eaaa3b35885…, a round trip the copying script refuses to ship without

Where the rows came from

The names are a compiled set and the CSV's own source column says who compiled each one. All 10 distinct values, over all 54,207 rows, including the seven rows of test data I left in the file and never took out:

World Checklist of Useful Pl. Spp. (Diazgranados et al., 2020)23,289
MPNS Import13,059
(no source given)10,272
UPWTA (Burkill, 2025)2,475
GRIN (USDA)2,412
SEPASAL database. (RBG Kew, 1999)1,848
Duke's Phytochem. & Ethnobot. Databases. (Duke, 1992-2016)761
Plants. Goldfrank's Tox Emergencies (Nelson, 2018)76
Ethnobot. & Nutritive Potential of Wild Food Pl. (Shad et al., 2013)8
Dummy Data: "Product" Use Cases7

the sibling vernacular file carries http://creativecommons.org/licenses/by-nc-sa/3.0 in a licence column of its own; this taxon file states no licence in its columns

And what they say

taxonStatus over the same rows. The empty bucket is exactly the set with no scientific name, which is exactly the set the index cannot reach.

Accepted40,084
(empty)9,874
Synonym4,005
Illegitimate163
Artificial Hybrid34
Invalid27
Unplaced10
Orthographic6
Misapplied4

pfh-autosuggest-simple. Five commits between the 6th and the 15th of September 2025, MIT licensed, ninety files that are not node_modules or .git and 478 MB of generated indexes sitting underneath them on top of 188 MB of input. The indexer is Python, shelled out to from Node by scripts/build-index.js after the CSV has been sorted on the target column, and the comment above that call explains why it happens in that order: Index using the sorted CSV as the source (so byte ranges are stable). The README describes the whole thing in one line as an SSTable of terms + Adaptive Prefix Index → fetch only relevant byte ranges, and the claim in IMPLEMENTATION.md of ~6355 prefix nodes for 44,331 unique terms still holds against the files as they sit on disk today.

Two of the rules I set myself for it are in .cursor/rules and I have kept both here: Use British spellings for new function names: normalise, optimise, initialise, and, in the error-handling section, Never fall back to mock data in error handling followed by Use real functionality instead of generating fake results. The README puts it plainly as No mock data is used; all results come from actual CSV byte ranges, which is a promise that is easy to make and awkward to keep, so I have kept it here by measuring rather than asserting: status codes and Content-Range headers come off the Response, wire bytes come off performance.getEntriesByName(url).transferSize and the one tier that is not fetched says so on every row it writes.

Source project: pfh-autosuggest-simple/plants-for-health, git remote github.com/davidrobertmcculloch/plants-for-health. Format and configuration from indexer.py (144, 165, 24–29, 122–125, 134, 178–187). Client algorithm from search-client.js (318, 351, 401, 443, 472). Debounce and the three-character floor from index.html (641–647, 847). Range behaviour and the gzip refusal from server.js (34–77) and .cursor/rules/byte-range-and-gzip.mdc. The first four demo queries and their stated reasons verbatim from test/data/test-data.json; the last two are mine. All figures regenerated 2026-07-22 by scripts/build-byte-range-data.mjs, which reads the three shipped files and writes every number above.

← back homeDR M^cCulloch · pfh-autosuggest-simple