rdf

A read-only RDF quad store and SPARQL 1.2 engine, in safe Rust but for one audited AVX2 kernel.

Data goes in once, by bulk load from N-Triples, Turtle or HDT — .gz and .bz2 included, decompressed as they are read, a bzip2 file on every core — and is never written again. That is enforced rather than intended: the loader writes its files under fixed names, so a destination that already holds a manifest is StoreError::DestinationHoldsStore, and rdf load --replace is how a caller who knows nobody has that store open says so. The constraint is what the rest of the design is built on: the store is assembled in a single pass, so every dictionary is sorted before an identity is handed out.

rdf estimate data.nt.bz2                    # how many triples, without unpacking it
rdf load  --output mystore data.nt data.hdt # bulk load, syntax from the extension
rdf query --store mystore query.rq          # answer a query
rdf explain --store mystore --actual q.rq   # the plan, with estimates beside actuals
rdf stat  mystore                           # what the store is made of
rdf dump  mystore                           # every quad back out as N-Triples
rdf bench --scale 20                        # LUBM-shaped generator and query suite
rdf bench --scale 100 --parallel 8          # what dividing a query is worth
rdf bench --dir store --queries q.ndjson    # answer somebody else's queries, report JSON
rdf serve --store mystore                   # answer queries over HTTP until stopped
#   built with --features rdf/console, that also serves a query console at /

The two properties everything rests on

A term is one 64-bit word, and two terms are identical exactly when their words are equal. rdf_model::same_term is ==, so join keys, DISTINCT and GROUP BY never materialise a term. This is why a value is encoded inline only when its lexical form is already canonical: "01"^^xsd:integer is a different RDF term from "1"^^xsd:integer, and encoding both inline would collide them.

Identity order is term order wherever TagSpan::orders_by_identity holds, and there ORDER BY is a radix sort over the identities themselves — no comparator, no dictionary read. Neither property would hold in a store that could be written to.

docs/CONCEPTS.md is where to start if RDF, SPARQL or column stores are new to you: every term the other documents use, each with a short explanation and an example.

ARCHITECTURE.md is the long version of both properties above and of everything they buy: the crate map, what happens to a byte between a load and an answer, the on-disk format, the planner's decisions, the executor's contract, the invariants in one place, and where a change of a given kind belongs. KNOWN-ISSUES.md is what this system is not — what was measured and rejected, and what is missing.

Layout

rdf-iri        RFC 3987 grammar and RFC 3986 resolution, asked by every parser
rdf-model      term identities, inline codecs, value semantics, comparison
rdf-ntriples   streaming N-Triples 1.2 parser, word-at-a-time scanning
rdf-turtle     resumable Turtle 1.2 parser, readable in as many pieces as it takes
rdf-hdt        streaming reader for HDT, the binary serialisation, divisible per file
rdf-io         file envelope, checksums, integer codecs, pages, buffer pool
rdf-store      dictionary, quad index, statistics, bulk loader
rdf-sched      morsel scheduler with hierarchical fair-share priority (no RDF in it)
sparql-eval    batches, expression evaluation, operators, planner, async engine
sparql-results SRX, SRJ, CSV and TSV writers
rdf            the command line
sparql-console the query console's files and examples  (only under --features rdf/console)
sparql-syntax  lexer, parser, abstract syntax        (pre-existing)
sparql-algebra `sparql12-query` section 18.2 and its translation (pre-existing)

Serving

rdf serve --store DIR answers SPARQL 1.2 Protocol query requests over HTTP against one store, opened once at start-up rather than once per request:

rdf serve --store ./store &
curl 'http://127.0.0.1:3030/sparql?query=ASK%20%7B%7D' \
     -H 'Accept: application/sparql-results+json'

Opening a store reads its sections whole when they fit in DEFAULT_RESIDENT_BUDGET, and rdf query pays that read every time it runs. But the reason worth having is the other one: the scheduler underneath is built for concurrent queries, and one query per process can never use it. A request may carry Rdf-Class: analytic, and that class is created with a weight of 1 against the interactive class's 64 (crates/rdf/src/serve.rs), so a report cannot starve a lookup.

An answer under --hold-back is held whole so that a query failing halfway is still a clean status code; past it the answer streams, and the engine's backpressure means a slow client parks its query rather than buffering for it. A request also has a bounded time to arrive: HEAD_TIMEOUT bounds the head and BODY_TIMEOUT what follows it, a body that misses it being a 408, which is what makes --max-connections a bound on connections rather than a count of sockets a client may hold for as long as it likes. There is no TLS and no authentication — --bind is DEFAULT_BIND, which is loopback, and anything exposed belongs behind a reverse proxy. See D7 and D8 in KNOWN-ISSUES.md.

The query console

Built in, rdf serve also answers a query console at /. It has an editor with SPARQL highlighting and an error gutter, and a results table that shortens IRIs against the query's own prefixes. Beside the table are an image grid and four charts drawn as inline SVG. And it ships the 344 examples query.wikidata.org publishes that this engine accepts:

cargo build --release --features rdf/console   # needs the Elm 0.19.1 compiler
rdf serve --store ./store                      # then open http://127.0.0.1:3030/

It is opt-in at build time, and off by default. The console is written in Elm, and building it needs the Elm compiler, which is not a Rust toolchain — so cargo build --workspace and cargo test --workspace need nothing but Rust, and a binary built without the feature answers a 404 at / and is otherwise identical. Note that --all-features therefore needs Elm too. The application lives in crates/sparql-console, a leaf crate; crates/rdf/src/serve/ui.rs holds the whole of the feature's effect on the endpoint, which is two function bodies.

The status line reports the engine's own time, not the round trip. Every answer carries a Server-Timing header holding what the server spent between having the query text and writing the last byte of the document — the parse, the plan, the wait for admission, the execution and the serialisation — with the planning broken out beside it. Hovering the line shows the round trip too; the gap between the two is the socket, the kernel and the browser.

The console is served from the port it queries, and that is a consequence of D7 rather than a convenience: the endpoint sends no CORS headers and is not going to, so a page served from anywhere else could not call /sparql. Same-origin needs no CORS header at all.

The examples come from compat/wikidata/, which this repository already holds for the compatibility suite. All 387, less the 43 this engine refuses — and that 43 takes two passes rather than one. Thirty-two come from the unparsed and unsupported buckets of compat/wikidata/expectations.tsv. Eleven more are listed by hand in crates/sparql-console/excluded.tsv, because a bucket only says something about an example query.wikidata.org answered itself, and it did not answer a hundred of them.

Neither number can drift quietly. Two tests in crates/rdf/tests/serve.rs run every bundled example through the endpoint, and fail if one is refused or if one that was excluded stops being refused. What the console draws and what it does not — Map is the notable absence — is D43 in KNOWN-ISSUES.md.

Measured

rdf bench --scale N generates a LUBM-shaped corpus of N universities, loads it, and runs the query suite. Per query it prints what planning took, what the run took, how many rows came back against what the planner estimated, what the query held at its peak, how many row groups were decoded and what the buffer pool hit. Then what a prefix count costs, what the same count over the span a FILTER left costs, and what a scan open and a seek cost. Then point-query latency under load.

rdf bench --queries FILE asks the other question: what does this engine do with somebody else's queries. One JSON object per line — {"id": ..., "sparql": ...}, as a workload capture or a gallery is written — answered against the store --dir names, which must already exist. A JSON report goes to standard output as each answer arrives, one object per query: its form, whether it answered, how long it took, how much of that was over before a row was read (prepare_ms), what the scheduler charged its own task, what it held against the same cap rdf query gives a query, how many row groups it decoded, and how many rows or triples came back. That charge is elapsed time on a worker rather than CPU time, and it is the query's own task and not the helpers it divided itself into — so on a workload where every query divides it reads a fifth of what the process spends, and --helpers 0 or /usr/bin/time is what answers the other question. The row groups move with none of that. They are what the query read the store for, so they are the same under --jobs 8 as under --jobs 1, and the same however many helpers the query divided itself over. That is what makes them the number to compare two plans by. A query that will not parse, will not plan or fails while running is recorded with what it reported and the run goes on, because a captured workload is expected to hold queries this engine does not answer.

Three options say how the file is answered rather than which one it is, and the third has three settings of its own. --deadline MS bounds one query, because a capture holds queries that read the whole store and one of them costs more than the thousand around it together. --helpers N is rdf query --max-parallelism under the name the report uses, so that what dividing a query is worth can be asked of a workload rather than of one query: reviews/REVIEW-2026-08-21-query-performance.md is that question asked of three hundred thousand captured Wikidata queries, and what the answers came to.

And --stage says where each query stops. plan stops at the plan and drops it, which isolates the half of a small query's cost that no row count explains. answer runs the operators and counts the rows. write goes the rest of the way: it writes the document into a sink, which resolves every cell through the dictionaries exactly as answering a caller does, and reports that as document_ms inside runtime_ms and beside it, the way prepare_ms already is. The three stages are three measurements rather than three readings of one, so compare a run against the same stage and not against another.

Read write before drawing a conclusion about where a query's time goes. answer stops one stage before the socket, and the stage it stops before is not a rounding error. SELECT ?item ?type WHERE { ?item wdt:P31 ?type } LIMIT 50000 against the Wikidata truthy store, one task, no helpers: 4.9 ms at --stage answer and 66.9 ms at --stage write, of which 64.5 ms is the document. The operators are a fifteenth of it. rdf bench --scale has always drained through the resolution and reported it as its own column; this half of the suite did not until write existed, so a profile taken through --queries before it measured a pipeline that stopped one stage short.

--jobs N is how many are answered at once, defaulting to the core count. The queries then contend, so each runtime_ms is what that query took while the others ran — a throughput number rather than a latency one, and --jobs 1 answers the other question. Note that raising it does less than it looks: max_parallelism already divides one query across every core, so on eight cores sixteen scan-shaped queries took 3.19 s at --jobs 1 and 2.79 s at --jobs 8, while the median query went from 201 ms to 1372 ms.

That last one runs ANALYTIC_STREAMS scans and PLANNING_STREAMS planning streams against the point queries, all charged to the analytic class (crates/rdf/src/bench/priority.rs); KNOWN-ISSUES P4a carries what it measured, at --scale 5 rather than either scale below. rdf stat reports where a store's bytes went, per file and per section, which is where bytes per quad comes from.

Everything either of them prints is a figure of one corpus at one scale, and --scale is the only knob on that corpus: what a scale comes to is generate::triple_count(scale), a multiple of one university's worth. So every figure here stands beside the scale it was taken at, and the command that takes it again is rdf bench --scale N --dir DIR, with rdf stat DIR for the bytes. Twenty is DEFAULT_BENCH_SCALE, what rdf bench alone generates. One eight-core Linux box that was not idle, one run a column.

	--scale 20	--scale 42
generated triples, generate::triple_count	948 540	1 991 934
quads loaded, duplicates gone	919 400	1 930 605
loading them, out of generated N-Triples	1.3 s	2.5 s
bytes a quad, rdf stat	11.52	10.93
P2 one property of one student	1.2 ms	1.1 ms
S1 student star, 100 000 and 210 000 rows	73 ms	157 ms
C1 complex, ten patterns, 270 860 and 568 671 rows	195 ms	471 ms
C1 at its peak	8 301 KiB	17 432 KiB
point-query median under the priority load	5.2 ms	7.0 ms
its 99th percentile	16.0 ms	17.5 ms

Each column is one run, and only half of it repeats: the rows, the estimates, the row groups and the peak bytes come out identical on a second run, the times do not — the same --scale 20 again gave S1 73 ms and C1 198 ms. Read the milliseconds that coarsely, and re-run them rather than cite them.

The bytes are the half that does repeat, and they moved: 5cab048 gave a dictionary column a codec of its own and took about 3.5% off a store. That is the right direction, which is exactly why the table went on saying 11.94 and 11.47 for two days. Both figures above are rdf bench --scale N --dir D followed by rdf stat D, re-run 2026-08-22 over 919 400 and 1 930 605 quads.

The bytes are the half that does repeat, and they moved: 5cab048 gave a dictionary column its own codec and took 3.5% off a store, which is the right direction and exactly why nobody noticed the table saying 11.94 and 11.47. Both figures above are rdf bench --scale N --dir D followed by rdf stat D, re-run 2026-08-22 over 919 400 and 1 930 605 quads.

Two figures the old table carried are missing because nothing in the suite prints them: there is no GROUP BY query in crates/rdf/src/bench/queries.rs, whose shapes are linear, star, snowflake, complex and point; and peak resident under a small budget is a load-side figure a test holds rather than a run reports. That test is the first of two things pinned rather than measured, and both hold at every scale:

A load holds its budget, not its corpus. crates/rdf-store/tests/peak_memory.rs loads a corpus whose dictionary is many times the budget and reads this process's high-water mark out of /proc/self/status: it must stay under PEAK_LIMIT_BYTES and under the dictionary the load wrote, which a load that assembled a section before writing it could not do.

A load that spills and a load that does not write the same bytes. crates/rdf-store/tests/streaming.rs compares whole store directories byte for byte, with the uuid fixed through LoadOptions::store_uuid. Twenty bytes per file are what a uuid would otherwise move: the sixteen holding it and the four holding the header checksum that covers them (crates/rdf-io/src/format.rs). A load derives that uuid afresh from the clock, the process and the destination, so that pairing one store's index with another's dictionary is caught.

Reading HDT was measured on the real thing: latest-all-08-Mar-2018.hdt, 53.6 GB, 5.73 billion triples. Opening it and dividing it into eight shares takes about ninety seconds, after which the loader reads and interns 303 000 triples/s on eight cores, all eight of them CPU-bound. A full load of that file is some hours of parsing and was not run; D18 in KNOWN-ISSUES.md says what the per-triple cost is made of and what would take it down.

Nothing on the read path checks a checksum — a scan borrows a page and verifies nothing, because a CRC per page is a constant per quad. rdf stat --verify reads every section whole and checks all of them, and is the answer to "did that disk take something with it". See D14 in KNOWN-ISSUES.md for what a store corrupted underneath this engine does without it.

Conformance

Four official W3C suites are vendored and run, all from one upstream commit, which each suite's own tests/w3c/README.md names:

Suite	Where	Passing
SPARQL evaluation	crates/sparql-eval/tests/w3c	564 of 575
SPARQL syntax	crates/sparql-syntax/tests/w3c	571 of 571
RDF Turtle	crates/rdf-turtle/tests/w3c	416 of 416
RDF N-Triples	crates/rdf-ntriples/tests/w3c	99 of 99

Every one of those totals is asserted by the suite that reports it — crates/sparql-eval/tests/w3c_eval.rs pins (575, 564, 11) — and each suite carries a known-failures.txt listing what does not pass, one line at a time with a reason, whose set must equal the set that fails. So a new failure breaks the build, and so does a fix nobody records.

Every one of them is deliberate, and they are one kind: a computed number written in a lexical form the expected file spells differently, for which SPARQL prescribes none. known-failures.txt groups them by cause and writes each cause out once — including agg-min-02, where it is the expected file that canonicalised the spelling the datum itself was given, and cast-decimal, which wants two incompatible spellings out of one code path. Nothing in the failure list is a defect this engine agrees is one.

Comparing two xsd:date terms used to be in that list and is not: KNOWN-ISSUES D3 has what = on a pair of dates now follows, and what about < is still an extension.

The last five of the N-Triples tests were bought rather than found: every IRIREF goes through rdf_iri::is_absolute, which cost 11% of an end-to-end load until rdf-iri was made to read an IRI in one pass instead of five, and then 32 bytes at a time. rdf bench --parse is the instrument and KNOWN-ISSUES D12 the record of every number. What it buys is that rdf-ntriples and rdf-turtle now agree completely about what an IRI is — crates/rdf-turtle/tests/subset.rs asserts it.

Building

One thing to install first. cargo audit and cargo fuzz are cargo subcommands rather than libraries, so no manifest can bring them in — but their versions are pinned like any other dependency, in [workspace.metadata.tools], and this reads them from there. It also unpacks the Elm compiler, which is not downloaded at all: vendor/elm/ holds it, and that directory's README says why a vendored binary was the answer and what makes its provenance checkable.

./install-tools.sh

Then the gate, before every commit:

cargo audit                                                              # before every build
cargo fmt -- --check
cargo clippy --workspace --all-targets -- -W clippy::pedantic -D warnings
RUSTDOCFLAGS='-D warnings' cargo doc --workspace --no-deps
cargo fmt --manifest-path fuzz/Cargo.toml -- --check                     # `fuzz/` is its own workspace
cargo audit --file fuzz/Cargo.lock
cargo test --workspace

The two fuzz/ steps are there because --workspace does not go into a second workspace: without them its lock file and its formatting are outside the gate. Its code is not — crates/*/tests/fuzz_regressions.rs compiles the harness as an ordinary integration test, which is what ARCHITECTURE.md section 12 explains.

None of the seven needs a second toolchain, and the query console is opt-in for that reason. Build it too when you touch it, in a second group after these, so that an opt-in feature is held to the same standard:

cargo clippy --workspace --all-targets --features rdf/console -- -W clippy::pedantic -D warnings
cargo test --workspace --features rdf/console
(cd crates/sparql-console/ui && elm-test)     # elm comes from vendor/elm/; node does not

All seven must be clean, in that order. There is no continuous integration here — it was removed on 2026-08-22, because this project is built locally — so that order is yours to keep rather than a runner's. Audit comes first because nothing may be built on a dependency with a known advisory, and the doc build sits between clippy and the suite because a link into an item that was renamed or made private is a reader sent nowhere, which neither of its neighbours can see. See AGENTS.md for the full conventions. The toolchain is pinned in rust-toolchain.toml, because clippy in pedantic mode with -D warnings turns a new lint in a new toolchain into a broken build nobody asked for.

Two greps and two benchmarks are the audit of the exceptions:

grep -rn 'allow(clippy' crates/       # six lines
grep -rn 'allow(unsafe_code)' crates/ # two lines, both an AVX2 block filter
rdf bench --iri                       # what the first of those two is worth
rdf bench --scan                      # and the second

Four of the six are in the hash implementations, where the single-letter names are the ones RFC 1321 §3.3 and FIPS 180-4 §6.1.2, §6.2.2 and §6.4.2 give the working variables. Renaming them would make the code harder to check against the specification. The other two are a rounding that is the point rather than a mistake, and a struct of bools a clap argument group has already made exclusive.

The two unsafe modules read 32 bytes at a time where the safe code reads eight: rdf-iri's reads an IRI behind two vpshufb lookups, and rdf-ntriples's finds the byte that ends a token. unsafe_code is deny rather than forbid for exactly those two. AGENTS.md §8.13 is what a third has to satisfy, and the two commands above are the measurement it demands — runnable rather than quoted. It sets no ratio to clear. It asks for the measurement and for the trade in writing, because a kernel's own speed-up says more about what sits behind a #[target_feature] boundary than about the kernel. KNOWN-ISSUES.md D12 and D44 are those two write-ups.

Everything else is written to vectorise on its own rather than with intrinsics. If you care about throughput, build with RUSTFLAGS='-C target-cpu=native'; the default stays portable.

Dependencies

Eleven dependencies are chosen. What those eleven pull in behind them is another forty, for fifty-one third-party crates altogether; cargo tree -p rdf -e normal is the check, and it prints all of them rather than the eleven. The comments in Cargo.toml say what each choice costs transitively. Everything else in Cargo.toml is test-only — tempfile and pretty_assertions — and is not linked into rdf.

Shipped	For
tokio	the async submission API, result channels and timers. The scheduler runs its own OS threads; tokio never sees CPU work
hyper, hyper-util	HTTP/1.1 for rdf serve, and the Body abstraction the engine's backpressure rests on
elm/* (eight direct, three behind them)	the query console, under --features rdf/console only. Not Rust and not in Cargo.lock, so cargo audit does not see them; crates/sparql-console/ui/elm.json pins them exactly, which Elm requires of a direct dependency
clap	the command line, where which option belongs to which subcommand is the shape of a struct rather than a table nobody checks
regex	REGEX and REPLACE. Chosen for having no backtracking: a pattern runs inside a scheduler morsel, and one that could take exponential time would hold a worker for an unbounded one
logos	the lexer in sparql-syntax
thiserror	error types
self_cell	the operator tree borrows the store and the plan while a scheduler task must be 'static
quick-xml	reading and writing SPARQL Results XML — one mechanic for the format, in both directions
bzip2	the .bz2 the large public dumps are published in. Its default backend is libbz2-rs-sys, a Rust rewrite of libbz2, so this is one crate and no C toolchain
flate2	gzip, on its rust_backend, which is miniz_oxide rather than zlib. Serial, because DEFLATE is: see KNOWN-ISSUES D31

Three things about that table get asked about.

Nothing reads RDF but this workspace. oxttl, oxrdf, oxrdfxml and oxiri were all here once and are all gone. The vendored suites' TriG and RDF/XML documents are read by harness-local readers under crates/sparql-eval/tests/w3c_support, which cover exactly the constructs the vendored files use and refuse everything else.

The hash functions go the other way. MD5 and the SHA family are implemented in crates/sparql-eval/src/expr/digest.rs rather than pulled in as five crates, and tested against the published vectors. Fixed, self-contained and fully specified is exactly when writing it yourself costs less than depending on someone else.

No audit exceptions. There is no .cargo/audit.toml, and cargo audit passes with nothing ignored. The one exception this workspace ever had left with the crate that needed it.

Not supported, by decision

SPARQL Update. The store is read-only and bulk load is the only write path.

SERVICE over the network. Federation means trusting a remote endpoint's answer, which this store cannot cost, cannot bound, and cannot schedule against the priorities everything else is built around. D1 in KNOWN-ISSUES.md says where a query that needs it belongs instead.

One service is answered, because it is not federation. SERVICE wikibase:label names no endpoint and sends no request: it asks for a term's label, alternative labels and description in a preference order of languages, and those are quads this store holds. It is the Wikidata Query Service's extension, 296 of the 387 queries in that service's own example gallery use it, and D1 records what this engine's reading of it does and does not promise.

Every input format except N-Triples, Turtle and HDT, which are the three rdf_store::RdfFormat names. N-Quads, TriG and RDF/XML appear only in the vendored test suites, read there by this workspace's own parsers and by the harness-local readers above; JSON-LD is read nowhere at all. HDT is in the list of three because it is the format the large public dumps are published in, and because reading it needs no character-level parsing: a file is a sorted dictionary and three packed integer arrays, so a graph arrives already deduplicated, already in order, and divisible across every core of one machine — which is more than either text syntax offers for a single file.