← ALL POSTS · /BLOG

CORPUS PACKS · IDENTITY RESOLUTION · ED25519 · BUILT 2026-09-26/27

ONE VULNERABILITY, THREE NAMES,
AND A PACK WE SIGN

The same Rust vulnerability arrives as RUSTSEC-2021-0003, GHSA-43w2-9j62-hq99 and CVE-2021-25900 — three records, in two databases that never talk to each other, none of them carrying everything the other knows. We built a tool that collapses them into one record, packs it into something portable and checksummed, and then signs the pack. The last part is the strangest finding of the project: in our survey of vulnerability databases people pip-install and OCI-pull every day, not one ships a signature a consumer can verify.

01 · THE PROBLEM

Two sources, each sure it is the whole story

We started with the sources everyone reaches for, and measured what each one actually carries for the Rust ecosystem:

Source (both CC0)RecordsHasMissing
osv.dev crates.io export2,856GHSA + RUSTSEC + MAL ids, CVSS scoreszero affected-function data; aliases never merged
RustSec advisory-db1,251curated vulnerable-function paths, categories, keywords, full advisory proseno GHSA-only records
GitHub advisory-database—the same crates.io recordsits extra fields are marked internal-use-only — we never read them

The alias problem is the one that hurts an agent most. osv.dev's export lists the same vulnerability twice — once under its GHSA id, once under its RUSTSEC id — with a shared CVE in each record's alias list, and no merge ever performed between them. Search it for “smallvec insert_many” and you get whichever filing happened to phrase it that way; the other half of the knowledge is a separate record you did not retrieve. RustSec fixes the merge by being one curated database, but it cannot see GHSA-only filings at all.

HackerOne, the source the original brief asked about, is out of scope for a different reason: report content there is unredistributable under its terms of service. A pack's author owns what goes in the pack, and this pack ships only sources whose licence lets us redistribute them.

02 · THE RECIPE

Identity resolution as a declarative file

We did not write a Rust-vulnerability importer. We wrote a corpus builder whose entire domain knowledge lives in a commented TOML recipe — because the next corpus is not advisories. The same tool with a different recipe indexes mailboxes, or code, or genomes; nothing about the pipeline knows what a vulnerability is.

The identity section is three lines:

[identity]
edges = [{ field = "id" }, { field = "aliases", each = true }]
canonical_source_order = ["rustsec", "osv"]

Union-find over those edge values does the rest, and it closes chains for free: a RustSec filing and a GHSA filing that share a CVE alias merge even with no direct edge between them, because each is transitively bound to the CVE. Measured on the real sources:

rust-vulns, measured off the committed buildCount
source records harvested (envelopes)4,107
records after identity resolution1,950
…merged from both sources1,221
canonical id: RUSTSEC / GHSA / MAL1,221 / 711 / 18
records carrying curated function paths270 (663 distinct paths)
withdrawn advisories (kept, flagged)124
packages covered · records JSONL bytes · cold build1,813 · 10,140,789 · ~34 s

The function paths are the field no export carries. RustSec's hand-curated affected.functions — which function in which crate is the vulnerable one — survives the merge unioned across sibling filings: one 2026 CVE filed against three libcrux-* crates becomes one record carrying all three crates' paths. For an agent asking “is this call the dangerous one?”, that is the whole difference between a hit and an answer.

Merge precedence is declared, not discovered: RustSec wins a disagreeing scalar because it owns the curated fields; arrays are always unioned; the merged record's licence is the most restrictive of its sources and sources[] names every contributor, so a merge is auditable after the fact. Withdrawn advisories are kept and flagged rather than dropped — filtering is the consumer's decision, not the pack's.

03 · THE PACK

A directory you can move, and a machine that refuses surprises

The build emits a self-contained directory: sharded records in canonical JSON order, a manifest with per-file SHA-256 checksums, the recipe verbatim as provenance, and suggestions — a proposed field mapping and join candidates computed from a sample, explicitly suggestions, not decisions, because the index infers the real mapping the way it does for any folder.

Two rules in the format are load-bearing. A manifest carries a format_version that readers refuse when they do not know it — a published pack outlives the release that built it. And installing one verifies every checksum before anything is materialized, so a truncated or edited pack is refused whole, not half-mounted:

$ xerj corpus add rust-vulns --from rust-vulns-pack.zip --verify-sig rust-vulns.pub
$ xerj corpus index rust-vulns
$ xerj code rust-vulns "smallvec insert_many buffer overflow"

Incremental rebuilds need no mode flag: every normalized record is stored content-addressed, so a re-run recomputes each key, sees the file, and skips. Only new or changed upstream records are written, and the pack re-emits deterministically from whatever the store holds — same bytes in, same pack out, which is what makes a signature over it meaningful.

04 · THE SIGNATURE

Checksums prove integrity. They do not prove origin.

Here is the attack that checksums cannot stop. Take a pack. Edit records — flip a fixed_versions string, say. Recompute every file's checksum and rewrite SHA256SUMS honestly. The pack now verifies perfectly against its own manifest. Every checksum agrees, because the checksums were never the thing being attacked — the pack's origin was.

We pinned exactly this in a test: a self-consistent rebuild passes every checksum check and fails the signature. That gap is the entire reason the pack carries one — a raw 64-byte ed25519 signature over SHA256SUMS, verified by xerj corpus add --verify-sig <pubkey> before a byte is materialized.

Before building it we surveyed what the existing ecosystem ships, and found the lane empty. None of these is a criticism of the projects — their data is good and their tooling is real; it is a statement about the format:

Database (our survey, 2026-09)DistributionIntegrity mechanism
Trivy DBOCI artifact, 6-hour buildschecksums in registry metadata — no consumer-verifiable signature
Grype / vunnelper-provider feedsunsigned
PrimeVulHugging Face datasetunsigned
VulnRAGresearch artifactunsigned

The design rules we landed on, each one because its absence would make the signature decorative:

What the signature does not do. It proves who built a pack, not that the pack is safe to open — a verified pack is exactly what the key holder built. And it is one release key in v1: rotation means a new keypair, a new CI secret, a committed .pub, and consumers choosing to trust the new one. We wrote the procedure down before we needed it.
05 · THE HONEST PART

What this is not

Not a security product. It is a searchable corpus of advisories for agents and humans, with no completeness claim beyond what its two sources carried at build time. It is not a scanner and will not tell you what is running in your production.

Freshness is a cron job, not a promise. The publish workflow rebuilds daily; between a source updating and the next run, the pack lags by up to a day. The README badge reads the build date off the release itself, and the same 30-day staleness refusal that governs every reference corpus governs ours — a workflow that stops running shows up as a stale pack, not a silently aging artifact.

Records, not an index. Installing the pack still costs every consumer the same CPU-minutes to index — the pre-indexed half of the corpus-hub idea is a separate, harder format problem (a published index needs a compatibility promise our internal segment format has never had to make).

The value hypothesis is unproven. Trivy's database gets pulled because scanners need it at scan time; an agent-facing vulnerability corpus has weaker daily pull, and the honest analogy is template packs that took years to compound. What we can say is what the pack uniquely carries — identity-resolved clusters, the curated function paths, a signature — and that the numbers above are measured, not projected.

06 · THE TRACE

Every number, pinned to a run

The counts in section 02 come from tools/packs/rust-vulns — a committed pack-stats.json whose provenance field says measured, with a test that fails the build if the README's table and the stats file drift apart, and another that fails if the stats describe any recipe other than the one shipped. The builder, the signature code, and the pack format are in the same repository under engine/crates/xerj-autoindex/src/harvest/, and the format is documented end to end in docs/CORPUS_PACKS.md.

The pack itself is a rolling GitHub Release — pack-rust-vulns — rebuilt and signed on a schedule, with the zip, the loose checksums, the signature, and a freshness file as assets. Build it yourself instead, from the recipe, if you would rather trust your own machine than our key; that is a supported path, not a workaround. If your agent works with Rust dependencies, point it at the pack and ask it the question a raw export cannot answer: which function is the vulnerable one?