Lance v14 reference#

Lance is an open columnar format for multimodal AI - "a columnar data format that is 100x faster than Parquet for random access." It is not one format but a stack of interoperating specs: a file format, a table format, index formats, catalog specs, and a namespace client spec. The Rust workspace at lance-format/lance implements all of them plus Python ( pylance) and Java bindings.

This skill tracks v14.0.0-beta.14 (the lance-format/lance git tag), the current development frontier; v13.0.0 is the stable pin, released 2026-10-07. Pin against tags, not main - Lance ships beta tags every few days and next-format encodings can change. Version landscape below.

Three layers of reference, load what the task needs:

• The deep reference - any concrete schema, parameter, proto, or constraint. Split by topic: File in references/CoversSectionsformat-file.mdWhat Lance is, the 27 crates, file format, data types1-4format-table.mdDataset layout, manifests, fragments, schema evolution, versioning/tags/branches, row IDs, transactions + OCC, MemWAL5-10indexes.mdVector / scalar / FTS / geo indexes, distributed builds11-12ops.mdObject store, capability matrix, source map13, 15, 16changelog-v7-v14.mdThe full v7 -> v14 delta14 Cross-references written as "section N" resolve through references/lance-reference.md.

• references/performance.md - ALL performance guidance. Part A routes to the official text and adds the source-derived changes upstream has not documented; Part B is field-verified remote-storage practice. Load for any performance, tuning, maintenance-cost, or "why is this slow" question.

• references/docs/ - a verbatim mirror of the official docs ( docs/src at the tracked tag): every guide, quickstart, and format spec, unedited. Load when you need the full official text. Directory map below.

references/maintenance.md covers refreshing this skill against a new upstream tag.

Lance vs LanceDB#

These are two different things and conflating them produces wrong answers.

• Lance - the format and engine. The lance-format/lance repo; the lance / lance-* Rust crates; pylance. It gives you datasets, the file/table format, indexes, commits, scans. Consumed directly by DuckDB, Polars, Ray, Spark, PyTorch, DataFusion, or your own Rust/Python code. This skill is about Lance.
• LanceDB - a separate database product ( lancedb/lancedb) built on top of Lance. It adds a query-builder API, an embedding registry, rerankers-as-API, multi-language SDK parity, and managed Cloud / Enterprise tiers. Not covered here.

The wider ecosystem (separate repos, own version lines, none covered here): Flink streaming writes ( lance-flink), PostgreSQL reads via pglance, a Cypher graph engine ( lance-graph), a dataset browser ( lance-data-viewer), agentic context management ( lance-context), and namespace catalogs for Hive, Polaris, Gravitino, Unity Catalog, and AWS Glue. The canonical docs site is lance.org. Generated per-language SDK docs live at lance-format.github.io/lance-python-doc for Python and javadoc.io for Java - the matching lance-format.github.io/lance-java-doc path 404s.

Linking the lance crate in Cargo.toml means you are using Lance directly - use this skill. For LanceDB internals, the storage layer underneath is still Lance, so this skill remains the authority for the format itself.

The wrapper can hide format features. LanceDB's create_table cannot enable stable row IDs; only pylance's write_dataset(enable_stable_row_ids=True) can. If a format-level capability matters to your design, check whether the wrapper exposes it before assuming the underlying format settles the question - and reach for pylance directly when it does not.

The crate workspace#

27 crate directories under rust/ at v14.0.0-beta.14 - #9534 vendored the jsonb codec as lance-jsonb (26 at v13.0.0). lance is the public entry point - Dataset, scanner, indexes, commits; everything else ( lance-table, lance-file, lance-encoding, lance-index, lance-io, lance-core, lance-datafusion, lance-linalg, lance-namespace*, ...) is a layer beneath it. Edition 2024, arrow 58, datafusion 54, pyo3 0.28; Python bindings need 3.10+. MSRV is 1.94.0 in the v14 line (#8997), up from 1.91.0 through v13.0.0. Full table with roles, versions, and every workspace dep in references/format-file.md section 2.

If you depend on anything below lance, expect moves. v11 deleted lance-encoding::version with no re-export ( LanceFileVersion and ConcreteFileVersion live in lance-file::version), removed lance_io::encodings, and gave each format its own versions/v2_{0,1,2,3} module (#8020-#8026). The transaction code moved to a rust/lance-table/src/transaction/ module tree (#8053/#8054/#8056); a lance::dataset::transaction shim still re-exports Operation, Transaction, TransactionBuilder, RewriteGroup and UpdateMap, so the common surface is unbroken - but a symbol the shim omits, or a citation of the old path, needs retargeting. Section 2.1.

File format versions#

The file format carries a single major.minor version. data_storage_version is set per dataset at creation - but since v12.0.0 it is no longer fixed once the dataset exists. It is the default for writes that omit a target: "Create and overwrite establish this default; append, update, merge-insert, and compaction do not change it." An existing V2 dataset can take "2.0", "2.1", "2.2" or "2.3" per operation without rewriting its other files (#8582-#8585), so one dataset can hold data files at several exact V2 versions. V1 and V2 still cannot be mixed. Section 3.

VersionStatusNotes0.1 ( legacy)read-onlyOriginal format; no longer writable2.0stableRemoved row groups; null support for lists/FSL/primitives2.1previous defaultAdaptive structural encodings; better integer/string compression; nulls in struct fields; better nested random access. Was the default from Lance 5.0.0 until v12.0.0-beta.15``2.2current default ( stable)Map type, Blob v2, VariablePackedStruct, larger mini-blocks. Required for Map and Blob v22.3unstable ( next)Sparse structural pages (auto-selected by the 2.3 writer), the generic block v5 codecs (#8325-#8333, all landed by v14.0.0-beta.9), and compressed mini-block offsets (#8331). Stable readers refuse it

stable resolves to 2.2 (#8657), and 2.2 is the enum #[default], so a dataset created without an explicit data_storage_version is written as 2.2 - Python's write_dataset inherits it. The docs still do not say so - format/file/versioning.md only calls stable an "alias for the default version" - so the code is the authority. next resolves to 2.3. Pin an explicit number for deterministic behavior across builds: anything relying on WriteParams::default() (tests, benches) silently changed format at the 2.2 switch.

On v12, a writer defaulting to 2.2 rejected legacy blob metadata outright. Since v13.0.0 (#9459) legacy blob metadata written to 2.2+ is converted to Blob v2, and a 2.0/2.1 table can append Blob v2 without a rewrite. Blob v2 tiers by size: 64 KiB inline, 4 MiB dedicated, packed .blob sidecars between (the pre-v14 docs said 16 KiB / 2 MiB; the code never did).

The release selectors ( LanceFileVersion) are a type distinct from the persisted identity ( ConcreteFileVersion). Details in references/format-file.md sections 3.1 and 3.6.

Version landscape#

The major is bumped by a bot, not a human: ci/publish_beta.sh re-roots at MAJOR+1 whenever any PR since the release root carries the GitHub breaking-change label - the marker is the label, not a conventional-commit !. A major bump therefore means "some labeled breaking change landed", not a redesign. It has fired on five consecutive lines: none of v9.1.0, v10.1.0, v11.1.0, v12.1.0 or v13.1.0 was released. 13.1 got further than the rest - v13.1.0-beta.1 and beta.2 were tagged (fury 13.1.0b1/ b2) before #9518 re-rooted it to 14

• so a 13.1 beta pin is a dead end, not an early 13.1.

Four recent lines did ship a final: v10.0.0 (2026-08-08), v11.0.0 (2026-08-30), v12.0.0 (2026-09-17) and v13.0.0 (2026-10-07). Each sits on a stabilization branch that is not an ancestor of main - normal for a Lance final. v13.0.0 is the 14 release root plus two release commits, so every v13-final change is also in every v14 beta.

MajorIts breaking themev14 (current, v14.0.0-beta.14)ScalarIndexPlugin::load_index takes index_version (#9518, re-rooted the major); jsonb vendored as lance-jsonb (#9534); compressed 2.3 mini-block offsets (#8331). Unlabeled: inline transaction moved to manifest field 23 plus Operation::Unknown (#9482). Delta belowv13 ( v13.0.0, 2026-10-07)WriteParams.file_writer_options (#9192); lazy page-metadata init with a per-page metadata cache key (#7465); json_extract/ json_get stop routing to JSON indices (#9101); BM25F (#7905); JSON accessor API (#9479); merge_columns field ids (#9547); duplicate-pair output (#9551); covering-fields spec split (#9159). Net-new: tagged FRI supported, fragment metadata tree spec, MinHash LSH. Delta belowv12 ( v12.0.0, 2026-09-17)WrappingObjectStore::wrap_paginated (no default); MemWAL ShardManifestStore narrowed; lance-namespace response objects; predecessor-conditioned external stores; namespace merge-insert key lists; caller-provided Writer / open_part removed (#9072). Unlabeled but bigger: stable -> 2.2 and the IVF_RQ 5-bit default. Net-new: mixed data-file versionsv11 ( v11.0.0, 2026-08-30)Fragment ids became a dataset-lifetime high-water mark; lance-file / lance-encoding reorganization; transaction code moved to lance-table. Net-new: covering indexes, merge_insert write_mode, row-address prefilterv10Blob APIs preserve null selections; cache keys became opaque BLAKE3 digests; async create_remapper; MemWAL renamed generation -> SSTablev9.1 (never released; renamed into v10)FTS/inverted creation took a block_size param. Net-new: Data Overlay Files (unstable, env-gated), sparse structural pages, lance-index-corev9Python 3.9 dropped; alter_columns fails fast when casting an indexed column; FM-Index proto rename; FTS defaults to on-disk format v2v8All index builds unified onto one segment-based lifecycle. Net-new: lance-derive, FM-Index, multi-bit IVF_RQ, approx_mode, TOS + GooseFSv7MemWAL, branches, the geo/RTree index, the lance-select crate, ICU FTS

v13.0.0 is the stable pin and what GitHub Releases marks Latest. crates.io carries finals only (newest lance 13.0.0); PyPI pylance is likewise at 13.0.0. A beta pin means a git dependency - beta wheels publish to fury.io ( https://pypi.fury.io/lance-format), which carries pylance-14.0.0b1 through b14.

Full per-tag deltas with every PR citation: references/changelog-v7-v14.md.

Manifest feature flags#

A reader or writer refuses a manifest that sets a bit it does not support. The supported set is FLAG_UNKNOWN - 1 minus whatever the build masks out, so the docs table is not the authority - rust/lance-table/src/feature_flags.rs is. Bits 0-6 are long-standing (deletion files, stable row IDs, the deprecated v2 marker, table config, base paths, disabled transaction file, and the env-gated data overlay files). Everything from bit 7 up arrived in the last four majors, and FLAG_UNKNOWN moved 1 << 9 (v12.0.0) -> 1 << 14 (v13.0.0) -> 1 << 16 (v14 line):

BitFlagStatus at v14.0.0-beta.147FLAG_COVERED_INDEX_METADATASupported since v11 final (#8535)8FLAG_MIXED_DATA_FILE_VERSIONSSupported since v12 final; sticky; a half-set manifest is a hard error9FLAG_FRAG_REUSE_WITH_STABLE_ROW_IDSDebug builds only - release builds refuse it. Docs say reader/writer Yes10FLAG_FRAGMENT_REUSE_INDEX (tagged FRI)Supported since v13.0.0 (#9420); sticky11FLAG_UNSTABLE_SPILLED_ROW_LINEAGEUnstable: debug builds or LANCE_ENABLE_UNSTABLE_SPILLED_ROW_LINEAGE12FLAG_FRAGMENT_TREEReserved; rejected until implemented, though the spec exists (#9060)13FLAG_INDEPENDENT_COVERING_FIELDSReserved and masked out. Docs present it as normative14FLAG_MANAGED_BLOBSv14 only; sticky; opt-in via lance.blob.managed=true since beta.1415FLAG_NON_REUSABLE_FIELD_IDSv14 only (#8658); writer-only, sticky

"Sticky" means the bit survives every later commit ( STICKY_PAIRED_FLAGS / STICKY_WRITER_FLAGS), so setting it once permanently raises the reader floor of the table.

Two pin windows open a table they should refuse - or brick it for older readers.

• Do not pin v11.0.0-beta.4 through beta.17. Those builds read bit 7 as a retired MemWAL flag they support, so they open a covering-index dataset instead of refusing it - wrong neighbours, no error.
• Do not pin v14.0.0-beta.4 through beta.13. Any ordinary Blob v2 write there sets FLAG_MANAGED_BLOBS, which is sticky, so v13.0.0 can never open the table again. #9891 (beta.14) made it explicit opt-in - "Ordinary Blob V2 writes currently activate FLAG_MANAGED_BLOBS and require newer readers" - but the beta.14 docs still describe automatic activation. Trust the code.

Covering indexes (#8535) - IndexMetadata.covering_fields (proto field 11) names the columns an index carries values for, so a query projecting only those is answered without a base-table take. Under the contract every released build implements, the carried columns are the trailing suffix of fields in emission order, and a column the index is both keyed on and carries appears twice: " fields = [1, 1] with covering_fields = [1] is keyed on 1 and carries 1." #9159 specified an independent contract behind bit 13 ( fields holds only keys), but the build masks that bit, so it is spec only. Invalidation is wide either way: any index whose fields include the updated column. V3 IVF auxiliary files can physically carry columns ("a reader discovers carried columns by exclusion, not by position"), bound by a covering_field_ids metadata key, and coverage is per-segment. Section 11.

The v13 final and v14 delta#

161 commits from v13.0.0-beta.4 to the v13.0.0 final, then 137 more on the v14 line to beta.14. 8 breaking-change-labeled PRs: #7905, #9479, #9551, #9547, #9159 in v13 final; #9518, #9534, #8331 in v14. Invariants: 27 crates (v14), 17 transaction ops ( Operation::Unknown, v14), CommitConfig.num_retries still 20 (in v14 the retry clock starts after catch-up, #9631), arrow 58 / datafusion 54.

The label missed the format break again. #9482 (v14.0.0-beta.2) moved the inline transaction from manifest field 21 to field 23, added Operation::Unknown so unknown operations decode instead of failing, and turned From conversions into TryFrom - labeled format-change, never breaking-change, so the bot did not count it. Exhaustive match on Operation fails to compile.

• merge_columns field ids come from the manifest (#9547) - the Rust max_field_id argument is gone. The bug it fixed made merged columns silently read back as null.
• BM25F cross-field search (#7905) - combined_fields / CombinedFieldsQuery scores one query across several FTS-indexed columns as a single field. Section 11.
• JSON API moved (#9479, #9534) - one accessor ( JsonTextStream::try_new), jsonb_to_json removed, and jsonb vendored as lance-jsonb with its own spec ( format/table/jsonb.md). The v13 rule still stands: only json_get_int / _float / _bool / _string reach a JSON index (#9101); json_extract / json_get silently full-scan.
• ScalarIndexPlugin::load_index takes index_version (#9518) - every third-party scalar index plugin fails to compile.
• Duplicate-pair enumeration (#9493) changed output order and cosine values in #9551.
• Default IVF probing moved, unlabeled. #9184 leaves probes unset so queries get adaptive bounds, #9467 recalibrated dot-product profiles, and #9798 (beta.12) added search_effort (0 to 1, resolved to 0.5; it overrides a fixed nprobes) and dropped the Python binding's DEFAULT_NPROBES = 1 minimum-probes floor. Latency and recall at default settings shift across these upgrades - re-benchmark rather than carry numbers forward.

Net-new format surface: tagged FRI is supported (bit 10, #9420), extending reuse to reclustering via StablePartition; the fragment metadata tree spec (#9060 - fragment_metadata.proto, Manifest field 22, a _bt/ directory; bit 12 still rejected); spilled row lineage replacing the retired DataFragment.external_* fields (#9253, hidden columns at field ids -3/-4/-5, unstable); non-reusable field ids (#8658, bit 15, max_allocated_field_id); and an experimental MinHash LSH index (type 12, #9076/#9114).

Behavior changes without a label: explicit cloud storage options now beat env vars (#9719); fast_search no longer adds _rowid (#9699); main is reserved as the default-branch alias (#7180); Restore conflicts with an UpdateConfig touching schema or field metadata (#9361); raw deep-Clone commits are rejected (#9555); the DataFusion memory pool is 150MB per effective partition - all cores when target_partitions is unset - with per-execution pools (#9183, #9408).

Fixes needing a rebuild, rewrite or repair, not just an upgrade: rewrite data hit by #9018, #9239 (NULL written as 0) or #9394; rebuild scalar indexes hit by #9421 (segment merge produced an empty index), #9352 (LABEL_LIST) or #9203; repair or restore after #9218, #9219, #9573, #9447, #9176, #9595, #9085; v14: #9760, #9524, #9620, #7706, and #9857 (MemWAL, beta.9-12 only). Address-domain indexes - ZoneMap, BloomFilter and FM-Index - still lose coverage of compacted fragments rather than claim them (v14 recalculates instead when a fragment reuse index is present, #9514), and one damaged under v10 or earlier needs recreating. Conditions for each in references/changelog-v7-v14.md.

Still not landed: the arrow 59 / DataFusion 55 / pyo3 0.29 upgrade. #8997 merged only as "prepare" ("Not in this PR: the dependency bumps and lockfiles"); the real bump is #9647, open and blocked on upstream releases. It also gates PyO3 RUSTSEC-2026-0176/0177, still outstanding.

Earlier majors in brief#

v13 betas - WriteParams.file_writer_options (#9192, struct literals fail to compile); lazy page-metadata init (#7465): StructuralFieldScheduler::initialize takes requested_ranges, and caching moved from per-column FieldDataCacheKey to per-page PageDataCacheKey, so every warm or persisted metadata cache cold-misses across the upgrade - the payoff is that "a cold point/range read's metadata IO is invariant to the column's total page count". The FRI got a versioned on-disk contract (#9136; stable row IDs and tagged FRI mutually exclusive outside debug builds). inline_optimization_enabled flipped true -> false (#9180). hf:// object store (#9236).

v12 - mixed data-file versions shipped (#8581-#8585); stable -> 2.2 (#8657); IVF_RQ defaults to 5 bits, ~4.4x the 1-bit index size (#8936; set num_bits=1 to opt out); WrappingObjectStore implementors must add wrap_paginated - "There is deliberately no default"

• and paged listing via read_dir_page means walking until the token is None, never until a page comes back short; namespace merge-insert keys have arity-dependent NULL semantics (a composite key's NULL "matches nothing"); lance-namespace pins now differ by language (Rust 0.13.0, Java 0.12.0 since #8979, Python >=0.13.0,<0.14 in v14 - re-check rather than carry forward).

v11 - fragment ids are a dataset-lifetime high-water mark (#8206): overwrite no longer restarts at 0, and datasets with duplicate ids read but no longer commit. LanceFileVersion lost Ord (#8027/#8028). Operation::Project / Merge gained preserves_nullability (#8347). The external-manifest protocol made object storage authoritative (#8499) - a retained ETag makes readers reject a good manifest with Manifest e_tag mismatch. #8101 silently duplicated rows on nullable primary keys under repeat merge_insert; existing duplicates must be removed by hand. Rebuild HNSW indexes hit by #8834 (v11) or #9053 (v13 betas).

Performance questions#

For anything performance-shaped - slow scans or searches, remote/object-storage cost, index maintenance cost, memory sizing, version bloat, benchmarking - load references/performance.md first. Part A routes to the official guidance plus the undocumented source-derived changes; Part B is field-verified practice against S3-compatible storage. The governing rule stays minimize remote calls - fewer commits, fewer scans, fewer round trips - because that is where the order-of-magnitude wins are. The official "Tuning remote scans" section gives a starting point for cross-region or public-internet access, where the cloud default of 64 concurrent requests is too aggressive: LANCE_IO_THREADS=8, fragment_readahead=1, batch_readahead=2, io_buffer_size=64MB. It is a legitimate second move once call volume is already minimized.

AMX-FP16 (#8540) is the one performance change that alters results, not just speed: where it engages, IVF partition assignment becomes exact instead of approximate, so recall improves and assignments differ from an older build. It is shape-gated ( float16 + dot, dimension >= 32, num_centroids >= 32). LANCE_DISABLE_AMX=1 disables it and reverts assignment to the approximate path too.

Two cache facts to know before tuning anything remote: Lance has no resident data cache (a Session holds only index and metadata caches, never decoded values, so repeated point reads re-pay object-store IO), and one Arc<Session> shared via DatasetBuilder::with_session lets datasets share it. Cold first search is dominated by paging indexes in - prewarm_index is the remedy, and for FTS it loads the whole inverted index into RAM. #7465 changed the metadata cache key shape, so the first run after a v13 upgrade re-pays that paging even against a warm cache.

Layout debt is version-dependent. Binary-copy compaction of small appends preserves tiny pages; a reader that pays per page (v12) turns that into tens of thousands of GETs per lookup. Re-encode append-heavy tables, and benchmark on a fresh copy of the real store - details and measurements in references/performance.md Part B.

Time travel is not an archive mechanism. Versions look like free history, but the default cleanup reclaims anything older than 7 days and cleanup is part of routine optimize - so a design that treats old versions as the durable record loses it on the first maintenance pass. Keep an explicit archive if you need one. The reverse also holds: delete alone does not erase data - old versions, unrewritten fragments, and indexes (deletes do not conflict with index creation) all keep it until rewritten, cleaned up, and rebuilt.

Official docs mirror#

references/docs/ mirrors docs/src of lance-format/lance at the tracked tag, verbatim - 49 markdown files plus 5 diagrams, all directly readable.

DirectoryFilesCoversguide/14CRUD, performance, object store, distributed write + indexing, JSON, tokenizers, data types, data evolution, blob, arrays, tags/branches, migration, observabilityquickstart/4First dataset, vector search, full-text search, versioningformat/1Spec-stack overviewformat/file/3Container spec, structural encodings + compression, format versionsformat/table/11 + 1 svgLayout, schema, transactions ( conflict-resolution matrix), versioning, row-id lineage, branch/tag, MemWAL, data overlay files, fragment metadata tree, JSONBformat/index/1 + 4 svgIndex lifecycle, fragment coverage, compaction interplay, covering contractsformat/index/scalar/11fts, fmindex, ngram, btree, bitmap, bloom_filter, label_list ( array_has_any/all), zonemap, rtree, json, minhash_lshformat/index/vector/1IVF / PQ / SQ / RQ / HNSW concepts and storage layoutformat/index/system/2Fragment reuse index, MemWAL system indexintegrations/1DataFusion SQL over Lance, incl. JSON functions

Not mirrored: docs/src/images/ and the format/*.png assets, so image links in the mirrored pages do not resolve - the prose is self-contained, and the five .svg diagrams are mirrored. Also out by design: community/, examples/, integrations/{index,pytorch,tensorflow}.md; and the landing stubs and contributor files ( format/AGENTS.md).

A whole tier of docs is not in this repo at all, so it cannot be mirrored and cannot be enumerated from a clone. docs/make-full-website.sh assembles format/catalog, format/namespace, and the integrations/{duckdb,huggingface,spark,ray,trino,context} sections at build time from six sibling repos with their own version lines - the checked-in integrations/index.md links spark/, duckdb and trino as if they were local, but those paths do not exist in the tree. Lance Context and the HuggingFace integration docs are whole nav sections that exist only on the built site. For any of those, read lance.org rather than this mirror. Protobuf message bodies are likewise expanded at build time from protos/ by mkdocs_protobuf, so the mirrored spec pages show %%% proto.message.X %%% placeholders where the site shows a rendered schema.