Upgrading
Edit pageThis guide takes a node from 1.0.0-alpha.9, 1.0.0-alpha.10 or 0.40.0 to 1.0.0. The short version: 1.0.0 searches with the strata engine, which does not read the indexes that those releases wrote with the Xapian engine, so your documents move over with a dump and a restore. Xapiand runs as a single node; the cluster returns in a later release.
Data compatibility
Section titled “Data compatibility”1.0.0 reads only indexes that the strata engine wrote. An index written by 0.40.0, 1.0.0-alpha.9 or
1.0.0-alpha.10 holds Xapian’s on-disk format, which 1.0.0 cannot open. A node started on such a data
directory exits with status 65 and names the first Xapian file it found (iamglass, flintlock,
postlist.* or *.glass, in the registry or in an index’s shard directory), and creates nothing. There is no
option to skip the check.
Carry the documents over instead: dump each index with the final Xapian-only release (the xapiand
branch) and restore it into 1.0.0. A dump holds every document’s data with its _id and _version; the
restore indexes the documents again, so the new node builds every term, value and number encoding itself
(see What changes). Index settings and explicit schemas are set
again on the new node, before the restore (see
Restore using different schema).
Upgrading from 1.0.0-alpha.9 and 1.0.0-alpha.10
Section titled “Upgrading from 1.0.0-alpha.9 and 1.0.0-alpha.10”Both alphas searched with the Xapian engine. The steps are the same as for 0.40.0, and the old data directory stays untouched until you are done.
-
Stop nothing yet: run the final Xapian-only release (the
xapiandbranch) against the old data directory, and dump every index. From the command line, with the same parameters you start the server with (at least-D):Terminal window xapiand -D /var/lib/xapiand --dump="twitter/tweet" --out="twitter.msgpack"Over HTTP,
DUMP /twitter/tweet/streams the same records (see Backups). Restore large indexes from the command line: aRESTOREover HTTP answers413for a body over 16 MiB, and for a chunked one. -
Stop the old node and let it shut down cleanly.
-
Start the 1.0.0 binary on a new, empty data directory (
-D). Do not point it at the old one. -
Create each index with its settings and schema, if it had explicit ones.
-
Restore every dump into the new directory. From the command line, with the new server’s parameters:
Terminal window xapiand -D /var/lib/xapiand-1.0 --restore="twitter/tweet" --in="twitter.msgpack"Over HTTP,
RESTORE /twitter/tweet/takes the body of the dump. -
Start the node and check each index: its document count against the old node’s, and the searches you depend on (see the list below).
The command line --dump and --restore exit once the data is written.
What changes in a restored index
Section titled “What changes in a restored index”The documents are the same; the engine around them is not. These are the changes a restored index shows, grouped by what you meet first.
Stored documents and numbers
Section titled “Stored documents and numbers”- Numbers compare exactly across types. An integer, an unsigned integer and a floating field encode to
the same bytes for the same number, so sorts, ranges and
_min/_maxagree with the true values, including integers above 2^53 (snowflake ids, nanosecond times, hashes), which the old encoding merged. A value that has no place in that order is refused instead of stored wrong: a NaN answers400, auint64of 2^63 or more given to an integer field answers400(it wrapped negative), and-0.0indexes as0. - Dates read differently at the edges. A new field whose first value is
2020-01-01T10:00:00.Z(a fraction dot with no digits) or has a fraction of more than 62 digits is typedtext;2020-12-15||+12Mis2021-12-15. - A document that repeats a key keeps one entry:
{"a":1,"a":2,"b":3,"c":4}is stored as{"a":2,"b":3,"c":4}. - Geospatial ids of points near a cell edge can differ. On builds that fuse multiply-adds (Apple silicon,
-mfma), 0.7% of random points sit in the neighboring trixel; the restore recomputes them. - Ids. See Identifiers move to the v6 format.
Schema and queries
Section titled “Schema and queries”_cjk_wordsis gone. The property is unknown, and queries no longer split CJK words; use_cjk_ngram._termsover an analysedtextfield answers400, and_termscounts a field’s stored values: keyword buckets keep their case (Redandredare two buckets)._termsand_valueskeep about_limit * 1.5 + 10buckets while they collect (_shard_size), so counts can fall short when a field has more distinct values; over geo fields they answer400._elite_setanswers400,nearestandfuzzy(withn_rset,n_eset,n_term,field,type) are ignored and the search runs as an ordinary one, andINFOon a document no longer lists its terms.- Text query strings follow precedence and refuse malformed input.
a AND b cis(a AND b) OR c;a NOT,a ()and(aanswer400. A lone-ain a field query answers400. - Some query shapes answer
400: a_synonymover a wildcard, a fuzzy word or a group; an array given to_scale_weight,_wildcardor_edit_distance; a_phraseor_nearover anything but words. - Sorting by a string metric is corrected.
Jaro,Jaro-Winkler,Sorensen-Diceand weightedLevenshteinfollow their standard definitions; sorting bysoundexagainstDICKSONXgivesDICKSONX,DIXON,DWAYNE. collapse_maxabove 1 returns the best documents of each group, where Xapian returned lower-ranked ones.- A search over several indexes weighs each index on its own statistics. Scores and the order of hits change against alpha.10; adding an index to a search no longer reorders the others’ hits.
- Index patterns are globs over path segments:
*,?and[...]never match a/, so writetest*/*ortest*/**for whattest*used to find belowtest1/._all, wildcards and exclusions work in the index list of a search.
Limits
Section titled “Limits”A request over a limit answers 400 (or the status named) with the limit in the message.
- Queries: 100 levels of operators, 4,096 terms and operators (an
_inor OR over more ids is refused), 256 phrases per AND, 45 levels of query DSL, 200 levels of body nesting. YAML anchors and aliases (&name,*name) answer400. - Aggregations: 65,535 live buckets per search and 16 levels of nesting.
- Bodies: 16 MiB buffered; a larger non-
storedbody, or a chunkedRESTORE, answers413. Astoredblob over 16 MiB streams and needs aContent-Length; aPOSTof one without an id answers400, and past 4 GiB413. - Lua scripts: per call 10^8 instructions, 5x10^8 units of work, 64 MiB of live data and 16x10^6 units in
one library call (
400); results over 16 MiB or 262,144 value units answer413. Calls of a script no longer share globals, an error a script raises answers400where it used to store the document untransformed, andload,dofile,loadfile,collectgarbage,string.pack,string.unpack,string.packsizeandstring.dumpare gone. - Concurrency: a search of an index already read by its cap of concurrent searches answers
503; a loop with 256 requests waiting answers503withRetry-After: 1.--max-search-indexes(100,000) bounds a search. - Paths:
-D "",PUT /a/../doc,/./doc, a path outside the data directory, and index names with NUL, control bytes, a backslash or malformed UTF-8 answer400or a usage error.a/b/..is the patha.
Running the node
Section titled “Running the node”- One node.
--solo,--cluster,--discovery-*,--xapian-port,--replica-port,--primary-node,--matchers,--flush-threshold,--replicas,--replicators,--remote-clients,--remote-servers,--replication-clients,--replication-servers,--discoverersand--database-stall-timeare removed, and so are--committers,--fsynchers,--http-servers,--http-clients,--bulk-preparers,--bulk-indexersand--max-database-readers. They are unknown arguments, so an old systemd unit or Docker command line exits with a usage error until you drop them.GET /has nocluster_name,nodesornum_committers, andversionshas noXapianorICU. :metricsdrops thexapiand_http_clients_*series andxapiand_committers_capacity/_pool_size; the#xapian_querydebug field is#engine_query. Update dashboards.- Shutdown is bounded: the node exits at most 35 s after the clients are done, with status 75 if it had to abandon work (see Stopping the node). A taken port ends the process with status 70.
- HTTP is stricter: a bare LF, obs-fold, a
Transfer-Encodingother than exactlychunkedand a badContent-Lengthclose the connection without a reply. - A write whose commit wait times out succeeds; the commit is left to the timer.
Red flags
Section titled “Red flags”Downgrading needs the old data directory. 0.40.0 cannot read an index that 1.0.0 wrote. Keep the 0.40.0 data directory, or a backup of it, until you no longer need to step back (see Rolling back).
Identifiers move to the v6 format. 1.0.0 mints compact UUID identifiers in a new format, cuuid v6: a UUIDv6-based, splitmix64-reconstructed id that decodes about 120x faster than the old v1 (no per-read Mersenne Twister) at the same wire size. Two consequences:
- New ids differ. Ids the server generates for you in 1.0.0 are v6, so both their
encoded (
~…) and canonical (xxxxxxxx-…) forms differ from what 0.40.0’s v1 generator would have produced. Any id you supply yourself is stored as given. - Old ids still resolve, but their canonical form may render differently. Ids
already in your data are matched by their stored bytes, so lookups, links, and the
~…encoded form are unchanged. However, if you read an old id back through a field-decoding representation (vanilla,urn,guid), 1.0.0 decodes it as v6 and produces a different UUID string than 0.40.0 did. Deployments using the default encoded (~…) representation see no difference; only the canonical reprs do, and only for pre-existing ids.
This is a deliberate breaking change for the major-version bump. If you need the exact
old behavior, start Xapiand with --legacy-ids: it mints v1 and takes the v6 codec
entirely out of the loop, so identifiers behave byte-for-byte as they did in 0.40.0.
It is also the safe rollback if the new format causes any surprise. To make old ids
render their original canonical form under the default mode, reindex the affected
documents so their ids are re-minted as v6.
Migrating a single node from 0.40.0
Section titled “Migrating a single node from 0.40.0”The steps are those of Upgrading from 1.0.0-alpha.9 and 1.0.0-alpha.10,
with the 0.40.0 data directory as the source. Run the final Xapian-only release (the xapiand branch) against it
for the dump.
Rolling back
Section titled “Rolling back”Stop 1.0.0 and start the 0.40.0 binary against the old data directory. Writes made on 1.0.0 since the upgrade live only in the new directory.