Skip to content

Internals

Edit page

This page is for contributors: where the code is, the conventions it follows everywhere, and the invariants that look like implementation details but hold the system up. For the runtime picture, start with the Architecture Overview; for the libraries, see Dependencies. The repository’s AGENTS.md is the full working guide.

src/ is Xapiand’s own code. Everything else, the search engine included, is a standalone library the build fetches (see Dependencies); a bug in a library is fixed in that library’s repository. The entry point is src/main.cc, and XapiandManager (src/manager.*) owns the node: the control loop, the HTTP service and the stop.

You’re touching…Start in
The HTTP API: routes and request handlerssrc/server/search_views.cc (the route table and the views), src/server/search_service.*, src/server/http_utils.h, src/server/mime_types.*
Index paths, patterns and settingssrc/endpoint.*, src/index_path.h, src/index_pattern.*, src/index_resolver_lru.*
The control loop, autocommit and the stopsrc/manager.*, src/control_plane.h, src/shutdown_wait.h
The index registry: pins, turns, choressrc/database/pool.*
Writes, commits and blob storage of a shardsrc/database/shard.*, src/database/data.*
The write-ahead logsrc/database/wal.*
Documents in and out, search, restoresrc/database/handler.*
Schemas and field typessrc/database/schema.*, src/database/schemas_lru.*, src/reserved/
The engine seamsrc/engine/ (the interface), src/engine/strata/ (strata and prospect behind it)
Query languagessrc/query_dsl.* (the DSL; the q= string through boolean-parser), src/engine/text.* (text fields through queryparser), src/engine/query.* (the query tree)
Aggregationssrc/aggregations/
Sorting, ranges and geospatial termssrc/multivalue/
Geospatial shapessrc/geospatial/ (the shapes; the mesh is the htm library)
Value encodingsrc/serialize*.{cc,h}, src/cast.*, src/length.*
Lua scriptssrc/script.*, src/lua/
Metricssrc/metrics.*, XapiandManager::server_metrics_impl
Command-line optionssrc/opts.*
Loggingsrc/log.h, src/logger_fwd.h (the engine is the logger library)
HTTPSsrc/tls.*

Tests: C++ unit and fuzz targets in tests/, end-to-end scripts in harness/ (harness/run_all.sh runs the suites), and the docs E2E, which runs every request example on this site against a node.

Xapiand reaches the engine through src/engine/: a shard interface (engine::Shard: writes, commits, transactions, reads, refresh, close), the document in the engine’s own form (engine::Document), the query tree (engine::Query) and search (engine::search), and the two ways a host runs work (engine::Host, engine::Chores). src/engine/strata/ implements it over a prospect index. A document is built in strata’s form from the start: the schema calls the document’s add_term, add_value and set_data, the shard applies that object, and the log serializes the same one, so nothing is converted on the way.

The index registry and its entries are the one place outside src/engine/ that names strata and prospect: DatabasePool is a strata catalog of prospect indexes (src/database/pool.*) and ShardEndpoint opens one (src/database/shard.cc), since opening, pinning and closing an index is the catalog’s job.

A build with -DENGINE_CENSUS=ON (off by default) counts the engine facilities each request uses at the seam and writes them to the file XAPIAND_ENGINE_CENSUS names (src/engine/census.*).

Never block a loop. A request, and the node’s own work, is a task on an event loop. A task that has to wait awaits a timer, an event, an index’s turn or a chore (device work on the WorkPool); it never calls sync_wait, waits on a condition variable, sleeps or makes a file call. engine::check_may_block aborts a debug build that does, and harness/async_check.sh --strict (a CI step) fails on any blocking primitive that harness/async_allowlist.tsv does not classify.

Chores have a class. query is work a request waits for (a refresh, an open, a blob read), flush is throughput work (a commit, a close, a blob write), and merge is background work (src/engine/chores.h). Each index runs its chores in a work group of its own on the one WorkPool (IndexChores, src/database/pool.h).

MsgPack is the universal value. JSON, YAML, MessagePack and internal objects are all MsgPack, the copy-on-write value of the msgpack library: copies share one reference-counted buffer and copy only on mutation. Reserved keys start with _ (src/reserved/).

Perfect-hash dispatch. Reserved words (the query DSL, aggregations, field types, casts) dispatch through compile-time perfect-hash tables (phf::make_phf, the constexpr-phf library) and a switch, not if-ladders. Adding a keyword extends both.

Logging macros L_*. Most category macros (L_CALL, L_DATABASE, …) compile to nothing; switch one on by editing its #define near the top of src/log.h (or of the file you are tracing) and rebuild. Arguments are evaluated lazily, so an expensive one costs nothing when the message is filtered out.

Order-preserving values. A value stored for sorting and ranges is encoded so that comparing its bytes compares the values. Change an encoding and you change what sorts and ranges return, so any new encoding keeps that order.

These look safe to “clean up” and are not:

  • A task holds one index’s turn at most, and takes no turn, open or drain while it holds a view. Either would let two tasks wait for each other. Debug builds count both per task and abort on a breach (src/engine/host.h).
  • The turn covers the apply, the revision and the log line. The log line is queued under the same turn as the apply, so the log’s order is the apply’s order (Turn, src/database/pool.h). Don’t queue a log line outside it.
  • Replay is idempotent. Each log line carries the version its document was written at, and replay applies it only as the next version of that document; a line further ahead means the log lost a line and fails the open (replay_applies, src/engine/strata/shard.cc). Don’t add a log operation that cannot be applied twice.
  • A read never takes the turn to read. It takes the turn only to refresh an index that has unrefreshed writes, then reads a view. A read that held the turn would make every write to its index wait for it.
  • Trixel ids encode the mesh path, 2 bits per level, big-endian. That is why a region on the sphere maps to contiguous integer ranges. Changing how an id is built or serialized breaks geospatial indexing silently (the htm library, Serialize::trixel_id in src/serialize.cc).
  • An index path is well-formed UTF-8 with no control bytes, NUL or backslash (src/index_path.h). The system calls that act on a path read it as a C string, and the storage layer cannot round-trip the rest.