Skip to content

API Conventions

Edit page

Xapiand uses a RESTful API exposed using JSON over HTTP.

When we talk about our API, we use terms like “REST” and “RESTful.” “REST” stands for Representational State Transfer.

The conventions listed in here can be applied throughout the REST API, unless otherwise specified.

You may see these standard HTTP methods referred to as CRUD, or Create, Read, Update, Delete. Although CRUD has roots in database operations, you can also map those operations to the standard HTTP methods. For example, use a POST request to create a new resource, a GET request to read or retrieve a resource, a PATCH or UPDATE request to edit a resource, and a DELETE request to delete a resource.

We do our best to use standard HTTP methods with accurate and well-known status codes in the Xapiand API, but here are some additions and deviations.

Additionally to the standard HTTP methods, we also use other custom methods such as UPDATE, for certain operations.


  • Safe Methods
    Requests that use safe HTTP methods won’t alter a resource at all. Examples are GET, OPTIONS and HEAD.

  • Idempotent Methods
    An idempotent HTTP method is a HTTP method that can be called many times without different outcomes. It would not matter if the method is called only once, twice or a hundred times over, the result should be the same. This only applies to the result, not the resource itself. Examples are DELETE, PUT.

  • Not Safe/Idempotent Methods
    This are requests that will alter the resource and potentially end up with different results every time. Examples are POST.

The Standard Methods, the ones we all are familiar with, have simpler and well-defined semantics but there is functionality that cannot be easily expressed via standard methods. Custom methods refer to such API methods.

Simply pass the required method as a non-standard HTTP method in the request.

Example:

INFO /some/resource/name

If your firewall rules don’t support non-standard HTTP methods like PATCH, UPDATE or DELETE, for example, you have two options:

  • Use HTTP Method Override
  • Use HTTP Method Mapping

You can use the X-HTTP-Method-Override (or HTTP-Method-Override) header. Pass the method you want to use in the X-HTTP-Method-Override header and use the POST method.

Example:

POST /some/resource/name
X-HTTP-Method-Override: INFO

Custom methods can use the following generic HTTP mapping: http://service.name:8880/some/resource/name:customMethod

To use method mappings pass the mapping in the URL and use HTTP POST verb since it has the most flexible semantics, except for methods serving as an alternative get or list which may use GET. Any other method answers 400 (“HTTP Mappings must use GET or POST method”).

Example:

GET /some/resource/name:info

That is the question we hear often. Onward to the answers! Historically, it’s common for URLs with a trailing slash to indicate a directory, and those without a trailing slash to denote a file:

  • http://example.com/foo (without trailing slash, conventionally a file)
  • http://example.com/foo/ (with trailing slash, conventionally a directory)

Source: Google WebMaster Central Blog - To slash or not to slash

To us, trailing slashes are important to distinguish between a path to an Index (a directory) and a path to a Document (a file).

The following will delete a single document from index /some/resource/name, the document with ID name:

DELETE /some/resource/name

Whilst the next one addresses the index /some/resource/path/. Xapiand does not delete indexes over HTTP: DELETE on an index path answers 501 Not Implemented (see Delete Index):

DELETE /some/resource/path/

A request is a verb and a path. The path names a document (it ends with an ID), an index (it ends with a /), a command (/index/:command) or the node (/). A verb the path does not accept answers 405 Method Not Allowed.

VerbOnWhat it does
GETnode, index, documentThe node, the index’s settings and schema, a document; GET /:metrics answers the metrics
HEADindex, document200 if it exists, 404 if not
SEARCH, COUNTone or many indexesSearch, or count the hits
INFOone index, documentStatistics of an index or document
PUTindex, documentCreate an index or index a document
POSTindexIndex a document with a generated ID (201 and Location)
UPDATE, PATCH, UPSERTindex, documentUpdate a document; UPSERT creates it when missing
DELETEdocumentDelete a document. On an index it answers 501, on / it answers 405
COMMITone indexCommit the index now (200)
CHECKone indexCheck the index’s files; answers {"errors": {<index>: <count>}}
DUMP, RESTOREone index, documentDump or restore its documents
WALone single-shard indexThe write-ahead log
FLUSH, QUIT/Administrative commands, only with --admin-commands
OPTIONSany200 with an Allow header
OPEN, CLOSEany429 Too Many Requests
501 Not Implemented
LOCK, UNLOCK, COPY, MOVE, PURGE, LINK, UNLINK, TRACE, CONNECTany405

Sources: the verb list and the route table, src/server/search_views.cc:187-218 and 548-602; the inline answers, :652-672.

OPTIONS /twitter/user/

A verb that is not served at that path answers 405:

LOCK /twitter/user/
COMMIT /twitter/user/John

COMMIT commits an index now instead of waiting for the autocommit (see Visibility and Durability):

COMMIT /twitter/user/

CHECK checks an index and answers how many errors it found in each shard:

CHECK /twitter/user/

FLUSH and QUIT are answered only at the root URL, and only when the node runs with --admin-commands; otherwise (and on any other path) they answer 405.

  • FLUSH commits and closes every open index and closes the client connections, and answers 200 when they are closed. FLUSH /?databases flushes only the indexes, FLUSH /?clients only the connections.
  • QUIT asks the node to shut down, answers 200 and closes the connection (see Stopping the node).

The API has no authentication: enable the option only on a node that untrusted clients cannot reach (see Options). Sources: src/server/search_views.cc:1497-1535 (flush_view, quit_view).

QUIT /
FLUSH /twitter/

The root of the API is the node itself. It names no index, so it has no documents: GET / answers with the node’s name, its server and library versions, its options and its state, and HEAD / answers 200 when the node is up.

GET /

A field selector reduces the answer, GET /.state answers just the node’s state, and GET /:info answers the same as GET /. GET /:metrics answers the node’s metrics in the Prometheus text format. A request that needs an index (PUT /, POST /, a search or a count) and names none answers 400; DELETE / answers 405.

GET /.server.name
PUT /

A root command that reads index metadata, such as /:stats or /:schema, names no index and answers 400 too.

GET /:stats

The Xapiand API can process JSON, MessagePack, or YAML objects. Choose the format with the request’s Content-Type header (application/json, application/x-msgpack, or application/x-yaml); JSON is the default when no Content-Type is given.

You can also send documents as YAML by setting the Content-Type to application/x-yaml. Xapiand parses the body into the same internal object it would from the equivalent JSON:

PUT /twitter/user/Jane?commit
Content-Type: application/x-yaml
name: Jane
age: 30
city: Turin
GET /twitter/user/Jane

JSON can have C-style /* */ block or single line // comments. Comments are allowed everywhere in the JSON document.

JSON can have trailing commas.


JSON or MessagePack fields in objects passed to Xapiand are expanded. For example, the following nested object:

{
"contact": {
"address": {
"country": {
"name": "Italy"
}
}
}
}

Is equivalent to:

{
"contact.address.country.name": "Italy"
}

SEARCH and COUNT run across multiple indexes, using simple test1,test2,test3 notation. Every other verb takes a single index (a list answers 400, “Method can only be used with single indexes”).

Searches and counts also take _all for all the indexes of the node, and globs over the segments of an index path (the parts between /), for example test*, *test, te*t, test-??, test-[0-9]* or logs/**/error-*, and - to exclude, for example test*,-test3:

  • A * stands for any run of characters inside one segment, none included. It never matches a /: test* matches test and test1, and not test/old (that is test*/*).
  • A ? stands for one character, in one segment: test-?? matches test-ab.
  • A [...] stands for one character of a class: [abc], a range [a-f], or every character but those with a leading ! or ^: [!0-9]. A [ with no closing ] is a plain [.
  • A segment that is ** stands for any number of segments, none included, and may be used more than once: **/some-?? matches some-ab at any depth, and logs/**/error-*/**/last matches logs/error-1/last, logs/a/error-1/b/c/last and so on. logs/** matches logs and everything below it.
  • _all is the same as **.
  • An item that starts with - removes the indexes it matches from the result, wherever it stands in the list.
  • A glob never matches an index whose name starts with a . (the node’s own indexes); name it to search it.
  • A glob that matches no index finds nothing: the search answers 200 with no hits.
  • A search over more than --max-search-indexes indexes (100,000 by default), counting the ones a glob expands to, answers 400; so does one whose patterns read more than that many directory entries to find them, in the directories of an index too. The search goes only into the directories a pattern can still match below: logs/2026/* reads logs and logs/2026, and no other directory. Once the node’s index names are built (its first pattern starts that), a pattern reads index names instead: each name under the pattern’s prefix that it reads counts against the same number, never more than the walk would read, so logs-2026* reads only the names that start with logs-2026, however many indexes the node holds. A search names at most 32 patterns (globs and exclusions together), and an item repeated counts once. A wide search reads its indexes in batches; see Searching many indexes.

To reach the indexes below a name, give the segments: test*/*, or test*/**.

Globs match the indexes that are on the node’s disk. They are the way to reach a great many indexes: a URL holds 64 KiB, which is a few thousand index names.

Indexes nest: twitter and twitter/user can both exist, the second inside the first. twitter* finds the first, twitter/* the second, and twitter/** or _all both. In a URL a ? starts the query string; write it %3F in an index pattern.

A wildcard or _all reads every index on the node, so a name that is not given no longer keeps an index out of a search: a proxy that allows a request by the first segment of its path has to treat ,, *, _all and a leading - as naming several indexes, and :count on a prefix tells whether names exist.

Four indexes, one holding two documents:

PUT /multi/alpha/one
{
"marker": "multiindex",
"name": "alpha one"
}
PUT /multi/beta/one
{
"marker": "multiindex",
"name": "beta one"
}
PUT /multi/gamma/one
{
"marker": "multiindex",
"name": "gamma one"
}
PUT /multi/gamma/two
{
"marker": "multiindex",
"name": "gamma two"
}
PUT /multi/old/delta/one
{
"marker": "multiindex",
"name": "delta one"
}

Search two of them by listing them:

SEARCH /multi/alpha,multi/beta/?q=marker:multiindex

Search the indexes directly under multi/ with a *, which stays in one segment:

SEARCH /multi/*/?q=marker:multiindex

Every index at any depth under multi/ with **:

SEARCH /multi/**/?q=marker:multiindex

An index at any depth by its last name:

SEARCH /**/delta/?q=marker:multiindex

A ? stands for one character, and a class for one of a set (a ? in a URL starts the query, so it is written %3F):

SEARCH /multi/alph%3F,multi/[bg]*/?q=marker:multiindex

A wildcard inside a name:

SEARCH /multi/g*/?q=marker:multiindex

Take indexes out with -:

SEARCH /multi/**,-multi/g*,-multi/old/*/?q=marker:multiindex

_all searches every index of the node:

SEARCH /_all/?q=marker:multiindex

A count takes the same patterns:

COUNT /multi/**/?q=marker:multiindex

A wildcard that matches nothing finds nothing:

SEARCH /multi/nomatch*/?q=marker:multiindex

Indexes nest: an index may hold others, and a glob finds each at its own depth. Make parent and parent/child:

PUT /parent/doc
{
"marker": "nestedindex",
"name": "parent doc"
}
PUT /parent/child/doc
{
"marker": "nestedindex",
"name": "child doc"
}

parent* is one segment, so it finds the parent only:

SEARCH /parent*/?q=marker:nestedindex

parent/* is the indexes in it, and parent/** the parent and everything in it:

SEARCH /parent/*/?q=marker:nestedindex
SEARCH /parent/**/?q=marker:nestedindex

_all is **: it finds both too.

SEARCH /_all/?q=marker:nestedindex
COUNT /parent*,parent/*/?q=marker:nestedindex

A failed request answers a JSON body with the status, its type and a message ({"status": 400, "type": "Bad Request", "message": "Bad Request: ..."}).

StatusMeaning
200 OKThe request succeeded and has a body.
201 CreatedPOST without an ID: the new document’s URL is in Location.
204 No ContentA write succeeded and echoes nothing (the default; see Quick-start).
400 Bad RequestThe request is malformed or past a limit; the message says which.
404 Not FoundThe document or index is not there.
405 Method Not AllowedThe verb is not served at that path.
406 Not AcceptableNo format in Accept can be served.
409 ConflictA write’s ?version does not match the document’s version; the body’s _version is the current one (0: no document).
412 Precondition FailedA strict index refuses a field or setting that has no type.
413 Payload Too LargeThe body is past the limit of the request.
501 Not ImplementedThe verb is known and not implemented here (OPEN, CLOSE, DELETE on an index).
503 Service UnavailableThe index is busy or the node holds too many requests; Retry-After says when to try again.

Sources: src/server/http_utils.h:48-135 (the mapping), src/server/search_views.cc:3182 (413), src/manager.cc:950,961 and the http library’s http_asio.h:1062-1066 (503).

Each limit answers 400 unless it says otherwise, and each is a constant of the code. The table repeats the architecture page’s with the verbs they apply to.

WhatLimitSource
A buffered body16 MiB (413)src/server/search_service.h:50
Nesting in a JSON, YAML or MessagePack body200 levelssrc/database/utils.h:143
A query100 levels, 4,096 clauses, 256 phrase or near groups joined by one ANDsrc/engine/query.h:94-98
Nesting of aggregations16 levelssrc/aggregations/aggregations.h:40
Live buckets in one search65,535src/aggregations/aggregations.h:39
Index patterns in one search32src/index_pattern.h:126
Indexes one search reads--max-search-indexes (100,000)src/opts.cc
A Lua script call10^8 instructions, 64 MiB of memorysrc/lua/processor.h:50-62

Past 32 index patterns a search answers 400:

SEARCH /p0*,p1*,p2*,p3*,p4*,p5*,p6*,p7*,p8*,p9*,p10*,p11*,p12*,p13*,p14*,p15*,p16*,p17*,p18*,p19*,p20*,p21*,p22*,p23*,p24*,p25*,p26*,p27*,p28*,p29*,p30*,p31*,p32*/?q=x

An aggregation nested 17 levels deep (the body is on one line) is refused:

SEARCH /bank/?limit=0
{"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}

The same aggregation at 16 levels runs:

SEARCH /bank/?limit=0
{"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}

A body that nests arrays or objects 201 levels deep is refused before it is read:

PUT /limits/deep
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]