API Conventions
Edit pageRESTful Features
Section titled “RESTful Features”Xapiand uses a RESTful API exposed using JSON over HTTP.
When we talk about our API, we use terms like “REST” and “RESTful.” “REST” stands for Representational State Transfer.
The conventions listed in here can be applied throughout the REST API, unless otherwise specified.
RESTful HTTP Methods
Section titled “RESTful HTTP Methods”You may see these standard HTTP methods referred to as CRUD, or Create, Read,
Update, Delete. Although CRUD has roots in database operations, you can also
map those operations to the standard HTTP methods. For example, use a POST
request to create a new resource, a GET request to read or retrieve a resource,
a PATCH or UPDATE request to edit a resource, and a DELETE request to
delete a resource.
Deviations from REST
Section titled “Deviations from REST”We do our best to use standard HTTP methods with accurate and well-known status codes in the Xapiand API, but here are some additions and deviations.
Additionally to the standard HTTP methods, we also use other custom methods
such as UPDATE, for certain operations.
HTTP Methods
Section titled “HTTP Methods”-
Safe Methods
Requests that use safe HTTP methods won’t alter a resource at all. Examples areGET,OPTIONSandHEAD. -
Idempotent Methods
An idempotent HTTP method is a HTTP method that can be called many times without different outcomes. It would not matter if the method is called only once, twice or a hundred times over, the result should be the same. This only applies to the result, not the resource itself. Examples areDELETE,PUT. -
Not Safe/Idempotent Methods
This are requests that will alter the resource and potentially end up with different results every time. Examples arePOST.
Custom Methods
Section titled “Custom Methods”The Standard Methods, the ones we all are familiar with, have simpler and well-defined semantics but there is functionality that cannot be easily expressed via standard methods. Custom methods refer to such API methods.
Simply pass the required method as a non-standard HTTP method in the request.
Example:
INFO /some/resource/nameIf your firewall rules don’t support non-standard HTTP methods like PATCH,
UPDATE or DELETE, for example, you have two options:
- Use HTTP Method Override
- Use HTTP Method Mapping
HTTP Method Override
Section titled “HTTP Method Override”You can use the X-HTTP-Method-Override
(or HTTP-Method-Override) header. Pass the method you want to use in the
X-HTTP-Method-Override header and use the POST method.
Example:
POST /some/resource/nameX-HTTP-Method-Override: INFOHTTP Method Mapping
Section titled “HTTP Method Mapping”Custom methods can use the following generic HTTP mapping:
http://service.name:8880/some/resource/name:customMethod
To use method mappings pass the mapping in the URL and use HTTP POST verb
since it has the most flexible semantics, except for methods serving as an
alternative get or list which may use GET. Any other method answers 400
(“HTTP Mappings must use GET or POST method”).
Example:
GET /some/resource/name:infoResource Paths
Section titled “Resource Paths”To slash or not to slash
Section titled “To slash or not to slash”That is the question we hear often. Onward to the answers! Historically, it’s common for URLs with a trailing slash to indicate a directory, and those without a trailing slash to denote a file:
http://example.com/foo(without trailing slash, conventionally a file)http://example.com/foo/(with trailing slash, conventionally a directory)
Source: Google WebMaster Central Blog - To slash or not to slash
Trailing slashes are important
Section titled “Trailing slashes are important”To us, trailing slashes are important to distinguish between a path to an Index (a directory) and a path to a Document (a file).
The following will delete a single document from index /some/resource/name,
the document with ID name:
DELETE /some/resource/nameWhilst the next one addresses the index /some/resource/path/. Xapiand does not
delete indexes over HTTP: DELETE on an index path answers 501 Not Implemented
(see Delete Index):
DELETE /some/resource/path/The Verbs
Section titled “The Verbs”A request is a verb and a path. The path names a document (it ends with an ID), an
index (it ends with a /), a command (/index/:command) or the node (/). A verb
the path does not accept answers 405 Method Not Allowed.
| Verb | On | What it does |
|---|---|---|
GET | node, index, document | The node, the index’s settings and schema, a document; GET /:metrics answers the metrics |
HEAD | index, document | 200 if it exists, 404 if not |
SEARCH, COUNT | one or many indexes | Search, or count the hits |
INFO | one index, document | Statistics of an index or document |
PUT | index, document | Create an index or index a document |
POST | index | Index a document with a generated ID (201 and Location) |
UPDATE, PATCH, UPSERT | index, document | Update a document; UPSERT creates it when missing |
DELETE | document | Delete a document. On an index it answers 501, on / it answers 405 |
COMMIT | one index | Commit the index now (200) |
CHECK | one index | Check the index’s files; answers {"errors": {<index>: <count>}} |
DUMP, RESTORE | one index, document | Dump or restore its documents |
WAL | one single-shard index | The write-ahead log |
FLUSH, QUIT | / | Administrative commands, only with --admin-commands |
OPTIONS | any | 200 with an Allow header |
OPEN, CLOSE | any | 429 Too Many Requests |
501 Not Implemented | ||
LOCK, UNLOCK, COPY, MOVE, PURGE, LINK, UNLINK, TRACE, CONNECT | any | 405 |
Sources: the verb list and the route table, src/server/search_views.cc:187-218 and
548-602; the inline answers, :652-672.
OPTIONS /twitter/user/A verb that is not served at that path answers 405:
LOCK /twitter/user/COMMIT /twitter/user/JohnCOMMIT commits an index now instead of waiting for the autocommit (see
Visibility and Durability):
COMMIT /twitter/user/CHECK checks an index and answers how many errors it found in each shard:
CHECK /twitter/user/Administrative Commands
Section titled “Administrative Commands”FLUSH and QUIT are answered only at the root URL, and only when the node runs
with --admin-commands; otherwise (and on any other path) they answer 405.
FLUSHcommits and closes every open index and closes the client connections, and answers200when they are closed.FLUSH /?databasesflushes only the indexes,FLUSH /?clientsonly the connections.QUITasks the node to shut down, answers200and closes the connection (see Stopping the node).
The API has no authentication: enable the option only on a node that untrusted clients
cannot reach (see Options). Sources:
src/server/search_views.cc:1497-1535 (flush_view, quit_view).
QUIT /FLUSH /twitter/The Node
Section titled “The Node”The root of the API is the node itself. It names no index, so it has no documents:
GET / answers with the node’s name, its server and library versions, its options
and its state, and HEAD / answers 200 when the node is up.
GET /A field selector reduces the answer, GET /.state answers just the node’s
state, and GET /:info answers the same as GET /. GET /:metrics answers the
node’s metrics in the Prometheus text format. A request that needs an index (PUT /,
POST /, a search or a count) and names none answers 400; DELETE / answers 405.
GET /.server.namePUT /A root command that reads index metadata, such as /:stats or /:schema, names no index and answers 400 too.
GET /:statsJSON, MessagePack, and YAML
Section titled “JSON, MessagePack, and YAML”The Xapiand API can process JSON, MessagePack, or YAML objects. Choose the format
with the request’s Content-Type header (application/json,
application/x-msgpack, or application/x-yaml); JSON is the default when no
Content-Type is given.
You can also send documents as YAML by setting the
Content-Type to application/x-yaml. Xapiand parses the body into the same
internal object it would from the equivalent JSON:
PUT /twitter/user/Jane?commitContent-Type: application/x-yaml
name: Janeage: 30city: TurinGET /twitter/user/JaneDeviations from JSON
Section titled “Deviations from JSON”Comments
Section titled “Comments”JSON can have C-style /* */ block or single line // comments. Comments are
allowed everywhere in the JSON document.
Trailing Commas
Section titled “Trailing Commas”JSON can have trailing commas.
Field Expansion
Section titled “Field Expansion”JSON or MessagePack fields in objects passed to Xapiand are expanded. For example, the following nested object:
{ "contact": { "address": { "country": { "name": "Italy" } } }}Is equivalent to:
{ "contact.address.country.name": "Italy"}Multiple Indexes
Section titled “Multiple Indexes”SEARCH and COUNT run across multiple indexes, using simple
test1,test2,test3 notation. Every other verb takes a single index (a list
answers 400, “Method can only be used with single indexes”).
Searches and counts also take _all for all the indexes of the node, and globs
over the segments of an index path (the parts between /), for example
test*, *test, te*t, test-??, test-[0-9]* or logs/**/error-*, and -
to exclude, for example test*,-test3:
- A
*stands for any run of characters inside one segment, none included. It never matches a/:test*matchestestandtest1, and nottest/old(that istest*/*). - A
?stands for one character, in one segment:test-??matchestest-ab. - A
[...]stands for one character of a class:[abc], a range[a-f], or every character but those with a leading!or^:[!0-9]. A[with no closing]is a plain[. - A segment that is
**stands for any number of segments, none included, and may be used more than once:**/some-??matchessome-abat any depth, andlogs/**/error-*/**/lastmatcheslogs/error-1/last,logs/a/error-1/b/c/lastand so on.logs/**matcheslogsand everything below it. _allis the same as**.- An item that starts with
-removes the indexes it matches from the result, wherever it stands in the list. - A glob never matches an index whose name starts with a
.(the node’s own indexes); name it to search it. - A glob that matches no index finds nothing: the search answers
200with no hits. - A search over more than
--max-search-indexesindexes (100,000 by default), counting the ones a glob expands to, answers400; so does one whose patterns read more than that many directory entries to find them, in the directories of an index too. The search goes only into the directories a pattern can still match below:logs/2026/*readslogsandlogs/2026, and no other directory. Once the node’s index names are built (its first pattern starts that), a pattern reads index names instead: each name under the pattern’s prefix that it reads counts against the same number, never more than the walk would read, sologs-2026*reads only the names that start withlogs-2026, however many indexes the node holds. A search names at most 32 patterns (globs and exclusions together), and an item repeated counts once. A wide search reads its indexes in batches; see Searching many indexes.
To reach the indexes below a name, give the segments: test*/*, or test*/**.
Globs match the indexes that are on the node’s disk. They are the way to reach a great many indexes: a URL holds 64 KiB, which is a few thousand index names.
Indexes nest: twitter and twitter/user can both exist, the second inside the
first. twitter* finds the first, twitter/* the second, and twitter/** or
_all both. In a URL a ? starts the query string; write it %3F in an index
pattern.
A wildcard or _all reads every index on the node, so a name that is not given
no longer keeps an index out of a search: a proxy that allows a request by the
first segment of its path has to treat ,, *, _all and a leading - as
naming several indexes, and :count on a prefix tells whether names exist.
Four indexes, one holding two documents:
PUT /multi/alpha/one
{ "marker": "multiindex", "name": "alpha one"}PUT /multi/beta/one
{ "marker": "multiindex", "name": "beta one"}PUT /multi/gamma/one
{ "marker": "multiindex", "name": "gamma one"}PUT /multi/gamma/two
{ "marker": "multiindex", "name": "gamma two"}PUT /multi/old/delta/one
{ "marker": "multiindex", "name": "delta one"}Search two of them by listing them:
SEARCH /multi/alpha,multi/beta/?q=marker:multiindexSearch the indexes directly under multi/ with a *, which stays in one segment:
SEARCH /multi/*/?q=marker:multiindexEvery index at any depth under multi/ with **:
SEARCH /multi/**/?q=marker:multiindexAn index at any depth by its last name:
SEARCH /**/delta/?q=marker:multiindexA ? stands for one character, and a class for one of a set (a ? in a URL starts the query, so it is written %3F):
SEARCH /multi/alph%3F,multi/[bg]*/?q=marker:multiindexA wildcard inside a name:
SEARCH /multi/g*/?q=marker:multiindexTake indexes out with -:
SEARCH /multi/**,-multi/g*,-multi/old/*/?q=marker:multiindex_all searches every index of the node:
SEARCH /_all/?q=marker:multiindexA count takes the same patterns:
COUNT /multi/**/?q=marker:multiindexA wildcard that matches nothing finds nothing:
SEARCH /multi/nomatch*/?q=marker:multiindexIndexes nest: an index may hold others, and a glob finds each at its own depth. Make parent and parent/child:
PUT /parent/doc
{ "marker": "nestedindex", "name": "parent doc"}PUT /parent/child/doc
{ "marker": "nestedindex", "name": "child doc"}parent* is one segment, so it finds the parent only:
SEARCH /parent*/?q=marker:nestedindexparent/* is the indexes in it, and parent/** the parent and everything in it:
SEARCH /parent/*/?q=marker:nestedindexSEARCH /parent/**/?q=marker:nestedindex_all is **: it finds both too.
SEARCH /_all/?q=marker:nestedindexCOUNT /parent*,parent/*/?q=marker:nestedindexStatuses
Section titled “Statuses”A failed request answers a JSON body with the status, its type and a message
({"status": 400, "type": "Bad Request", "message": "Bad Request: ..."}).
| Status | Meaning |
|---|---|
200 OK | The request succeeded and has a body. |
201 Created | POST without an ID: the new document’s URL is in Location. |
204 No Content | A write succeeded and echoes nothing (the default; see Quick-start). |
400 Bad Request | The request is malformed or past a limit; the message says which. |
404 Not Found | The document or index is not there. |
405 Method Not Allowed | The verb is not served at that path. |
406 Not Acceptable | No format in Accept can be served. |
409 Conflict | A write’s ?version does not match the document’s version; the body’s _version is the current one (0: no document). |
412 Precondition Failed | A strict index refuses a field or setting that has no type. |
413 Payload Too Large | The body is past the limit of the request. |
501 Not Implemented | The verb is known and not implemented here (OPEN, CLOSE, DELETE on an index). |
503 Service Unavailable | The index is busy or the node holds too many requests; Retry-After says when to try again. |
Sources: src/server/http_utils.h:48-135 (the mapping), src/server/search_views.cc:3182
(413), src/manager.cc:950,961 and the http library’s http_asio.h:1062-1066 (503).
Limits
Section titled “Limits”Each limit answers 400 unless it says otherwise, and each is a constant of the
code. The table repeats the
architecture page’s with the verbs they apply to.
| What | Limit | Source |
|---|---|---|
| A buffered body | 16 MiB (413) | src/server/search_service.h:50 |
| Nesting in a JSON, YAML or MessagePack body | 200 levels | src/database/utils.h:143 |
| A query | 100 levels, 4,096 clauses, 256 phrase or near groups joined by one AND | src/engine/query.h:94-98 |
| Nesting of aggregations | 16 levels | src/aggregations/aggregations.h:40 |
| Live buckets in one search | 65,535 | src/aggregations/aggregations.h:39 |
| Index patterns in one search | 32 | src/index_pattern.h:126 |
| Indexes one search reads | --max-search-indexes (100,000) | src/opts.cc |
| A Lua script call | 10^8 instructions, 64 MiB of memory | src/lua/processor.h:50-62 |
Past 32 index patterns a search answers 400:
SEARCH /p0*,p1*,p2*,p3*,p4*,p5*,p6*,p7*,p8*,p9*,p10*,p11*,p12*,p13*,p14*,p15*,p16*,p17*,p18*,p19*,p20*,p21*,p22*,p23*,p24*,p25*,p26*,p27*,p28*,p29*,p30*,p31*,p32*/?q=xAn aggregation nested 17 levels deep (the body is on one line) is refused:
SEARCH /bank/?limit=0
{"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}The same aggregation at 16 levels runs:
SEARCH /bank/?limit=0
{"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"},"_aggs":{"a":{"_values":{"_field":"gender"}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}A body that nests arrays or objects 201 levels deep is refused before it is read:
PUT /limits/deep
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]