Skip to content

Store Content API

Edit page

The Store API allows adding additional content to documents and store such content in the Index Storage.

The Index Storage is designed to put files in volumes much in the way Facebook’s Haystack 1 works; once there a file enters the storage it can’t really be deleted/modified from the volume, but instead, if a change is needed, a new file blob will be written to the volume. Storage is envisioned to be used when there are files you need to store which you know won’t be changing often or at all.

Assuming there is a PNG file called sample.png in the working directory, lets add those to the storage using PUT or UPDATE (if the document already exists) and passing a content type:

PUT /assets/sample
Content-Type: image/png
@sample.png

And getting it is just a matter of retreiving it using the GET HTTP method:

GET /assets/sample
Accept: image/png

Or by visiting the link to it with your web browser: http://localhost:8880/assets/sample

Use PUT or UPDATE with a different Content-Type to add new content to the same document. For example, the following adds a PDF and a JPEG from files called sample.pdf and sample.jpg, respectively:

UPDATE /assets/sample
Content-Type: application/pdf
@sample.pdf

This time we also include the .jpg selector as a File Extension:

UPDATE /assets/sample.jpg
Content-Type: image/jpeg
@sample.jpg

Then you can get either of them requesting the appropriate content type in the Accept header:

GET /assets/sample
Accept: application/pdf
GET /assets/sample
Accept: *

If passing a file extension, the default content type is obtained from the mime.types file (usually in /usr/local/share/xapiand/mime.types).

For example, this will return the content with image/png content type of the document with ID sample:

GET /assets/sample.png

Or visiting the link to the PNG content with your web browser: http://localhost:8880/assets/sample.png

You can get the information about the document as usual:

INFO /assets/sample

The result (partially shown) has the available content types listed inside data

{
"docid": 1,
"data": [
{
"content_type": "application/msgpack",
"type": "inplace"
},
{
"content_type": "image/png",
"type": "stored",
"volume": 0,
"offset": 512,
"size": "154.3KiB"
},
{
"content_type": "application/pdf",
"type": "stored",
"volume": 0,
"offset": 60063,
"size": "154.7KiB"
},
{
"content_type": "image/jpeg",
"type": "stored",
"volume": 0,
"offset": 143195,
"size": "8.7KiB"
}
], ...
}

To remove stored content by storing an empty object:

UPDATE /assets/sample
Content-Type: image/jpeg
Content-Length: 0

A body of up to 16 MiB is buffered like any other write. A PUT or POST whose Content-Type is a stored type (not JSON, YAML or MessagePack) and whose Content-Length is past 16 MiB is streamed to the data storage as it arrives, so the node does not hold it in memory; a stored blob can be as large as 4 GiB less a kilobyte. The request must carry its Content-Length. Any other request whose body is past 16 MiB answers 413 Payload Too Large.

Sources: src/server/search_service.h:50 (the 16 MiB cap), src/server/search_views.cc:3094-3106 (when a body streams), :3180-3185 (413) and src/database/pool.h:157 (STORED_UPLOAD_LIMIT).


1 Finding a needle in Haystack: Facebook’s photo storage.