Version 1.1.0 of 23 November 2016
Use VelocyPack as body. Content-Type is "application/vpack"
This is not a request / response protocol. It is symmetric (in principle). Messages can be sent back and forth, pipelined, multiplexed, uni-directional or bi-directional.
It is possible that a message generates
-
no response
-
exactly one response
-
multiple responses
The VelocyStream does not impose or specify or require one of the above behaviors. The application must define a behavior in general or on a per-request basis, see below. The server and client must then implement that behavior.
The consumer (client or server) will deal with messages. A message consists of one or more VelocyPacks (or in some cases of certain parts of binary data). How many VelocyPacks are part of a message is completely application dependent, see below for ArangoDB.
It is possible that the messages are very large. As messages can be multiplexed over one connection, large messages need to be split into chunks. The sender/receiver class will accept a vector of VelocyPacks, split them into chunks, send these chunks over the wire, assemble these chunks, generates a vector of VelocyPacks and passes this to the consumer.
In order to allow reassemble chunks, each package is prefixed by a small header. A chunk is always at least 24 bytes long. The byte order is ALWAYS little endian. The format of a chunk is the following, regardless on whether it is the first in a message or a subsequent one:
| name | type | description |
|---|---|---|
| length | uint32_t | total length in bytes of the current chunk, including this header |
| chunkX | uint32_t | chunk/isFirstChunk (upper 31bits/lowest bit), details see below |
| messageId | uint64_t | a unique identifier, it is the responsibility of the sender to generate such an identifier (zero is reserved for not set ID) |
| messageLength | uint64_t | the total size of the message. |
| Binary data | binary data blob | size b1 |
Clarification: "chunk" and "isFirstChunk" are combined into an unsigned 32bit value. Therefore it will be encoded as
uint32_t chunkX
and extracted as
chunk = chunkX >> 1
isFirstChunk = chunkX & 0x1
For the first chunk of a message, the low bit of the second uint32_t is set, for all subsequent ones it is reset. In the first chunk of a message, the number "chunk" is the total number of chunks in the message, in all subsequent chunks, the number "chunk" is the current number of this chunk.
The total size of the data package is (24 + b1) bytes. This number is stored in the length field. If one needs to send messages larger than UINT32_MAX, then these messages must be chunked. In general it is a good idea to restrict the maximal size to a few megabytes.
Notes:
When sending a (small) message, it is important (for performance reasons) to ensure that only one TCP packet is sent. For example, by using sendmmsg under Linux (https://blog.cloudflare.com/how-to-receive-a-million-packets/)
Implementors should nevertheless be aware that in TCP/IP one cannot enforce this and so implementations must always be aware that some part of the network stack can split packets and the payload might arrive in multiple parts!
For an ArangoDB client, the request is of the following format, the array is a VelocyPack array:
[
/* 0 - version: */ 1, // [int]
/* 1 - type: */ 1, // [int] 1=Req, 2=Res,..
/* 2 - database: */ "test", // [string]
/* 3 - requestType: */ 1, // [int] 0=Delete, ...
/* 4 - request: */ "/_api/collection", // [string\]
/* 5 - parameter: */ { force: true }, // [[string]->[string]]
/* 6 - meta: */ { x-arangodb: true } // [[string]->[string]]
]
Body (binary data)
If database is missing (entry is null), then "_system" is assumed.
type:
1 = Request
2 = Response (final response for this message id)
3 = Response (but at least one more response will follow)
1000 = Authentication
requestType:
0 = DELETE
1 = GET
2 = POST
3 = PUT
4 = HEAD (not used in VPP)
5 = PATCH
6 = OPTIONS (not used in VPP)
For example:
The HTTP request
http://localhost:8529/_db/test/_admin/echo?a=1&b=2&c[]=1&c[]=3
With header:
X-ArangoDB-Async: true
is equivalent to
[
1, // version
1, // type
"test", // database
1, // requestType GET
"/_admin/echo", // request path
{ // parameters
a: 1,
b: 2,
c: [ 1, 3 ]
},
{ // meta
x-arangodb-async: true
}
]
The request is a message beginning with one VelocyPack. This VelocyPack
always contains the header fields, parameters and request path. If the
meta field does not contain a content type, then the default
"application/vpack" is assumed and the body will be one or multiple
VelocyPack object.
The response will be
[
1, // 0 - version
2 or 3, // 1 - type
400, // 2 - responseCode
{ etag: "1234" } // 3 - meta: [[str]->[str]]
]
Body (binary data)
Request can be pipelined or mixed. The responses are mapped using the "messageId" in the header. It is the responsibility of the sender to generate suitable "messageId" values.
The default content-type is "application/vpack".
A connection can be authenticated with the following message:
[
1, // version
1000, // type
"plain", // encryption
"admin", // user
"plaintext", // password
]
or
[
1, // version
1000, // type
"jwt", // encryption
"abcd..." // token
]
The response is
{ "error": false }
if successful or
{
"error": true,
"errorMessage": "MESSAGE",
"errorCode": CODE
}
if not successful, and in this case the connection is closed by the server.
One can acquire a JWT token in the same way as with HTTP using the
open, unauthenticated route /_open/auth with the same semantics as
in the HTTP version. In this way, the complete authentication can be
done in a single session via JWT.
In general the content-type will be VPP, that is the body is an object stored as VelocyPack.
Sometimes it is necessary to respond with unstructured data, like text, css or html. The body will be a VelocyPack object containing just a binary attribute and the content-type will be set accordingly.
The rules are as follows.
Request: Content-Type
-
"application/json": the body contains the JSON string representation -
"application/vpack": the body contains a velocy pack
There are some handler that allow lists of JSON (seperared by newline). In this case we also allow multiple velocy packs without any separator.
Request: Accept
-
"application/json": send a JSON string representation in the body, if possible -
"application/vpack": send velocy pack in the body, if possible
If the request asked for "application/json" or "application/vpack" and
the handler produces something else (i.e. "application/html"), then the
accept is ignored.
If the request asked "application/json" and the handler produces
"application/vpack", then the VPACK is converted into JSON.
If the request asked "application/vpack" and the handler produces
"application/json", then the JSON is converted into VPACK.
Similar to HTTP with the exception: the "Accept" header is not supported
and "application/json" will always be converted into
"application/vpack". This means that the body contains one or more
velocy-packs. In general it will contain one - notable exception being
the import.
If the handler produces something else (i.e. "application/html"), then
The body will be a binary blob (instead of a velocy-pack) and the
content-type will be set accordingly.
The first bytes sent after a connection (the "client" side - even if the program is bi-directional, there is a server listening to a port and a client connecting to a port) are
VST/1.1\r\n\r\n
(11 Bytes)