Protocol
The edytor wire format and the helpers edytor/protocol exports for writing your own sync client or server.
The shipped provider and room speak a small binary protocol over WebSocket (and over BroadcastChannel between tabs). Read this page if you are writing your own server, a relay on another platform, or a client in another environment. Everything you need is exported from edytor/protocol, which runs in Node and Workers.
Frames
Every frame is three parts, written with lib0 variable-length integers:
varuint GENERATION | varuint messageType | payload
GENERATION names the engine, the wire protocol and the document schema together: PROTOCOL_VERSION * 1000 + SCHEMA_VERSION, which is 14011 in this release (generationWord(SCHEMA_VERSION); 14010 at schema generation 10, and so on down). Check it before decoding anything else and drop frames of any other generation: a v13 yjs peer’s first word is its message type (0 to 3), and another edytor schema writes 14000 + schema. readProtocolVersion(decoder) reads and checks it.
| Type | Constant | Direction | Payload |
|---|---|---|---|
0 |
messageSync |
both | A sync subtype, then a varUint8Array. See below. |
1 |
messageAwareness |
both | A varUint8Array holding an awareness update. |
2 |
messageAuth |
server to client | Subtype 0 (messagePermissionDenied), then a varString reason: a refused write. The reason read-only is not terminal: send it for each write of a read-only socket you drop; the provider emits permission-denied, keeps syncing and stops counting unsaved updates. Any other reason fails a provider that has not synced. Subtype 1 (messageReadOnly, no payload) is the read-only notice: send it when a read-only socket joins; the provider sets readOnly and stops counting, without an event. A provider older than this subtype reports it on message-error and keeps going. Subtype 2 (messageAccess, then a varString) is the socket’s level (write, suggest, comment, read), sent as it joins. Subtype 3 (messageEpoch, then a varuint) is the room’s epoch, sent instead of anything else to a socket that dialed an older one: see re-seeds. |
3 |
messageQueryAwareness |
client to server | None. Asks for everyone’s presence. |
4 |
messageSaved |
server to client | The server’s state vector (varUint8Array), then optionally the acknowledged deletes. |
5 |
messageChunk |
server to client | One piece of a frame too large to send whole. |
6 |
messageComments |
both | A comment subtype, then its body. See Comments. |
7 |
messageRebase |
client to server | A varUint8Array: the client’s whole document of an earlier epoch, its unsaved edits in it. See re-seeds. |
Sync
| Subtype | Constant | Payload | Meaning |
|---|---|---|---|
0 |
messageYjsSyncStep1 |
state vector | “This is what I have.” |
1 |
messageYjsSyncStep2 |
update | “Here is what you lack.” |
2 |
messageYjsUpdate |
update, v1 encoding | A live edit. |
The handshake is symmetric. On connect, each side sends Step 1. A side that receives Step 1 answers with Step 2 and, if the other side holds something it lacks, with its own Step 1. After one exchange each way both hold the same state, so no periodic resync is needed. A Step 2 carries a state and an Update one transaction, both in the engine’s one update encoding (the columnar one upstream Yjs calls v2, since schema generation 6), which sync.writeSyncStep2, Y.encodeStateAsUpdate and the document’s update event write and sync.applyRemote reads.
Awareness
An awareness update is a count, then for each entry a varuint client id, a varuint clock and a JSON string state (null means the client left). readAwarenessEntries and writeAwarenessEntries convert between bytes and { clientID, clock, state } arrays without an Awareness instance, which matters on a server that must not hold timers.
Saved
A server sends messageSaved after it has durably stored what a sync message brought. The body is the server’s state vector and, when the acknowledged message carried deletes, an update with no structs holding the deletes of that message the server stored. A client that knows only the state vector ignores the second field.
- Writing and reading it. Write it with
sync.writeSaved(encoder, doc, deletes, unstored?)and read it withsync.readSaved(decoder). By default it leaves out every deletedocholds pending (of an item it does not have yet); a server that stores such deletes, as the edytor room does, passes only the ones it did not store asunstored. - Deletes. The provider counts a delete as saved only once a
messageSavednames it, so a delete the server keeps in memory, or drops, stays unsaved. - Unsolicited acknowledgements. The edytor room also sends a
messageSavedto the other sockets when a message releases updates that were waiting for a missing dependency, with the deletes that waited with them, so the clients that wrote them see them stored; a client should accept one at any time.
Chunks
A chunk sequence is a start frame (0, then the total length), any number of part frames (1, then a varUint8Array), and an end frame (2). Both directions chunk: the provider sends a frame larger than its maxFrameBytes as a sequence (an offline backlog at reconnect), and a server must reassemble it per connection; the edytor room does, and closes a socket with 1011 when a part or end arrives with no sequence started.
chunkFrame(bytes, maxFrameBytes?)splits one complete frame (it returns the frame itself when it fits).createChunkReader(maxBytes?, admit?)returns a per-connection reader that you feed each chunk body and that returns the whole frame onend,nullbefore. A sequence announcing more thanmaxBytesthrowsChunkError(withtotalandlimit) at itsstart; a part or end with no start throws it too. The default limit isMAX_FRAME_BYTES(32 MiB).- The reader assembles a sequence into one buffer of the announced size, allocated at the
start, and itsbufferedproperty is that size while the sequence is in flight (0 otherwise). admit(total), asked at eachstartbefore the allocation, may throw to refuse the sequence or returnfalseto skip it (its parts are dropped and itsendreturnsnull). The edytor room uses it to skip a read-only socket’s sequences and to bound what all its sockets buffer together.
Re-seeds
A room that re-seeds its document starts a new epoch, which the document holds (root epoch, attr n; none before the first re-seed: epoch 0). A client names its document’s epoch in its dial’s query string, epoch=<n>, or epoch=new for a document that holds nothing yet. The room closes every socket 4410 at a re-seed, its reason the new epoch, and answers a dial:
- Its own epoch, or
new: the usual join. - An earlier epoch whose cut it keeps, from a writer:
messageAuthsubtype3(messageEpoch) with its epoch, and nothing else; it reads only amessageRebasefrom that socket (chunked like any large frame, up to twice the document quota), merges the edits it holds that the room did not have when it re-seeded, admits what that writes as the user’s update (its quotas:4413), then closes it4410. Nothing is relayed to it. From any other socket:4410at once. - An epoch older than the cuts kept, or a later one:
4410, its reason the room’s epoch. None, in a room that re-seeded:1008(refused: epoch).
A socket of the current epoch whose update holds an older epoch’s content is closed 4410 too.
The provider hands its document over when it hears messageEpoch, and redials once when a 4410 finds it with unsaved updates (they are handed over then); after that, 4410 is a refusal of its document. Only the room writes the epoch root.
Comments
A document’s comment threads are not in the document: the room keeps them, and tells the sockets that ask about each change. The subtypes:
| Subtype | Name | Direction | Body |
|---|---|---|---|
0 |
commentsSubscribe |
client to server | A varuint: 1 to start hearing the changes, 0 to stop. |
1 |
commentsSnapshot |
server to client | A varString of JSON, { seq, threads }: every thread, and the sequence number of the last change. The answer to a subscribe. |
2 |
commentsChange |
server to client | A varString of JSON, one change: { type, thread, comment, user, at, seq }. |
The room sends comment messages only to a socket that subscribed, so a client that never asks never receives one: older clients are unaffected and the generation word is unchanged. The subscription is part of the socket’s attachment, kept through hibernation, and a reconnected socket subscribes again. A client must not send a snapshot or a change: the room refuses one as malformed (1008), as it refuses any message type it does not know, so subscribe only to a room that keeps comments. Subscribes share the socket’s presence rate. Write and read them with writeCommentsSubscribe(encoder, on), writeCommentsSnapshot(encoder, snapshot), writeCommentsChange(encoder, change) and readCommentsMessage(decoder).
A change’s seq grows by one per change in the document; each thread carries the seq of its last change as rev. A client merges what it hears by rev: an older thread never replaces a newer one, and a snapshot removes the threads it lacks unless a change after its seq made them. The shipped provider subscribes while a listener watches the room (watchComments, what createCommentsClient uses).
Exported helpers
All from edytor/protocol:
| Export | Use |
|---|---|
bindCrdt(Y).sync |
writeSyncStep1, writeSyncStep2, writeUpdate, lacks(doc, stateVector), applyRemote(doc, update, origin), writeSaved, readSaved. |
frame(type, write) |
Build a frame: the generation word, the type, then whatever write(encoder) writes. |
GENERATION, generationWord, PROTOCOL_VERSION, SCHEMA_VERSION, readProtocolVersion |
The generation gate. |
messageSync, messageAwareness, messageAuth, messageQueryAwareness, messageSaved, messageChunk, messageComments |
Message types. |
commentsSubscribe, commentsSnapshot, commentsChange, writeCommentsSubscribe, writeCommentsSnapshot, writeCommentsChange, readCommentsMessage |
The comment messages. |
messageYjsSyncStep1, messageYjsSyncStep2, messageYjsUpdate |
Sync subtypes. |
messagePermissionDenied, writePermissionDenied(encoder, reason) |
The auth reply to a refused write. |
messageReadOnly, writeReadOnly(encoder) |
The read-only notice at the join. |
readAwarenessEntries, writeAwarenessEntries, applyAwarenessUpdate, encodeAwarenessUpdate |
The awareness codec. |
chunkFrame, createChunkReader, MAX_FRAME_BYTES, ChunkError |
Chunked frames. |
CLOSE |
The close codes the room and its router send, by name (refused 1008, fault 1011, invalidDocument 4400, expired 4401, denied 4403, replicaTaken 4409, quota 4413; see admission). |
createDecoder, readVarUint, readVarUint8Array, writeVarUint8Array |
The lib0 helpers for frame bodies. |
GENERATION_RECORD, GenerationMismatchError |
Tag and check stored containers of this generation. |
bindCrdt(Y).doc.purge, .horizonOf, .restoreTo |
The shipped room’s purge (purge(doc, facade, { at, sv }), one transaction of yours, sv a state vector the room stored before at) and restore (restoreTo(doc, facade, json)), for a room of your own; facade is bindCrdt(Y).doc.create(doc), which publishes no author. |
applyRemote applies an update unless it writes a foreign schema stamp. It returns { applied, problem, discarded? }: refuse the frame when problem is set.
A minimal server loop
This is the core of a relay that stores and acknowledges, for any platform with WebSockets. Authorization, identity checks and persistence are yours to add; the shipped DocumentRoom does all of them.
import * as Y from 'edytor/crdt';
import {
bindCrdt, frame, readProtocolVersion, createDecoder, readVarUint, readVarUint8Array,
writeVarUint8Array, readAwarenessEntries, writeAwarenessEntries,
messageSync, messageAwareness, messageSaved, messageYjsSyncStep1, messageYjsSyncStep2
} from 'edytor/protocol';
const { sync, createDoc } = bindCrdt(Y);
const doc = createDoc(); // restore it from your storage: Y.applyUpdate(doc, stored)
const sockets = new Set<WebSocket>();
doc.on('update', (update: Uint8Array, origin: unknown) => {
// Store `update` durably here, then relay it.
const out = frame(messageSync, (e) => sync.writeUpdate(e, update));
for (const ws of sockets) if (ws !== origin) ws.send(out);
});
export function onOpen(ws: WebSocket) {
sockets.add(ws);
ws.send(frame(messageSync, (e) => sync.writeSyncStep1(e, doc)));
}
export function onMessage(ws: WebSocket, bytes: Uint8Array) {
const decoder = createDecoder(bytes);
if (!readProtocolVersion(decoder)) return ws.close(1008, 'generation');
const type = readVarUint(decoder);
if (type === messageSync) {
const subtype = readVarUint(decoder);
const payload = readVarUint8Array(decoder);
if (subtype === messageYjsSyncStep1) {
ws.send(frame(messageSync, (e) => sync.writeSyncStep2(e, doc, payload)));
if (sync.lacks(doc, payload)) ws.send(frame(messageSync, (e) => sync.writeSyncStep1(e, doc)));
return ws.send(frame(messageSaved, (e) => sync.writeSaved(e, doc)));
}
// A Step 2 and an Update carry the same encoding.
const update = payload;
const { applied, problem } = sync.applyRemote(doc, update, ws);
if (problem) return ws.close(1008, 'refused: schema'); // final: the provider stops
if (!applied) return ws.close(1011, 'internal error'); // a fault: the provider redials
ws.send(frame(messageSaved, (e) => sync.writeSaved(e, doc, Y.decodeUpdate(update).ds)));
} else if (type === messageAwareness) {
const entries = readAwarenessEntries(readVarUint8Array(decoder));
const out = frame(messageAwareness, (e) => writeVarUint8Array(e, writeAwarenessEntries(entries)));
for (const other of sockets) if (other !== ws) other.send(out);
}
}
Send large frames through chunkFrame if your platform limits message size, and remember the latest presence entry per client to greet newcomers, as the shipped room does. If your room purges, only it may write the document’s horizon root: strip a client update that writes it, as the shipped room does. Close with 1008 or a 4xxx code (other than 4401) only for a refusal the client should not retry: the provider stops dialing. Close with 1011 for a fault of your server, and refuse an upgrade you turn away by accepting it and closing it, not with an HTTP error, which a browser reports as 1006 (unreachable, redialed).
Never attach a document on the server
Do not call createDocument, loadDocument or attachDocument on the document a server holds. They publish the local author into the document’s replicated attribution data, so the server would appear as an author in every client. A server works on the raw CRDT document and the sync helpers only. Make that document with bindCrdt(Y).createDoc(), before it integrates anything, never with a bare new Y.Doc(): createDoc gives it the rule every edytor document carries (when two writers create one block id at once, the losing node keeps its text), and a document without it deletes that text where it integrates the race, then stores and relays the delete.
To read or edit the content server-side (for indexing, exports or bots), load a separate copy from the stored bytes with loadDocument and send your edits through a normal client connection, or apply them to that copy and send its update like any other client.
Compatibility
- Peers of different generations never exchange edits: each side drops the other’s frames and reports
'protocol-mismatch'. - A v13
yjspeer ory-websocketserver cannot join a room. An opaque relay that forwards frames byte for byte without decoding them works, but it cannot acknowledge saves or chunk frames. - Clients that predate
messageSavedandmessageChunkreport them through'message-error'and otherwise sync small documents normally. - A room that predates
messageCommentscloses a socket that subscribes (1008,malformed): the comments plugin subscribes only through its client, so upgrade the room before you list the plugin.