Skip to content
Edytor
Esc
↑↓navigate↵open⌘Jpreview
On this page

Protocol

The edytor wire format and the helpers edytor/protocol exports for writing your own sync client or server.

The shipped provider and room speak a small binary protocol over WebSocket (and over BroadcastChannel between tabs). Read this page if you are writing your own server, a relay on another platform, or a client in another environment. Everything you need is exported from edytor/protocol, which runs in Node and Workers.

Frames

Every frame is three parts, written with lib0 variable-length integers:

varuint GENERATION | varuint messageType | payload

GENERATION names the engine, the wire protocol and the document schema together: PROTOCOL_VERSION * 1000 + SCHEMA_VERSION, which is 14011 in this release (generationWord(SCHEMA_VERSION); 14010 at schema generation 10, and so on down). Check it before decoding anything else and drop frames of any other generation: a v13 yjs peer’s first word is its message type (0 to 3), and another edytor schema writes 14000 + schema. readProtocolVersion(decoder) reads and checks it.

Type Constant Direction Payload
0 messageSync both A sync subtype, then a varUint8Array. See below.
1 messageAwareness both A varUint8Array holding an awareness update.
2 messageAuth server to client Subtype 0 (messagePermissionDenied), then a varString reason: a refused write. The reason read-only is not terminal: send it for each write of a read-only socket you drop; the provider emits permission-denied, keeps syncing and stops counting unsaved updates. Any other reason fails a provider that has not synced. Subtype 1 (messageReadOnly, no payload) is the read-only notice: send it when a read-only socket joins; the provider sets readOnly and stops counting, without an event. A provider older than this subtype reports it on message-error and keeps going. Subtype 2 (messageAccess, then a varString) is the socket’s level (write, suggest, comment, read), sent as it joins. Subtype 3 (messageEpoch, then a varuint) is the room’s epoch, sent instead of anything else to a socket that dialed an older one: see re-seeds.
3 messageQueryAwareness client to server None. Asks for everyone’s presence.
4 messageSaved server to client The server’s state vector (varUint8Array), then optionally the acknowledged deletes.
5 messageChunk server to client One piece of a frame too large to send whole.
6 messageComments both A comment subtype, then its body. See Comments.
7 messageRebase client to server A varUint8Array: the client’s whole document of an earlier epoch, its unsaved edits in it. See re-seeds.

Sync

Subtype Constant Payload Meaning
0 messageYjsSyncStep1 state vector “This is what I have.”
1 messageYjsSyncStep2 update “Here is what you lack.”
2 messageYjsUpdate update, v1 encoding A live edit.

The handshake is symmetric. On connect, each side sends Step 1. A side that receives Step 1 answers with Step 2 and, if the other side holds something it lacks, with its own Step 1. After one exchange each way both hold the same state, so no periodic resync is needed. A Step 2 carries a state and an Update one transaction, both in the engine’s one update encoding (the columnar one upstream Yjs calls v2, since schema generation 6), which sync.writeSyncStep2, Y.encodeStateAsUpdate and the document’s update event write and sync.applyRemote reads.

Awareness

An awareness update is a count, then for each entry a varuint client id, a varuint clock and a JSON string state (null means the client left). readAwarenessEntries and writeAwarenessEntries convert between bytes and { clientID, clock, state } arrays without an Awareness instance, which matters on a server that must not hold timers.

Saved

A server sends messageSaved after it has durably stored what a sync message brought. The body is the server’s state vector and, when the acknowledged message carried deletes, an update with no structs holding the deletes of that message the server stored. A client that knows only the state vector ignores the second field.

  • Writing and reading it. Write it with sync.writeSaved(encoder, doc, deletes, unstored?) and read it with sync.readSaved(decoder). By default it leaves out every delete doc holds pending (of an item it does not have yet); a server that stores such deletes, as the edytor room does, passes only the ones it did not store as unstored.
  • Deletes. The provider counts a delete as saved only once a messageSaved names it, so a delete the server keeps in memory, or drops, stays unsaved.
  • Unsolicited acknowledgements. The edytor room also sends a messageSaved to the other sockets when a message releases updates that were waiting for a missing dependency, with the deletes that waited with them, so the clients that wrote them see them stored; a client should accept one at any time.

Chunks

A chunk sequence is a start frame (0, then the total length), any number of part frames (1, then a varUint8Array), and an end frame (2). Both directions chunk: the provider sends a frame larger than its maxFrameBytes as a sequence (an offline backlog at reconnect), and a server must reassemble it per connection; the edytor room does, and closes a socket with 1011 when a part or end arrives with no sequence started.

  • chunkFrame(bytes, maxFrameBytes?) splits one complete frame (it returns the frame itself when it fits).
  • createChunkReader(maxBytes?, admit?) returns a per-connection reader that you feed each chunk body and that returns the whole frame on end, null before. A sequence announcing more than maxBytes throws ChunkError (with total and limit) at its start; a part or end with no start throws it too. The default limit is MAX_FRAME_BYTES (32 MiB).
  • The reader assembles a sequence into one buffer of the announced size, allocated at the start, and its buffered property is that size while the sequence is in flight (0 otherwise).
  • admit(total), asked at each start before the allocation, may throw to refuse the sequence or return false to skip it (its parts are dropped and its end returns null). The edytor room uses it to skip a read-only socket’s sequences and to bound what all its sockets buffer together.

Re-seeds

A room that re-seeds its document starts a new epoch, which the document holds (root epoch, attr n; none before the first re-seed: epoch 0). A client names its document’s epoch in its dial’s query string, epoch=<n>, or epoch=new for a document that holds nothing yet. The room closes every socket 4410 at a re-seed, its reason the new epoch, and answers a dial:

  • Its own epoch, or new: the usual join.
  • An earlier epoch whose cut it keeps, from a writer: messageAuth subtype 3 (messageEpoch) with its epoch, and nothing else; it reads only a messageRebase from that socket (chunked like any large frame, up to twice the document quota), merges the edits it holds that the room did not have when it re-seeded, admits what that writes as the user’s update (its quotas: 4413), then closes it 4410. Nothing is relayed to it. From any other socket: 4410 at once.
  • An epoch older than the cuts kept, or a later one: 4410, its reason the room’s epoch. None, in a room that re-seeded: 1008 (refused: epoch).

A socket of the current epoch whose update holds an older epoch’s content is closed 4410 too.

The provider hands its document over when it hears messageEpoch, and redials once when a 4410 finds it with unsaved updates (they are handed over then); after that, 4410 is a refusal of its document. Only the room writes the epoch root.

Comments

A document’s comment threads are not in the document: the room keeps them, and tells the sockets that ask about each change. The subtypes:

Subtype Name Direction Body
0 commentsSubscribe client to server A varuint: 1 to start hearing the changes, 0 to stop.
1 commentsSnapshot server to client A varString of JSON, { seq, threads }: every thread, and the sequence number of the last change. The answer to a subscribe.
2 commentsChange server to client A varString of JSON, one change: { type, thread, comment, user, at, seq }.

The room sends comment messages only to a socket that subscribed, so a client that never asks never receives one: older clients are unaffected and the generation word is unchanged. The subscription is part of the socket’s attachment, kept through hibernation, and a reconnected socket subscribes again. A client must not send a snapshot or a change: the room refuses one as malformed (1008), as it refuses any message type it does not know, so subscribe only to a room that keeps comments. Subscribes share the socket’s presence rate. Write and read them with writeCommentsSubscribe(encoder, on), writeCommentsSnapshot(encoder, snapshot), writeCommentsChange(encoder, change) and readCommentsMessage(decoder).

A change’s seq grows by one per change in the document; each thread carries the seq of its last change as rev. A client merges what it hears by rev: an older thread never replaces a newer one, and a snapshot removes the threads it lacks unless a change after its seq made them. The shipped provider subscribes while a listener watches the room (watchComments, what createCommentsClient uses).

Exported helpers

All from edytor/protocol:

Export Use
bindCrdt(Y).sync writeSyncStep1, writeSyncStep2, writeUpdate, lacks(doc, stateVector), applyRemote(doc, update, origin), writeSaved, readSaved.
frame(type, write) Build a frame: the generation word, the type, then whatever write(encoder) writes.
GENERATION, generationWord, PROTOCOL_VERSION, SCHEMA_VERSION, readProtocolVersion The generation gate.
messageSync, messageAwareness, messageAuth, messageQueryAwareness, messageSaved, messageChunk, messageComments Message types.
commentsSubscribe, commentsSnapshot, commentsChange, writeCommentsSubscribe, writeCommentsSnapshot, writeCommentsChange, readCommentsMessage The comment messages.
messageYjsSyncStep1, messageYjsSyncStep2, messageYjsUpdate Sync subtypes.
messagePermissionDenied, writePermissionDenied(encoder, reason) The auth reply to a refused write.
messageReadOnly, writeReadOnly(encoder) The read-only notice at the join.
readAwarenessEntries, writeAwarenessEntries, applyAwarenessUpdate, encodeAwarenessUpdate The awareness codec.
chunkFrame, createChunkReader, MAX_FRAME_BYTES, ChunkError Chunked frames.
CLOSE The close codes the room and its router send, by name (refused 1008, fault 1011, invalidDocument 4400, expired 4401, denied 4403, replicaTaken 4409, quota 4413; see admission).
createDecoder, readVarUint, readVarUint8Array, writeVarUint8Array The lib0 helpers for frame bodies.
GENERATION_RECORD, GenerationMismatchError Tag and check stored containers of this generation.
bindCrdt(Y).doc.purge, .horizonOf, .restoreTo The shipped room’s purge (purge(doc, facade, { at, sv }), one transaction of yours, sv a state vector the room stored before at) and restore (restoreTo(doc, facade, json)), for a room of your own; facade is bindCrdt(Y).doc.create(doc), which publishes no author.

applyRemote applies an update unless it writes a foreign schema stamp. It returns { applied, problem, discarded? }: refuse the frame when problem is set.

A minimal server loop

This is the core of a relay that stores and acknowledges, for any platform with WebSockets. Authorization, identity checks and persistence are yours to add; the shipped DocumentRoom does all of them.

import * as Y from 'edytor/crdt';
import {
  bindCrdt, frame, readProtocolVersion, createDecoder, readVarUint, readVarUint8Array,
  writeVarUint8Array, readAwarenessEntries, writeAwarenessEntries,
  messageSync, messageAwareness, messageSaved, messageYjsSyncStep1, messageYjsSyncStep2
} from 'edytor/protocol';

const { sync, createDoc } = bindCrdt(Y);
const doc = createDoc(); // restore it from your storage: Y.applyUpdate(doc, stored)
const sockets = new Set<WebSocket>();

doc.on('update', (update: Uint8Array, origin: unknown) => {
  // Store `update` durably here, then relay it.
  const out = frame(messageSync, (e) => sync.writeUpdate(e, update));
  for (const ws of sockets) if (ws !== origin) ws.send(out);
});

export function onOpen(ws: WebSocket) {
  sockets.add(ws);
  ws.send(frame(messageSync, (e) => sync.writeSyncStep1(e, doc)));
}

export function onMessage(ws: WebSocket, bytes: Uint8Array) {
  const decoder = createDecoder(bytes);
  if (!readProtocolVersion(decoder)) return ws.close(1008, 'generation');
  const type = readVarUint(decoder);
  if (type === messageSync) {
    const subtype = readVarUint(decoder);
    const payload = readVarUint8Array(decoder);
    if (subtype === messageYjsSyncStep1) {
      ws.send(frame(messageSync, (e) => sync.writeSyncStep2(e, doc, payload)));
      if (sync.lacks(doc, payload)) ws.send(frame(messageSync, (e) => sync.writeSyncStep1(e, doc)));
      return ws.send(frame(messageSaved, (e) => sync.writeSaved(e, doc)));
    }
    // A Step 2 and an Update carry the same encoding.
    const update = payload;
    const { applied, problem } = sync.applyRemote(doc, update, ws);
    if (problem) return ws.close(1008, 'refused: schema'); // final: the provider stops
    if (!applied) return ws.close(1011, 'internal error'); // a fault: the provider redials
    ws.send(frame(messageSaved, (e) => sync.writeSaved(e, doc, Y.decodeUpdate(update).ds)));
  } else if (type === messageAwareness) {
    const entries = readAwarenessEntries(readVarUint8Array(decoder));
    const out = frame(messageAwareness, (e) => writeVarUint8Array(e, writeAwarenessEntries(entries)));
    for (const other of sockets) if (other !== ws) other.send(out);
  }
}

Send large frames through chunkFrame if your platform limits message size, and remember the latest presence entry per client to greet newcomers, as the shipped room does. If your room purges, only it may write the document’s horizon root: strip a client update that writes it, as the shipped room does. Close with 1008 or a 4xxx code (other than 4401) only for a refusal the client should not retry: the provider stops dialing. Close with 1011 for a fault of your server, and refuse an upgrade you turn away by accepting it and closing it, not with an HTTP error, which a browser reports as 1006 (unreachable, redialed).

Never attach a document on the server

Do not call createDocument, loadDocument or attachDocument on the document a server holds. They publish the local author into the document’s replicated attribution data, so the server would appear as an author in every client. A server works on the raw CRDT document and the sync helpers only. Make that document with bindCrdt(Y).createDoc(), before it integrates anything, never with a bare new Y.Doc(): createDoc gives it the rule every edytor document carries (when two writers create one block id at once, the losing node keeps its text), and a document without it deletes that text where it integrates the race, then stores and relays the delete.

To read or edit the content server-side (for indexing, exports or bots), load a separate copy from the stored bytes with loadDocument and send your edits through a normal client connection, or apply them to that copy and send its update like any other client.

Compatibility

  • Peers of different generations never exchange edits: each side drops the other’s frames and reports 'protocol-mismatch'.
  • A v13 yjs peer or y-websocket server cannot join a room. An opaque relay that forwards frames byte for byte without decoding them works, but it cannot acknowledge saves or chunk frames.
  • Clients that predate messageSaved and messageChunk report them through 'message-error' and otherwise sync small documents normally.
  • A room that predates messageComments closes a socket that subscribes (1008, malformed): the comments plugin subscribes only through its client, so upgrade the room before you list the plugin.

Was this page helpful?