The room
What the DocumentRoom Durable Object enforces, how it stores and acknowledges edits, and the limits to plan for.
DocumentRoom is a Durable Object that coordinates one document: deploy one object per document, reached through routeDocumentSocket. This page describes what the room checks on every message, how it stores the document, and the settings and limits you can tune.
One room per document
routeDocumentSocket(request, env.ROOMS, documentId, authorize) opens the room named documentId (env.ROOMS.getByName(documentId)). The room keeps the live document in memory, stores it in the object’s SQLite storage, and relays edits and presence between the sockets connected to it. Clients are the ordinary createWebsocketSync or WebsocketProvider.
Admission
Every frame a socket sends is checked in this order. A refused frame is never applied, stored or relayed.
- Generation. The frame’s first word must be this build’s generation (see the protocol). A client of another edytor generation is refused before anything is decoded.
- Access. An edit from a read-only socket is dropped with a
permission-deniedreply (read-only). The socket stays open. When it joins, it gets a read-only notice instead (auth subtype1), which is not a refusal. - Identity. Content an update adds under a client id owned by another user, or under an unregistered id that already has content, is stripped from the frame, and the rest of the frame is applied: the sender stays connected. See replicas. So is a client’s content when it writes another client’s per-writer block mark (
del.<n>,wd.<n>), and a delete of another user’s mark is dropped (loggedmark; see validate edits). - Schema. An update that writes a foreign schema stamp is refused. The same check covers updates waiting for a missing dependency: when an update would release a waiting one that carries a forged stamp, the waiting updates are discarded, the new update is applied, and its sender stays connected. Honest updates discarded this way come back at their writer’s next sync.
A refused socket is closed with code 1008 and reason refused: <reason> (generation, replica, schema, malformed, identity, container). Only bytes the room cannot decode, a text frame other than the keepalive ping, or an unknown message type count as malformed. A refusal is final: the provider stops dialing instead of reconnecting. The other closes a dial can get:
| Close | When | Final? |
|---|---|---|
4403 |
authorize denied the dial: it never reaches the room, routeDocumentSocket closes it (see authorization). |
Yes |
4400 invalid document id |
The id is empty, ., .., over 256 characters or with a lone surrogate; closed before authorize runs. |
Yes |
4401 |
authorize returned { expired: true }. |
No: the one 4xxx code the provider retries, with the params it re-reads. |
4409 replica bound to another user |
The dial’s replica belongs to another user; the dial is accepted, then closed. | Yes |
A dial your own Worker turns away before routeDocumentSocket gets the same kind of close from closedSocket(code, reason), never an HTTP error, which the browser reports as 1006 and the provider redials.
A write past a quota is closed with 4413 and reason quota: <name>; it is not applied, and the provider stops dialing too.
A fault of the room itself is never a refusal. A storage error (writing a client-id registration, say) closes the socket with 1011 and reason storage failure; an error of the CRDT engine or of a send closes it with 1011 and reason internal error, after the live document is rebuilt from the stored rows. The provider redials, and the handshake resends what the room does not hold. A fault that persists (a full store, an update the engine cannot apply) does not make it redial every 100 ms: each 1011 doubles its backoff, and emits unreachable, until the room acknowledges its edits again.
Identity and presence
- The socket’s identity (
user,replica,readOnly,expiresAt) is saved in the socket’s attachment, so it survives hibernation. - Each relayed presence state that names an actor names the socket’s verified user as its
actor.id, whatever the client wrote (attribution trust). - Presence is relayed without an
Awarenessinstance (its timer would keep the object awake). A socket may publish presence only for its own client id; entries for other ids are dropped. - A new socket receives the presence of everyone connected. A socket that closes without saying goodbye is announced as gone, also after the room woke from hibernation.
- An accepted presence entry is also sent back to its sender. A client alone in a room therefore hears from it at every presence renewal and is never closed as silent.
- The room relays presence within a budget of frames,
EDYTOR_MAX_PRESENCE_FANOUTa second over all its sockets together (2,000, with a one-second burst). Within the budget, each entry goes out at once, in its own frame, as it arrives (ten editors each at 20 Hz stay within it).- Why a budget: relaying one entry costs a frame for each socket, so a room where everyone moves a caret sends a number of frames that grows with the square of its sockets. 50 editors at the client’s 20 Hz would send 50,000 a second.
- Past it, entries wait per recipient, only the newest of each client kept, and each recipient’s go out together in one frame as soon as the budget has a frame for it, at the end of a later message the room handles (it schedules no timer). Recipients are served in turn, so each one hears from every client at a lower rate rather than some not at all.
- A departure goes out at once, and drops that client’s waiting entries everywhere, so no older entry follows it out.
- What waits is in memory: after a wake it is gone, and clients renew their presence within 15 seconds.
- In the soak, 50 clients all at 20 Hz went from 5.6 million frames in two minutes, a saturated core and an acknowledgement p99 of 16 s to 0.77 million frames, a third of a core and a p99 of 43 ms.
- Keepalive: the provider sends a text
pingafter 15 seconds of silence, and the room answerspongthrough the runtime’s WebSocket auto-response (ctx.setWebSocketAutoResponse), without waking the object. The room installs the pair when it starts, unless your object already set one; it then answerspingfrom its message handler. The auto-response applies to every socket of the object, yours included. - The presence snapshot lives in memory. After a wake it refills as clients renew their presence, every 15 seconds.
- Presence has quotas of its own, which never close a socket (read-only sockets keep presence, within them):
- An entry whose state, as JSON, is larger than
EDYTOR_MAX_PRESENCE_BYTES(16 KiB) is ignored: not kept, not relayed, logged as apresencerefusal ({ user, quota: 'size', bytes, limit }). The socket’s previous entry stays. - Presence messages (entries and queries) draw from a token bucket of their own,
EDYTOR_MAX_PRESENCE_PER_SECONDper socket (50, with a ten-second burst), apart from the sync messages’ update rate. A view publishes at most every 50 ms, so only a misbehaving client reaches it. Past it, a query is dropped, and an entry is held: the socket’s newest entry waits for the next token and is relayed at the end of any socket’s next message (the room schedules no timer), with the other entries released then, in one frame. Both are loggedpresence({ user, quota: 'rate', limit }), once per burst: the socket’s first message past the rate is one entry, its next message within the rate ends the burst, andrefusalCounts.presencecounts every message past it.
- An entry whose state, as JSON, is larger than
Storage
The room stores the document in its own SQLite tables (Room internals lists them all, with the alarm’s tasks):
- Every integrated update is appended as one record, split into rows under the platform’s 2 MB row limit, in one transaction. Only then is it broadcast to the other sockets. Update records are stored in the v1 update encoding, the smaller one for an edit.
- A delete of an item the room does not hold yet (a deletion of text a restore lost, for example) is stored as a waiting record and applies when the item arrives. At most
MAX_WAITING_DELETES(1,024) ranges wait: see Waiting deletes. - After
EDYTOR_COMPACT_AFTERupdate records (500 by default), the room replaces all rows with one snapshot of its live document (plus one record of the deletes still waiting: those that applied, and those reclaimed, are dropped), atomically, once the message that made it due is acknowledged. The snapshot is the document’s healed state: what each keystroke wrote is merged and deleted content is collected, except text a peer’s undo may copy back, which the room keeps as every editing replica does. It never includes updates still waiting for a missing dependency: they are stored when it arrives. On a real editing trace (automerge-paper, 259,778 edits) that is 1.6 MB instead of the 6.7 MB a merge of every stored update keeps, and a woken room loads it in about a sixth of the time. A compaction that fails is logged as astoragerefusal and retried later; the edit is already stored and nobody is disconnected. - Every record holds an update in the engine’s one (columnar) encoding. The snapshot is then gzip-compressed in place (
CompressionStream): the trace above stores 224 KB. The compression runs after the compaction (it is asynchronous, the acknowledgement path is not) and rewrites the snapshot’s rows only while they are still the stored snapshot;compressed()resolves once it is stored. A woken room inflates the snapshot before it accepts any message. A snapshot that does not inflate refuses every socket (1008,refused: container). - Memory never runs ahead of storage. If an append fails, nothing is relayed or acknowledged, the live document is rebuilt from the stored rows as a restart would, and the sender’s socket is closed with
1011. Its provider reconnects and the handshake resends the edit. If the append of the room’s own edit fails (transact),transactthrows the storage error and the edit is lost: nothing resends it.- After the room’s own failed append, the room rebuilds from its rows too, but unlike after a client’s failed append it keeps the updates it holds waiting for a missing dependency. A rebuild replaces the room’s
docandfacade. - If reading the rows fails, during that rebuild or when the room starts, it has no document until a dial reads them again: sockets are closed with
1011(room unavailable), their providers redial, and each dial retries the read. - The rebuild happens once per outage. Until storage takes an append again, the room asks it first (a probe append, rolled back) before it applies any frame that would write, and closes the sender with
1011without applying the frame, so its memory stays what it stored. A frame that writes nothing (a join of a client with nothing new) is served from memory meanwhile. The outage ends at the first probe or append that succeeds.
- After the room’s own failed append, the room rebuilds from its rows too, but unlike after a client’s failed append it keeps the updates it holds waiting for a missing dependency. A rebuild replaces the room’s
- A woken room rebuilds the document from its rows before it accepts any message. A room whose rows read but cannot be restored (another edytor generation, a torn record) refuses every socket with
1008and reasonrefused: container, and the object keeps answering. See generation cutover. - The client-id registrations used for identity are stored next to the document, and handed to
onSavewith it. Compaction drops the registrations of ids that hold no content and belong to no open socket (a page that loaded and never wrote); such an id registers again at its next dial or write. - Deleted content is kept for undo and for edits made concurrently elsewhere, then purged: content deleted more than 30 days ago (
EDYTOR_PURGE_AFTER_DAYS) is removed from the rows and from every client by the room’s own edit. A version history (two versions a day, in KV, R2, the room’s own storage or a store of your own) keeps those 30 days restorable. See History and purge.
Waiting deletes
Any writer may delete any content, but storing deletes that wait adds durability and a per-update cost, which a bound caps. Every update the room applies re-reads the waiting deletes, so at most MAX_WAITING_DELETES (1,024) ranges wait.
- Past the bound. A frame whose deletes of unheld items would pass it has those deletes dropped, logged as a
waitingrefusal, and the rest of the frame applied. That includes deletes of the frame’s own items that wait (for a missing origin) and the attribute values its waiting changes replace: when they take the room past the limit, all of the frame’s new waiting deletes are dropped the same way. - The sender sees it. The acknowledgement leaves the dropped deletes out, so the sender’s
savedstaysfalseand its next connection resends them; until one is stored, the deleted content shows if its author returns. - How long one waits. A waiting delete lasts until its item arrives, but compaction reclaims those of a client id that no user registered and no open socket holds (a made-up id: nothing can deliver its items), in memory too; the registrations of ids with waiting deletes are kept.
- Registered ids are kept. Deletes of a registered id’s items are kept, as they may be a relayer’s deletes of text its author will bring back, so a writer can still fill the limit with deletes of its own or another user’s future items. Frames past the limit then keep logging
waitingrefusals: calldropWaitingDeletes()to empty it. - Restores. An author missing from the restored registry (a snapshot saved before that author first dialed, or a bare
updatewith noreplicas) is unknown until it dials again, so a compaction before then reclaims the deletes of that author’s text. Waiting deletes armonSavelike any edit, and a room restored from itsonSavecopy stores the waiting deletes it carries as a waiting record again, so they are acknowledged and reclaimed the same way.
Edge cases.
- The attribute value that an attribute change of the same update replaces, while that change itself still waits for a missing dependency, is not a waiting record: that delete waits with the change, in memory, so the attribute is never emptied.
- If a later update would make such a delete remove the schema stamp, every waiting delete is discarded (the room also drops a pending forged stamp that way).
Store before acknowledge
Every sync message a client sends is answered, after the room has stored what it integrated, with a saved frame: the room’s state vector and, when the message deleted content, the deletes of that message the room stored, applied or waiting for their items (never one it dropped or keeps in memory only). The provider turns these into its saved and unsaved state. A client shows “saved” only for what is on disk.
A client that catches up receives what the room has stored, never updates still waiting for a missing dependency.
Catch-up of large documents
Cloudflare limits a WebSocket message to 32 MiB. A frame the room sends that is larger than EDYTOR_MAX_FRAME_BYTES (32 MiB by default) goes out as a sequence of chunks, and the provider applies it only once the sequence is complete. Smaller frames are sent whole, so clients without chunk support still sync small documents.
The provider does the same the other way (its maxFrameBytes option, 32 MiB by default): an offline backlog larger than one message reaches the room at reconnect as a chunk sequence, which the room reassembles per socket, then admits like any frame (the frame quota applies to the whole, and a sequence announcing more is refused at its first chunk). With the default quotas no frame the room accepts needs it (the frame quota is below one message); it matters once you raise them. A chunk with no sequence started (the room woke between two chunks and lost the ones it had) closes the socket with 1011 (chunk sequence lost): the provider redials and resends.
The room’s memory for chunks is bounded:
- A sequence is assembled into one buffer of the size its first chunk announces, allocated then, never into parts copied at the end.
- The sequences in flight on all sockets together take at most
EDYTOR_MAX_BUFFERED_BYTES(twice the frame quota by default, never less than one frame). A sequence that would pass it closes its socket with1011(room busy), logged as aquotarefusal (quota: 'buffer'): the room is busy, the sender is not at fault, so its provider redials with backoff and resends. A closed socket releases its buffer. - A read-only socket’s sequence is never buffered: its start is answered with the
read-onlydenial, its chunks are dropped, and the socket stays, as for any write it sends. - A sequence’s start counts against the update rate, like a sync message.
metrics().buffered reports the sequences in flight and the bytes buffered for them.
Quotas
Three limits keep one client from filling the object’s storage or keeping it busy. Each is a vars setting (and an attachRoom option), and a socket past one is closed with 4413 (quota: document, quota: rate or quota: frame), its write not applied, logged as a quota refusal with { user, quota, … }:
| Variable | Option | Default | Limit |
|---|---|---|---|
EDYTOR_MAX_DOCUMENT_BYTES |
maxDocumentBytes |
2,097,152 (2 MiB) | The document’s size: what its records hold, uncompressed, plus the updates it holds waiting for a dependency. A frame that would take it past the limit is refused once the room has compacted its update records (they collapse into the live state). A frame that deletes at least as much content as it adds always applies, so a full document can still be trimmed. |
EDYTOR_MAX_UPDATES_PER_SECOND |
maxUpdatesPerSecond |
50 | Sync messages per second and socket, sustained, with a burst of ten seconds’ worth (a reconnect, a paste). The editor sends one per transaction; a script driving the provider should batch its edits into transactions. |
EDYTOR_MAX_INBOUND_FRAME_BYTES |
maxInboundFrameBytes |
4,194,304 (4 MiB) | One frame a socket sends, reassembled from its chunks; a chunked frame announcing more is refused at its first chunk, before anything is buffered. It must hold a whole document (a client seeding an empty room sends it in one frame): raise it with the document quota. The platform caps one message at 32 MiB. |
The quota closes the socket rather than dropping the frame: the sender’s later edits build on the one dropped, so the room would hold them waiting, in memory, for good, and never acknowledge them. It is final rather than redialed, because a redial resends the same edits through the handshake and meets the same quota. The edits stay in the client’s document and its local copy, and saved stays false.
Size the document quota for the room’s memory. An isolate has 128 MB, and the live document takes far more memory than it stores. Measured with bench/room-memory.mjs (the room’s document and its block index, loaded from a compacted snapshot, in Node):
| Document | Stored | Heap | Heap per stored byte |
|---|---|---|---|
| 10,000 blocks of one short line (24 characters) | 0.93 MB | 65 MB | 73 |
| 2,000 typed paragraphs (60 characters, a transaction a word) | 0.30 MB | 15 MB | 53 |
| 2,000 paragraphs with a bold and an italic run | 1.0 MB | 20 MB | 20 |
| 5,000 paragraphs of prose (400 characters) | 2.3 MB | 32 MB | 15 |
Since schema generation 10 the stored encoding writes each string and each client id once, so the same documents store a third to a half less, and take a tenth less memory: the bytes of heap per stored byte went up (they were 18 to 46; short blocks now read 72 to 77 from one run to the next). The default stays 2 MiB, so every document a room admitted before is admitted, smaller. At 2 MiB the worst case, a long list of short blocks (about 23,000 of them), takes about 150 to 160 MB in Node; a quota of 1.25 MiB keeps it near 100 MB, as the default did before.
Workers run V8 with pointer compression and Node does not, so a room needs less than these figures: the soak read about 20 bytes of heap per stored byte in workerd, through its inspector, for a document 50 writers typed, split and merged (18 in its run near the quota; 27 in Node for typed paragraphs). That is the live heap after a full collection: before one, the soak saw peaks several times higher. A document of long text can take a higher quota: at 20 bytes a byte, 4 MiB is about 84 MB. Leave room for the rest: a compaction and a catch-up each encode the whole document once more, and chunked frames take up to EDYTOR_MAX_BUFFERED_BYTES. A room past its memory is reset by the platform and restarts from its rows, so a quota set too high shows as sockets closing and redialing, not as a refusal. Your own server edits (transact) are not limited.
The quota is the largest document you accept. A busy document’s stored size is mostly its editing history, not its content, and the room gives that back by re-seeding it: a document whose content itself nears the quota is what the quota refuses. Pick it per deployment from the documents you expect (the table above; see limitations), and watch for documents nearing it. When the document reaches EDYTOR_DOCUMENT_WARNING of the quota (documentWarning, 0.8 by default, above 0 and at most 1), the room logs size ({ bytes, limit, share }) once, and again only after the document went back under it (a compaction, a purge) or the room woke. metrics() reports documentBytes against documentLimit. What to do then is yours: raise the quota, ask users to split the page, or archive it.
Inline images count at their size. Without an upload, the image plugin stores each image in the document as a data: URL of up to 1 MiB (MAX_INLINE_IMAGE_BYTES), so two or three of them reach the default document quota, and the 4413 that follows is final: the provider stops and every later edit stays unsaved. An inline image takes about its own size in memory, not the 15 to 77 times of the table, so for documents that hold images give the image plugin an upload that stores the file and answers its URL, or raise EDYTOR_MAX_DOCUMENT_BYTES (and the frame quota) by the images’ size.
Re-seeding
Every edit leaves CRDT structure in the document (the identity of each run of typed text, the deleted text’s place, each block ever registered, attribution), so a busy document grows with its edits, not its content; the purge gives back about a tenth of it. The room rebuilds such a document from its content: the same blocks (ids, kinds, data, text, marks, the document’s own data) and their attribution (createdBy, the contributors, lastChangedBy), with fresh identities, so it stores what its content takes. On the soak’s document, a fresh seed stores a quarter of it after 4,000 edits and a fifth after 16,000. This is a re-seed, and each one starts the document’s next epoch (epoch on the room and in metrics()).
When it runs. Once the stored document passes half its quota (at), a re-seed is due when nobody has written for ten minutes (idleMs); each write pushes it back. It runs only when the new document gives back at least 40 % (gain); when it would not, the room logs { "edytor": "reseed", "skipped": "gain" } and does not try again before the document grew by another quarter of at. Past 80 % of the quota (force) it waits for no idle period, still not before that growth. One that fails is due again at the first change a minute later or more. While a shared suggestion is pending it waits for the next idle period; past force it runs and drops them (suggestions in its log entry). reseed(), also over RPC, runs one now, whatever it gives back. Set reseed: { at, idleMs, gain, force, graceDays } (ReseedOptions) to tune it, reseed: false (EDYTOR_RESEED=off, or reseeding() in a DocumentRoom subclass) to never re-seed.
What a client sees. The room closes every socket with 4410, its reason the new epoch. A client with nothing unsaved hears it as a refusal (SyncRefusedError with code 4410, through onSyncRefused): its document belongs to the previous epoch and never syncs again. An <Edytor> that owns its document (room with server, or sync) builds itself again on a new one, which loads the new epoch: the selection, the scroll position and the undo history start over; with your own document, create a new one when that refusal comes. createWebsocketSync deletes its local copy then (once every tab let go of it). The version history is the document’s JSON, so its versions still restore.
Offline edits. A client names its document’s epoch when it dials (?epoch=<n>, or new for a document that holds nothing yet; the document holds it, written by the re-seed). With a local copy, it dials once the copy is loaded, so the epoch it names is the copy’s. A client of an earlier epoch that may write is told the room’s and hands over its whole document: the edits the room did not have when it re-seeded are merged into a copy of the current document, and what that writes reaches the document as one update of that user, through every check an update meets: the quotas (past the document quota it is refused 4413, and the edits stay in the client’s copy), its access at that moment, validate and the locks, attribution (the blocks it changed get it as lastChangedBy, its history slot records it). Then the room closes it 4410 like the others. A client that still had an edit in flight when the room re-seeded does the same: it dials again to hand it over.
What merges. Text edits merge as they would have without the re-seed; when both sides split, merged or deleted the same blocks meanwhile, a few characters can land otherwise (see limitations). Its pending suggestions are dropped, never accepted. What a user’s hand-overs merged is kept per user: a second copy of the same edits (another tab) merges nothing, and a tab’s edits that build on another tab’s still land; another user’s upload never counts as theirs. A rebase rebuilds the document in the room, so it runs at most once per user every ten seconds: past that, the room closes the dial 1011 before asking for anything, and the provider dials again later. A hand-over it cannot read is dropped (4410); a fault of the room’s own storage meanwhile closes 1011, and the provider hands over again. One still in flight when the room re-seeds again merges all the same. A view built again after a re-seed loads the room’s document: it never seeds its value, as with requireHydration. Each user’s name and color come along to the new epoch.
What the room keeps. A reader or a commenter has no edit to merge, and a suggest-only user’s edits are all suggestions: closed 4410 at once (a suggest-only user’s offline work is dropped). The room keeps what the merge needs, the cut (about 7 to 8 % of the document’s stored size), for 30 days (graceDays), then the re-seed’s task drops it. A client of an epoch older than the cuts kept, or of a later one than the room’s (a backup restored over it), is closed 4410 at once: its edits cannot merge, and it loads the room’s document. Nothing is relayed to a socket of an older epoch, and a socket of the current epoch that sends an older epoch’s content (a tab that took it from another tab before reloading) is closed 4410 too.
Upgrading. A client before 7.0 names no epoch: a room that re-seeded refuses it (1008, refused: epoch) and its edits stay in its local copy. Update the clients first: they work with rooms that never re-seeded. A host Worker that builds its own upgrade request must keep the dial’s query string (routeDocumentSocket forwards the request’s URL). Only the room writes the epoch: a client’s write of it is stripped (mark).
Hibernation
The room schedules no timers: each entry point runs with setTimeout and setInterval disabled (the exported noTimers helper), because a Durable Object with a pending timer cannot hibernate. Sockets stay connected while the object sleeps; identity comes back from the socket attachments and the document from storage.
Block roles
Edits the room makes itself (transact) obey the block roles of the bundled rich-text, code, image and columns plugins (defaultSemantics): an edit those roles refuse in a view, such as merging into a divider, moving a code line out of its code block, merging a code block’s first line into it or moving a block directly into a columns layout, is refused on the server, and the room shows a columns layout by the same rules as a view.
- Keyboard policies stay in the view. What a plugin adds on top (Delete before a code block removes an empty block and does nothing otherwise) belongs to the view: the server applies the plain operation.
- Other kinds. Rooms whose clients define other void or island kinds pass their own rules; see Block roles on the server.
- Client edits are relayed as they come: the room does not check their roles. To check client edits against your own rules (locks, kinds, sizes), validate them: the room undoes a frame you deny.
Settings
Set these as vars in wrangler.jsonc. All are optional.
| Variable | Default | Description |
|---|---|---|
EDYTOR_MAX_ROW_BYTES |
1,995,904 | Largest stored row. Can only be lowered. |
EDYTOR_MAX_FRAME_BYTES |
33,554,432 (32 MiB) | Largest frame sent whole; larger ones are chunked. Can only be lowered. |
EDYTOR_COMPACT_AFTER |
500 | Update records before the room compacts its rows into one snapshot. |
EDYTOR_SAVE_AFTER |
2000 | ms between the first unsaved change and onSave. |
EDYTOR_MAX_DOCUMENT_BYTES |
2,097,152 (2 MiB) | The document quota. Can be raised. |
EDYTOR_DOCUMENT_WARNING |
0.8 | The share of the document quota at which the room logs size (quotas). |
EDYTOR_MAX_UPDATES_PER_SECOND |
50 | The update rate quota, per socket. Can be raised. |
EDYTOR_MAX_INBOUND_FRAME_BYTES |
4,194,304 (4 MiB) | The frame quota. Can be raised. |
EDYTOR_MAX_PRESENCE_BYTES |
16,384 (16 KiB) | The largest presence state a socket may publish, as JSON. |
EDYTOR_MAX_PRESENCE_PER_SECOND |
50 | Presence messages per second and socket, a bucket apart from the update rate (presence). |
EDYTOR_MAX_PRESENCE_FANOUT |
2,000 | Presence frames per second the room sends over all its sockets; past it, entries wait per recipient and go out together (presence). |
EDYTOR_MAX_COMMENT_BYTES |
4,194,304 (4 MiB) | The bytes the room’s comments hold at most; a thread or reply past it is refused 413. |
EDYTOR_MAX_COMMENT_REQUESTS_PER_SECOND |
2 | Comment requests per second and user over HTTP, and comment messages per socket, a ten-second burst (comments). |
EDYTOR_MAX_BUFFERED_BYTES |
twice the frame quota | The chunk sequences in flight on all sockets together (catch-up). Never below the frame quota. |
EDYTOR_LOG |
off silences the room’s log. |
|
EDYTOR_PURGE_AFTER_DAYS |
the retention, else 30 | Days after which deleted content is purged; off never purges. |
EDYTOR_HISTORY_RETENTION_DAYS |
30 | Days a version is kept. |
EDYTOR_HISTORY_TIME_ZONE |
UTC |
The IANA time zone of the history’s half-day slots. |
EDYTOR_LOCKS |
The data key that locks a block to the user it names, e.g. lockedBy. Unset: no locks. |
|
EDYTOR_RESEED |
off never re-seeds a document. |
The rooms blocks move to are the EDYTOR_ROOMS binding, a Durable Object namespace (usually the room’s own). The history’s store is EDYTOR_HISTORY: a KV namespace or an R2 bucket binding, or the variable "room" for the room’s own storage. Without it, the room keeps no history.
{
"vars": { "EDYTOR_COMPACT_AFTER": "200" }
}
Values that are not positive integers are ignored. Pass your own Env type to the class if you extend it: DocumentRoom<Env>. With attachRoom, pass the same settings as options. To load, save or edit the document from your own code, see Your own Durable Object.
Compaction over RPC
compact() is also callable over Durable Object RPC, for example from an admin route or your own scheduled job. It returns the number of rows left:
const { rows } = await env.ROOMS.getByName(documentId).compact();
Compaction stores the live document as one snapshot. It collects deleted content the CRDT no longer needs, keeps what it does need (the structure of deleted items, and text a peer’s undo may copy back), and never loses an edit: a room restarted from its rows holds exactly the document it held before.
dropWaitingDeletes(), also over RPC, drops every waiting delete, stored or kept in memory with a waiting attribute change, and returns how many ranges it dropped. Use it when waiting refusals show the limit is full. A dropped delete’s content shows again if its author returns; an attribute value a waiting change replaces is still replaced if that change applies.
const { ranges } = await env.ROOMS.getByName(documentId).dropWaitingDeletes();
Diagnostics
refusals holds the newest 100 refusals (MAX_REFUSALS) of the running instance, each with its reason and detail; refusalCounts counts every refusal by reason. Both live in memory and restart empty; read them inside the object, or return them from an RPC method of your own.
| Reason | What it records | detail |
|---|---|---|
storage |
A failed append, compaction or registration. | |
internal |
An error of the engine or a send. | |
waiting |
The deletes of unheld items a frame carried past MAX_WAITING_DELETES; the rest of the frame applied. |
{ user, ranges } |
quota |
A write past a quota: the socket closed 4413, or 1011 for quota: 'buffer'. |
{ user, quota, … } |
presence |
A presence entry past the size quota, or the start of a socket’s burst of presence messages past the rate: one entry per burst, refusalCounts.presence counting each message. The socket stays; see presence. |
{ user, quota: 'size' | 'rate', … } |
mark |
Another writer’s per-writer block marks a frame wrote or deleted, stripped. | { user, writers, ranges } |
forged |
Attribution entries a frame may not write or delete: another replica’s c/<n> binding, another user’s u/<id> profile; removed, see attribution trust. |
{ user, keys } |
suggest-only |
What a suggest-only user’s frame wrote that is no suggestion’s, stripped (writers) or dropped (deletes); the rest applied, see access levels. | { user, writers, ranges } |
history |
A version the room skipped, or a time zone it does not know. | { key, bytes, limit } past its store’s value limit, { key, error } when the store failed |
epoch |
A dial of an epoch the room does not serve: none, in a room that re-seeded (closed 1008), or one older than the cuts kept or later than the room’s (closed 4410); or an older epoch’s content on a socket of the current one (closed 4410). |
{ user, epoch, room }, { user, epoch } |
Four more entries are not refusals:
| Entry | What it records | detail |
|---|---|---|
denied |
A frame validate denied, undone by the room. |
{ user, touched } |
access |
The room ended or changed a user’s access: expired, closed (closeUser), or the level setAccess set (read, comment, suggest, write, none); see revoking access. |
{ user, access, sockets } |
orphan |
An unowned id claimed by its dial (after a restore without a registry, or a relay). | |
relayed |
Content under an id the room held nothing of, delivered by another replica and kept unowned (see replicas). |
Metrics and log
metrics(), also over RPC, returns the room’s counters since the instance started (a wake starts them again):
| Field | Content |
|---|---|
documentBytes |
The document’s size as the document quota measures it: its records, uncompressed. |
documentLimit |
The document quota (maxDocumentBytes). |
storedBytes, records, rows |
What the rows take in storage (a compressed snapshot at its compressed size), the logical records (the generation record included) and their rows. |
updateRecords |
Update records since the last compaction (it runs at EDYTOR_COMPACT_AFTER). |
waitingBytes |
Updates held in memory waiting for a missing dependency. |
sockets |
The document’s open sockets. |
compaction |
Compactions run: count, totalMs, maxMs, lastMs, and lastBytes (the last snapshot’s size). |
fold |
Client sync frames folded (admitted, applied, stored, relayed, validated, acknowledged): count, totalMs, maxMs, lastMs. |
fanOut |
Frames sent to other sockets (messages, a chunk counting as one) and their bytes. |
quotaHits, validationDenials |
Writes refused by a quota, frames validate denied. |
refusals |
Every refusal by reason (as refusalCounts). |
history |
Versions written and skipped, and the last key written. |
purge |
Purges run, the last horizon purged, and what they deleted (removed, emptied, marks, records, candidates, claims, merged). |
epoch |
How many times the document was re-seeded. |
since |
When the instance started. |
const metrics = await env.ROOMS.getByName(documentId).metrics();
lastUpdated(), also over RPC, returns when the room last stored a change (ms since the epoch; null for a room that has stored none since it began recording them, with this release), from one row of its meta table (edytor_meta with attachRoom). Clients reach it without opening the document through lastUpdated, which routeDocumentSocket answers.
Times are what the runtime’s clock reports. Workers advance the clock only across I/O, so synchronous work such as a compaction often reads 0 ms in production; sizes and counts are exact.
The room also logs one JSON line per event below with console.log, which Workers Logs and wrangler tail collect. Set EDYTOR_LOG to off to silence it, override protected log(entry) in a DocumentRoom subclass to send entries elsewhere, or pass log to attachRoom (false for none).
| Event | Entry |
|---|---|
| A compaction | { "edytor": "compaction", "ms", "bytes", "records", "rows" } |
| A quota hit | { "edytor": "quota", "user", "quota" } |
The document reaching EDYTOR_DOCUMENT_WARNING of its quota |
{ "edytor": "size", "bytes", "limit", "share" } |
| A denial | { "edytor": "denied", "user", "touched" } |
| A fault | { "edytor": "fault", "reason", "detail" } |
| A version written | { "edytor": "history", … } |
| A restore | { "edytor": "restore", … } |
| A client whose roles differ from the room’s (development builds; see Block roles on the server) | { "edytor": "semantics", "user", "kinds" } |
| A purge (see History and purge) | { "edytor": "purge", … } |
| A re-seed, or one skipped | { "edytor": "reseed", "epoch", "before", "after", "suggestions", "ms" }; { "edytor": "reseed", "skipped": "gain", "bytes", "estimate" } or { …, "skipped": "suggestions", "bytes", "pending" } |
| An offline client’s edits merged after a re-seed | { "edytor": "rebase", "user", "from", "epoch", "status", "bytes", "ms" } (status: applied, what validation reverted or attribution stripped included; noop when it held nothing new; refused when the admission closed it, a quota or a fault, or the socket was read-only; failed when the room could not read or rebase it) |
Load
The repository’s soak harness (bench/soak/) runs many real clients against one room and injects faults while they edit. Each client is a headless document (createDocument) on the shipped WebsocketProvider. It types at its caret, presses Backspace and Enter, merges, moves blocks, sets marks and data, and cuts text (the mix is bench/soak/edits.mjs). It also publishes presence after each edit, and at 20 Hz while it drags a selection (10 % of its time). Clients drop their socket about once a minute (a tenth of those drops are page loads, with a new replica synced from nothing), and go offline for 10 to 60 seconds about twice an hour, still editing, then replay their edits. Every run also forces one offline session and one page load, so even a short run replays and reloads.
Locally the room is the shipped DocumentRoom in Miniflare, with the default quotas, the default compaction threshold and its history in its own storage. Faults run on a schedule:
| Fault | How | Every |
|---|---|---|
| Forced compaction | compact() over RPC, during traffic |
90 s |
| Hibernation | The instance’s memory dropped, sockets kept (workerd:unsafe eviction) |
4 min |
| Eviction | The instance dropped with its sockets closed | 6 min |
| Reset | ctx.abort(), as on an exceeded limit or a deploy |
7 min |
| Storage outage | Every update append fails, for 2 s (a SQLite trigger) | 5 min |
| Failing compaction | Every compaction fails, for 30 s | 8 min |
The harness measures:
- the ack latency of every edit outside an offline session: from the edit to the room’s
messageSavedthat covers it, across reconnects. Edits made while the socket was connected and synced are reported apart from those made while it was down or not yet synced (a dropped socket, or a fault’s redial and backoff), which wait for the next connection; - the replay time of each offline session: from the reconnect to the ack of its last edit;
- the room’s
metrics(); - the isolate’s heap, read through workerd’s inspector after a full garbage collection (and, every 10 s, without one), and workerd’s resident memory;
- the final document’s stored size, against the same JSON seeded fresh and against the document after the room’s purge at a horizon of now (
bench/soak/size.mjs).
At the end every client comes online and settles. Convergence holds when every client’s JSON, the room’s own read() and a fresh joiner’s document are equal, and still equal after the room is evicted and loaded from what it stored.
Results
Measured with 50 clients for 20 minutes, on an Apple M4 Pro laptop shared with other jobs (load average 12 to 55, median 25). Room and clients ran on the same machine. Sizes are in MB of 10^6 bytes (the reports store MiB). The run is soak-wu16-followup-20min.json, after the three changes below; the figures of the first run, before them, are in parentheses:
| Measure | Result |
|---|---|
| Edits | 22,615 applied, about 19 a second; 3.2 million frames fanned out, 1.15 GB (6.5 million, 1.2 GB: presence now waits and goes out together past its budget) |
| Ack latency, edits made while connected | 21,639 edits: p50 5.4 ms, p90 20 ms, p99 613 ms, p99.9 3.2 s, max 4.5 s (p99 691 ms, max 57 s) |
| Ack latency, edits made while disconnected | 666 edits (a dropped socket, a fault’s redial and backoff; offline sessions apart): p50 880 ms, p99 3.0 s, max 3.6 s (max 60 s) |
| Ack latency, 10 s windows with no fault | Edits made while connected, in the 30 windows that no fault started in the last 40 s of (70 s for a failing compaction): p99 51 ms in the median window, 688 ms at worst (1.3 s) |
| Offline replays | 22 sessions: p50 58 ms, max 107 ms |
| Compaction | 51 automatic, p50 15 ms, max 62 ms. 13 forced over RPC, 43 ms p50 and 139 ms max round trip |
| Room heap, live (after a full GC) | 36 MB with a 1.86 MB document once the clients were idle: about 19 bytes of heap per stored byte; 33 MB after an eviction while every client caught up |
| Room heap, before collection | 125 MB at its peak, with 171 MB allocated by V8 (121 MB, 190 MB) |
| Client heap (after a full GC) | 56 MB a client at 1.86 MB stored, about 30 bytes a byte |
| Stored size | 1.83 MB for 402 blocks and 20,132 characters, 74 bytes an edit; the same JSON seeded fresh: 0.16 MB; purged at a horizon of now: 1.52 MB, 17 % less (5 % before the purge dropped merged blocks’ attribution) |
| Convergence | 50 of 50 clients, the room, a joiner and the reloaded room equal |
| Restarts | 12 (4 hibernations, 3 evictions, 3 resets, the final eviction, and the first start), all injected |
Every fault healed without intervention:
- Hibernation, eviction and reset. Each one closed or kept the sockets, the providers redialed, and the room loaded again from its rows. The windows after them show p99 20 ms to 2.7 s.
- Storage outage. Each 2 s outage rebuilt the document once (four rebuilds in the run; before, one for every refused frame, about 50 an outage). The other frames that would write were closed
1011unapplied, 337 of them, as the providers redialed and resent with backoff: the window of each outage shows p99 3.0 to 3.9 s, the next one under 500 ms. - Failing compaction. Edits were stored and acknowledged throughout, and the next compaction after the fault went through.
No edit waited more than 4.5 s (the first run had edits unsaved for a minute after the last reset). The room did not refuse anything (quotaHits 0). Its only refusals were the injected storage faults and one relayed note.
A run near the document quota (50 clients at 2 edits a second each for 5 minutes, no faults; 25,056 edits, before the changes) ended at 1.93 MB with 34 MB of live heap once the clients were idle (about 18 bytes a byte), and 54 MB right after an eviction while all 50 clients caught up again. Ack latency was p50 4.2 ms and p99 153 ms, and the run converged.
Presence at 20 Hz for everyone. Rerun on the same machine (50 clients all dragging at 20 Hz, 2 edits a second each, 2 minutes, no faults; soak-wu16-followup-presence-*.json): before the fan-out budget, the room relayed 5.6 million frames, its core ran at 80 to 100 %, and acknowledgements took p50 22 ms, p99 16 s, max 28 s. With it, 0.77 million frames, the core at about 35 %, and p50 3.8 ms, p99 43 ms, max 0.8 s. Both converged.
Memory. The live heap stays under 70 % of an isolate’s 128 MB in every run, but only the heap after a full collection. Before collection it reached 125 MB (171 MB allocated) in the 20-minute run: past 70 %, and, allocated, past the limit itself. That peak did not come from rebuilding the document: with one rebuild per outage it came all the same, and it grew over the 30 s before the room hibernated (42, 79, 104, 125 MB) as V8 deferred its major collection, which Miniflare’s isolate, with no memory limit, lets it do. Whether a deployed room, which V8 collects under its limit, rides out such a peak is not verified. A room the platform resets restarts from its rows, as the injected resets above did.
What a client holds. A soak client takes about 30 bytes of heap per stored byte after a full collection (56 MB at 1.86 MB). bench/soak/heap.mjs measures it layer by layer on a soak document (16 writers, 6,000 edits, 0.34 MB stored): the engine’s document takes 6.9 MB of the replica’s 8.9 MB, about 440 bytes for each of its 16,000 structs (the item, its id, its content and the content’s array, the types’ maps), and the block index 2.4 MB. The engine’s struct layout is the owner to shrink; no change short of the engine’s own representation would gain much.
Findings to plan around:
- Stored size follows edits, not content. The 20-minute document stores 1.83 MB, about 74 bytes for each edit, while its JSON seeded fresh takes 0.16 MB. That growth is the CRDT structure each edit leaves, and most of it stays for the life of the document; no deleted item still holds its content (the engine keeps only a placeholder).
bench/soak/size.mjsreplays the mix in memory and reports what each kind of struct costs (bytes): with 50 writers, about 60 bytes an edit when this run was made, of which the attribution records’ contributor entries took 29 %, the text’s mark operations 16 %, the text 11 % and the replacedlastChangedByvalues 8 %. Since schema generation 10, which writes each string and each client id once per update, the same mix stores about 22 bytes an edit (55 just before it).- Each Enter registers a block that stays after a merge (the block it merged into shows its text), and a split copies the contributors of the block it splits, so in a document many people edit every block lists most of them. The purge now deletes the attribution records of blocks merged away before its horizon, which halves those entries: at a horizon of now it gives back 17 % of the 20-minute document (12 % with the in-memory mix), from 5 %.
- A document that many people edit all day can still reach the default 2 MiB quota in days, whatever its visible length. Raise
EDYTOR_MAX_DOCUMENT_BYTESfor such documents: the live heap above leaves room. A shorterEDYTOR_PURGE_AFTER_DAYSgives back less than it costs in history and undo.
- Presence costs grow with the square of the room. The fan-out budget keeps the room’s acknowledgements fast, but each client still decodes every other client’s presence: in a crowded room, raise the views’
throttleor narrow what they share (see sharing less at scale).
Run it
pnpm soak:smoke # 6 clients, 90 s, every fault on a compressed schedule, one offline session and one page load
pnpm soak # 50 clients, 20 minutes, the schedule above
node bench/soak/run.mjs --clients 100 --minutes 60 --ops 0.4 --drag 0.1 --faults on
node bench/soak/run.mjs --clients 50 --minutes 2 --ops 2 --drag 1 --faults off # presence at 20 Hz for everyone
node bench/soak/size.mjs --edits 4000 --without split,merge # the stored size, in memory
node --expose-gc bench/soak/heap.mjs --replicas 16 --edits 6000 # a client's heap, layer by layer
run.mjs starts the room locally unless it gets --server. It writes its report to bench/results/soak-<time>.json: the settings, the latency percentiles, a sample every 10 s (room heap, CPU, memory, document size, records, sockets, compactions, refusals), the faults and their answers, the forced offline session and page load, the room’s starts, the stored size and the convergence. A run that does not converge also writes every client’s state vector and the distinct documents next to it. It exits with code 1 when it does not converge, or when the forced offline session or page load did not happen. The reports of the runs above are bench/results/soak-wu16-followup-20min.json (and the first run, soak-wu16-20min.json), soak-wu16-followup-presence-baseline.json and soak-wu16-followup-presence-worst.json (before and after the presence budget), soak-wu16-near-quota.json, and soak-size-*.json for the in-memory sizes (soak-size-50-replicas.json with the cost of each kind of struct). The near-quota report comes from an earlier version of the harness: it keeps every edit’s latency in one figure, made while connected, and measures no stored size.
A multi-hour soak on staging
A laptop shares its cores with the clients, so run the long soak against a staging deployment:
- Deploy the soak Worker to a staging account, never to production:
bench/soak/wrangler.jsoncdeploysbench/soak/worker.ts, the shipped room asSoakRoom, with routes for metrics, its document, compaction, storage faults andctx.abort(). First set a token. Without it the Worker refuses every request:wrangler secret put SOAK_TOKEN -c bench/soak/wrangler.jsonc wrangler deploy -c bench/soak/wrangler.jsonc - From one or more machines near the region (100 clients takes about 8 GB for the clients’ documents), run:
The operator routes get the token in annode bench/soak/run.mjs --server wss://edytor-soak.<account>.workers.dev/rooms \ --token <SOAK_TOKEN> --room soak-staging-1 --clients 100 --threads 8 --minutes 360Authorizationheader. A socket cannot send one, so its URL carries?token=, and the token shows in request logs:wrangler tailprints it, and Workers Logs, which the configuration leaves off, would store it. Use a fresh token for each soak. Against a deployed room, the hibernation and eviction faults are skipped (they need Miniflare); the reset, both storage faults and forced compaction run. The heap is not readable there: read the room’s memory and CPU in the Worker’s metrics in the dashboard, and its log withwrangler tail -c bench/soak/wrangler.jsonc. - Compare the report with the results above. Pass when it converges, nothing restarts but the injected faults,
quotaHitsis 0 unless the document reached its quota, and ack p99 outside fault windows stays under a second. - Delete the staging Worker afterwards (
wrangler delete -c bench/soak/wrangler.jsonc).
Generation cutover
A container of another schema generation is refused (1008, refused: container); bytes are never converted in place, and versions in the history written by another generation do not read. To move such a document across:
- Before deploying the new version, make sure
onSavehas mirrored the document as JSON (value). - Deploy. Rooms of the old generation refuse every socket (
1008,refused: container);failureis aGenerationMismatchError. - Call
reset()on the room, for example from an admin route. It deletes the document’s tables in one transaction and starts again fromonLoad. - Have
onLoadreturn the saved JSON. The room seeds it as the new generation’s document.
await env.ROOMS.getByName(documentId).reset();
reset() throws for any other room: it only replaces a container of another generation.
Limits
- A document is capped at 2 MiB stored by default (
EDYTOR_MAX_DOCUMENT_BYTES), about 4,500 paragraphs of prose or 23,000 one-line blocks: the room’s memory, not its storage, is the limit (see quotas). One frame is capped at 4 MiB (EDYTOR_MAX_INBOUND_FRAME_BYTES), and one message at 32 MiB by the platform; both sides chunk larger frames. The image plugin keeps inline images under 1 MiB: upload larger ones. - Document ids are 1 to 256 characters, not
.or.., with no lone surrogate; user ids are 1 to 256 characters with no lone surrogate. - Presence refills over up to 15 seconds after the room wakes.
- One document per room. Rooms do not share state; listing, searching or aggregating documents is up to your application.
- No auth rules ship with it.
authorizeis yours; see Authorization. - A busy document grows with its edits, until the room re-seeds it. Each edit leaves CRDT structure (a block registered for each Enter, attribution, placements, marks, replaced values): about 22 bytes an edit in the soak’s mix, of which the purge gives back about a tenth. Past half the quota, a re-seed brings the document back to what its content takes; every client reloads it once.
- Deleted content goes after 30 days (
EDYTOR_PURGE_AFTER_DAYS), from the room and every client; a version or an undo brings it back within them. See History and purge.
For the frame format, or to write a room of your own, see the protocol.