celld & Durable Object Fleet Architecture

1. How celld Works Under the Hood

celld is a lightweight, self-hostable Durable Object-style database engine written in Go. Instead of running a heavy consensus cluster (Raft, Paxos, etcd, or ZooKeeper), celld gives every tenant, user, room, or workflow its own isolated SQLite database—called a cell—and uses S3-compatible object storage with conditional writes (If-Match / If-None-Match) as its sole coordination authority.

S3 Bucket Layout & ownership.json Schema

s3://my-celld-bucket/prod/
├── fleet/
│   └── capacity-v1.json                           # Published every 5s for cell-count rebalancing
└── cells/
    └── tenant_acme/
        ├── ownership.json                         # CAS lease record (guarded by S3 ETag)
        └── ltx/
            ├── e1/
            │   ├── 00000001-00000001.ltx          # Epoch 1 initial snapshot / frames
            │   └── 00000002-00000015.ltx          # Compacted LTX range
            └── e2/                                # Created after node-b took over in Epoch 2
                └── 00000016-00000016.ltx

When a node claims or renews ownership of tenant_acme, it updates cells/tenant_acme/ownership.json using the S3 object ETag as a compare-and-swap token:

{
  "cell": "tenant_acme",
  "owner": "node-a",
  "advertise_url": "http://10.0.1.11:8081",
  "epoch": 2,
  "lease_expires_at": "2026-10-02T12:30:15Z",
  "max_txid": 16
}

Interactive S3 CAS Ownership & Split-Brain Fencing Simulator

Step through commits, inject a network partition to trigger a CAS lease takeover, and test what happens when the partitioned zombie node attempts to write with a stale epoch.

celld HTTP API Contract

2. How to Set Up & Run celld (Local & Production)

You can run a complete 3-node celld cluster locally against MinIO, Tigris, Cloudflare R2, or AWS S3. Below is a copy-pasteable 3-node launch script and a hardened production systemd service unit.

Local 3-Node Fleet Launch Script

#!/usr/bin/env bash
set -euo pipefail

# Build celld from source
git clone https://github.com/superfly/celld.git
cd celld && go build -o ./bin/celld ./cmd/celld

export CELLD_BUCKET_URL="s3://celld-local/dev"
export CELLD_S3_ENDPOINT="http://127.0.0.1:9000"
export CELLD_S3_REGION="us-east-1"
export CELLD_DURABILITY="fleet"
export CELLD_OTEL="1"

# Launch 3 peers on :8081, :8082, :8083
for i in 1 2 3; do
  mkdir -p "/tmp/celld/node-${i}"
  CELLD_NODE_ID="node-${i}" \
  CELLD_LISTEN="127.0.0.1:808${i}" \
  CELLD_ADVERTISE_URL="http://127.0.0.1:808${i}" \
  CELLD_DATA="/tmp/celld/node-${i}" \
  CELLD_PEERS="http://127.0.0.1:8081,http://127.0.0.1:8082,http://127.0.0.1:8083" \
  ./bin/celld &
done

# Execute a parameterized transaction against cell "org_42" on any node
curl -sS -X POST http://127.0.0.1:8081/cells/org_42/sql \
  -H "Content-Type: application/json" \
  -d '{
    "statements": [
      {"sql": "CREATE TABLE IF NOT EXISTS events (id TEXT PRIMARY KEY, ts INTEGER, kind TEXT)"},
      {"sql": "INSERT INTO events (id, ts, kind) VALUES (?, ?, ?)", "params": ["ev_101", 1759404000, "signup"]}
    ]
  }'

Production systemd Unit with Graceful /drain

# /etc/systemd/system/celld.service
[Unit]
Description=celld SQLite Cell Fleet Daemon
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=celld
Group=celld
EnvironmentFile=/etc/celld/celld.env
ExecStart=/usr/local/bin/celld
# Step down cleanly on stop/restart so peers receive final LTX frames (RPO=0)
ExecStop=/usr/bin/curl -fsS -X POST http://127.0.0.1:8080/drain
TimeoutStopSec=30
Restart=on-failure
RestartSec=2
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

3. Load Balancing celld & Handling Hot Cells

Load balancing a stateful actor fleet has two distinct layers: Ingress Request Routing (how HTTP requests reach the node that currently owns a cell) and Cell Placement Rebalancing (how cells are distributed across nodes).

• Layer 1 — Stateless L7 Round-Robin + Peer Tunneling: Because every celld node knows the owner of active cells (or looks up ownership.json) and proxies non-local requests over its internal HTTP/2 mesh, a standard round-robin load balancer works immediately with zero state synchronization. In an N-node cluster, (N − 1) / N of uncached requests incur a ~0.5–1.0 ms intra-VPC hop (X-Cell-Proxied: 1).
• Layer 2 — Cell-ID Consistent Hashing / Client Affinity: For ultra-low-latency workloads, L7 proxies (Envoy/HAProxy/Cloudflare Workers) can hash the /cells/{cell}/ URI path segment—or clients can cache the owner’s advertise_url from GET /cells/{cell}—reducing peer proxy hops to near zero except immediately after a failover.
• The Rebalancer Caveat (Count vs Load): Every 5 seconds, celld nodes publish their owned cell counts to fleet/capacity-v1.json and rebalance so each node holds roughly TotalCells / N cells. It does not rebalance by CPU or request rate. If a single “celebrity” cell receives 4,000 writes/sec, migrating that cell to another node still funnels all 4,000 writes into a single SQLite writer lock, triggering 503 cell busy.

Production Load-Balancer Configs (Caddy & Envoy)

# Option A: Caddyfile — Stateless round-robin with active /fleet/status health probes
celld.internal.example.com {
    reverse_proxy 10.0.1.11:8081 10.0.1.12:8081 10.0.1.13:8081 {
        lb_policy round_robin
        health_uri /fleet/status
        health_interval 3s
        health_timeout 1s
        fail_duration 10s
    }
}

# Option B: Envoy Ring-Hash on X-Cell-ID header to minimize X-Cell-Proxied hops
route_config:
  routes:
    - match: { prefix: "/cells/" }
      route:
        cluster: celld_fleet
        hash_policy:
          - header: { header_name: "x-cell-id" }

4. Architecting Large Applications with Durable Objects: Choosing the Right Grain

The most consequential architectural decision in any Durable Object or celld system is cell granularity (what a single actor instance represents). A single SQLite Durable Object can comfortably sustain ~300–1,000 SQL transactions/sec.

5. Accounting & Querying Usage Across a Fleet of Durable Objects

Suppose you have 10,000 active AgentSessionDO instances executing LLM calls and SQLite writes, and you need to: (1) enforce hard real-time credit/token quotas per organization, and (2) query usage and performance across the entire fleet (e.g., “Which 50 tenants consumed the most tokens in the last hour?”).

If every AgentSessionDO makes a synchronous RPC to a central BillingDO on every event, that central DO saturates at ~800 req/s. Below is an interactive calculator and the four production code patterns that solve this cleanly.

Pattern 1: The Credit / Quota Lease Pattern (Real-Time Enforcement)

Instead of calling `TenantQuotaDO` on every action, each worker DO leases a block of `N = 500` credits and stores the remaining lease balance in its own local SQLite database. Local deductions happen in `0 ms` with zero network hops; when the local lease drops below 20%, the worker DO tops up asynchronously:

// Worker Durable Object: deducts locally from leased credits in SQLite
export class WorkspaceSessionDO extends DurableObject {
  async consumeCredits(cost: number): Promise<{ allowed: boolean; remaining: number }> {
    const row = this.ctx.storage.sql
      .exec("SELECT balance FROM local_lease WHERE id = 1")
      .one() as { balance: number } | null;
    let balance = row?.balance ?? 0;

    // Top up a 500-credit block from TenantQuotaDO only when local lease is insufficient
    if (balance < cost) {
      const quotaStub = this.env.TENANT_QUOTA.get(
        this.env.TENANT_QUOTA.idFromName(this.tenantId)
      );
      const granted = await quotaStub.leaseCredits(Math.max(500, cost * 2));
      if (granted < cost) return { allowed: false, remaining: balance };
      balance += granted;
    }

    balance -= cost;
    this.ctx.storage.sql.exec(
      "INSERT INTO local_lease (id, balance) VALUES (1, ?) ON CONFLICT(id) DO UPDATE SET balance = excluded.balance",
      balance
    );
    return { allowed: true, remaining: balance };
  }
}

Pattern 2: Write-Behind `alarm()` Rollups (Durable Billing Ledger)

For exact billing and audit ledgers, accumulate high-frequency events inside each DO’s local SQLite table (`unflushed_usage`) and schedule a Durable Object `alarm()` every 30 seconds to flush a single pre-aggregated delta row to your billing queue or central ledger:

export class AgentSessionDO extends DurableObject {
  async recordUsage(model: string, inputTokens: number, outputTokens: number) {
    this.ctx.storage.sql.exec(
      `INSERT INTO unflushed_usage (model, reqs, in_tok, out_tok)
       VALUES (?, 1, ?, ?)
       ON CONFLICT(model) DO UPDATE SET
         reqs = reqs + 1,
         in_tok = in_tok + excluded.in_tok,
         out_tok = out_tok + excluded.out_tok`,
      model, inputTokens, outputTokens
    );

    const currentAlarm = await this.ctx.storage.getAlarm();
    if (currentAlarm === null) {
      await this.ctx.storage.setAlarm(Date.now() + 30_000); // Flush once every 30s
    }
  }

  async alarm() {
    const rows = this.ctx.storage.sql
      .exec("SELECT model, reqs, in_tok, out_tok FROM unflushed_usage")
      .toArray();
    if (rows.length === 0) return;

    await this.env.BILLING_QUEUE.send({
      cellId: this.ctx.id.toString(),
      flushedAt: Date.now(),
      deltas: rows
    });
    this.ctx.storage.sql.exec("DELETE FROM unflushed_usage");
  }
}

Pattern 3: Async Telemetry & Columnar Lakehouse (`CELLD_OTEL=1` / Analytics Engine)

Never fan out read RPCs across thousands of DOs to render an analytics dashboard. Emit non-blocking telemetry points from each DO (or enable `CELLD_OTEL=1` in `celld`) and query the columnar sink with SQL:

// 1. Inside Cloudflare Durable Object: non-blocking writeDataPoint (0ms latency)
this.env.USAGE_ANALYTICS.writeDataPoint({
  indexes: [this.tenantId],
  blobs: [this.cellId, modelName, region],
  doubles: [1, inputTokens, outputTokens, elapsedMs]
});

// 2. For self-hosted celld (CELLD_OTEL=1): query cross-fleet JSONL/Parquet logs with DuckDB
SELECT
  cell_id,
  count(*) AS total_tx,
  sum(rows_written) AS total_rows_written,
  quantile_cont(duration_ms, 0.99) AS p99_ms
FROM read_json_auto('/var/log/celld/otel-*.jsonl')
WHERE ts >= now() - INTERVAL '1 hour'
GROUP BY cell_id
ORDER BY total_rows_written DESC
LIMIT 50;

Pattern 4: Hierarchical CQRS Index DOs (Consistent Entity Listings)

Analytics pipelines have a few seconds of ingestion delay. When your application UI needs a **strongly consistent list** of active child DOs (for example, listing every active chat room or agent in an organization), maintain a parent `WorkspaceIndexDO` that stores only lightweight metadata rows updated when a child DO is created, renamed, or archived:

export class WorkspaceIndexDO extends DurableObject {
  registerChildRoom(roomId: string, title: string, createdAt: number) {
    this.ctx.storage.sql.exec(
      `INSERT INTO rooms (room_id, title, status, updated_at)
       VALUES (?, ?, 'active', ?)
       ON CONFLICT(room_id) DO UPDATE SET title = excluded.title, updated_at = excluded.updated_at`,
      roomId, title, createdAt
    );
  }

  listActiveRooms() {
    return this.ctx.storage.sql
      .exec("SELECT room_id, title, updated_at FROM rooms WHERE status = 'active' ORDER BY updated_at DESC LIMIT 100")
      .toArray();
  }
}