Real-Time Updates Architecture¶

Updated: 2026-04-10 (SSE race condition fix per ADR-039)

Delivering low-latency UI changes for worker lifecycle, metrics, lablet definitions, and lablet instances using Server-Sent Events (SSE).

Overview¶

The application provides a push-based real-time channel so the browser reflects backend changes immediately without manual refresh. This is implemented with Server-Sent Events (SSE) for simplicity, broad browser support, and one-directional event flow.

Endpoint¶

GET /api/events/stream

Query Parameters (filtering - per ADR-013):

Examples:

# All events (default)
GET /api/events/stream

# Events for specific workers
GET /api/events/stream?worker_ids=worker-1,worker-2

# Only metrics events
GET /api/events/stream?event_types=worker.metrics.updated

# Combined filtering
GET /api/events/stream?worker_ids=worker-1&event_types=worker.metrics.updated,worker.status.updated

Response:

• Content-Type: text/event-stream
• Includes initial connected event and heartbeat every 30 seconds.
• Closes gracefully with system.sse.shutdown on application stop.

Event Types¶

Core Events (existing)¶

Worker Template Events (ADR-013)¶

Lablet Definition Events (ADR-013)¶

Lablet Session Events (ADR-013, ADR-039)¶

Flow Diagram¶

sequenceDiagram
    autonumber
    participant Browser
    participant SSE as SSEEventRelay
    participant Batch as BatchingBuffer
    participant Int as InternalController
    participant Dom as Domain Events
    participant WR as WorkerReconciler

    Browser->>SSE: Establish EventSource (w/ filters)
    SSE-->>Browser: connected
    loop Heartbeat every 30s
        SSE-->>Browser: heartbeat
    end
    WR->>Int: POST /api/internal/workers/{id}/metrics
    Int->>Dom: Emit CMLWorkerTelemetryUpdatedDomainEvent
    Dom->>Batch: Add to metrics batch buffer
    Note over Batch: Buffer: 100ms or 10 events
    Batch->>SSE: Flush batched metrics
    SSE-->>Browser: worker.metrics.updated (batched)
    Int->>Dom: Emit CMLWorkerStatusUpdatedDomainEvent
    Dom->>SSE: Broadcast immediately (non-batched)
    SSE-->>Browser: worker.status.updated

Event Batching (ADR-013)¶

High-frequency metrics events are batched to reduce browser overhead:

Batching Configuration¶

class BatchingConfig:
    ENABLED = True
    MAX_BATCH_SIZE = 10           # Flush after N events
    FLUSH_INTERVAL_MS = 100       # Flush after N milliseconds
    BATCHED_EVENT_TYPES = [
        "worker.metrics.updated",
        "lablet-instance.metrics.updated",
    ]

Batched vs Non-Batched Events¶

Batch Payload Format¶

{
  "batch_id": "batch-123",
  "event_type": "worker.metrics.updated",
  "count": 5,
  "events": [
    {"worker_id": "w1", "cpu_percent": 45.2, "collected_at": "..."},
    {"worker_id": "w2", "cpu_percent": 32.1, "collected_at": "..."},
    ...
  ],
  "batched_at": "2026-01-19T10:30:00.123Z"
}

Server-Side Filtering (ADR-013)¶

Filtering is performed server-side before event transmission to reduce bandwidth:

class SSEEventRelay:
    async def should_send_to_client(
        self, event: dict, client: SSEClient
    ) -> bool:
        # Check worker_ids filter
        if client.worker_id_filter:
            worker_id = event.get("worker_id")
            if worker_id and worker_id not in client.worker_id_filter:
                return False

        # Check event_types filter
        if client.event_type_filter:
            if event["type"] not in client.event_type_filter:
                return False

        return True

Frontend Client¶

sse-client.js provides:

• Auto reconnection (exponential backoff capped at 30s)
• Status callbacks (connected, reconnecting, disconnected, error)
• Event routing (on(eventType, handler)) for UI modules (workers.js)
• Toast notifications for key lifecycle events
• Graceful cleanup on page lifecycle events:
• beforeunload: Closes connection when page is refreshed, closed, or navigated away
• visibilitychange: Maintains connection when tab is hidden, reconnects if disconnected when tab becomes visible
• freeze/resume: Handles mobile browser backgrounding and page lifecycle states

Status Badge¶

A dynamic badge (Realtime: <status>) shows live connection state:

Server Components¶

Extending Events¶

1. Publish a new domain event or hook existing handler.
2. In handler: await get_sse_relay().broadcast_event("my.event", { ... })
3. In UI: sseClient.on('my.event', data => {/* update UI */})

Performance & Scalability¶

Current design targets moderate connection counts (< hundreds). Future enhancements:

• Backpressure metrics & queue size monitoring
• Client pruning (idle timeout) in hosted service
• Filtering by role or region (add authorization context to generator)
• Optional upgrade path: WebSocket or HTTP/2 for bi-directional communications

Error Handling¶

• Dropped events logged if queue put times out (>0.1s)
• Reconnection attempts escalate delay until capped
• UI continues functioning with last known data while reconnecting
• Page lifecycle handling:
• Connections are intentionally closed on page unload to prevent stale connections
• Auto-reconnection is suppressed during intentional disconnects (refresh, navigation)
• Reconnection resumes when page becomes visible again after being hidden

Testing & Debugging¶

• Use browser DevTools → Network → events/stream to inspect raw event frames
• Temporarily add console logs in handlers for verbose tracing
• Simulate failures by stopping backend; observe badge transition

Future Improvements¶

Explicitly Rejected:

• ❌ Resumable streams (event ID + Last-Event-ID): Clients get current state on reconnect via REST; no replay needed
• ❌ Controller-direct Redis: Violates ADR-001; all events flow through control-plane-api domain events

FAQ¶

Why SSE instead of WebSockets? One-directional updates reduce complexity; no need for server-side subscriptions or bidirectional commands. Browsers handle proxy/load-balancer quirks more predictably with standard HTTP.

How are lab updates delivered? WorkerReconciler syncs labs and reports via /api/internal/workers/{id}/labs. The command broadcasts worker.labs.updated events.

How do reconnections work? When a client reconnects, it establishes a new stream and receives the current snapshot via REST API (worker list, etc.). No event replay is provided; this simplifies implementation and matches the current UI architecture.

Can I disable real-time updates? Remove the SSE connection call in workers.js or provide a settings flag to skip initializing the stream for certain deployments.

Related:

• Worker Monitoring - Background metrics collection
• ADR-001: API-Centric State Management
• ADR-013: SSE Protocol Improvements
• ADR-039: SSE Race Condition Fix