LabletSession Lifecycle Flow¶

Overview¶

This document describes the complete lifecycle of a LabletSession from reservation through execution to archival, showing how all four microservices collaborate via etcd watch streams and the Control Plane API as the single source of truth (ADR-001).

Key Architectural Principles:

• Control Plane API (CPA) is the sole MongoDB writer — all state mutations go through it
• etcd is the cross-service coordination bus — state projection, watch streams, leader election, capacity tracking
• Watch + Poll dual-mode — etcd watches provide reactivity; periodic polling guarantees consistency against missed events
• Pipeline fire-and-check (ADR-034) — long-running multi-step phases (INSTANTIATING, COLLECTING, GRADING, STOPPING) run as managed asyncio.Task inside a LifecyclePhaseHandler; the reconciler never blocks — it checks handler status on each cycle
• Lab reuse over reimport — labs are wiped (not deleted), preserving topology for ~20s reuse vs ~90s fresh import

Service Responsibilities¶

All services are leader-elected — only one instance per service reconciles at a time.

State Machine¶

Status Enum¶

class LabletSessionStatus(str, Enum):
    PENDING         = "PENDING"          # Session created, awaiting scheduling
    SCHEDULED       = "SCHEDULED"        # Worker assigned, awaiting timeslot
    INSTANTIATING   = "INSTANTIATING"    # Pipeline: lab import + start + LDS provision
    READY           = "READY"            # Lab BOOTED + LDS provisioned, awaiting user
    RUNNING         = "RUNNING"          # User logged in, session active
    COLLECTING      = "COLLECTING"       # Evidence capture pipeline (configs, screenshots, pcaps)
    GRADING         = "GRADING"          # Grading engine evaluating evidence
    STOPPING        = "STOPPING"         # Teardown pipeline: stop + wipe + archive
    STOPPED         = "STOPPED"          # Teardown complete, awaiting final archival
    ARCHIVED        = "ARCHIVED"         # Terminal: session preserved for historical records
    TERMINATED      = "TERMINATED"       # Terminal: force-terminated from any state
    EXPIRED         = "EXPIRED"          # Terminal: timeslot elapsed

Transition Diagram¶

stateDiagram-v2
    [*] --> PENDING: User creates session
    PENDING --> SCHEDULED: resource-scheduler assigns worker
    PENDING --> TERMINATED: Force terminate

    SCHEDULED --> INSTANTIATING: lablet-controller (timeslot approach)
    SCHEDULED --> TERMINATED: Force terminate

    INSTANTIATING --> READY: Pipeline completes (mark_ready step)
    INSTANTIATING --> EXPIRED: Timeslot elapsed
    INSTANTIATING --> TERMINATED: Force terminate / pipeline exhausted

    READY --> RUNNING: LDS CloudEvent or manual transition
    READY --> EXPIRED: Timeslot elapsed
    READY --> TERMINATED: Force terminate

    RUNNING --> COLLECTING: Assessment trigger
    RUNNING --> STOPPING: Timeslot ended / manual
    RUNNING --> EXPIRED: Timeslot elapsed
    RUNNING --> TERMINATED: Force terminate

    COLLECTING --> GRADING: Evidence pipeline completes
    COLLECTING --> STOPPING: Manual override
    COLLECTING --> EXPIRED: Timeslot elapsed
    COLLECTING --> TERMINATED: Force terminate

    GRADING --> STOPPING: Grading pipeline completes
    GRADING --> EXPIRED: Timeslot elapsed
    GRADING --> TERMINATED: Force terminate

    STOPPING --> ARCHIVED: Teardown pipeline completes (archive step)
    STOPPING --> TERMINATED: Force terminate

    ARCHIVED --> TERMINATED: Cleanup
    EXPIRED --> TERMINATED: Cleanup

Valid Transitions¶

End-to-End Sequence Diagram¶

sequenceDiagram
    autonumber
    box rgba(100,149,237,0.1) User Layer
    actor User
    participant UI as UI (Bootstrap SPA)
    end

    box rgba(144,238,144,0.1) Control Plane
    participant CP as Control Plane API
    participant etcd as etcd
    participant Mongo as MongoDB
    end

    box rgba(255,215,0,0.1) Controllers
    participant Sched as Resource Scheduler
    participant LC as Lablet Controller
    participant WC as Worker Controller
    end

    box rgba(255,182,193,0.1) External Systems
    participant CML as CML Labs API
    participant LDS as Lab Delivery System
    end

    Note over User,LDS: Phase 1: Reservation (→ PENDING)

    User->>UI: Request Lab Session
    UI->>CP: POST /api/v1/sessions
    CP->>Mongo: Save LabletSession (PENDING)
    CP->>etcd: PUT /sessions/{id} = {status: PENDING, ...}
    CP-->>UI: 201 Created

    Note over Sched,etcd: Phase 2: Scheduling (PENDING → SCHEDULED)

    etcd-->>Sched: Watch event: session PENDING
    Sched->>Sched: PlacementEngine.schedule()
    Sched->>CP: POST /api/internal/sessions/{id}/schedule
    CP->>Mongo: Update (SCHEDULED, worker_id, capacity reserved)
    CP->>etcd: PUT /sessions/{id} = {status: SCHEDULED, ...}
    CP-->>Sched: 200 OK

    Note over LC,CML: Phase 3: Instantiation (SCHEDULED → INSTANTIATING → READY)

    etcd-->>LC: Watch event: session SCHEDULED
    LC->>LC: _handle_scheduled() — timeslot check
    LC->>CP: POST /api/internal/sessions/{id}/start-instantiation
    CP->>Mongo: Update (INSTANTIATING)
    CP->>etcd: PUT /sessions/{id} = {status: INSTANTIATING}

    etcd-->>LC: Watch event: session INSTANTIATING
    LC->>LC: _handle_instantiating() — fire-and-check
    LC->>LC: Start LifecyclePhaseHandler (instantiate pipeline)

    Note right of LC: Pipeline runs as asyncio.Task

    LC->>CML: Import/reuse lab topology
    LC->>CP: Allocate ports, bind LabRecord
    LC->>CML: Start lab, poll until BOOTED
    LC->>LDS: Create LDS session + device mapping
    LC->>CP: POST /api/internal/sessions/{id}/mark-ready
    CP->>Mongo: Update (READY)
    CP->>etcd: PUT /sessions/{id} = {status: READY}

    Note over User,LDS: Phase 4: Active Session (READY → RUNNING)

    etcd-->>LC: Watch event: session READY
    LC->>LC: _handle_ready() — verify lab BOOTED

    User->>LDS: Login via IFRAME
    LDS-)LC: CloudEvent: lds.session.started
    LC->>CP: POST /api/internal/sessions/{id}/transition (RUNNING)

    Note right of CP: Or via ADR-040: LDS → CPA direct
    LDS-)CP: CloudEvent: io.lablet.lds.session.running.v1
    CP->>CP: session.mark_running() (direct aggregate mutation)

    CP->>Mongo: Update (RUNNING)
    CP->>etcd: PUT /sessions/{id} = {status: RUNNING}

    Note over LC,CML: Phase 5: Assessment (RUNNING → COLLECTING → GRADING)

    rect rgba(255,200,100,0.15)
        Note right of LC: Assessment flow (stubs in Sprint D)
        LC->>CP: Transition → COLLECTING
        LC->>LC: _handle_collecting() — fire-and-check
        Note right of LC: collect_evidence pipeline (stubs)
        LC->>CP: Transition → GRADING
        LC->>LC: _handle_grading() — fire-and-check
        Note right of LC: compute_grading pipeline (stubs)
    end

    Note over LC,CML: Phase 6: Teardown (→ STOPPING → ARCHIVED)

    LC->>CP: Transition → STOPPING (releases capacity)
    CP->>etcd: PUT /sessions/{id} = {status: STOPPING}

    etcd-->>LC: Watch event: session STOPPING
    LC->>LC: _handle_stopping() — fire-and-check
    LC->>LC: Start LifecyclePhaseHandler (teardown pipeline)

    LC->>CML: stop_lab() + poll until STOPPED
    LC->>LDS: archive_session()
    LC->>CML: wipe_lab() (preserve topology for reuse)
    LC->>CP: Record lab run + transition → ARCHIVED

    CP->>Mongo: Update (ARCHIVED)
    CP->>etcd: PUT /sessions/{id} = {status: ARCHIVED}

Phase Details¶

Phase 1: Reservation (→ PENDING)¶

Owner: control-plane-api Trigger: User API call

1. User selects a LabletDefinition and timeslot in the UI
2. UI calls POST /api/v1/sessions with definition_id, timeslot_start, timeslot_end
3. CPA executes CreateLabletSessionCommand → persists LabletSession aggregate to MongoDB
4. CPA publishes session state to etcd at /sessions/{id}
5. Domain event lablet_session.created.v1 is emitted

POST /api/v1/sessions
Content-Type: application/json

{
  "definition_id": "def-abc123",
  "timeslot_start": "2026-01-19T14:00:00Z",
  "timeslot_end": "2026-01-19T16:00:00Z",
  "reservation_id": "ext-reservation-456"
}

Phase 2: Scheduling (PENDING → SCHEDULED)¶

Owner: resource-scheduler Trigger: etcd watch on /sessions/{id} where status=PENDING (+ polling fallback)

The LabletSessionScheduler (a WatchTriggeredHostedService) reconciles PENDING sessions:

1. Fetches the LabletDefinition for resource requirements
2. Runs PlacementEngine.schedule() against:
   • RUNNING workers list
   • Real-time capacity from etcd (/capacity/{worker_id})
   • Worker templates (for scale-up decisions)
3. Decision outcomes:
   • assign → calls CPA ScheduleLabletSessionCommand → SCHEDULED (worker capacity reserved)
   • scale_up → requests new worker provisioning, requeues session
   • wait → requeues for next cycle (no suitable worker)

POST /api/internal/sessions/{session_id}/schedule
X-API-Key: {internal_api_key}
Content-Type: application/json

{
  "worker_id": "worker-xyz789",
  "allocated_ports": {}
}

Phase 3: Instantiation (SCHEDULED → INSTANTIATING → READY)¶

Owner: lablet-controller Trigger: etcd watch on /sessions/{id} status changes Pattern: Pipeline fire-and-check delegation (ADR-034)

Step 1: SCHEDULED → INSTANTIATING¶

_handle_scheduled() checks whether the timeslot is approaching (or is on-demand). When ready, calls CPA StartInstantiationCommand.

Step 2: Pipeline Execution¶

_handle_instantiating() uses the fire-and-check pattern:

reconcile cycle 1:  No handler exists → start LifecyclePhaseHandler
reconcile cycle 2:  Handler is_running → return success (self-driving)
     ...
reconcile cycle N:  Handler finished → check result:
    • completed/partial → success
    • failed → check retry budget → restart or terminate
    • None (crash) → restart handler

The LifecyclePhaseHandler wraps a PipelineExecutor in a managed asyncio.Task. The executor runs the instantiate pipeline defined in the seed YAML as a DAG (topological sort via graphlib):

Pipeline Executor Features¶

• DAG-based execution — steps sorted via graphlib.TopologicalSorter based on needs declarations
• skip_when expressions — evaluated via simpleeval with $SESSION, $DEFINITION, $STEPS context
• Per-step retry — configurable max_retries + retry_delay_seconds with backoff
• Per-step timeout — enforced via asyncio.wait_for()
• Progress persistence — after each step, calls CPA UpdateInstantiationProgressCommand
• Crash-resilient resumability — on restart, completed/skipped steps are restored from CPA progress

Lab Resolution Strategy¶

_step_lab_resolve attempts lab reuse before fresh import:

1. If cml_lab_id is empty → query CPA for WIPED/STOPPED labs on this worker matching node count
   • Priority: WIPED (~20s reuse) > STOPPED (wipe first, ~30s)
2. Fallback: fresh import via CML import_lab() (~90s)

Phase 4: Active Session (READY → RUNNING)¶

Owner: lablet-controller Trigger: etcd watch

READY State¶

_handle_ready() performs a health check — verifies the lab is still BOOTED on the CML worker. The READY → RUNNING transition is triggered externally:

• LDS CloudEvent (lds.session.started) — user logged into the IFRAME
• Manual API call — admin transitions via POST /api/v1/sessions/{id}/transition

RUNNING State¶

_handle_running() monitors the active session:

• Verifies lab state is still healthy (BOOTED on CML)
• Checks timeslot hasn't elapsed
• Observes resource utilization
• On timeslot end or manual trigger → transition to STOPPING (or COLLECTING for assessment flow)

Phase 5: Assessment (RUNNING → COLLECTING → GRADING)¶

Owner: lablet-controller Pattern: Pipeline fire-and-check (same as instantiation)

COLLECTING — Evidence Pipeline¶

_handle_collecting() uses fire-and-check with the collect_evidence pipeline:

GRADING — Scoring Pipeline¶

_handle_grading() uses fire-and-check with the compute_grading pipeline:

Phase 6: Teardown (→ STOPPING → ARCHIVED)¶

Owner: lablet-controller Pattern: Pipeline fire-and-check (ADR-034 Sprint D)

When a session transitions to STOPPING (capacity is released by CPA), _handle_stopping() starts a LifecyclePhaseHandler with the teardown pipeline:

The _step_archive handler internally calls _record_lab_run_completed() before transitioning to ARCHIVED (AD-PIPELINE-011: no separate step for run recording).

Alternative Paths¶

Timeslot Expiry (→ EXPIRED)¶

The TimeslotWatcherService runs every 10 seconds, scanning for sessions with elapsed timeslots via CPA's GET /api/internal/lablet-sessions/imminent-deadlines. For each match, it writes an etcd trigger key to initiate watch-based reconciliation.

Additionally, _reconcile_inner() performs an early timeslot expiry check before status routing: if the timeslot has elapsed and the session is not in a terminal/stopping state, it calls ExpireLabletSessionCommand immediately.

EXPIRED is reachable from: INSTANTIATING, READY, RUNNING, COLLECTING, GRADING.

Force Termination (→ TERMINATED)¶

Reachable from any state via TerminateLabletSessionCommand. Side effects:

• Releases allocated ports back to worker pool
• Releases worker capacity
• Unbinds LabRecord from session

Reconciliation Architecture¶

Watch + Poll Dual-Mode¶

The lablet-controller's LabletReconciler extends WatchTriggeredHostedService:

┌──────────────────────────────────────────────────────────────┐
│                     LabletReconciler                          │
│                                                               │
│  ┌─────────────┐     ┌──────────────────────────────────┐    │
│  │ etcd Watch   │────▶│   _reconcile_inner(instance)      │   │
│  │ /sessions/*  │     │                                    │   │
│  └─────────────┘     │  1. Early timeslot expiry check     │   │
│                       │  2. Per-session lock                 │   │
│  ┌─────────────┐     │  3. Worker IP validation              │   │
│  │ Poll Timer   │────▶│  4. Status routing:                  │   │
│  │ (fallback)   │     │     SCHEDULED → _handle_scheduled()  │   │
│  └─────────────┘     │     INSTANTIATING → _handle_inst..()  │   │
│                       │     READY → _handle_ready()           │   │
│                       │     RUNNING → _handle_running()        │   │
│                       │     COLLECTING → _handle_collecting()  │   │
│                       │     GRADING → _handle_grading()        │   │
│                       │     STOPPING → _handle_stopping()      │   │
│                       └──────────────────────────────────────┘    │
│                                                                    │
│  Sub-services (started on leader election):                        │
│  • LabsRefreshService — periodic CML lab discovery                 │
│  • LabActionWatcherService — reactive lab pending actions          │
│  • ContentSyncWatcherService — definition content sync             │
│  • TimeslotWatcherService — proactive deadline detection (10s)     │
└────────────────────────────────────────────────────────────────────┘

Fire-and-Check Pattern¶

Used for INSTANTIATING, COLLECTING, GRADING, and STOPPING:

┌──────────────────────────────────────────────────────────┐
│              Fire-and-Check Delegation                     │
│                                                            │
│  Reconciler checks _active_handlers[handler_key]:          │
│                                                            │
│  ┌─ No handler exists ──────────────────────────────────┐ │
│  │  1. Get pipeline_def from LabletDefinition YAML       │ │
│  │  2. Build PipelineContext ($SESSION, $DEFINITION)      │ │
│  │  3. Create PipelineExecutor with step_dispatcher       │ │
│  │  4. Wrap in LifecyclePhaseHandler (asyncio.Task)       │ │
│  │  5. Store in _active_handlers                          │ │
│  └───────────────────────────────────────────────────────┘ │
│                                                            │
│  ┌─ Handler exists + is_running ─────────────────────────┐ │
│  │  → return success (pipeline is self-driving)           │ │
│  └───────────────────────────────────────────────────────┘ │
│                                                            │
│  ┌─ Handler finished ────────────────────────────────────┐ │
│  │  result.status == "completed"/"partial" → success      │ │
│  │  result.status == "failed":                            │ │
│  │    retries < max → restart handler (retry)             │ │
│  │    retries >= max → terminate session                  │ │
│  │  result is None (crash) → restart handler              │ │
│  └───────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘

External Systems Integration¶

Domain Events¶

CloudEvents Consumed¶

API Endpoints Summary¶

Public Endpoints (User-Facing via CPA)¶

Internal Endpoints (Service-to-Service via CPA)¶

Key Code Locations¶

Related Documentation¶

• ADR-001: API-Centric State Management
• LCM Platform Architecture
• Lablet Controller Architecture
• Resource Scheduler Architecture
• Worker Controller Architecture
• Control Plane API Architecture