LabRecord Architecture — Implementation Plan¶

1. Executive Summary¶

This document is the implementation plan for the LabRecord Architecture as designed in the LabRecord Architecture Design. It continues the phase numbering from the MVP Implementation Plan (Phases 0–6) and maps the 26 architecture gaps (G1–G26) to concrete, dependency-ordered implementation phases.

Scope¶

Elevate LabRecord from a passive CML lab sync-snapshot to a first-class independent AggregateRoot with:

• 16-state lifecycle (§4), M:N binding to LabletInstance via LabletLabBinding (§5)
• LabletRecordRun cross-aggregate execution mapping with LDS + grading state (§3.4)
• RuntimeBinding abstraction for CML/K8s/Pod/BareMetal runtimes (§4.1)
• Session-centric UI with LDS IFRAME and grading integration (§9)
• Lab discovery v2 with adoption flow (§7)

Design Decisions on Feature Flags¶

Phase Overview¶

Phase 7:  LabRecord Domain Foundation     ✅  (Architecture §3, §4, §5)  — completed 2026-02-11
Phase 8:  LabRecord API & CQRS            ✅  (Architecture §8.1–8.6)  — completed 2026-02-13
Phase 9:  Lab Discovery V2 & Reuse        ✅  (Architecture §7)          — completed 2026-02-11
Phase 10: Labs Frontend                    ✅  (Architecture §9.4)          — completed 2026-02-12
Phase 11: LabletRecordRun & Session Model  ✅  (Architecture §3.4, §8.7–8.10)  — completed 2026-02-13
Phase 12: LDS Session Integration         ⬜  (Architecture §8.8, §9.5)
Phase 13: Grading Integration             ⬜  (Architecture §8.9, §9.6)
Phase 14: Advanced Features               ⬜  (Architecture §10.2 Phase F)

Gap-to-Phase Mapping¶

2. Phase Dependencies¶

flowchart TD
    subgraph MVP["Phases 0–6 (Complete / In Progress)"]
        P6["Phase 6: SSE & Frontend ✅/🔄"]
    end

    subgraph LabRecord["LabRecord Architecture"]
        P7["Phase 7: Domain Foundation ✅"]
        P8["Phase 8: API & CQRS"]
        P9["Phase 9: Discovery V2 & Reuse"]
        P10["Phase 10: Labs Frontend"]
        P11["Phase 11: LabletRecordRun & Session Model"]
        P12["Phase 12: LDS Integration"]
        P13["Phase 13: Grading Integration"]
        P14["Phase 14: Advanced Features"]
    end

    P6 --> P7
    P7 --> P8
    P7 --> P9
    P8 --> P9
    P8 --> P10
    P9 --> P10
    P10 --> P11
    P11 --> P12
    P12 --> P13
    P11 --> P14

Critical path: Phase 7 → 8 → 9 → 10 → 11 → 12 → 13

Parallel opportunities:

• Phase 9 (controller intelligence) can start once Phase 7 enums/VOs are merged, even if Phase 8 API work is in progress
• Phase 10 (frontend) requires Phase 8 API + Phase 9 discovery to be functional

3. Phase 7: LabRecord Domain Foundation¶

Status: ✅ Complete (2026-02-11) Goal: Establish LabRecord as a first-class aggregate with typed status, value objects, state machine, and M:N binding entity. Architecture Ref: §3 Domain Model, §4 Aggregate Design, §5 Relationship Model Gaps Addressed: G1, G2, G3, G8, G13, G16, G17

3.1 Current State¶

3.2 Tasks¶

3.3 Specifications¶

P7-1: LabRecordStatus Enum (Architecture §4.2–4.3)¶

class LabRecordStatus(CaseInsensitiveStrEnum):
    DISCOVERED = "discovered"
    IMPORTING = "importing"
    DEFINED = "defined"
    STARTING = "starting"
    QUEUED = "queued"
    BOOTED = "booted"
    PAUSED = "paused"
    STOPPING = "stopping"
    STOPPED = "stopped"
    WIPING = "wiping"
    WIPED = "wiped"
    DELETING = "deleting"
    DELETED = "deleted"
    ARCHIVED = "archived"
    ERROR = "error"
    ORPHANED = "orphaned"

CML state mapping (migration):

Valid transitions: see Architecture §4.3.

P7-11: LabRecord Aggregate Refactor¶

The existing LabRecord aggregate retains its AggregateRoot + event-sourcing pattern but gains:

• status: LabRecordStatus replacing raw CML state strings
• runtime_binding: RuntimeBinding replacing worker_id + lab_id strings
• topology_spec: LabTopologySpec for structured topology
• external_interfaces: list[ExternalInterface] derived from node tags
• revision: int + revision_history: list[LabRevision]
• run_history: list[LabRunRecord]
• pending_action / pending_action_at / pending_action_error fields
• Transition guard: _validate_transition(from_status, to_status) using valid transitions table

P7-17: LabletDefinition Attribute Additions¶

# In LabletDefinitionState
lab_reuse_enabled: bool = False     # Allow wipe+restart instead of fresh import
multi_lab_enabled: bool = False     # Allow M:N lab bindings (multi-site topologies)

These flags drive controller behavior in Phase 9 (lab reuse resolution) and Phase 14 (multi-lab binding).

3.4 Acceptance Criteria¶

• [x] LabRecordStatus enum with 16 states exists in lcm_core
• [x] RuntimeEnvironmentType enum exists in lcm_core
• [x] BindingRole and BindingStatus enums exist in lcm_core
• [x] LabletRecordRunStatus, LdsSessionStatus, GradingStatus enums exist in lcm_core (prep for Phase 11)
• [x] All 5 value objects created: RuntimeBinding, ExternalInterface, LabTopologySpec, LabRevision, LabRunRecord
• [x] LabRecord aggregate uses typed LabRecordStatus with guarded transitions
• [x] 20 domain events defined (exceeds Architecture §4.4 target of 16)
• [x] LabletLabBinding entity with BindingRole and BindingStatus
• [x] LabletLabBindingRepository (ABC + Motor impl)
• [x] LabRecordReadModel in lcm-core with all state fields
• [x] LabletDefinition has lab_reuse_enabled and multi_lab_enabled attributes
• [x] LabletInstanceState has lab_bindings: list[str]
• [x] lab_discovery_v2_enabled setting exists (default false)
• [x] All existing LabRecord tests still pass (backward compat) — 731 passed, 1 pre-existing failure
• [x] New unit tests: state machine (60 cases), binding lifecycle (20 cases), VOs (26 cases)
• [x] Domain resilience hardening: R1 OCC, R2 CMLWorker transitions, R4 clear event, R5 stale timeout, R6 freshness guard (33 additional tests)

4. Phase 8: LabRecord API & CQRS¶

Status: ✅ Complete (30/30 tasks) | Completed: 2026-02-13 | Tests: 140 new (52 command + 27 query + 61 integration) Goal: Full CQRS command/query surface and BFF controller for LabRecord lifecycle management. Architecture Ref: §8.1–8.6 Backend API Gaps Addressed: G6, G7, G12, G14 Depends on: Phase 7

4.1 Current State¶

4.2 Tasks¶

4.3 Specifications¶

P8-1: DiscoverLabRecordsCommand¶

Replaces SyncLabRecordsCommand (kept for backward compat via delegation). Adds:

• Status tracking via LabRecordStatus (DISCOVERED for new, mapped state for existing)
• Topology change detection (SHA-256 checksum → new LabRevision if changed)
• Orphan detection (DB labs not in CML scan → mark ORPHANED, don't auto-delete)
• SSE events: lab.discovered, lab.status.updated, lab.topology.updated, worker.labs.synced

P8-23: LabRecordsController Endpoints (Architecture §8.1)¶

4.4 Acceptance Criteria¶

• [x] 14 CQRS commands implemented (self-contained: request + handler per file)
• [x] 8 CQRS queries implemented (7 new + GetWorkerLabsQuery pre-existing)
• [x] LabRecordsController serves all 16 BFF endpoints per Architecture §8.1
• [x] InternalController serves 9 internal endpoints per Architecture §8.2 (+ 1 existing sync)
• [x] ControlPlaneApiClient has methods for lab discovery, status update, binding (9 new methods)
• [x] 10 SSE event types emitted for lab lifecycle (§8.6) — 13 new handlers + 3 legacy
• [ ] Existing SyncLabRecordsCommand still works (delegates to DiscoverLabRecordsCommand) — deferred to Phase 9 (discovery integration)
• [x] Unit tests: ≥1 per command (14+), ≥1 per query (8+) — 52 command tests + 27 query tests passing
• [x] API integration tests: 61 structural tests (22 BFF routes + 12 internal routes + 27 request models)

5. Phase 9: Lab Discovery V2 & Reuse¶

Status: ✅ Complete | Completed: 2026-02-11 | Tasks: 12/12 | Tests: 60 new (26 discovery + 34 resolution/reuse) Goal: Evolve lab discovery in lablet-controller to use typed LabRecord lifecycle, and add lab reuse logic to the reconciler. Architecture Ref: §7 Discovery & Synchronisation Gaps Addressed: G4, G5, G15 Depends on: Phase 7 (enums, VOs), Phase 8 (CPA client methods + internal API)

5.1 Current State¶

5.2 Tasks¶

5.3 Specifications¶

P9-5: Lab Resolution Strategy (Architecture §5.4)¶

1. Resource Scheduler assigns LabletInstance to Worker W
2. Lablet Controller checks: does Worker W have a LabRecord
   matching the LabletDefinition topology?
   a. YES and status=WIPED → Bind to existing LabRecord, start lab
   b. YES and status=STOPPED → Wipe first, then start
   c. NO → Import fresh from LabletDefinition.topology_yaml
3. Create LabletLabBinding(role=PRIMARY, status=ACTIVE)
4. On timeslot end: Release binding, wipe lab (don't delete → reuse)

Guard: Only executes if definition.lab_reuse_enabled == True. Otherwise, always cold-import.

P9-1: LabDiscoveryService¶

When LAB_DISCOVERY_V2_ENABLED=true:

1. Scan — For each running worker, fetch all labs from CML API
2. Diff — Compare against existing LabRecords via ControlPlaneApiClient
3. Create — New labs → POST /internal/lab-records/discover → status=DISCOVERED
4. Update — Known labs → sync status, detect topology changes via checksum
5. Orphan — DB labs not on CML → mark ORPHANED (don't auto-delete)
6. Emit — SSE events via CPA for UI real-time updates

When LAB_DISCOVERY_V2_ENABLED=false: delegate to legacy SyncLabRecordsCommand.

5.4 Acceptance Criteria¶

• [x] LabDiscoveryService replaces LabsRefreshService (old service kept behind flag)
• [x] Discovery creates LabRecords with proper LabRecordStatus (not raw strings)
• [x] Topology change detection produces new LabRevision entries
• [x] Orphan labs marked ORPHANED (not auto-deleted)
• [x] LabletReconciler resolves existing labs before importing (when lab_reuse_enabled)
• [x] Reuse path: WIPED lab → start (~20s) vs fresh import (~90s)
• [x] Bindings created/released during instance lifecycle
• [x] LAB_DISCOVERY_V2_ENABLED flag controls discovery path
• [x] Unit tests: ≥15 for discovery (26 actual), ≥10 for resolution/reuse (34 actual)

6. Phase 10: Labs Frontend¶

Status: ✅ Complete | Completed: 2026-02-12 | Tasks: 8/10 (P10-9, P10-10 deferred) Goal: Dedicated Labs management page in the UI for admin operations on LabRecords. Architecture Ref: §9.4 Labs Management Page Gaps Addressed: G9, G10 Depends on: Phase 8 (API endpoints), Phase 9 (discovery producing LabRecords)

6.1 Tasks¶

6.2 Files Created/Modified¶

New files (4):

Modified files (8):

6.3 Acceptance Criteria¶

• [x] "Labs" page accessible from main navigation (pill tab between Workers and System)
• [x] Lab records listed in table with status badges, filter by worker/status/bound/search
• [x] Collapsible summary metric cards (Total, Running, Stopped, Wiped, Discovered, Errors)
• [x] Lab detail modal shows Overview, Topology, Revisions, Bindings tabs
• [x] Action buttons work: Start, Stop, Wipe, Clone, Export, Delete, Archive (context-sensitive per status)
• [x] SSE events update labs page in real-time (14 event types mapped)
• [x] 16 LabRecordStatus states have distinct badge colors and icons
• [x] make build-ui exits 0 (verified — Parcel build succeeds in 3.27s)

7. Phase 11: LabletRecordRun & Session Model¶

Status: ✅ Complete (25/25 tasks) | Started: 2026-02-12 | Completed: 2026-02-13 Goal: Create the LabletRecordRun cross-aggregate entity and Sessions page for session-centric UX. Architecture Ref: §3.4 LabletRecordRun, §8.7–8.10 Run/LDS/Grading API, §9.1–9.3 Session UI Gaps Addressed: G18, G19, G22, G23 Depends on: Phase 10 (Labs frontend functional)

7.1 Tasks¶

7.2 Files Created/Modified¶

New backend files (9):

New frontend files (7):

Modified frontend files (10):

7.3 Specifications¶

P11-1: LabletRecordRun Entity (Architecture §3.4)¶

A cross-aggregate execution mapping linking:

• Who — lablet_instance_id (scheduled timeslot)
• What — lab_record_id (CML lab)
• When — started_at / ended_at runtime window
• Where — allocated_ports (resolved, frozen at run start)
• Why — session_part_id + form_qualified_name
• How — LDS session + grading session state

Status lifecycle: PROVISIONING → ACTIVE → PAUSED → ENDING → ENDED → FAULTED

P11-4: Port Mapping Resolution¶

Resolves port allocations from three sources:

1. LabRecord.external_interfaces (parsed from CML node tags)
2. CML Worker IP (EC2 instance reachable address)
3. LabletInstance.allocated_ports (existing port mapping)

Frozen at run creation for LDS/grading stability.

7.4 Acceptance Criteria¶

• [x] LabletRecordRun entity persists in MongoDB with all fields from Architecture §3.4
• [x] PortMappingResolutionService resolves and freezes port allocations
• [x] BFF endpoints: GET/POST /api/lablet-record-runs, GET /api/lablet-record-runs/{id}, POST .../end
• [x] Sessions page shows session list with status, candidate, location, timeslot
• [x] Session detail page shows SessionPart accordion with LabletInstance + LabletRecordRun cards
• [x] Port mapping table displays resolved device access endpoints
• [x] Worker Detail Modal Labs tab enhanced with binding info (carried from P10-8)
• [x] Lablet Instance cards show lab binding info (carried from P10-9)
• [x] Vitest tests for Phase 10 + Phase 11 web components (136 frontend tests, carried from P10-10)
• [x] make build-ui exits 0

8. Phase 12: LDS Session Integration¶

Status: ✅ Complete Goal: LDS IFRAME integration in Session Detail page with postMessage bridge. Architecture Ref: §8.8 LDS Session API, §9.5 LDS IFRAME Gaps Addressed: G20, G24, G26 Depends on: Phase 11 (LabletRecordRun entity, Sessions page) Completed: 2025-07-08 — Backend: 5 LDS commands + 1 query + adapter + controller + SSE + 30 unit tests. Frontend: 2 IFRAME components + SSE wiring + API client + store updates + session page integration. Build passes.

8.1 Tasks¶

8.2 Acceptance Criteria¶

• [x] LDS session provisioned/started/paused/resumed/ended via LabletRecordRun API
• [x] LDS IFRAME renders in Session Detail page with login URL
• [x] postMessage bridge supports lcm:pause, lcm:resume, lcm:end (parent→LDS)
• [x] postMessage bridge handles lds:status, lds:grade_request, lds:timer_update (LDS→parent)
• [x] CML Dashboard IFRAME renders for admin/proctor view
• [x] SSE events update run/LDS status in real-time
• [x] make build-ui exits 0

9. Phase 13: Grading Integration¶

Status: ⬜ Not Started Goal: Grading pipeline via LabletRecordRun with score report display. Architecture Ref: §8.9 Grading API, §9.6 Grading Integration Gaps Addressed: G21, G25 Depends on: Phase 12 (LDS integration — grading triggers from LDS events) Note: This phase also addresses the deferred MVP Phase 5 (FR-2.6.1, FR-2.6.2, FR-2.2.7) through the LabletRecordRun model rather than directly on LabletInstance.

9.1 Tasks¶

9.2 Specifications¶

Grading Trigger Flow (Architecture §9.6.1)¶

GradingEngine Adapter¶

Translates LCM's LabletRecordRun context into GradingEngine's Session/SessionPart/Pod model:

• LabletRecordRun.allocated_ports → Pod.Devices[].Interfaces[]
• LabRecord.topology_spec.nodes → Pod.Devices[] with label mapping
• Device labels must match between CML topology, content.xml, and grade.xml

9.3 Acceptance Criteria¶

• [ ] Grading triggered via API, LDS postMessage, or auto-trigger on LDS end
• [ ] GradingEngine adapter translates LabletRecordRun ports to Pod.Devices format
• [ ] Score report displayed inline (summary panel) and full (IFRAME)
• [ ] Grade/Submit/Reread actions available in Session Detail
• [ ] CloudEvent grading.completed updates LabletRecordRun grading state
• [ ] SSE events for grading lifecycle update UI in real-time
• [ ] make build-ui exits 0
• [ ] Addresses deferred MVP Phase 5 requirements (FR-2.6.1, FR-2.6.2, FR-2.2.7)

10. Phase 14: Advanced Features¶

Status: ⬜ Not Started Goal: Post-MVP enhancements: multi-lab support, topology visualization, K8s runtime. Architecture Ref: §10.2 Phase F Gaps Addressed: G11 Depends on: Phase 13 (core LabRecord architecture complete)

10.1 Tasks¶

10.2 Acceptance Criteria¶

• [ ] Multi-lab: LabletInstance can bind >1 LabRecord (guarded by multi_lab_enabled)
• [ ] Each item individually scoped — acceptance criteria defined when task is started

11. Migration Strategy (Architecture §11)¶

11.1 Data Migration (applies across Phases 7–9)¶

Step 1 (Phase 7): Add status field to existing LabRecords with CML state mapping:

Step 2 (Phase 8): Migration command to create LabletLabBinding records for existing instances:

• For each LabletInstance with non-null cml_lab_id:
• Find or create LabRecord matching (worker_id, lab_id)
• Create LabletLabBinding(role=PRIMARY, status=ACTIVE)

Step 3: Deprecate LabletInstance.state.cml_lab_id (keep read-only for backward compat).

11.2 Backward Compatibility¶

• SyncLabRecordsCommand continues to work (delegates to DiscoverLabRecordsCommand when v2 enabled)
• cml_lab_id field kept on LabletInstanceState but new code reads from LabletLabBinding
• All existing API responses include both cml_lab_id and new lab_bindings array

12. Risk Register¶

13. Revision History¶