Implementation Status¶

Executive Summary¶

This document tracks implementation progress against the Lablet Resource Manager Requirements Specification v0.3.0. It is organized into two parts:

1. MVP Phase Progress \u2014 Delivery-focused sections tracking what was built in each phase
2. Requirements Tracking Matrix \u2014 Requirement-focused matrices showing per-service status for every FR/NFR

Status Legend¶

Component Legend¶

Overall Progress¶

Phase 0: Domain Prerequisites    ████████████████████ 100%  ✅ Complete (2026-02-08)
Phase 1: Worker Foundation       ████████████████████ 100%  ✅ Complete (2026-02-09)
Phase 2: Resource Scheduling     ████████████████████ 100%  ✅ Complete (2026-02-10)
Phase 3: Auto-Scaling            ████████████████████ 100%  ✅ Complete (2026-02-08)
Phase 4: LDS Integration         ████████████████████ 100%  ✅ Complete (2026-02-10)
Phase 5: Grading Integration     ░░░░░░░░░░░░░░░░░░░░   0%  ⬜ Deferred to Phase 7
Phase 6: SSE & Frontend          █████████████████░░░  85%  🔄 In Progress (G1+G3+G4+G5+F1+F2+F3+F8 done)
Phase 7: LabRecord Domain        ████████████████████ 100%  ✅ Complete (2026-02-11)
Phase 8: LabRecord API & CQRS    ████████████████████ 100%  ✅ Complete (2026-02-11)
Phase 9: Lab Discovery V2        ████████████████████ 100%  ✅ Complete (2026-02-11)
Phase 10: Labs Frontend          ████████████████████ 100%  ✅ Complete (2026-02-11)
Phase 11: LabletRecordRun        ████████████████████ 100%  ✅ Complete (2026-02-11)
Phase 12: LDS Session Integr.    ████████████████████ 100%  ✅ Complete (2026-02-12)

Session Entity Model Migration   ████████████████████ 100%  ✅ Complete (2026-02-20)

Phase 0: Domain Prerequisites¶

Status: ✅ Complete | Completed: 2026-02-08 | Tasks: 8/8 | Tests: 21 new (210 total domain) Plan Reference: MVP Implementation Plan §3 | Bootstrap: PHASE_0_BOOTSTRAP.md

Changes Delivered¶

Files Modified¶

Phase 1: Worker Foundation¶

Status: ✅ Complete | Completed: 2026-02-09 | Tasks: 6/8 core tasks (P1-7, P1-8 deferred to Phase 2) | Tests: 26 new (387 total CPA) Plan Reference: MVP Implementation Plan §4 | Bootstrap: PHASE_1_BOOTSTRAP.md

Changes Delivered¶

Files Created¶

Files Modified¶

Capacity Flow (Implemented)¶

sequenceDiagram
    participant RS as resource-scheduler
    participant CPA as control-plane-api
    participant Agg as CMLWorker Aggregate
    participant etcd as etcd

    RS->>CPA: POST /api/instances/{id}/schedule (worker_id)
    CPA->>Agg: Validate worker RUNNING + capacity
    CPA->>Agg: schedule(instance_id, worker_id)
    CPA->>CPA: mediator.execute(AllocateCapacityCommand)
    Agg->>Agg: assign_lablet_instance() + allocate_ports()
    Agg-->>CPA: WorkerInstanceAssignedDomainEvent
    CPA->>etcd: PUT /lcm/workers/{id}/capacity
    CPA-->>RS: 200 OK {status: SCHEDULED}

    Note over RS,etcd: On Termination
    CPA->>CPA: mediator.execute(ReleaseCapacityCommand)
    Agg->>Agg: release_ports() + remove_lablet_instance()
    Agg-->>CPA: WorkerInstanceRemovedDomainEvent
    CPA->>etcd: PUT /lcm/workers/{id}/capacity (updated)

Phase 2: Resource Scheduling¶

Status: ✅ Complete | Completed: 2026-02-10 | Tasks: 6/6 core tasks | Tests: 41 new (77 total resource-scheduler) Plan Reference: MVP Implementation Plan §5 | Bootstrap: PHASE_2_BOOTSTRAP.md

Changes Delivered¶

Files Created¶

Files Modified¶

Key Design Decisions¶

• Overlay pattern: SchedulerHostedService fetches etcd capacity once per cycle, passes to PlacementEngine. PlacementEngine stays pure/testable \u2014 no etcd dependency.
• Graceful fallback: If etcd is unavailable, PlacementEngine falls back to API declared - allocated capacity data automatically.
• Base class reuse: ReconciliationHostedService already provides exponential backoff (1s→60s); Phase 2 adds escalation for persistent failures only.
• Rejection granularity: _filter_eligible_workers() returns rejection counts by category to support intelligent scale-up decisions.

Phase 3: Auto-Scaling¶

Status: ✅ Complete | Completed: 2026-02-08 | Tasks: 9/9 (P3-8, P3-9 partial \u2014 unit-tested, no cross-service E2E) | Tests: 44 new Plan Reference: MVP Implementation Plan §6 | Bootstrap: PHASE_3_BOOTSTRAP.md

Changes Delivered¶

Bugfix: Discovery State Sync (AD-21)¶

Files Created¶

Files Modified¶

Phase 4: LDS Integration¶

Status: ✅ Complete | Completed: 2026-02-10 | Tasks: 10/10 code tasks + staging validation | Tests: 57 new (44 LC + 13 CPA) Plan Reference: MVP Implementation Plan §7 | Bootstrap: PHASE_4_BOOTSTRAP.md

Changes Delivered¶

Files Created¶

Files Modified¶

Key Design Decisions¶

• AD-P4-01: Atomic mark-ready \u2014 Single MarkInstanceReadyCommand handles INSTANTIATING→READY with all LDS fields (session_id, login_url, launch_url) atomically. Avoids split-brain from separate LDS-set + transition commands.
• AD-P4-02: Device mapping strategy \u2014 _build_device_access_list() maps CML lab nodes to DeviceAccessInfo using allocated port mappings. Static helper for testability.
• AD-P4-03: Multi-instance session matching \u2014 HandleSessionStartedCommand matches lds_session_id across all READY instances (not just by instance_id) since LDS fires session.started by session, not instance.
• CommandHandlerBase adoption \u2014 All Phase 4 handlers extend CommandHandlerBase + CommandHandler[TCmd, TResult], providing consistent cloud event bus injection.

Phase 6: SSE & Frontend Readiness¶

Status: 🔄 ~85% Complete | Started: 2026-02-09 | Tasks: 8/15 (G1, G3, G4, G5, F1, F2, F3, F8 done) | Tests: 121 new (53 G4 + 59 G5 + 9 F2) Plan Reference: MVP Implementation Plan §9 |

Sub-Phase A: SSE & Backend Readiness¶

Sub-Phase B: Frontend Implementation¶

Files Created (F1+F3+F8)¶

Files Modified (F1+F3+F8)¶

Files Created (G3+G4+G5)¶

Files Modified (G4+G5+G3)¶

Key Design Decisions¶

• AD-16: Navbar converted to dropdowns — Lablets and Workers tabs converted from direct links to Bootstrap 5 dropdowns to accommodate sub-views (Reservations, Capacity Dashboard). Consistent with existing System dropdown pattern.
• Scheduler API proxy — SchedulerPage uses /scheduler/ URL prefix, proxied through nginx to the resource-scheduler admin API. Separate api/scheduler.js client module.

Phase 7: LabRecord Domain Foundation¶

Status: ✅ Complete | Completed: 2026-02-11 | Tasks: 23/23 | Tests: 106 new (60 state machine + 20 binding + 26 VOs) Plan Reference: LabRecord Implementation Plan §3 | Architecture: LabRecord Architecture Design

Objective¶

Establish LabRecord as a first-class aggregate with typed status, value objects, state machine, and M:N binding entity. Closes gaps G1, G2, G3, G8, G13, G16, G17.

Key Deliverables¶

Files Created¶

Key Design Decisions¶

• Removed LAB_RECORD_LIFECYCLE_ENABLED flag — Cross-cutting concern; the new typed status replaces raw strings unconditionally.
• Feature flags as definition attributes — lab_reuse_enabled and multi_lab_enabled set per-LabletDefinition, not system-wide, enabling gradual rollout.
• 20 domain events — Exceeded the Architecture §4.4 target of 16 to provide finer-grained lifecycle observability.
• Domain resilience hardening — Added OCC guards, stale timeout detection, freshness validation, and transition safety across CMLWorker and LabRecord aggregates.

Phase 8: LabRecord API & CQRS¶

Status: ✅ Complete | Completed: 2026-02-13 | Tasks: 30/30 | Tests: 140 new (52 command + 27 query + 61 integration) Plan Reference: LabRecord Implementation Plan §4 | Architecture: LabRecord Architecture §8.1–8.6

Objective¶

Full CQRS command/query surface and BFF controller for LabRecord lifecycle management. Closes gaps G6, G7, G12, G14.

Key Deliverables¶

Files Created¶

Files Modified¶

Key Design Decisions¶

• AD-22: LabRecordsController replaces LabsController — Unified endpoint surface under /labrecords/ prefix. Old LabsController removed from package exports. All 16 endpoints follow Architecture §8.1 contract.
• AD-23: POST consistency for all mutations — All state-changing operations (start, stop, wipe, delete, clone, archive, bind, unbind) use POST method for consistency, even though some could semantically be DELETE.
• Self-contained CQRS pattern — Each command/query file contains both the request dataclass and its handler class, following the established project convention.
• Structural integration tests — Controller tests validate instantiation, route paths, HTTP methods, and Pydantic request models rather than full HTTP-level integration, matching the established test_lablet_controllers.py pattern.

Phase 9: Lab Discovery V2 & Reuse¶

Status: ✅ Complete | Completed: 2026-02-11 | Tasks: 12/12 | Tests: 60 new (26 discovery + 34 resolution/reuse) Plan Reference: LabRecord Implementation Plan §5 | Architecture: LabRecord Architecture §7

Objective¶

Evolve lab discovery in lablet-controller to use typed LabRecord lifecycle, add lab reuse logic to the reconciler, and implement binding/run tracking. Closes gaps G4, G5, G15.

Key Deliverables¶

Files Created¶

Files Modified¶

Key Design Decisions¶

• AD-24: Wipe-for-reuse pattern — During STOPPING, labs are wiped but NOT deleted. Labs remain on the worker in WIPED state, available for reuse by future instances. This reduces instantiation time from ~90s (cold import) to ~20s (restart wiped lab).
• AD-25: Graceful binding/run tracking — Binding creation, release, and run recording are all graceful operations. Failures are logged but never block the main instantiation/stopping flow.
• AD-26: Dual-mode discovery — LabDiscoveryService supports both V2 (typed statuses, checksums, orphan detection) and legacy sync modes, controlled by a single feature flag.

Phase 10: Labs Frontend¶

Status: ✅ Complete | Completed: 2026-02-12 | Tasks: 8/10 (P10-9, P10-10 deferred to Phase 11) | Tests: 0 (Vitest infrastructure deferred) Plan Reference: LabRecord Implementation Plan §6 | Architecture: LabRecord Architecture §9.4

Objective¶

Dedicated Labs management page in the UI for admin operations on LabRecords. Closes gaps G9, G10 (partial — P10-9 deferred).

Key Deliverables¶

Deferred Tasks¶

Files Created¶

Files Modified¶

Key Design Decisions¶

• AD-27: LcmDataTable with inline btn-group actions — Labs table uses the same btn-group btn-group-sm inline action pattern as Workers, with .lcm-row-action class for event delegation via LcmDataTable._onDelegatedClick(). Context-sensitive buttons per status (e.g., Start only for STOPPED/WIPED/DEFINED).
• AD-28: LabDetailModal with 4 tabs (Overview/Runs/Topology/Revisions) — Tabs lazy-load data on activation. Overview tab shows identity, status, worker, resources, and binding count inline. Runs/Topology/Revisions tabs fetch via dedicated API endpoints.
• AD-29: Instance-level annotations for LabRecordState — Neuroglia JsonSerializer._deserialize_object() uses get_type_hints() which requires class-level annotations. LabRecordState fields annotated at class level (not just __init__). Defensive getattr patterns removed.

Phase 11: LabletRecordRun & Session Model¶

Status: ✅ Complete | Completed: 2026-02-13 | Tasks: 25/25 | Tests: 102 backend (pytest) + 136 frontend (Vitest) Plan Reference: LabRecord Implementation Plan §7 | Architecture: LabRecord Architecture §3.4, §8.7–8.10, §9.1–9.3

Objective¶

Create the LabletRecordRun cross-aggregate runtime entity, Sessions page for session-centric UX, and complete deferred Phase 10 tasks (binding UI, Vitest infrastructure). Closes gaps G10, G18, G19, G22, G23.

Key Deliverables¶

Deferred Phase 10 Tasks Completed¶

Files Created¶

Files Modified¶

Key Design Decisions¶

• AD-30: Vitest in CPA UI for plain JavaScript — Added Vitest test infrastructure directly to control-plane-api ui/ package (separate from lcm_ui TypeScript tests). Config uses jsdom environment for DOM testing. Component tests mock EventBus and API modules.
• AD-31: Binding cross-reference in WorkerDetailsModal — Labs tab cross-references CML-native labs with LabRecords by lab_id, fetches bindings for matching records, and displays active/released bindings inline.
• AD-32: Lazy-load pattern for LabletInstanceCard runs — Active runs loaded via listRuns({ lablet_instance_id }) only for non-terminal instances, with graceful failure handling. Keeps initial card render fast.

Bug Fixes¶

• CreateLabletRecordRunCommand handler: self.not_found("LabletLabBinding", binding_id) passed a string instead of the class, causing AttributeError: 'str' object has no attribute '__name__' at runtime. Fixed to self.not_found(LabletLabBinding, binding_id).

Phase 12: LDS Session Integration¶

Status: ✅ Complete | Completed: 2025-07-08 | Tasks: 16/17 (P12-16 Vitest deferred) | Tests: 30 backend (pytest) Plan Reference: LabRecord Implementation Plan §8 | Architecture: LabRecord Architecture §9.5

Objective¶

Integrate LDS (Lab Delivery System) session lifecycle with LabletRecordRuns. Provides IFRAME-based LDS session embedding with postMessage bridge, CML admin dashboard, and full CQRS backend for LDS session provisioning/start/pause/resume/end operations.

Key Deliverables¶

Files Created¶

Files Modified¶

Key Design Decisions¶

• AD-33: postMessage bridge protocol for LDS IFRAME — Parent→LDS messages prefixed lcm: (pause, resume, end), LDS→Parent messages prefixed lds: (status, grade_request, timer_update). Origin validation on both sides. Follows architecture §9.5 specification.
• AD-34: SSE broadcasting from command handlers — LDS events broadcast directly from command handlers via SSEEventRelay.broadcast_event() (not via domain events), since LabletRecordRun is a plain Entity not an AggregateRoot. Consistent with Phase 11 pattern.
• AD-35: Dual IFRAME panels (LDS + CML) — LDS session IFRAME has full postMessage bridge for interactive session control. CML dashboard IFRAME is read-only with no postMessage (proctor monitoring only). Both follow LcmGrafanaPanel IFRAME loading pattern (spinner, error state, 8-second fallback timeout).
• AD-36: P12-16 Vitest deferred — IFRAME component testing requires jsdom/happy-dom environment with postMessage simulation. Deferred to avoid blocking Phase 12 completion. Can be added incrementally.

Session Entity Model Migration (MVP Phase 7)¶

Status: ✅ Complete | Completed: 2026-02-20 | Sub-Phases: 11/12 (7I deferred) | Tests: 933 pass across 4 services (CPA 589, RS 92, WC 100, LC 152) Plan Reference: MVP Implementation Plan §10 | Execution Plan: phase-7-session-migration.md ADRs: ADR-020, ADR-021, ADR-022

Objective¶

Replace the LabletInstance/LabletRecordRun/LabletLabBinding entity model with a consolidated LabletSession aggregate + 3 child entities (UserSession, GradingSession, ScoreReport) per ADR-020/021/022. Consolidate runtime state, eliminate redundant entities, and prepare the codebase for Phase 5 (Grading Integration).

Sub-Phase Completion¶

Key Deliverables¶

Architectural Decisions (AD-P7-01 through AD-P7-08)¶

Verification Results¶

Requirements Tracking Matrix¶

The following sections track per-service implementation status for every functional and non-functional requirement from the Requirements Specification. These tables are organized by requirement area (FR/NFR), independent of implementation phases.

FR-2.1: LabletDefinition Management¶

Port Allocation (ADR-004)¶

FR-2.2: LabletSession Lifecycle (formerly LabletInstance)¶

FR-2.2.2–2.2.4: LabletSession Attributes & Operations (formerly LabletInstance)¶

FR-2.2.5: LDS Integration¶

FR-2.2.6: CloudEvent Handling¶

FR-2.2.7: CollectAndGrade Command¶

FR-2.3: Scheduling & Reservations¶

FR-2.4: Worker Capacity Management¶

FR-2.5: Auto-Scaling¶

FR-2.6: Assessment & Grading Integration¶

CloudEvents (FR-2.6.3)¶

Background Jobs Migration (ADR-011)¶

Status: ✅ Complete \u2014 All background jobs migrated from APScheduler to controller-based reconciliation loops.

ADR-036 Resource Hierarchy Migration¶

Status: 🔄 In Progress — Batches A–G ✅ Complete, Batch I ⬜ Not Started Plan Reference: ADR-036 Phase 2 Implementation Plan

Promoting all managed aggregates into the ResourceState → TimedResourceState hierarchy defined in lcm_core. Provides uniform state_history, desired_status, timeslot, and managed_lifecycle support across all resource types.

Remaining gap: LabletDefinitionState is the only managed aggregate still extending raw AggregateState[str]. Batch I will promote it to TimedResourceState (Layer 2) — definitions are time-bounded templates that expire when their timeslot ends. desired_status initially unused (None), reserved for future reconciliation. created_by → owner_id mapping.

NFR-3: Non-Functional Requirements (Post-MVP)¶

These requirements are tracked for production readiness. Some are partially addressed through ongoing development (observability, security), while others require dedicated hardening effort after MVP completion.

Blocking Dependencies¶

Revision History¶