Lablet Controller Guide¶

Overview¶

The Lablet Controller is a Kubernetes-style controller that manages the lifecycle of LabletInstances on CML Workers. It handles lab import, startup, collection, and teardown operations.

Key Concept: A LabletInstance is a composite resource consisting of:

• CML Lab: Infrastructure running on a CML worker (managed via CML API)
• LabSession: User-facing session in LDS (managed via LabDeliverySPI)

Both components must be provisioned for a LabletInstance to be considered fully operational.

Architecture¶

See Lablet Controller Architecture for detailed design.

Core Responsibilities¶

Key Flows¶

Lab Provisioning Flow¶

sequenceDiagram
    participant RS as Resource Scheduler
    participant etcd as etcd (State Store)
    participant LC as Lablet Controller
    participant CML as CML Worker

    RS->>etcd: Write: Instance assignment
    LC->>etcd: Watch: Instance assignments
    etcd-->>LC: Notify: New assignment
    LC->>CML: POST /api/v0/labs: Import lab
    CML-->>LC: 201: lab_id
    LC->>CML: PUT /api/v0/labs/{id}/state: START
    CML-->>LC: 200: Started
    LC->>etcd: Write: Instance status=READY

LDS Session Provisioning Flow¶

sequenceDiagram
    participant LC as Lablet Controller
    participant CML as CML Worker
    participant LDS as Lab Delivery System
    participant S3 as S3/MinIO

    Note over LC: LabletInstance in INSTANTIATING

    LC->>CML: Start lab (CML API)
    CML-->>LC: Lab running

    LC->>S3: Fetch content.xml
    S3-->>LC: Device definitions

    LC->>LDS: create_session_with_part()
    LDS-->>LC: session_id, login_url

    LC->>LDS: set_devices(session_id, devices)
    LDS-->>LC: OK

    Note over LC: Transition to READY (awaiting user login)

User Login Flow (CloudEvent)¶

sequenceDiagram
    participant User
    participant LDS as Lab Delivery System
    participant CPA as Control-Plane-API
    participant DB as MongoDB

    User->>LDS: Login via login_url
    LDS->>LDS: Validate session token
    LDS->>CPA: POST /api/cloudevents<br/>type: session.started
    CPA->>DB: Lookup by lds_session_id
    CPA->>CPA: Validate state == READY
    CPA->>DB: Update: READY → RUNNING
    CPA-->>LDS: 202 Accepted

Port Allocation¶

flowchart LR
    A[Instance Scheduled] --> B{Ports Needed?}
    B -->|Yes| C[Reserve from Worker Pool]
    C --> D[Assign to Instance]
    D --> E[Configure Console Access]
    B -->|No| E
    E --> F[Lab Started]

API Endpoints¶

Configuration¶

Key environment variables:

CML Labs API Integration¶

The Lablet Controller uses these CML endpoints:

LDS Integration¶

The Lablet Controller integrates with the Lab Delivery System (LDS) to provision user-facing sessions. See ADR-018 LDS Integration for architectural decision.

LabDeliverySPI¶

The LabDeliverySPI abstraction provides:

Device Mapping¶

Device access info is derived from:

1. content.xml - Device labels and definitions
2. cml.yaml - Node topology with device_label annotations
3. Port Allocation - Assigned external ports
4. DeviceAccessInfo - Final payload sent to LDS

@dataclass
class DeviceAccessInfo:
    name: str           # Device label
    protocol: str       # ssh, telnet, https
    host: str           # Worker hostname
    port: int           # Allocated port
    uri: str | None     # Optional URI
    username: str       # Device credentials
    password: str       # Device credentials

Content Refresh¶

When a LabletDefinition is versioned:

1. Content updated in S3 bucket
2. LCM API triggers refresh
3. Lablet Controller calls refresh_content(form_qualified_name)
4. LDS refreshes content from S3

See FR-2.1.6 for requirements.

Collect and Grade¶

When the user completes the lab, external systems trigger the CollectAndGradeCommand:

sequenceDiagram
    participant Exam as Exam System
    participant CPA as Control-Plane-API
    participant LC as Lablet Controller
    participant CML as CML Worker
    participant GE as Grading Engine

    Exam->>CPA: POST /api/instances/{id}/collect-and-grade
    CPA->>CPA: Validate state == RUNNING
    CPA->>CPA: Transition: RUNNING → COLLECTING
    CPA-->>Exam: 202 Accepted

    LC->>LC: Observe COLLECTING state
    LC->>CML: Extract node configs
    CML-->>LC: Config artifacts

    LC->>LC: Transition: COLLECTING → GRADING
    LC->>GE: Submit artifacts + rubric
    GE-->>LC: grading_score

    LC->>CPA: Store grading_score
    LC->>LC: Transition: GRADING → STOPPING

See FR-2.2.7 for requirements.

Related Documentation¶

• Lablet Controller Architecture
• ADR-018 LDS Integration
• FR-2.2.5 LabSession Provisioning
• FR-2.2.6 CloudEvent Handling
• FR-2.2.7 Collect and Grade
• ADR-017 Lab Operations via Lablet Controller
• Worker Controller Guide - For CML System API