Solution Overview¶

Operator / Author summary¶

The platform turns authored lab content into graded delivery sessions for candidates. Everything operable is modelled as a timed resource with a declared desired state and an observed actual state, reconciled continuously. Two responsibilities are kept strictly apart:

• CPA is the session manager + control plane. It is the only place operators and students enter. It keeps the catalog of definitions, drives each Session (and its Parts) through their lifecycle phases, runs the plumbing (pick a host, find the pod, open ports, register with the delivery system), and hosts the unified resource dashboard.
• SE is the automation engine. Whenever a phase needs something done to the lab — apply changes post-init, collect device output, grade it, produce a report — CPA hands that work to SE. SE talks to the live devices through ROC (and other adapters) and sends results back.

A session may be single-part (the classic Lablet) or multi-part (e.g. an expert exam with DES / DOO / AI-DOO parts), and each part may run 0..N pods of any PodType on any HostType. The same lifecycle/reconciliation machinery applies at every level of the tree.

If a session is stuck provisioning, look at CPA + the LCM controllers. If grading or a report is wrong, look at SE + the content's grading rules.

Architect detail¶

Responsibility map¶

Two sources of truth, by design: CPA treats Mosaic as the SSOT for what content exists (resolved per form-qualified name); SE treats RustFS as the SSOT for the content bytes it executes. The sync flow is what keeps them consistent.

Resource management plane¶

The control plane is a single reconciliation pattern applied recursively. Every resource — Session, SessionPart, PodInstance, Host/Worker — is a TimedResource with a desired_status, a status, a Timeslot, and a ManagedLifecycle. A generic reconcile framework (observe → diff → act → record) is specialised by a per-type manager for each kind. Intent cascades down the tree (operator/scheduler → Session → Part → Pod); observed status bubbles up.

Ownership of reconciliation:

See resource-model.md for the full tree and reconciliation framework, session-model.md for definitions and session profiles, and ui-resource-dashboard.md for the dashboard.

C4 — System context¶

C4Context
    title System Context — Lablet delivery platform

    Person(operator, "Operator", "Triggers syncs, schedules and supervises sessions")
    Person(student, "Candidate", "Consumes a delivery session")
    Person(author, "Content author", "Publishes lab content")

    System_Boundary(lcm, "Lablet Cloud Manager") {
        System(cpa, "Control-Plane API", "Session manager + single entry point")
        System(se, "Scenario Engine", "Automation engine: collect / evaluate / report")
    }

    System_Ext(mosaic, "Mosaic", "Content authoring & publishing (catalog SSOT)")
    System_Ext(roc, "ROC", "RADkit device interaction service")
    System_Ext(lds, "LDS", "Lab Delivery System (student access)")
    System_Ext(keycloak, "Keycloak", "Identity provider")

    Rel(author, mosaic, "Publishes content")
    Rel(operator, cpa, "Syncs labs, runs sessions", "HTTPS")
    Rel(student, cpa, "Joins session", "HTTPS")
    Rel(cpa, se, "Triggers jobs per phase", "HTTP / CloudEvents")
    Rel(cpa, mosaic, "Resolves & downloads content", "HTTPS")
    Rel(cpa, lds, "Registers delivery", "HTTP")
    Rel(se, roc, "Delegates device collection", "HTTP")
    Rel(cpa, keycloak, "Authn/Authz", "OIDC")

    UpdateLayoutConfig($c4ShapeInRow="2", $c4BoundaryInRow="1")

C4 — Container view¶

C4Container
    title Container View — services and stores

    Person(operator, "Operator")

    System_Boundary(lcm, "Lablet Cloud Manager") {
        Container(cpa, "Control-Plane API", "FastAPI + Neuroglia", "Catalog, sessions, phase ordering, UI")
        Container(labletctl, "lablet-controller", "Python service", "Content-sync ingester + lifecycle reconciler")
        Container(sched, "resource-scheduler", "Python service", "Timeslots + worker allocation")
        Container(workerctl, "worker-controller", "Python service", "EC2/CML worker lifecycle")
        Container(se, "scenario-engine", "FastAPI + Neuroglia", "Jobs, scenarios, grading, reports")

        ContainerDb(cpadb, "CPA MongoDB", "MongoDB", "SessionDefinition, Form, Session")
        ContainerDb(sedb, "SE MongoDB", "MongoDB", "Job, JobDefinition, reports")
        ContainerDb(blob, "RustFS / S3", "Object store", "Canonical content bytes (SE SSOT)")
        ContainerDb(etcd, "etcd", "KV / watch", "CPA <-> controller coordination")
    }

    System_Ext(mosaic, "Mosaic", "Catalog SSOT")
    System_Ext(roc, "ROC", "RADkit devices")
    System_Ext(lds, "LDS", "Student access")

    Rel(operator, cpa, "Uses", "HTTPS")
    Rel(cpa, cpadb, "Reads/writes")
    Rel(cpa, etcd, "Writes desired state / watches")
    Rel(labletctl, etcd, "Watches / reconciles")
    Rel(labletctl, mosaic, "Downloads package", "HTTPS")
    Rel(labletctl, blob, "Uploads content", "S3")
    Rel(labletctl, lds, "Triggers content sync", "HTTP")
    Rel(labletctl, se, "Triggers content sync", "HTTP")
    Rel(cpa, sched, "Requests allocation", "HTTP / events")
    Rel(sched, workerctl, "Requests worker ops", "HTTP / events")
    Rel(cpa, se, "Submits jobs", "HTTP")
    Rel(se, sedb, "Reads/writes")
    Rel(se, blob, "Reads content", "S3")
    Rel(se, roc, "Collects device output", "HTTP")
    Rel(se, cpa, "Job results", "CloudEvents")

    UpdateLayoutConfig($c4ShapeInRow="2", $c4BoundaryInRow="1")

Integration style¶

• CPA → SE (commands): synchronous HTTP for triggering a job (submit_job, sync_content). CPA never blocks on job completion.
• SE → CPA (results): asynchronous CloudEvents consumed by CPA's integration-event handlers, which advance the session phase.
• CPA → controllers: desired-state via etcd; controllers reconcile and report progress back through CloudEvents / phase-progress commands.
• SE → ROC: synchronous HTTP using the (session/bucket, bulk_cmd_uuid) contract (POST /devices, POST /execute/bulk, GET/DELETE /execute/bulk/{uuid}).

Why this split¶