Worker Templates¶

Version: 1.0.0 (February 2026) Scope: Resource Scheduler (template selection) + Control Plane API (template management) Phase: Phase 3 - Auto-Scaling

1. Overview¶

Worker templates define predefined EC2 instance configurations for CML workers. They serve as the capacity model for:

• Placement decisions — matching lablet instances to appropriately-sized workers
• Scale-up provisioning — selecting which EC2 instance type to launch
• Cost optimization — choosing the cheapest template that satisfies requirements

Templates are managed as configuration (not user-created entities) per ADR-007. They are loaded from YAML on startup and seeded to MongoDB.

2. Template Lifecycle¶

flowchart TD
    YAML["config/worker_templates.yaml<br/>(5 template definitions)"] --> Load

    subgraph startup["Application Startup"]
        Load["WorkerTemplateService<br/>load_templates_from_yaml()"]
        Validate["Validate & parse<br/>each template definition"]
        Seed["seed_templates_async()<br/>upsert by name"]
        Load --> Validate --> Seed
    end

    Seed --> MongoDB[("MongoDB<br/>worker_templates collection")]

    MongoDB --> Query["Query at runtime"]

    Query --> Placement["PlacementEngine<br/>_select_template()"]
    Query --> ScaleUp["RequestScaleUpCommand<br/>_resolve_template()"]
    Query --> API["API queries<br/>list/get templates"]

    style YAML fill:#FFF9C4,stroke:#F57F17
    style MongoDB fill:#E3F2FD,stroke:#1565C0
    style startup fill:#E8F5E9,stroke:#2E7D32

Startup Seeding (WorkerTemplateSeederHostedService)¶

The WorkerTemplateSeederHostedService runs during application startup:

1. Reads config/worker_templates.yaml
2. Parses each template definition into WorkerTemplate aggregates
3. Upserts by name — existing templates are updated, new ones created
4. Logs count of seeded templates

3. Capacity Model¶

Template Definitions¶

WorkerCapacity Value Object¶

The WorkerCapacity dataclass is an immutable value object used to represent and compare capacity:

@dataclass(frozen=True)
class WorkerCapacity:
    cpu_cores: int
    memory_gb: int
    storage_gb: int
    max_nodes: int | None = None

Key operations:

4. Template Selection Algorithm¶

The placement engine uses a 3-tier fallback strategy to select a template for scale-up:

flowchart TD
    Start["_select_template(requirements)"] --> T1

    subgraph tier1["Tier 1: Cost-Optimized"]
        T1["Get enabled templates"] --> T1F["Filter: capacity >= requirements"]
        T1F --> T1S["Sort by cost_per_hour ASC"]
        T1S --> T1C{"Matches?"}
    end

    T1C -->|Yes| Result1["✅ Cheapest viable template"]

    T1C -->|No| T2

    subgraph tier2["Tier 2: Largest Available"]
        T2["Get all enabled templates"] --> T2S["Sort by CPU cores DESC"]
        T2S --> T2C{"Any templates?"}
    end

    T2C -->|Yes| Result2["⚠️ Largest template + warning"]

    T2C -->|No| T3

    subgraph tier3["Tier 3: Hardcoded Fallback"]
        T3{"Required CPU?"} -->|">= 32"| Metal["metal"]
        T3 -->|">= 16"| Large["large"]
        T3 -->|">= 4"| Medium["medium"]
        T3 -->|"< 4"| Small["small"]
    end

    style tier1 fill:#E8F5E9,stroke:#2E7D32
    style tier2 fill:#FFF3E0,stroke:#E65100
    style tier3 fill:#FFEBEE,stroke:#C62828
    style Result1 fill:#4CAF50,color:white
    style Result2 fill:#FF9800,color:white

TemplateSelection Result¶

The select_optimal_template_async method returns a TemplateSelection dataclass:

@dataclass
class TemplateSelection:
    template: WorkerTemplate        # Selected template
    match_reason: str               # Why this template was chosen
    excess_capacity: WorkerCapacity  # Unused capacity above requirements
    cost_ranking: int               # 0 = cheapest matching option

Headroom Selection¶

For workloads expecting spikes, select_template_with_headroom_async adds a capacity buffer:

$$ \text{adjusted_capacity} = \text{required_capacity} \times \left(1 + \frac{\text{headroom\%}}{100}\right) $$

Default headroom is 20%, meaning a requirement of 10 CPU cores would search for templates with ≥ 12 cores.

5. Domain Model¶

WorkerTemplate Aggregate¶

flowchart TD
    subgraph agg["WorkerTemplate (AggregateRoot)"]
        direction TB
        State["WorkerTemplateState"]
        Events["Domain Events"]
    end

    subgraph fields["State Fields"]
        direction TB
        F1["name: str (unique)"]
        F2["instance_type: Ec2InstanceType"]
        F3["capacity: WorkerCapacity"]
        F4["cost_per_hour_usd: float"]
        F5["ami_name_pattern: str"]
        F6["enabled: bool"]
    end

    subgraph events["Events"]
        direction TB
        E1["WorkerTemplateCreatedDomainEvent"]
        E2["WorkerTemplateUpdatedDomainEvent"]
        E3["WorkerTemplateDisabledDomainEvent"]
        E4["WorkerTemplateEnabledDomainEvent"]
    end

    State --> fields
    Events --> events

    style agg fill:#E3F2FD,stroke:#1565C0

Event-Sourced State Transitions¶

Ec2InstanceType Enum¶

Instance type mapping from friendly names:

6. WorkerTemplateService API¶

Service Methods¶

Custom Exceptions¶

7. YAML Configuration Format¶

Templates are defined in config/worker_templates.yaml:

templates:
  - name: medium                     # Unique identifier
    description: Medium worker...    # Human-readable description
    instance_type: medium            # Friendly name or AWS type
    ami_name_pattern: 'CML-*'        # AMI name filter pattern
    capacity:
      cpu_cores: 2                   # Logical CPU cores
      memory_gb: 4                   # Memory in GB
      storage_gb: 100                # Root volume in GB
      max_nodes: 15                  # Soft limit on CML nodes
    cost_per_hour_usd: 0.0416       # Approximate hourly cost
    enabled: true                    # Available for provisioning

Instance Type Resolution¶

The _create_template_from_dict method resolves instance types through a 3-step chain:

1. Friendly name → micro, small, medium, large, metal → mapped to Ec2InstanceType enum
2. AWS instance type → t3.small, m5zn.metal → parsed as enum value
3. Fallback mapping → m5.xlarge → mapped to closest enum member

8. Configuration Reference¶