Router & Dispatch

Blocking load balancer with reservation-based scheduling, health monitoring, and async queuing.

The router is a blocking load balancer. Requests wait until a model is free — no skipping, no rejecting, no timeouts from contention. All plugins get served. Models are shared infrastructure.

Synchronous Dispatch

The simplest way to call an LLM. The router selects the best model for the task automatically.

response = engine.dispatch(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Analyze this text..."},
    ],
    task="extraction",              # determines which model gets picked
    max_tokens=2048,
    temperature=0.7,
)
print(response)  # model's response text

Reservation-Based Scheduling Rust

The canonical path for model selection. Atomically sweeps expired reservations, checks capacity, scores candidates, and reserves.

Select and Reserve

sel = engine.select_and_reserve({
    "task_class": "extraction",     # extraction|reasoning|synthesis|generation|evaluation
    "min_tier": "small",            # small|medium|large
    "min_context": 4096,
    "thinking": False,
    "execution": "stateless",       # stateless|session|exclusive
    "latency": "medium",            # low|medium|high
    "determinism": False,
    "typical_tokens": 1024,
    "priority": 2,                  # 0=critical, 1=high, 2=normal, 3=bulk
    "plugin_id": "research",
})

if sel:
    print(f"Reserved {sel['model']} (id={sel['reservation_id']})")
else:
    print("No model available")

Task Descriptor Fields

Record Result (preferred release path)

engine.record_result_v2(sel["reservation_id"], success=True, latency_ms=250.0)

Release Without Result (fallback)

Sweep Expired Reservations

Health Management

The router tracks model health automatically based on request results.

Health State Transitions

Async Queue

For bulk work, use the async queue. Tasks are submitted and processed in the background across all available models.

Submit

task_id = engine.queue_submit(
    prompt="Extract beliefs from this document...",
    task="extraction",
    priority=2,                     # 0=critical, 1=high, 2=normal, 3=bulk
    system_prompt="You are a belief extraction engine.",
    max_tokens=4096,
)

Poll Result

result = engine.queue_result(task_id)
# {
#     "status": "pending" | "running" | "complete" | "error",
#     "response": "...",        # only when complete
#     "latency_ms": 1234,       # only when complete
#     "error": "...",            # only when error
# }

Queue Status

status = engine.queue_status()
# {
#     "depth": 5,               # items waiting
#     "active": 2,              # currently processing
#     "completed": 47,          # total completed
#     "workers": 3,             # background worker count
#     "by_priority": {0: 0, 1: 1, 2: 3, 3: 1},
# }

Model Info

models = engine.get_model_status()
for m in models:
    print(f"{m['name']}: {m['health']} ({m['in_flight']} in-flight)")
    print(f"  {m['tokens_per_second']} tok/s, p95={m['p95_latency']}ms")

Plugin Scoping & Fairness

Control which models a plugin can use and ensure fair access across plugins.

engine.set_plugin_scope(
    "research",
    allowed=["local-*", "cloud-*"],
    preferred=["local-primary"],
    blocked=["small-*"],
)

Exclusive Locks & Session Stickiness

For workflows requiring model consistency:

Embeddings

Vector embedding utilities for semantic operations.