Runtime Pipelines

Inference in Tribunus Compute is a deterministic state machine over six precompiled pipeline stages. Each stage has a fixed memory budget, known shapes, and prequalified backend regions from the compute image. The scheduler guarantees that no single pipeline can starve the others, and that interactive latency targets are met even under high batch concurrency.

1. Token Intake

The token intake pipeline is the entry point for every inference request. It performs radix-tree prefix matching against the KV cache, allowing requests with shared prefixes to reuse previously computed key-value tensors. The radix tree is compacted at every insert — common prefixes are merged, and cache entries are reference-counted so they are evicted only when all dependent sequences complete.

The tokenizer is embedded in the compute image itself. Rather than loading a separate tokenizer model at runtime, the compute image bundles the precompiled tokenizer vocabulary, merge rules, and sentencepiece/unigram model as a read-only mmap’d table. This eliminates a source of startup latency and guarantees tokenizer consistency across deployments.

2. Prefill

The prefill stage processes the prompt in a single forward pass, computing the initial KV cache entries. It uses chunked flash attention to split long prompts into contiguous blocks that fit within the backend’s SRAM budget, reducing HBM bandwidth by materializing attention scores block by block without writing the full N x N matrix.

KV pages are written exactly once during prefill. Each page stores a fixed number of key-value heads (configurable per model, default 16). The page table is initialized with write-once semantics — once a prefill page is committed, it is immutable until eviction. This guarantees that decode never races with prefill on the same page.

3. Decode

The decode pipeline runs the autoregressive generation loop. It does not compile or JIT kernels — it replays precompiled CUDA Graphs (NVIDIA) or Metal Indirect Command Buffers (Apple Silicon). CUDA Graphs capture the entire kernel launch DAG as a single unit, eliminating driver overhead for the repeated GEMM + attention + activation pattern of autoregressive decoding.

A weight-staging ring buffer keeps model weights in the fastest available memory tier. For NVIDIA backends, weights are staged in SRAM with double buffering so that one weight tile is consumed while the next is prefetched. For Apple Silicon, the unified memory architecture means weights are already resident — the ring acts as a prefetch schedule for the GPU texture cache.

4. KV Cache Management

KV cache management runs as a concurrent pipeline on a dedicated schedule so it never blocks generation. It maintains two tiers: a hot tier for recently accessed sequences (resident in HBM/unified memory) and a compressed tier for older sequences (4-bit or 2-bit quantized with per-head scale factors).

Migration between tiers is driven by generation counters. Each KV page carries a counter incremented on every decode access. When a page’s counter falls below a configurable threshold, the management pipeline copies it to the compressed tier and marks the hot tier slot as available. On re-access, the page is decompressed back to the hot tier with a single kernel call.

5. Speculative Decoding

The speculative decoding pipeline generates draft tokens asynchronously across multiple devices and verifies them in batch. On Apple Silicon hardware, the pipeline uses a three-device hierarchy: ANE generates draft tokens (high throughput, lower quality), CPU runs a small verification model, and GPU performs the final tree verification.

The pipeline speculates a tree of draft sequences rather than a single chain. A draft model proposes K continuations at each step; the verification model scores all K in a single forward pass. Accepted branches continue speculation; rejected branches are discarded and the verifier backfills the correct token. Tree speculation achieves higher acceptance rates than single-chain speculation because the oracle (target model) can choose among multiple plausible continuations.

6. Output Streaming

The output pipeline manages token-by-token delivery to the caller. Logits are computed using Apple’s Accelerate framework (on Apple Silicon) or cuBLAS (on NVIDIA), applying top-k or top-p filtering directly on the logit buffer without materializing the full vocabulary distribution.

Stop detection runs in the output pipeline using a precompiled Aho-Corasick automaton. The automaton is built at compile time from the user-provided stop-words list and bundled into the compute image. Each decoded token is tested against the automaton in O(n) time independent of the number of stop patterns. On a match, the pipeline signals the scheduler to terminate generation immediately.

Tokens are streamed to the caller over Server-Sent Events (SSE) with configurable buffering. The output pipeline also handles tool-calling requests: when the model emits a function-call token, the pipeline pauses token streaming, executes the tool via a pre-registered sandbox, injects the tool result as a new prompt segment, and resumes generation from the updated KV state.