0

Design a multi-tenant fine-tuning and LoRA serving platform

by Frontier Interview· about 3 hours ago· ⑂ 0 remixes

Question: Design a multi-tenant fine-tuning and LoRA serving platform

Companies: Mistral AI

Difficulty: Hard

Problem statement

A model provider lets customers fine-tune its open-weight models on their own data through an API, then call the fine-tuned models through the same chat endpoint. Most use LoRA adapters. Design the platform: dataset validation, scheduling training jobs on shared GPUs, evaluating results before deployment, and serving thousands of fine-tuned variants on shared base-model replicas with near-base latency. Work out adapter sizes, how many fit per GPU, load times and training time.

Requirements

A model provider lets customers fine-tune its open-weight models on their own data through an API and then call their fine-tuned models like any other. Most customers fine-tune with LoRA (small low-rank adapter matrices on top of frozen base weights). Design the platform: training jobs and serving thousands of fine-tuned variants without a dedicated GPU for each.

Functional requirements

  • Upload a dataset, validate it, start a fine-tuning job with a few hyperparameters, watch progress, get a model id.
  • Evaluate the result before it can be deployed (quality against the base model, safety checks).
  • Serve every fine-tuned model through the same chat API, billed per token, with the same latency as the base model.
  • Isolate customers: data, adapters and usage are private to the customer.

Non-functional requirements

  • Fine-tuning a typical job (10M tokens, 3 epochs) finishes in under an hour once scheduled.
  • Serving: thousands of adapters on a shared fleet; time to first token within about 20% of the base model even for an adapter that was not in GPU memory.
  • GPU utilization high on both sides; idle adapters cost almost nothing.

Back-of-the-envelope estimation

  • Base model: 24B parameters in bf16 is 48 GB of weights; two 80 GB GPUs per replica with tensor parallelism leave room for the KV cache.
  • Adapter size: rank 16 on the 7 projection matrices of 40 layers. Each matrix gets r × (d_in + d_out) parameters, about 16 × 13,000 ≈ 210,000 on average, so 7 × 40 × 210,000 ≈ 58M parameters, about 117 MB in bf16. 1% of the base model.
  • Adapters per GPU: reserving 12 GB of GPU memory gives about 100 resident adapters. 5,000 customer adapters do not fit, so most live in host memory (a 1 TB host holds about 8,500) and in object storage.
  • Load times: host to GPU over PCIe at about 25 GB/s: 117 MB in about 5 ms. Object storage to host at about 1 GB/s: about 120 ms. So a cold adapter costs a few milliseconds if it is in host memory and a noticeable fraction of a second if not.
  • Training: about 6 × N × D FLOPs (the backward pass through frozen weights still costs about the same as full training): 6 × 24e9 × 3e7 tokens = 4.3e18. On 8 GPUs at about 1e15 FLOP/s and 40% utilization: about 22 minutes.

API design

  • POST /v1/files (JSONL chat examples) → file_id; validation runs asynchronously.
  • POST /v1/fine_tuning/jobs {base_model, training_file, validation_file?, hyperparameters: {epochs, learning_rate, rank}} → job_id
  • GET /v1/fine_tuning/jobs/{id} → status, metrics (train and validation loss), fine_tuned_model when done
  • POST /v1/chat/completions {model: "ft:base:org:suffix:id", messages}: the same endpoint as base models

Data model

  • datasets(id, org_id, uri, rows, tokens, validation_report)
  • ft_jobs(id, org_id, base_model, dataset_id, hyperparameters, status, gpu_hours, metrics, adapter_id)
  • adapters(id, org_id, base_model, uri, rank, size_bytes, eval_report, status: pending_eval | deployable | blocked)
  • usage(org_id, model, tokens_in, tokens_out, hour)

High-level design

Build it one requirement at a time. Each drawing adds to the one before; the shaded boxes are what that step adds.

1) Fine-tune: a job queue in front of GPU trainers

The fine-tuning API validates the dataset (format, size, token counts, content policy) and queues a job. A scheduler packs jobs onto GPU nodes by size and priority. Trainers load the frozen base model, train only the adapter, report metrics, and write the adapter to an adapter store.

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"cust","x":0,"y":0,"width":180,"height":70,"label":{"text":"Customer"}},
 {"type":"rectangle","id":"ftapi","x":340,"y":0,"width":180,"height":70,"label":{"text":"Fine-tune API"}},
 {"type":"rectangle","id":"sched","x":680,"y":0,"width":180,"height":70,"label":{"text":"Job scheduler"}},
 {"type":"rectangle","id":"train","x":1020,"y":0,"width":180,"height":70,"label":{"text":"GPU trainers"}},
 {"type":"ellipse","id":"store","x":1350,"y":-5,"width":220,"height":80,"label":{"text":"Adapter store"}},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":160,"height":0,"start":{"id":"cust"},"end":{"id":"ftapi"},"label":{"text":"dataset"}},
 {"type":"arrow","id":"a2","x":520,"y":35,"width":160,"height":0,"start":{"id":"ftapi"},"end":{"id":"sched"},"label":{"text":"jobs"}},
 {"type":"arrow","id":"a3","x":860,"y":35,"width":160,"height":0,"start":{"id":"sched"},"end":{"id":"train"}},
 {"type":"arrow","id":"a4","x":1200,"y":35,"width":150,"height":0,"start":{"id":"train"},"end":{"id":"store"}}
]}

2) Serve many adapters on shared base-model replicas

Every replica serves the base model plus whichever adapters its requests need. A router sends requests for the same adapter to the same few replicas, so adapters stay warm. Each model server keeps hot adapters in a GPU slot pool, warm ones in host memory, and loads cold ones from the adapter store; a single batch can mix requests for different adapters.

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-80,"text":"Shaded: added in this step","fontSize":16},
 {"type":"rectangle","id":"cust","x":0,"y":0,"width":180,"height":70,"label":{"text":"Customer"}},
 {"type":"rectangle","id":"ftapi","x":340,"y":0,"width":180,"height":70,"label":{"text":"Fine-tune API"}},
 {"type":"rectangle","id":"sched","x":680,"y":0,"width":180,"height":70,"label":{"text":"Job scheduler"}},
 {"type":"rectangle","id":"train","x":1020,"y":0,"width":180,"height":70,"label":{"text":"GPU trainers"}},
 {"type":"ellipse","id":"store","x":1350,"y":-5,"width":220,"height":80,"label":{"text":"Adapter store"}},
 {"type":"rectangle","id":"app","x":0,"y":320,"width":180,"height":70,"label":{"text":"Customer app"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"api","x":340,"y":320,"width":180,"height":70,"label":{"text":"Chat API"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"router","x":680,"y":320,"width":180,"height":70,"label":{"text":"Adapter-aware\nrouter"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"serve","x":1020,"y":320,"width":180,"height":70,"label":{"text":"Model servers"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":160,"height":0,"start":{"id":"cust"},"end":{"id":"ftapi"},"label":{"text":"dataset"}},
 {"type":"arrow","id":"a2","x":520,"y":35,"width":160,"height":0,"start":{"id":"ftapi"},"end":{"id":"sched"},"label":{"text":"jobs"}},
 {"type":"arrow","id":"a3","x":860,"y":35,"width":160,"height":0,"start":{"id":"sched"},"end":{"id":"train"}},
 {"type":"arrow","id":"a4","x":1200,"y":35,"width":150,"height":0,"start":{"id":"train"},"end":{"id":"store"}},
 {"type":"arrow","id":"a5","x":180,"y":355,"width":160,"height":0,"start":{"id":"app"},"end":{"id":"api"},"label":{"text":"requests"}},
 {"type":"arrow","id":"a6","x":520,"y":355,"width":160,"height":0,"start":{"id":"api"},"end":{"id":"router"}},
 {"type":"arrow","id":"a7","x":860,"y":355,"width":160,"height":0,"start":{"id":"router"},"end":{"id":"serve"}},
 {"type":"arrow","id":"a8","x":1430,"y":75,"width":-250,"height":245,"start":{"id":"store"},"end":{"id":"serve"},"label":{"text":"adapters"}}
]}

3) Gate deployment on evaluation, meter usage

A finished adapter is not deployable until an evaluation step compares it with the base model on the customer's validation set and runs safety checks (fine-tuning can remove refusal behavior). Usage is metered per organization and model for billing and quotas.

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-80,"text":"Shaded: added in this step","fontSize":16},
 {"type":"rectangle","id":"cust","x":0,"y":0,"width":180,"height":70,"label":{"text":"Customer"}},
 {"type":"rectangle","id":"ftapi","x":340,"y":0,"width":180,"height":70,"label":{"text":"Fine-tune API"}},
 {"type":"rectangle","id":"sched","x":680,"y":0,"width":180,"height":70,"label":{"text":"Job scheduler"}},
 {"type":"rectangle","id":"train","x":1020,"y":0,"width":180,"height":70,"label":{"text":"GPU trainers"}},
 {"type":"rectangle","id":"eval","x":1020,"y":160,"width":180,"height":70,"label":{"text":"Eval gate"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"ellipse","id":"store","x":1350,"y":-5,"width":220,"height":80,"label":{"text":"Adapter store"}},
 {"type":"rectangle","id":"app","x":0,"y":320,"width":180,"height":70,"label":{"text":"Customer app"}},
 {"type":"rectangle","id":"api","x":340,"y":320,"width":180,"height":70,"label":{"text":"Chat API"}},
 {"type":"ellipse","id":"meter","x":320,"y":475,"width":220,"height":80,"label":{"text":"Usage meter"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"router","x":680,"y":320,"width":180,"height":70,"label":{"text":"Adapter-aware\nrouter"}},
 {"type":"rectangle","id":"serve","x":1020,"y":320,"width":180,"height":70,"label":{"text":"Model servers"}},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":160,"height":0,"start":{"id":"cust"},"end":{"id":"ftapi"},"label":{"text":"dataset"}},
 {"type":"arrow","id":"a2","x":520,"y":35,"width":160,"height":0,"start":{"id":"ftapi"},"end":{"id":"sched"},"label":{"text":"jobs"}},
 {"type":"arrow","id":"a3","x":860,"y":35,"width":160,"height":0,"start":{"id":"sched"},"end":{"id":"train"}},
 {"type":"arrow","id":"a9","x":1110,"y":70,"width":1,"height":90,"start":{"id":"train"},"end":{"id":"eval"}},
 {"type":"arrow","id":"a10","x":1200,"y":180,"width":200,"height":-110,"start":{"id":"eval"},"end":{"id":"store"},"label":{"text":"approved"}},
 {"type":"arrow","id":"a5","x":180,"y":355,"width":160,"height":0,"start":{"id":"app"},"end":{"id":"api"},"label":{"text":"requests"}},
 {"type":"arrow","id":"a6","x":520,"y":355,"width":160,"height":0,"start":{"id":"api"},"end":{"id":"router"}},
 {"type":"arrow","id":"a7","x":860,"y":355,"width":160,"height":0,"start":{"id":"router"},"end":{"id":"serve"}},
 {"type":"arrow","id":"a8","x":1480,"y":75,"width":-280,"height":265,"start":{"id":"store"},"end":{"id":"serve"},"label":{"text":"adapters"}},
 {"type":"arrow","id":"a11","x":430,"y":390,"width":1,"height":85,"start":{"id":"api"},"end":{"id":"meter"}}
]}

Final design

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"cust","x":0,"y":0,"width":180,"height":70,"label":{"text":"Customer"}},
 {"type":"rectangle","id":"ftapi","x":340,"y":0,"width":180,"height":70,"label":{"text":"Fine-tune API"}},
 {"type":"rectangle","id":"sched","x":680,"y":0,"width":180,"height":70,"label":{"text":"Job scheduler"}},
 {"type":"rectangle","id":"train","x":1020,"y":0,"width":180,"height":70,"label":{"text":"GPU trainers"}},
 {"type":"rectangle","id":"eval","x":1020,"y":160,"width":180,"height":70,"label":{"text":"Eval gate"}},
 {"type":"ellipse","id":"store","x":1350,"y":-5,"width":220,"height":80,"label":{"text":"Adapter store"}},
 {"type":"rectangle","id":"app","x":0,"y":320,"width":180,"height":70,"label":{"text":"Customer app"}},
 {"type":"rectangle","id":"api","x":340,"y":320,"width":180,"height":70,"label":{"text":"Chat API"}},
 {"type":"ellipse","id":"meter","x":320,"y":475,"width":220,"height":80,"label":{"text":"Usage meter"}},
 {"type":"rectangle","id":"router","x":680,"y":320,"width":180,"height":70,"label":{"text":"Adapter-aware\nrouter"}},
 {"type":"rectangle","id":"serve","x":1020,"y":320,"width":180,"height":70,"label":{"text":"Model servers"}},
 {"type":"arrow","id":"a1","x":180,"y":35,"width":160,"height":0,"start":{"id":"cust"},"end":{"id":"ftapi"},"label":{"text":"dataset"}},
 {"type":"arrow","id":"a2","x":520,"y":35,"width":160,"height":0,"start":{"id":"ftapi"},"end":{"id":"sched"},"label":{"text":"jobs"}},
 {"type":"arrow","id":"a3","x":860,"y":35,"width":160,"height":0,"start":{"id":"sched"},"end":{"id":"train"}},
 {"type":"arrow","id":"a9","x":1110,"y":70,"width":1,"height":90,"start":{"id":"train"},"end":{"id":"eval"}},
 {"type":"arrow","id":"a10","x":1200,"y":180,"width":200,"height":-110,"start":{"id":"eval"},"end":{"id":"store"},"label":{"text":"approved"}},
 {"type":"arrow","id":"a5","x":180,"y":355,"width":160,"height":0,"start":{"id":"app"},"end":{"id":"api"},"label":{"text":"requests"}},
 {"type":"arrow","id":"a6","x":520,"y":355,"width":160,"height":0,"start":{"id":"api"},"end":{"id":"router"}},
 {"type":"arrow","id":"a7","x":860,"y":355,"width":160,"height":0,"start":{"id":"router"},"end":{"id":"serve"}},
 {"type":"arrow","id":"a8","x":1480,"y":75,"width":-280,"height":265,"start":{"id":"store"},"end":{"id":"serve"},"label":{"text":"adapters"}},
 {"type":"arrow","id":"a11","x":430,"y":390,"width":1,"height":85,"start":{"id":"api"},"end":{"id":"meter"}}
]}

Deep dives

Why LoRA changes the serving problem

A fully fine-tuned 24B model is another 48 GB to keep on GPUs; serving 5,000 of them would need thousands of mostly idle GPUs. With LoRA the base weights are shared, and each customer model adds about 117 MB. The forward pass computes x·W once for the whole batch and adds each request's own x·A·B (a low-rank product, about 1% extra compute). Batching kernels that gather each request's adapter (segmented or "multi-LoRA" matrix multiplies) let one batch mix many adapters, so an adapter with one request a minute costs almost nothing.

The adapter memory hierarchy

Three tiers: GPU slots (about 100 per replica), host memory (thousands), object storage (all). On a miss, the server loads the adapter into a free slot (5 ms from host) while the request waits in the queue; it must never evict an adapter that an in-flight batch is using. The batch builder also caps how many distinct adapters one batch carries, since the gather kernels slow down as adapter count grows.

from collections import OrderedDict, deque
from typing import Deque, Dict, List, Tuple


class AllSlotsPinned(Exception):
    pass


class AdapterCache:
    """GPU slots for LoRA adapters: LRU eviction, never evicting one in use."""

    def __init__(self, slots: int, host_load_ms: float = 5.0) -> None:
        self.slots = slots
        self.host_load_ms = host_load_ms
        self.resident: "OrderedDict[str, int]" = OrderedDict()   # adapter -> in-flight count
        self.hits = self.misses = 0
        self.load_ms = 0.0

    def acquire(self, adapter: str) -> bool:
        """Pin an adapter for a batch; returns True on a cache hit."""
        if adapter in self.resident:
            self.resident[adapter] += 1
            self.resident.move_to_end(adapter)
            self.hits += 1
            return True
        if len(self.resident) >= self.slots:
            victim = next((a for a, pins in self.resident.items() if pins == 0), None)
            if victim is None:
                raise AllSlotsPinned(adapter)
            del self.resident[victim]
        self.resident[adapter] = 1
        self.misses += 1
        self.load_ms += self.host_load_ms
        return False

    def release(self, adapter: str) -> None:
        if self.resident.get(adapter, 0) <= 0:
            raise ValueError(f"{adapter} is not pinned")
        self.resident[adapter] -= 1


def next_batch(queue: Deque[Tuple[str, str]], max_requests: int, max_adapters: int) -> List[Tuple[str, str]]:
    """Take requests in arrival order, but at most `max_adapters` distinct
    adapters per batch; requests for other adapters wait for the next batch
    (they keep their place, so nobody starves)."""
    batch: List[Tuple[str, str]] = []
    adapters: Dict[str, None] = {}
    skipped: Deque[Tuple[str, str]] = deque()
    while queue and len(batch) < max_requests:
        req_id, adapter = queue.popleft()
        if adapter in adapters or len(adapters) < max_adapters:
            adapters[adapter] = None
            batch.append((req_id, adapter))
        else:
            skipped.append((req_id, adapter))
    queue.extendleft(reversed(skipped))
    return batch

from collections import deque as _deque


def test_lru_eviction_skips_pinned_adapters():
    cache = AdapterCache(slots=2)
    assert cache.acquire("a") is False and cache.acquire("b") is False
    cache.release("b")
    assert cache.acquire("c") is False              # evicts b (unpinned), not a (in use)
    assert "a" in cache.resident and "b" not in cache.resident
    try:
        cache.acquire("d")                          # a and c both pinned
        assert False
    except AllSlotsPinned:
        pass
    cache.release("a")
    cache.release("c")
    assert cache.acquire("c") is True
    assert (cache.hits, cache.misses, cache.load_ms) == (1, 3, 15.0)


def test_batches_cap_distinct_adapters_without_starving():
    q = _deque([("r1", "a"), ("r2", "b"), ("r3", "c"), ("r4", "a"), ("r5", "d")])
    assert next_batch(q, max_requests=10, max_adapters=2) == [("r1", "a"), ("r2", "b"), ("r4", "a")]
    assert list(q) == [("r3", "c"), ("r5", "d")]    # waiting requests keep their order
    assert next_batch(q, max_requests=10, max_adapters=2) == [("r3", "c"), ("r5", "d")]

def test_routing_by_adapter_raises_hit_rate():
    import random
    rng = random.Random(0)
    adapters = [f"ad{i}" for i in range(300)]
    weights = [1 / (i + 1) for i in range(300)]        # a few popular adapters, a long tail
    reqs = rng.choices(adapters, weights=weights, k=20000)

    def run(route):
        servers = [AdapterCache(slots=40) for _ in range(4)]
        for i, ad in enumerate(reqs):
            s = servers[route(i, ad)]
            s.acquire(ad)
            s.release(ad)
        hits = sum(s.hits for s in servers)
        return hits / len(reqs)

    random_routing = run(lambda i, ad: i % 4)
    sticky_routing = run(lambda i, ad: int(ad[2:]) % 4)
    assert sticky_routing > random_routing + 0.15

The routing test shows the point of an adapter-aware router: spreading requests for the same adapter across all replicas makes every replica cache every popular adapter; sending each adapter to a home replica (with a second replica for hot adapters, and overflow when a replica is busy) multiplies effective cache capacity by the number of replicas.

Scheduling fine-tuning jobs

Jobs vary from minutes to hours. Pack them by GPU count (most LoRA jobs fit on one 8-GPU node), with per-organization quotas so one customer cannot fill the cluster, and preemption: training jobs checkpoint every few minutes and yield to higher-priority work, including serving capacity at traffic peaks (the same GPUs can do both). Validation runs before a job takes a GPU, so a malformed dataset fails in seconds, not after an hour in the queue.

Evaluation and safety

Every adapter is evaluated before deployment: validation loss against the base model (did it learn, did it overfit?), task metrics if the customer supplies them, and a fixed safety suite, since a small amount of fine-tuning data can weaken refusals. Adapters that fail are blocked with a report; borderline ones need review. Datasets themselves are screened for policy violations.

Isolation and data handling

Datasets and adapters are encrypted per organization; only that organization's keys can call ft:…:org:… models. Customer data is never used to train shared models unless they opt in. Training jobs run in containers without network egress beyond the storage they need.

Trade-offs

  • LoRA versus full fine-tuning. LoRA is cheap to train, store and serve and usually close in quality; full fine-tuning can do better on large domain shifts but needs dedicated replicas. Offer full fine-tuning as a premium tier with reserved capacity.
  • Adapter count per batch. More distinct adapters per batch improves fairness and cache use but slows the gather kernels; cap it and let the router keep batches coherent.
  • Merged weights for hot adapters. A customer with heavy steady traffic is cheaper to serve with the adapter merged into a dedicated copy of the weights (no extra per-token compute); move them when their traffic justifies it.
  • Shared GPUs for training and serving. Sharing raises utilization but needs preemption and checkpointing; separate pools are simpler to operate.

Interviewer expectations

Mid-level (L4)

  • A job pipeline: dataset upload and validation, queued training, status and metrics, and a model id that works in the chat API.
  • Knows what LoRA is and why it makes adapters small.
  • Basic per-organization isolation and usage metering.

Senior (L5)

  • Works out adapter size, adapters per GPU, load times per tier and training time with numbers.
  • Serves many adapters on shared replicas with mixed-adapter batching, an LRU slot pool that never evicts in-use adapters, and an adapter-aware router.
  • Gates deployment on evaluation, including safety regressions from fine-tuning.

Staff (L6+)

  • Designs the scheduling economics: quotas, preemption, sharing GPUs between training and serving, moving heavy users to merged weights.
  • Quantifies the routing and caching gains and the cost of cold starts at the tail.
  • Covers data governance (encryption, no cross-customer training, egress controls) as first-class requirements.

Sign in to comment, vote or remix.