Design a VM bandwidth rate limiter

Cap the network bandwidth each virtual machine in a cloud region may use, enforcing per-tenant limits across many hosts without adding latency to packets.

We are building the system that caps the network bandwidth of every VM in a cloud region: a control plane that stores each VM's bandwidth policy and pushes it to the hosts, and a per-host data plane (agent plus hypervisor or NIC shaper) that enforces it on every packet without adding measurable latency.

Clarifying questions worth asking first: is the limit per VM, per vNIC, or per tenant across VMs? Egress only, or ingress too? Can a VM borrow idle bandwidth from its host? Do we control the NIC (SmartNIC or SR-IOV) or only a software vSwitch? How fast must a policy change take effect?

Functional requirements

• Set a bandwidth policy per VM: guaranteed rate (floor), maximum rate (ceiling), burst size, and optional separate egress/ingress values.
• Optional tenant aggregate cap: all VMs of one tenant on one host, or across the region, share a ceiling.
• Enforce egress by shaping (queue and pace) and ingress by policing or shaping at the host.
• Work-conserving: a VM may use idle host bandwidth up to its ceiling, never below its floor when it needs it.
• Policy changes (resize, plan upgrade) apply within seconds without restarting the VM.
• Expose per-VM counters: bytes, packets, drops, time throttled.

Non-functional requirements

• Data-path overhead under 5 µs per packet p99 and no added latency for VMs under their limit.
• Line-rate on 100–200 Gbps host NICs with 50+ VMs per host.
• Accuracy within ~2% of the configured rate over a 1 s window.
• Hosts keep enforcing the last-known-good policy if the control plane is unreachable.
• Isolation: a noisy VM cannot push a neighbour below its guaranteed rate.

Back-of-the-envelope estimation

• Region: 50k hosts, 50 VMs each → 2.5M VMs; policy record ~200 B → 500 MB total. Fits in one replicated store.
• Policy churn: VM creates, deletes and resizes at ~500/s peak → 500 host updates/s, each a few hundred bytes.
• Data plane per host: 100 Gbps at 1500 B MTU ≈ 8.3M packets/s; at 64 B worst case ~150M pps. Per-packet budget at 8M pps on 8 cores ≈ 1 µs per packet per core, so no locks or syscalls on the hot path.
• Token bucket state per VM per direction: ~64 B. 50 VMs × 2 → 6.4 KB, sits in L1/L2.
• Counters: 2.5M VMs × 10 metrics every 10 s → 2.5M points/s into telemetry, aggregated on host first.

Public (tenant-facing, via the compute API):

PUT /v1/vms/{vm_id}/bandwidth-policy
{ "egress":  { "guaranteed_mbps": 2000, "max_mbps": 10000, "burst_kb": 1024 },
  "ingress": { "max_mbps": 10000, "burst_kb": 2048 },
  "priority": "standard" }
→ 202 { "policy_version": 41, "status": "pending" }

GET /v1/vms/{vm_id}/bandwidth-policy
→ 200 { ...policy, "policy_version": 41, "applied_version": 41, "host_id": "h-1832" }

Control plane to host agent (gRPC, server-streaming so pushes are immediate):

rpc WatchPolicies(WatchRequest{host_id, last_seen_revision})
    returns (stream PolicyUpdate{revision, upserts[VmPolicy], deletes[vm_id], full_snapshot bool})
rpc AckApplied(AckRequest{host_id, revision, failures[{vm_id, error}]}) returns (Empty)

Supporting endpoints: GET /v1/vms/{vm_id}/bandwidth-stats?window=1m (bytes, drops, throttled_ms); POST /internal/hosts/{host_id}/resync to force a full snapshot; GET /internal/hosts/{host_id}/capacity for the placement service.

Control plane store: a strongly consistent replicated store (etcd/Spanner-style, or Postgres with a regional replica) because placement and admission control need read-your-writes and versioning. Size is small (~500 MB), so consistency matters more than scale.

Revision: one monotonic counter per host partition; every change bumps it so agents can resume from last_seen_revision.

Host agent: the current policy set persisted to local disk (/var/lib/bwagent/policies.pb plus revision) so it survives agent and control-plane restarts. In-kernel/NIC state: per-VM token buckets and HTB classes keyed by vNIC id.

flowchart LR
  U[Tenant or compute API] --> PS[Policy service]
  PS --> DB[(Policy store)]
  PS --> AC[Admission control]
  PL[Placement service] --> AC
  DB --> FO[Fan-out service]
  FO -->|"gRPC stream"| HA[Host agent]
  HA --> DISK[(Local snapshot)]
  HA --> DP[Data plane shaper]
  VM1[VM] --> DP
  DP --> NIC[Physical NIC]
  NIC --> FAB[Region fabric]
  FAB --> NIC
  DP --> TEL[Telemetry]

Flow 1: policy configuration and distribution.

1. The compute API writes the policy; admission control checks that the host's sum of guaranteed rates stays under NIC capacity × headroom, in the same transaction.
2. The store bumps the host's revision; the fan-out service (sharded by host_id, ~5k hosts per instance) pushes the delta on the host's open stream.
3. The agent writes the new set to disk, programs the data plane (tc/HTB, OVS meters, or SmartNIC rate-limiter tables), and acks the revision. Median end-to-end ~1 s.

Flow 2: egress shaping.

1. A packet leaves the VM's virtio queue into the vSwitch or NIC queue for that vNIC.
2. The per-VM class checks its token bucket; tokens available → transmit now (no delay); otherwise the packet waits in that VM's queue until the scheduler releases it.
3. Only the over-limit VM queues; its own TCP sees the delay and backs off. Other VMs are unaffected.

Flow 3: ingress policing.

1. A packet arrives at the host NIC, is classified to the destination vNIC by MAC/VNI.
2. The per-VM ingress bucket checks it; conforming packets go to the VM, excess is dropped (policing) or briefly queued (shaping with a small queue).
3. Drops cause the remote sender's TCP to slow down; persistent abuse (e.g. UDP flood) is reported to fabric-level DDoS protection.

Why it is hard, and HTB mechanics. Bandwidth is a shared, perishable resource: bytes not sent this millisecond are gone, so a static split wastes capacity, and a plain per-VM cap can still let 50 VMs together exceed the NIC. Hierarchical Token Bucket fixes both. Tree: root (NIC rate, e.g. 95 Gbps leaving headroom) → optional tenant class → per-VM leaf. Each leaf has rate (guaranteed) and ceil (max). Scheduling per dequeue:

This is work-conserving: when neighbours are idle a VM can go to its ceiling; when everyone is busy each gets at least its rate. Bucket refill is lazy: tokens = min(burst, tokens + rate × (now − last)) computed on dequeue, no timers per VM. Linux HTB tops out at a few Gbps per qdisc because of a global lock, so at 100 Gbps use multi-queue designs: per-CPU EDT (earliest departure time) pacing with BPF (stamp each packet with a departure time from the VM's rate, let the fq qdisc release it), or offload to NIC hardware.

Ingress vs egress asymmetry. Egress is easy: the packets are ours before they hit the wire, so we can queue and pace them and TCP sees smooth delay. Ingress is hard: by the time a packet reaches the host it has already used the NIC and the fabric link; dropping it saves the VM's CPU but not the wire. Options:

Pick host policing with a generous burst (~2× BDP: 10 Gbps × 1 ms RTT = 1.25 MB, so ~2.5 MB) plus ECN marking above 80% of the bucket, and rely on fabric DDoS protection for floods. Internal east-west traffic between our own hosts can use receiver-driven credits for exactness.

Admission control and overcommit. Guarantees only mean something if the sum of guaranteed rates on a host stays under capacity. Placement treats egress and ingress Gbps as schedulable resources next to vCPU and memory: sum(guaranteed) ≤ 0.9 × NIC per direction, while ceil may be overcommitted (e.g. 3×) because it is best-effort. A resize that breaks the invariant returns 409 or triggers live migration. A tenant region-wide cap is split into per-host shares by recent usage, re-balanced every ~10 s by the control plane (same idea as distributed API rate limiting, but the hot path stays local).

SR-IOV bypass and hardware offload. With SR-IOV the VM writes directly to a NIC virtual function, so the software vSwitch never sees the packet and cannot shape it. Enforcement must move into the NIC: per-VF rate limiters (ip link set ... vf N max_tx_rate), or a SmartNIC/DPU (Nitro, Azure Boost-style) running the shaper in hardware. Hardware gives line rate with zero host CPU but offers fewer classes (often 128–1024 limiters per NIC), coarser granularity and no full HTB borrowing, so the agent maps policy to whatever the NIC exposes and simulates borrowing by adjusting VF ceilings every ~100 ms from usage counters.

Control-plane failure and bursts. The data plane never calls out per packet, so a control-plane outage only freezes changes. The agent keeps last-known-good from disk on restart, reports applied_revision lag, and on reconnect resumes from its revision (full snapshot if the gap is too large). New VMs on a host that cannot reach the control plane get a conservative default (e.g. 1 Gbps ceiling) rather than unlimited. For micro-bursts, the bucket depth sets tolerance: too small (e.g. 1 MTU) and TCP slow start and TSO 64 KB segments get clipped; too big and a VM can blast tens of MB at line rate and overflow the NIC or ToR buffer. Size burst at ~1–2 ms of the VM's ceiling (1.25–2.5 MB at 10 Gbps), and use a second small "peak" bucket at line rate to cap instant bursts.

• HTB with borrowing uses idle capacity well but makes a VM's performance depend on its neighbours; strict caps are predictable but waste bandwidth.
• Software shaping is flexible (any hierarchy, quick to change) but costs host CPU; NIC offload is free at line rate but limited in classes and features.
• Ingress policing is cheap and local but wastes fabric bandwidth and hurts TCP with drops; sender-side enforcement is exact but needs cross-host coordination.
• Strong consistency in the policy store slows writes slightly but keeps admission control correct; the data plane is fully local, so it never pays that cost.
• Last-known-good keeps traffic flowing during outages but delays urgent downgrades (e.g. an abusive tenant) until the control plane is back.
• Large bursts help TCP and short transfers; small bursts protect shared switch buffers.
• Per-host splitting of a regional tenant cap is cheap but can be briefly over or under the global number between rebalances.

Mid-level (L4)

• Separates control plane (policies) from data plane (per-packet enforcement on the host).
• Uses a per-VM token bucket and explains rate vs burst.
• Pushes policies to hosts and caches them locally; no per-packet remote calls.

Senior (L5)

• Explains HTB guaranteed/ceiling and work-conserving borrowing.
• Distinguishes egress shaping from ingress policing and why ingress is weaker.
• Adds admission control on guaranteed rates, versioned policy sync with acks, and last-known-good on failure.
• Gives per-packet CPU budget numbers and burst sizing.

Staff (L6+)

• Handles 100 Gbps limits of software qdiscs (EDT, multi-queue, offload) and SR-IOV bypass.
• Designs regional tenant caps with per-host shares and rebalancing.
• Discusses ECN or receiver-driven credits, noisy-neighbour isolation, and how to roll out shaper changes safely across 50k hosts.