Matmul performance modeling on a fixed GPU
Estimate wall-clock time for matrix multiplications of varying shapes on a given accelerator from arithmetic intensity and memory movement, then extend the model to a multi-layer pipeline.
Assume an H100-class GPU: 989 TFLOP/s dense BF16, 3.35 TB/s HBM, 132 SMs, ~4 µs kernel launch.
from dataclasses import dataclass
@dataclass
class GPU:
peak_flops: float = 989e12
bandwidth: float = 3.35e12
sms: int = 132
launch_s: float = 4e-6
tile: int = 128
def matmul_time(m, k, n, gpu=GPU(), dtype_bytes=2, waves=False):
"""-> {"flops", "bytes", "intensity", "bound", "seconds"}"""
def layer_time(tokens, d, ffn_mult=4, gpu=GPU()): ...
def pipeline_time(stage_s, stages, microbatches): ... # -> (seconds, bubble fraction)
Sign in to see the worked answer
The full answer, chapter by chapter, with what interviewers expect at each level. Free with an account: one full answer per company.