Inference loop with continuous batching over a mock model

You get a model function that takes a batch of sequences and returns one next token for each. Write the serving loop: keep a fixed number of batch slots full, retire a sequence when it hits its token limit or a stop token, and immediately refill freed slots from the request queue.