Batching service around an embedding model
Put a service around an embedding model: many callers send texts at once, the service groups them into batches for the model, returns each caller its vectors, and stops without losing work. A variant limits every batch both by number of texts and by total tokens.
No answer yet. Write the first one; the best answer gets endorsed.
