Pack fine-tuning samples into fixed-length sequences
For supervised fine-tuning, pack many tokenized samples of different lengths into sequences of a fixed maximum length so little space is wasted. Return the packed sequences with the boundaries and attention or position information needed so samples do not attend to one another. Discuss the packing heuristic and samples longer than the limit. Only the title of this report was public.
No answer yet. Write the first one; the best answer gets endorsed.
