Sort a 500 GB CSV by one column with 16 GB of RAM
A CSV file of about 500 GB must be sorted by one of its columns on a machine with 16 GB of memory. Describe the external merge sort: cutting the file into chunks that fit in memory, sorting each, writing runs, then a k-way merge with a heap. Discuss I/O cost, how many runs to merge at once, parsing quoted CSV fields, and alternatives such as loading it into a database or spreading the work across machines.
No answer yet. Write the first one; the best answer gets endorsed.
