Stream deduplication with near-duplicate suppression
Pass through a stream of strings, keeping only the first copy of each. Then also drop strings that are near-duplicates of something already kept, above a similarity threshold, and discuss the cost of exact sets versus approximate similarity methods.
No answer yet. Write the first one; the best answer gets endorsed.