Single-pass cleaning pipeline for a million crawled web documents

A take-home: input is a set of Parquet files with about a million documents crawled from public web pages. Write a Python pipeline that reads the input in a single pass and writes out the cleaned documents. Choose and justify the cleaning steps (for example filtering low-quality or non-text pages, normalization and deduplication), keep memory bounded, and report what each step removed.