Design a job scheduler for ETL pipelines
Design a system that runs ETL jobs on schedules and as dependency graphs: defining pipelines, triggering, distributing work to workers, retries, backfills, monitoring and avoiding duplicate runs. Only the title of this report was public.
MicrosoftSystem designNo answer yet. Write the first one; the best answer gets endorsed.