The Revenge of the Data Scientist
Summary
The article argues that data scientists remain essential even as LLMs take on more routine tasks, focusing on how to evaluate AI systems through data-driven experiments. It highlights five common eval pitfalls (generic metrics, unverified judges, bad experimental design, bad data and labels, automating too much), connections to core data-science activities, and practical guidance like treating judges as classifiers and grounding synthetic data in real production data.