AI Outperforms Human Prompt Engineering

In a head-to-head comparison, automated prompt optimization significantly outperformed manual human effort on a binary classification task.

The human prompt engineer (Schulhoff, lead author and experienced practitioner) spent 20 hours refining a prompt. The automated tool DSPy generated a superior prompt in 10 minutes. After minor adjustments, the AI-generated prompt achieved an F1 score approaching 0.6, meaningfully higher than the manually crafted version.

DSPy works by treating prompting as a programming problem: it generates candidate examples and explanations, then iteratively optimizes based on performance feedback. The paradigm treats prompts as optimizable artifacts rather than fixed instructions.

This doesn’t mean human expertise is irrelevant, understanding what to optimize for and how to evaluate outputs still requires human judgment. But the mechanical work of iterating on prompt text may increasingly shift to automated systems.

Related: [None yet]