Non-Monotonic Error Propagation

Non-monotonic error propagation occurs when improvements to one part of an ML system decrease the overall system quality.

In traditional software, fixing a bug improves the system. In ML systems with multiple interacting models, improving one model can make the whole system worse, because the rest of the system was tuned to accommodate the original model’s particular failure modes.

This happens when:

• Downstream models have learned to compensate for upstream model errors
• Evaluation metrics for individual components don’t capture system-level behavior
• Models introduce new, previously unseen mistakes that break interactions with components that depend on them

The practical implication: you can’t optimize ML components independently. System-level evaluation must accompany component-level improvements, and “better” on a benchmark doesn’t guarantee “better” in production.

This is a concrete example of Goodhart’s Law applied to ML systems, optimizing a component against its local metric can undermine the global objective.

Related: 05-atom—model-entanglement