Hidden Feedback Loops

Two systems can influence each other indirectly through the world, even when they share no direct data connection.

Direct feedback loops, where a model influences its own training data, are at least recognizable as a statistical challenge. Hidden loops are harder: the influence path runs through real-world behavior rather than data pipelines.

Example: Two systems determine different facets of a web page. Improving one changes user behavior, which changes the performance metrics of the other. Neither system “knows” about the dependency.

More concerning: two models from different organizations can create hidden loops. Stock prediction models from competing firms may influence each other’s bidding behavior. Improvements (or bugs) in one propagate to the other through market dynamics.

These loops may exist between completely disjoint systems, making them extremely difficult to detect or model.

Related:, 05-atom—undeclared-consumers, 05-molecule—ml-technical-debt-taxonomy