Execution agents that learn to punish

That trading agents can reach supra-competitive outcomes without being told to cooperate is not new. Christos Spyridon Koulouris and Carlo Campajola go after the harder question: whether what the agents learn amounts to collusion in the economic sense, which requires a mechanism that deters defection rather than merely a comfortable equilibrium.

The setting is a two-player, finite-horizon Almgren-Chriss liquidation game, the standard model for splitting a large order over time against price impact. Independent agents trained with proximal policy optimisation, each able to see prices and actions within the episode, achieved execution costs below the Nash benchmark.

Testing for a punishment

To find out whether a mechanism was holding that together, the authors first trained an agent against the mean learned liquidation schedule to construct a profitable deviation, then forced that deviation's first trade on one of the original agents. The opponent responded by accelerating its own liquidation. That response more than cancelled the deviator's gain in every run and in both player roles, while leaving the punisher's own average payoff materially unchanged compared with not punishing at all.

The last clause is what makes the result uncomfortable. Retaliation that costs the retaliator is a threat that may not be carried out; retaliation that costs nothing is a stable deterrent. The authors note that less punitive and more profitable liquidation plans were available, and the trained policies did not take them. They also track the behaviour through training: matched deviations and extra selling rise and then decline, while the final policies keep an effective punitive response.

Two checks are stated formally, and both hold for the deviations tested: that the punishment outweighs the gain from deviating, and that the change in trading behaviour is large enough to account for the loss imposed. For anyone deploying learned execution agents, the finding is that behaviour regulators would call collusive can emerge from independent training, with no communication and no shared objective.