On August 1 2012, the New York Stock Exchange started to record significantly unusual levels of activity at 0930, as the markets opened. Trade rates were running at 30% above normal. By the end of the day one trader alone, Knight Capital Group Inc, had burnt over $440 million. The trades damaged market values for the whole day and almost destroyed Knight completely.

Technical analysts Nanex have posted a great analysis of the pattern of trades that enabled them to identify the likely origin of the trades, and to present a reasonable theory that seems to fix the facts: Knight seem to have released an internal trade testing application onto their production servers. It tested the live NYSE market. It lost money, because making money was not a requirement for testing.

It is hard to come up with a test environment for software that acts in a realistic manner, especially when you do not want to let the software itself be aware that it is being tested (because that will change its behaviour, and then it isn’t a proper test, is it?). It is also hard to construct tests that have to change the system state, when testing things that write to databases for example. And if you do accidentally run the test in a live environment, you can always recover from backups, right?

No, not always. Not $440 million’s worth of real-world money …