Empty beds and queasy buses: the traps of optimising everything

About 95% of adult hospital beds in England were occupied at the height of last year’s flu season, the Guardian reports. If hospitals were hotels, that level of occupancy would be good business, and we would commend them for their efficiency. But do we really want a hospital to have so few beds in reserve that it is one bad flu week away from treating patients in corridors?

This is the subtle thing about efficiency. We usually assume that more of it must be better. But we can become so focused on improving one thing that we lose sight of the others. You can try to arrange your calendar so that every minute is accounted for, but you will soon start running over and end up behind. A gap between meetings might seem like inefficiency, but it is what allows your schedule to recover when something unexpected happens.

Mathematicians have a name for the pursuit of efficiency: optimisation. Give us something to minimise or maximise – cost, time, fuel, profit – and we will try to find the best possible answer. For most of human history, people optimised by common sense. Computers have enormously expanded what can be effectively tackled, from airline schedules to supply chains. The gains for society have been extraordinary. But there are at least three traps to beware of.

The hospital bed story illustrates the first: weakening a system’s robustness by squeezing out too much slack. An empty bed can be redundant on a quiet Tuesday night, but it is somewhere precious to put the next patient when flu cases surge. If you judge a system by its performance under ordinary conditions, you can lose the ability to cope with extraordinarily challenging ones.

A second trap is having a too-narrow definition of “best”. Milton Keynes, designed by urban planners in the late 1960s, has an American-style grid system. As the city was built out, roundabouts rather than traffic lights became a defining feature of its road junctions; today there are about 130 of them. The roundabouts were great for traffic flow, but they came with a now-notorious cost. On the coach from Oxford to Cambridge, an hour in, you get great views of Milton Keynes and a busload of queasy passengers as the coach negotiates one roundabout after another. Traffic flow is easy to measure; passenger comfort is harder to quantify. Optimisation gives you the best answer to the question you asked, but can deliver unwelcome answers to questions you failed to consider.

A third trap appears when attempts to optimise collide. On 6 May 2010, US financial markets gave a dramatic demonstration. In the middle of an already falling market, the S&P 500 suddenly plunged another 5% in just five minutes. Within 10 minutes, the market had recovered. Hundreds of individual securities briefly traded at absurdly low prices. One trigger was an automated program selling 75,000 futures contracts, worth more than $4bn. Other automated traders reacted nearly instantly, their responses fed back on one another, and liquidity evaporated. The algorithms’ actions may have been individually sensible in isolation, but they combined to produce collective turmoil.

AI is one of the most spectacular triumphs of optimisation in history. During training, enormous numbers of parameters are repeatedly adjusted so that a model gets better at predicting language, writing computer programs, answering scientific questions and giving legal advice. Unfortunately, AI now has the potential to make all three traps more treacherous than ever.

The first trap could tempt organisations to dispense with the human oversight that looks redundant in the age of quicker-witted algorithms. Why retain experienced staff when a machine can handle the routine chores? But in cutting that human capacity, you may lose the ability to spot and handle exceptional circumstances.

The second arises if we regard intelligence as whatever scores highest on a leaderboard. We are remarkably good at building AI systems that do well on measures such as user satisfaction, but we are finding it much harder to optimise them for judgment or empathy. Researchers have found that training algorithms on human preferences can encourage AI to sycophantically agree with us rather than tell us uncomfortable truths.

Then there is the third. AI systems are increasingly interacting with one another, often at blinding speeds and in vast numbers – especially in the era of agentic AI – and we worry their interactions could generate unpredictable, dangerous behaviour. Recently, an OpenAI agent broke into a non-public Medicare statistics portal in Australia, while public logs suggested that hundreds of other agents were coordinating attempts to gain access to Australian government health data. Although those attempts have not been shown to be part of the Medicare breach, they revealed something disturbing: hundreds of agents apparently working together in unauthorised ways.

As optimisation becomes more powerful, avoiding these traps becomes more important. The challenge is not simply finding the best answer. It is deciding what “best” really means – and knowing what not to optimise away. Steven Strogatz and Alex Townsend are mathematics professors at Cornell University and the authors of Big Math (John Murray).