DWDaniel WallerEngineering. Leadership. Stories.
All Factory Floor stories

Lessons from the Factory Floor / 22

When Your Backup Plan Breaks Too

"There's been a loud bang from the generator." Not exactly the radio call you want when an entire site relies on one generator for power.

We had a remote storage site alongside the railway line. It was no longer used for manufacturing, but it still held finished products waiting for customer deliveries. And without electricity, we couldn't operate the site.

When we arrived, coolant was everywhere. The generator's radiator had burst, and the engine had shut down.

After investigating, we found the cause. The drive belt for the coolant pump had snapped. Without coolant circulating, the engine had overheated until the radiator gave way.

We had no power, deliveries were waiting, and the generator wasn't going to be repaired that afternoon. So we arranged a temporary replacement.

It was the middle of winter, and by eight o'clock that evening we were still outside in freezing temperatures getting everything connected and running. Eventually, we got the site back online. Problem solved. Or so we thought.

Two days later, the temporary generator broke down. I'm not joking. The water pump had failed. We'd lost our original generator, brought in a replacement to keep the business operating, and now that had failed as well.

Fortunately, the hire company responded quickly and had an engineer on site almost immediately.

Meanwhile, we'd ordered a replacement radiator and drive belt for our original generator. Rather than replacing the whole unit, we carried out the repairs ourselves, recommissioned it and returned it to reliable service.

But that experience stayed with me. When the original generator failed, our immediate priority was getting power restored. And we'd achieved that. What we hadn't expected was for the contingency plan to fail almost immediately afterwards.

It's easy to feel you've got a problem under control once you've put a backup in place. But a backup is still another piece of equipment, and equipment can fail.

As an engineering manager, you can plan for breakdowns, have suppliers ready and arrange temporary equipment. But you also need to consider what happens when the solution you've put in place doesn't work.

That doesn't mean having endless contingency plans for every possible failure. It means understanding how exposed the business remains, even after the immediate problem appears to be solved.

Because sometimes getting through the first breakdown is only half the job. The real challenge comes when Plan B decides it doesn't fancy working either.

Have you ever had a contingency plan fail almost as soon as you'd put it in place?