All articles
Operations by Ismail Karankin

MTTR at 3am: Cutting Pages for Routine Network Faults

Many overnight pages trigger a familiar runbook. Fast relief comes from finding repeatable gaps and moving recovery out of the on-call queue.

Illustration of MTTR and on-call burden

MTTR splits into diagnosis time and execution latency. Diagnosis time covers finding the fault and choosing a remedy. Execution latency covers the wait between the fault and active investigation: alert delivery, access, navigation, and context. On familiar network faults, that waiting period can make up more than half the total repair time.

At 3:12am, a BGP peer flaps. The page reaches the on-call engineer at 3:14am. They log in at 3:19am, reach the device at 3:22am, and run the fix at 3:24am. Total resolution time: 12 minutes. Diagnosis took about 3 minutes. The remaining 9 minutes went to notification delay, login friction, tool navigation, and loading context. This is a routine operations pattern, not an exceptional one.

Which 3am Pages Belong to a Runbook

Some overnight incidents need judgment. Examples include ambiguous failures across several devices, unfamiliar fault patterns, and risky production changes without a clear rollback. Those pages earn a human response because experience can change the outcome. The rest are often requests to carry out a known procedure.

A large share of pages involve repeatable work. A BGP session drops on a connection already known to be unstable. Port errors cross a documented threshold where marginal SFP behavior is familiar. An OSPF adjacency cycles on a link pair with a configuration mismatch recorded after an earlier incident. The fault has precedent, and the remedy is already known. The on-call engineer is following instructions, not investigating an unknown failure.

We spoke with a network operations team at a regional managed service provider overseeing 150 managed devices across six client sites. Over a three-month review, about 65% of after-hours pages involved conditions the team had seen before and had already documented in runbooks. Their page count reflected routing policy more than network complexity: every alert went to a person, whether the response called for judgment or simple execution.

MTTR Math

A single MTTR figure can hide two very different populations. Novel incidents tend to take longer and vary widely because their causes are unclear. Routine incidents should resolve quickly, yet often do not. Both groups pass through the same human approval and execution path, so the easy cases inherit the delays of the hard ones.

Break MTTR apart by fault type, then isolate the waiting period before work begins. That is where routine cases offer the clearest gain. A BGP peer flap resolved in 18 minutes from the first alert may need only 3 minutes of diagnosis. The other 15 minutes are pre-diagnosis overhead. Removing that delay is a workflow change, not a demand for faster engineers.

A practical target combines detection, diagnosis, and the repair itself. A BGP soft reset takes seconds to run. If the system recognizes the peer flap within 30 seconds and completes the reset within 60 seconds of detection, the result is achievable. The limiting factor is workflow design, not device hardware.

On-Call Fatigue Has a Cost

Routine pages cost more than the minutes spent resolving them. Repeated interruptions wear down attention and can reduce the quality of response when a difficult incident finally arrives.

An engineer who handled four BGP resets and interface bounces overnight is less ready for a complex 3am event. Lost sleep accumulates across rotations. High page volume also makes on-call roles harder to retain, increasing the chance that a less experienced engineer covers an incident that calls for deeper judgment.

Low signal quality creates another hazard. When the pager is crowded with routine alerts, an engineer may decide, "this is probably the same BGP thing again" before reading the details. That shortcut usually works for familiar faults. If the alert is the rare complex case, the assumption adds another delay to an already difficult response.

What Automation Changes for Routine Faults

Autonomous remediation changes how the team spends attention, not just the duration of each incident. Routine cases can move through a controlled repair path while engineers remain available for failures that require judgment.

When known fault categories are repaired automatically, each engineer receives fewer pages and a more manageable on-call rotation. Rest improves. Human responders can also give fuller attention to the incidents that remain because routine noise no longer fills the channel.

Postmortems gain value when review time goes to incidents that reveal something new, rather than the fifth recurrence of an identical BGP flap. A cleaner incident queue makes patterns easier to see and gives the team a stronger learning loop.

Incorrect automation can apply the wrong change, extend an outage, or create a second fault. That risk sets clear engineering requirements: score confidence before acting, make each repair reversible, and log every change so a human can inspect and roll it back. These properties are part of the production design, not optional safeguards added later.

Where to Start

Do not begin by automating the entire network. First identify the three to five fault categories behind most after-hours pages, then confirm that each has a tested and documented runbook.

For each category, ask two questions. How certain is the fault classification from the available signal? Can the repair be reversed quickly if that classification is wrong? Favorable answers make autonomous execution a good candidate. Those cases can bring MTTR down to minutes rather than 15-25 minutes and remove the pages from the human queue.

The practical aim is a smaller, safer on-call queue, not a fully autonomous operations center. Start with a modest set of repeatable faults, measure the results, and expand when the evidence supports it. Lower MTTR and fewer overnight interruptions should show up quickly.

Cut Routine Pages Before They Reach Your On-Call Team

PhysicalLayer executes remediation actions on known fault patterns, cutting the volume of pages that actually need a human.