The first ten minutes of a customer outage

One of the people I work with told me: "When you open up the server, figure out how long it's been like that." In other words, collect data before you decide anything.

The order below is a summary of the path the people next to me followed during a few real outages. The order itself matters; if you look in the wrong place first, the remaining ten minutes are wasted.

0. Don't reboot

The first rule, and the only one worth writing down: don't reboot first.

It sounds backwards, because most outages go away when you reboot. But going away doesn't mean the cause was fixed — only that the symptom disappeared.

What you lose when you reboot:

The logs explaining why the server got into that state
Disk, memory and network state
The last error messages of the services that crashed
Who changed what, and when

If you have ten minutes, collect evidence first, then think. Collecting evidence is faster than rebooting.

1. Scope it: one thing, or everything?

The first question isn't technical. What is the customer saying, and what are you seeing?

"The site won't open" → network, disk, or DNS?
"It's slow" → is one request slow, or is everything slow?
Are other people in the same building affected too?

That last one is the most valuable question. If it's a single server, the problem is on the server; if it's several, it's the network or something local. It cuts your ten minutes in half.

Then try from your own machine:

ping server
ss -tulpn | grep -E ':(80|443|22)\b'

If you can reach the server and nothing is listening on 80/443, the problem isn't the network — it's on the server.

2. The physical layer: is it really on?

You don't check the power cable on a virtual machine. On a physical server, you do.

Is the power light on?
Is the disk activity light blinking (solid on is as suspicious as never on)?
Are the fans spinning?

Lights on means the server has power, not that it's working. If the lights are on but the disk light isn't, the drive may be the problem.

3. What happened since boot

If the server was just started, the real question is "what happened in the last hour?"

uptime                      # how long it's been up
systemctl --failed          # services that failed
journalctl -p err -b        # errors since THIS boot
journalctl -p err --since "2 hours ago"

uptime prints three numbers, and the one you care about most is the load average. Compare it with the number of cores.

4 cores, load 0.5 → relaxed
4 cores, load 12 → overloaded, a queue is building up

Don't forget the core count; without it the number means nothing:

nproc

4. Disk — the silent killer

A server with a full disk doesn't tell you. It can't, because there may be no room left to write a log.

df -h                  # space used
df -i                  # inodes used — easy to miss

df -i is separate because you can run out of file entries while plenty of space still shows as free. This really happens on servers that have been up for a long time.

Check where the logs went:

du -sh /var/log/* | sort -h | tail
journalctl --disk-usage

5. Memory — did the kernel kill something?

free -m
journalctl -k | grep -i "killed process"

The second command is worth its weight in gold. When memory runs out, Linux picks processes and kills them. It's usually the noisiest one, and from the outside it just looks like "no resources".

If dmesg contains "Out of memory" or "oom-kill", you've found your cause.

6. Network — is the port listening?

ss -tulpn | grep 8080

If something looks "up" but the port isn't listening, the service isn't running. "The site won't open" really means "the service isn't up", and those are not the same thing.

7. What changed

The most useful step, and the most skipped: finding out who changed what, and when.

last                        # who logged in
history | tail -50          # who did what (in the active session)
ls -lt /etc/                # when config files changed
cat /var/log/apt/history.log    # when packages were installed
crontab -l                  # scheduled jobs

If the server worked yesterday and doesn't today, something changed. ls -lt /etc/ shows it in one command: if /etc/hosts changed last night, you've probably found the cause.

8. Certificates — they look like network failures

I'm writing this one separately because it's the most misleading: an expired TLS certificate looks like a network failure.

The user says "I can't connect", you look at the network, and the network is fine. The browser is actually rejecting the certificate.

To check:

openssl s_client -connect site:443 -servername site </dev/null 2>/dev/null \
  | openssl x509 -noout -dates

See whether the end date has passed. Let's Encrypt certificates are valid for 90 days and are renewed continuously. If renewal isn't running, the site shuts itself off after 90 days.

If you run out of time

Sometimes ten minutes isn't enough, the customer is in a hurry, and you have to reboot. Even then, save something first:

journalctl -b > /tmp/before.txt
df -h > /tmp/before-df.txt

Then reboot. You'll lose much less.

The real point of this post

One sentence: measure first, then touch. Every piece of information you collect before you touch anything is worth more than anything you can recover afterwards.

I'd heard the same thing earlier as "read first, then act". Now I understand why it gets repeated so often: in software, every command can be undone; on a physical server, every move is permanent.