Hands-on Lab·LPIC-2

Finding what is consuming the machine

Topic 200 asks you to measure resource usage and predict future needs. This guide puts a real CPU load on the machine and names the process responsible, replaces it with a real I/O load and shows every number changing to a different set, then reads back the record sysstat has been keeping for days without anyone installing an agent.

201: Capacity Planning Guide 27 of 29 Intermediate

OSUbuntu 26.04 LTS
Kernel7.0.0-30-generic
systemd259
Block devicesone 50 GB disk
TimeAbout 26 min

One host. Nothing is installed - sysstat, pidstat and iostat are already present, and both loads are disposable scripts removed at the end.
Server Name	IP Address	OS	Roles	CPU	RAM	HDD
LPIC2-A01	192.168.0.78	Ubuntu 26.04 LTS	Primary service host - BIND, Apache, Samba, Postfix. Topic 204 RAID runs on loop devices; this machine has no spare disk	2 Core	4 GB	50 GB

This guide includes

Use this when a machine is slow and you need to say what is causing it, or when someone asks how long the current disk will last. This matters because the numbers that say a machine is busy do not say what is making it busy, and because a prediction needs history that has to have been collected before you were asked.

naming the process behind a load with pidstat -u rather than guessing from top
telling a CPU-bound machine from an I/O-bound one by wa and %util
reading an empty iostat -xz table as a measurement rather than a failure
explaining why niced CPU time appears as %nice and not %user
reading yesterday's and last week's figures with sar -f
turning a size and a rate into a date, which is what capacity planning means

Before you start

load-memory-and-what-the-numbers-mean

A quiet machine, described in numbers
The baseline, and two loops that will spoil it.
bash Example session
```
$nproc; echo "CPUs"; uptime; vmstat 1 3 | tail -22CPUs 10:06:47 up 1 day, 23:38,  1 user,  load average: 0.32, 0.47, 0.26 0  0      0 1970692 145816 1065396    0    0     0     0  160   74  0  0 100  0  0  0 0  0      0 1970692 145816 1065396    0    0     0     0  147   90  0  0 100  0  0  0$sudo tee /tmp/xk0cpu.sh > /dev/null <<'SH'#!/bin/shecho $$ > /tmp/xk0cpu$1.pidwhile :; do :; doneSH$sudo chmod +x /tmp/xk0cpu.sh; sudo setsid /tmp/xk0cpu.sh 1 > /dev/null 2>&1 & sudo setsid /tmp/xk0cpu.sh 2 > /dev/null 2>&1 & sleep 3; cat /tmp/xk0cpu1.pid /tmp/xk0cpu2.pid; echo "--- two busy loops, one per CPU"6273262734--- two busy loops, one per CPU
```
Note
A baseline is the measurement people skip and then wish they had. "Load is 1.5" means nothing without knowing it is normally 0.1 on a two-CPU box.

nproc is why the load number is readable at all. A load average of 2 on two CPUs is fully committed; the same 2 on thirty-two CPUs is nearly idle. Always pair the two.

vmstat 1 3 and discard the first line - it is an average since boot, not a current reading. Every guide on this path takes the tail for that reason.

A while :; do :; done loop is the cheapest honest CPU load there is. It needs no package, uses one core, and dies the instant it is killed. The scripts write their own $$ so the PID file names the process that is actually spinning.

Expected result2 CPUs, load average: 0.12, 0.11, 0.09, and vmstat reporting id 100 - then two PIDs, one busy loop per CPU.

Success conditionYou have a recorded baseline to compare everything else against.
Busy is not a diagnosis
The load is real. Now say what is causing it.
bash Example session
```
$sleep 60; uptime; vmstat 1 3 | tail -2; echo "--- us user, sy system, id idle, wa waiting on I/O" 10:07:53 up 1 day, 23:39,  2 users,  load average: 1.55, 0.79, 0.38 2  0      0 1990188 145824 1065424    0    0     0     0 2064   72 100  0  0  0  0  0 2  0      0 1990188 145824 1065424    0    0     0     0 2066   67 100  0  0  0  0  0--- us user, sy system, id idle, wa waiting on I/O$pidstat -u 1 2 | tail -4Average:      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  CommandAverage:        0     61153    0.00    0.50    0.00    0.00    0.50     -  kworker/0:1-eventsAverage:        0     62732  100.00    0.00    0.00    0.50  100.00     -  xk0cpu.shAverage:        0     62734  100.00    0.00    0.00    0.50  100.00     -  xk0cpu.sh$ps -eo pid,ni,comm,%cpu,etimes --sort=-%cpu | head -4; echo "--- ni is the nice value, and it is why iostat reports this as %nice and not %user"    PID  NI COMMAND         %CPU ELAPSED  62734   5 xk0cpu.sh       99.7      68  62732   5 xk0cpu.sh       99.6      68  62544   0 systemd          0.0      70--- ni is the nice value, and it is why iostat reports this as %nice and not %user
```
Note
us 100 says the CPU is doing userspace work and wa 0 says nothing is waiting on disk. That pair alone halves the search before any per-process tool is opened.

pidstat -u 1 2 is the answer top gives less clearly. It prints a per-process table plus an Average block over the interval, which is scriptable and does not depend on catching the right moment on screen. pidstat -p follows one process.

Load 1.55 after a minute, from two pinned CPUs. Load average is a decaying average - the one-minute figure was still climbing toward 2 and the five-minute figure was far behind. A load average read too early understates a load that just started, and overstates one that just ended.

NI 5 matters for the next step. These loops inherited a nice value from the session that started them, and niced CPU time is accounted separately from ordinary user time - which is why iostat is about to report %nice 100.00 and %user 0.00 on a machine that is completely busy.

Expected resultload average: 1.55 and vmstat showing us 100, id 0, wa 0 - pidstat naming two xk0cpu.sh at 100.00 %usr each - and ps showing both at NI 5.

Success conditionYou can attribute a load to named processes with a number beside each.
A CPU-bound machine has an idle disk
The measurement that rules something out.
bash Example session
```
$iostat -xz 1 2 | tail -4; echo "--- -z omits idle devices, so an empty table IS the measurement" Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util  --- -z omits idle devices, so an empty table IS the measurement
```
Note
An empty table is the result, not a failure. -z tells iostat to omit devices with no activity in the interval, and every device qualified. The machine is at 100% CPU and the disk is doing nothing.

Which is a genuine diagnosis. "The server is slow" has just been narrowed to a CPU problem, and no amount of faster storage will help it.

Without -z the same table lists every loop device and dm target at 0.00, which is a screen of zeros to read past. -x adds the extended columns - %util, await, aqu-sz - which are the ones worth knowing.

Learn %util and await as a pair. %util is the proportion of time the device had at least one request in flight; await is how long each took. High %util with low await is a device working steadily; high await is a device struggling.

Expected resulta device header with no rows underneath it at all.

Success conditionYou can rule the disk out with one command and no ambiguity.

The other kind of slow

Kill the first load, prove it is gone, and start a completely different one.

bash Example session

$sudo kill $(cat /tmp/xk0cpu1.pid) $(cat /tmp/xk0cpu2.pid); sleep 3; ps -eo comm --sort=-%cpu | head -2; echo "--- the CPU load is gone before the next one starts"COMMANDsystemd--- the CPU load is gone before the next one starts$sudo tee /tmp/xk0io.sh > /dev/null <<'SH'#!/bin/shecho $$ > /tmp/xk0io.pidwhile :; do dd if=/dev/zero of=/var/tmp/xk0io.dat bs=1M count=800 oflag=direct status=none; doneSH$sudo chmod +x /tmp/xk0io.sh; sudo setsid /tmp/xk0io.sh > /dev/null 2>&1 & sleep 8; sudo iostat -xz 1 2 | tail -4dm-0             0.00      0.00     0.00   0.00    0.00     0.00 10368.00 5308416.00     0.00   0.00    0.13   512.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    1.32  74.20sda              0.00      0.00     0.00   0.00    0.00     0.00 10367.00 5308415.50     0.00   0.00    0.13   512.05    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    1.37  74.20$vmstat 1 3 | tail -2; echo "--- wa is time the CPU spent with nothing to do but wait for the disk" 1  0      0 1990164 146008 1065444    0    0     0 6983828 12346 14433  1 23 45 31  0  0 0  1      0 1990216 146008 1065444    0    0     0 8508416 14395 17510  0 27 45 27  0  0--- wa is time the CPU spent with nothing to do but wait for the disk$sudo pidstat -d 1 2 | tail -3; echo "--- named by process, and it is not a busy loop this time"Average:      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  CommandAverage:        0     63121      0.00 6963200.00      0.00       0  xk0io.shAverage:        0     63559      0.00 217600.00      0.00       0  dd--- named by process, and it is not a busy loop this time

Note

Every number moved to a different place. us fell from 100 to 1, wa rose from 0 to 31, and the device table filled up. Same complaint, opposite cause, and the two are not confusable once you know which columns to read.

wa is not idle and it is not busy. It is time the CPU had nothing to run because every runnable task was waiting on the disk. A machine with high wa gets no faster with more CPU.

sy 23 is the kernel doing the writing. I/O costs system time, which is why an I/O-bound machine shows a system figure a CPU-bound one does not.

pidstat -d is the per-process I/O table, and it names both the script and the dd it spawns. This is the command for "something is hammering the disk and I do not know what" - iotop is the interactive equivalent and is not installed by default.

The first load was killed and verified gone before the second started. Two loads at once would have produced numbers that prove nothing, which is the usual flaw in a benchmark.

Expected resultthe CPU load gone - then %util 74.20 with over 10,000 writes per second - vmstat reporting wa 31, us 1, sy 23 - and pidstat -d naming xk0io.sh and dd.

Success conditionYou can recognise an I/O-bound machine and name the process writing.

The record that was being kept the whole time
No agent, no dashboard, and days of history already on disk.
bash Example session
```
$ls /var/log/sysstat/ | tail -4; grep -E "^ENABLED" /etc/default/sysstat; systemctl list-timers 'sysstat*' --no-pager | head -3sa30sa31sar29sar30ENABLED="true"NEXT                            LEFT LAST                          PASSED UNIT                  ACTIVATESMon 2026-08-31 10:10:00 UTC 1min 42s Mon 2026-08-31 10:00:17 UTC 7min ago sysstat-collect.timer sysstat-collect.serviceTue 2026-09-01 00:00:00 UTC      13h Mon 2026-08-31 00:00:17 UTC  10h ago sysstat-rotate.timer  sysstat-rotate.service$sar -u | tail -309:50:17 AM     all      0.00      0.00      0.02      0.00      0.00     99.9810:00:17 AM     all      0.00      0.00      0.01      0.01      0.00     99.98Average:        all      0.07      0.00      0.06      0.01      0.00     99.86$sar -r | tail -2; echo "--- memory, from the same collection, with no agent installed"10:00:17 AM   2009908   2970708    198132      5.69    145672    991928    309168      4.13    387540    804108       100Average:      2007491   2966489    202667      5.82    144846    991000    310393      4.15    386531    803445       101--- memory, from the same collection, with no agent installed
```
Note
This is the answer to "it was slow last night and nobody was watching". sysstat writes a binary record every ten minutes into /var/log/sysstat/sa
, keeps a month by default, and costs nothing.

ENABLED="true" in /etc/default/sysstat is the thing to check first on a Debian-family machine - the package installs with collection OFF on some releases, and sar then reports it cannot open today's file. On RPM systems the equivalent is the sysstat package's own timer.

Two timers, two jobs. sysstat-collect takes a sample; sysstat-rotate starts the next day's file. /usr/lib/sysstat/sa1 1 1 takes one immediately if you want a sample without waiting.

sar -u CPU, -r memory, -d disk, -n DEV network, -b I/O, -q load. All read the same file, so one collection answers every question. -f /var/log/sysstat/sa29 picks a different day, and -s 22:00 -e 23:00 narrows to an hour.

Expected resultdaily files sa30, sa31 and text reports sar29, sar30 - ENABLED="true" - sysstat-collect.timer firing every ten minutes and sysstat-rotate.timer daily - then per-interval CPU lines with a daily Average, and the same for memory.

Success conditionYou can produce yesterday's figures on a machine nobody instrumented.
Turning a measurement into a prediction
Three days of averages, and the arithmetic that makes them a date.
bash Example session
```
$for f in $(ls /var/log/sysstat/sa[0-9]* | tail -3); do echo -n "$(basename $f)  "; sar -u -f $f 2>/dev/null | tail -1; donesa29  Average:        all      0.01      0.00      0.03      0.01      0.00     99.94sa30  Average:        all      0.04      0.01      0.05      0.03      0.00     99.87sa31  Average:        all      0.07      0.00      0.06      0.01      0.00     99.86$for f in $(ls /var/log/sysstat/sa[0-9]* | tail -3); do echo -n "$(basename $f)  "; sar -r -f $f 2>/dev/null | tail -1 | awk '{print "memused%", $5}'; donesa29  memused% 5.34sa30  memused% 5.77sa31  memused% 5.82$df -h /var | tail -1; sudo du -sh /var/tmp/xk0io.dat 2>/dev/null; echo "--- a size and a rate are what turn a measurement into a date"/dev/mapper/ubuntu--vg-ubuntu--lv   48G  7.6G   38G  17% /673M	/var/tmp/xk0io.dat--- a size and a rate are what turn a measurement into a date
```
Note
This is what Topic 200.2 is asking for. A capacity answer is never "is it full" - it is "when will it be", and that needs two points in time. One reading is a number; three are a direction.

The arithmetic is deliberately simple: take the used figure now, take it from a week ago, divide the difference by the days, and divide the remaining space by that. Growth that is not linear makes it optimistic, which is the right direction for an estimate to be wrong in.

A daily Average: line is the right granularity for a trend and the wrong one for an incident. Averaging hides the spike - for a specific event, read the interval lines with -s and -e instead.

Memory rising 5.34 to 5.82 over three days is not alarming and is still the point. A trend spotted at 5% is a conversation; the same trend spotted at 95% is an outage. The value of the record is that it existed before anyone needed it.

Watch inodes on the same schedule - sar does not record them, so df -i belongs in whatever does. A filesystem can hit 100% with most of its bytes free.

Expected resultdaily CPU averages rising 0.01 → 0.04 → 0.07, memused 5.34 → 5.77 → 5.82 - and the filesystem at 17% with a single 673M file in it.

Success conditionYou can state a trend from evidence instead of an impression.
Putting the machine back
Both loads gone, the file gone, the load average falling.
zsh Example session
```
$sudo kill -- -$(cat /tmp/xk0io.pid) 2>/dev/null; sleep 2; sudo pkill -x dd 2>/dev/null; sleep 2; pgrep -c -x dd; echo "dd processes left"; sudo rm -f /var/tmp/xk0io.dat /tmp/xk0cpu*.pid /tmp/xk0io.pid /tmp/xk0cpu.sh /tmp/xk0io.sh0dd processes left$sleep 45; uptime; df -h /var | tail -1; ls /tmp/xk0* 2>&1 | tail -1; ps -eo comm --sort=-%cpu | head -2 10:09:07 up 1 day, 23:40,  1 user,  load average: 0.67, 0.72, 0.39/dev/mapper/ubuntu--vg-ubuntu--lv   48G  7.0G   38G  16% /zsh:2: no matches found: /tmp/xk0*COMMANDsystemd
```
Note
kill -- -PID kills the process GROUP, which is what a writer that spawns a fresh dd every few seconds needs. Killing the script alone leaves the current dd running to completion, so the group is killed and then pgrep -c -x dd is asked to confirm.

pkill -x dd matches the process name exactly, not the command line. pkill -f would have matched this very session's own command line - which is a genuinely easy way to disconnect yourself.

Load average lags recovery. It reads 0.67 with the machine already idle, because the one-minute average still contains the last minute. Waiting is the fix, and it is why the closing check also looks at the process list rather than trusting the number.

The 5.9 GB file went with it, restoring /var to 16%. A capacity page that leaves several gigabytes on the machine has not finished.

Expected result0 dd processes left, the filesystem back to 16%, load average: 0.67 and falling, no matches found: /tmp/xk0*, and nothing but systemd at the top of the CPU list.

Success conditionThe host carries no trace of either load.

Troubleshooting

The machine is slow and top does not obviously say why.

Why: Instantaneous per-process figures are hard to read at the moment they matter.

Fix:pidstat -u 1 5 prints a per-process table plus an Average over the interval. Pair it with vmstat 1 5: us high means CPU, wa high means disk, sy high means kernel work such as I/O or context switching.
It is unclear whether the disk or the CPU is the bottleneck.

Why: Both present as "slow", and the load average cannot distinguish them - it counts uninterruptible tasks too.

Fix:iostat -xz 1 3. An empty device table means the disk is idle and the problem is elsewhere. A %util near 100 with a high await means the device is the limit; confirm the process with pidstat -d 1 3.
sar reports it cannot open today's data file.

Why: The sysstat package is installed but collection is disabled - the Debian-family default on several releases.

Fix:Set ENABLED="true" in /etc/default/sysstat, then systemctl enable --now sysstat sysstat-collect.timer. /usr/lib/sysstat/sa1 1 1 writes a sample immediately so you do not wait ten minutes to test it.
Someone asks how long the current disk will last.

Why: There is no history, so there is no rate - only a current percentage.

Fix:sar -f /var/log/sysstat/sa<DD> for previous days if collection was running. Otherwise start collecting now and say so: a prediction needs two points in time, and the first one has to have been taken before the question was asked.
iostat shows the CPU as %nice and you expected %user.

Why: The processes have a non-zero nice value, and niced CPU time is accounted separately.

Fix:ps -eo pid,ni,comm --sort=-%cpu shows the nice values. Both are userspace time - add them together when asking how busy the machine is.
A load-generating process will not die.

Why: It respawns children, so killing the parent leaves the current child running.

Fix:kill -- -<pid> kills the whole process group, which works when the process was started with setsid. Confirm with pgrep -c -x <name> rather than assuming, and prefer pkill -x to pkill -f so you do not match your own session.

Official sources

← All LPIC-2 guides

Finding what is consuming the machine

This guide includes

Before you start

A quiet machine, described in numbers

Busy is not a diagnosis

A CPU-bound machine has an idle disk

The other kind of slow

The record that was being kept the whole time

Turning a measurement into a prediction

Putting the machine back

Troubleshooting

Official sources