Hands-on Lab·LPIC-2

Load, memory, and what the numbers mean

Topic 200 is capacity planning, and almost all of it is interpretation rather than commands. This guide reads load, memory, CPU and disk from a deliberately idle machine and is specific about what each number does NOT mean - because the commonest capacity mistakes are misreadings, not missing tools.

201: Capacity Planning Guide 10 of 29 Intermediate

OSUbuntu 26.04 LTS
Kernel7.0.0-30-generic
systemd259
Block devicesone 50 GB disk
TimeAbout 21 min

One host, deliberately idle. Every command on these pages reads a counter or a pseudo-file; nothing is configured and no process is signalled.
Server Name	IP Address	OS	Roles	CPU	RAM	HDD
LPIC2-A01	192.168.0.78	Ubuntu 26.04 LTS	Primary service host - BIND, Apache, Samba, Postfix. Topic 204 RAID runs on loop devices; this machine has no spare disk	2 Core	4 GB	50 GB

This guide includes

Use this for capacity planning, which is interpretation rather than commands. This matters because load average is not a percentage - and the memory figure most people read is the wrong one.

reading load average, which is not a percentage
reading memory, and the number people misread
sampling where the time goes, with three tools that share one trap
finding the two separate ways a disk fills up

Before you start

the-lab-a-service-needs-a-client

Load average is not a percentage
Two numbers together, or neither means anything.
bash Example session
```
$uptime; echo "---"; nproc; echo "--- load is runnable+uninterruptible tasks, NOT a percentage" 06:58:29 up 20:29,  1 user,  load average: 0.11, 0.12, 0.09---2--- load is runnable+uninterruptible tasks, NOT a percentage$cat /proc/loadavg; echo "--- 1min 5min 15min running/total lastpid"0.11 0.12 0.09 1/144 10453--- 1min 5min 15min running/total lastpid
```
Note
Load is meaningless without the core count, and that is the whole step. It counts tasks that are runnable or in uninterruptible sleep - not a percentage of anything. Load 4 on a four-core machine is fully busy and healthy; load 4 on one core means three tasks are waiting their turn.

The three numbers are one, five and fifteen minute averages, and the SHAPE matters more than any one of them. Rising means the problem is arriving; falling means it has passed. A single reading tells you almost nothing.

Uninterruptible sleep is the part people miss. A process blocked on a slow disk or a hung NFS mount counts towards load while using no CPU at all. That is why a machine can show load 20 and 100% idle CPU simultaneously - and why load alone never identifies the bottleneck.

/proc/loadavg adds what uptime hides: 1/144 is runnable tasks over total tasks, and the last field is the most recently created PID. A runnable count consistently above your core count is the genuine CPU-saturation signal.

Expected resultload average: 0.11, 0.12, 0.09 on a machine with 2 CPUs, and 1/144 runnable out of total tasks.

Success conditionYou can say whether a load figure is high without guessing.
Memory, and the number people misread
free is fine. available is the answer.
bash Example session
```
$free -h; echo "--- available is the number that matters, not free"               total        used        free      shared  buff/cache   availableMem:           3.3Gi       518Mi       2.3Gi       4.1Mi       781Mi       2.8GiSwap:          3.8Gi          0B       3.8Gi--- available is the number that matters, not free$grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|^Cached|SwapTotal|SwapFree)' /proc/meminfoMemTotal:        3480376 kBMemFree:         2387232 kBMemAvailable:    2954824 kBBuffers:           77488 kBCached:           679708 kBSwapTotal:       4003836 kBSwapFree:        4003836 kB$cat /proc/sys/vm/swappiness; echo "--- how eagerly the kernel swaps, 0-100"60--- how eagerly the kernel swaps, 0-100
```
Note
available is bigger than free, and that is the point. Free memory is memory doing nothing, which is memory wasted. The kernel fills it with page cache, and available is the honest estimate of what a new process could have - free plus the cache it could reclaim. Alarming on low free is the classic beginner monitoring alert.

781 MiB of buff/cache on an idle machine is the kernel doing its job. It caches file contents so a second read is instant, and gives that memory back the moment something needs it.

Swap in use is not itself a problem; swap ACTIVITY is. A page swapped out once and never touched again costs nothing. What hurts is continuous paging, which shows in vmstat's si/so columns rather than in free.

Swappiness 60 is the default and is often changed badly. It is a preference for reclaiming anonymous memory against page cache, 0 to 100 - and 0 does not disable swap, it just makes the kernel avoid it until it must. Lowering it to 10 on a database host is common; setting it to 0 as a habit is cargo cult.

A machine that has swapped and recovered still shows swap used until those pages are touched. swapoff -a && swapon -a forces them back in - and briefly needs the memory to do so.

Expected result3.3Gi total, 518Mi used, 2.3Gi free but 2.8Gi available, 781Mi in buff/cache - and swappiness 60 with no swap in use.

Success conditionYou can answer "is this machine short of memory" correctly.

Where the time goes

Three sampling tools, and one shared trap.

bash Example session

$vmstat 1 3 2>/dev/null | tail -3; echo "--- the FIRST line is an average since boot, ignore it" 0  0      0 2389936  77488 723256    0    0    14    20  121    0  0  0 100  0  0  0 0  0      0 2395256  77488 723348    0    0     0     0  135   68  0  0 100  0  0  0 0  0      0 2402484  77488 723348    0    0     0     0  133   67  0  0 100  0  0  0--- the FIRST line is an average since boot, ignore it$iostat -x 1 2 2>/dev/null | tail -6 || echo "iostat not installed - sysstat package"loop1            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00loop2            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00sda              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00sdb              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00$sar -u 1 2 2>/dev/null | tail -3 || echo "sar has no data yet - sysstat collection may be off"06:58:34 AM     all      0.00      0.00      0.00      0.00      0.00    100.0006:58:35 AM     all      0.00      0.00      0.00      0.00      0.00    100.00Average:        all      0.00      0.00      0.00      0.00      0.00    100.00

Note

The first line of vmstat and iostat is an average since BOOT, not a sample. It is almost always wrong for the question you are asking, and quoting it is the most common mistake in this objective. Always take at least two samples and discard the first - that is why every command here asks for more than one.

The columns worth knowing in vmstat: r runnable, b blocked, si/so swap in and out, wa iowait. High wa with low us means the CPU is waiting on storage, which points at the disk rather than the processor.

iostat -x gives the useful columns: %util how busy the device is, await average wait per request in milliseconds, aqu-sz queue depth. On modern SSDs %util near 100 is no longer proof of saturation - the device handles parallel requests - so await is the better signal.

sar is the only one that answers about YESTERDAY. vmstat and iostat sample now; sar reads the historical data sysstat collected on a timer. When an incident is over and someone asks what the load was at 3am, sar is the tool - and it only works if collection was already enabled, which is the point of turning it on before you need it.

Expected resultvmstat showing 100 idle with no swap activity, iostat with every device at zero, and sar reporting 100% idle across two samples.

Success conditionYou can sample CPU and disk over an interval rather than guessing.

Two ways a disk fills up
Blocks and inodes are separate limits.
bash Example session
```
$df -h / /boot | column -tFilesystem                         Size  Used  Avail  Use%  Mounted  on/dev/mapper/ubuntu--vg-ubuntu--lv  48G   6.9G  39G    16%   //dev/sda2                          2.0G  101M  1.7G   6%    /boot$df -i / | column -t; echo "--- inodes: a filesystem can be full with space left"Filesystem                         Inodes   IUsed   IFree    IUse%  Mounted  on/dev/mapper/ubuntu--vg-ubuntu--lv  3145728  117327  3028401  4%     /--- inodes: a filesystem can be full with space left$sudo -n du -xh --max-depth=1 /var 2>/dev/null | sort -h | tail -5824K	/var/backups30M	/var/log164M	/var/cache190M	/var/lib383M	/var
```
Note
A filesystem can be full with gigabytes free, and df -h will not show it. Inodes are allocated when the filesystem is created and each file - however small - consumes one. A mail spool or a cache full of tiny files exhausts them first, and the error is "No space left on device" with df -h showing 40% used. df -i is the check, and remembering it exists is the objective.

You cannot add inodes to an existing ext4 filesystem. The count is fixed at mkfs time. The fix is deleting files or recreating the filesystem with mkfs.ext4 -i to change the bytes-per-inode ratio. XFS allocates inodes dynamically and does not have this failure.

du -x stays on one filesystem, which matters when hunting for space: without it, du / descends into every mount and gives you a total nobody asked for. --max-depth=1 then lets you walk down one level at a time rather than reading thousands of lines.

df and du disagreeing is usually a deleted file still held open. du walks directory entries; df asks the filesystem. A log deleted while the daemon has it open frees nothing until the process closes it - lsof +L1 finds them, and restarting the service reclaims the space.

Expected result48G root at 16% used; 3145728 inodes with 4% used; and /var totalling 383M with /var/lib and /var/cache largest.

Success conditionYou can diagnose a full filesystem that still reports free space.

Troubleshooting

A high load average is reported as a CPU problem.

Why: Load counts processes running OR waiting - including waiting on disk. A load of 8 on 2 CPUs may be entirely IO.

Fix:Compare against nproc, then look at what is waiting: vmstat 1 shows the b column for blocked processes and wa for IO wait. /proc/pressure/io says how much time was actually lost.
free shows little free memory and the machine is called short of RAM.

Why: Linux uses spare memory for cache. The number that matters is available, not free - 2.3Gi free against 2.8Gi available here.

Fix:Read the available column. Cache is reclaimed on demand, so a machine with almost no free memory and plenty available is healthy.
A filesystem reports free space and writes still fail.

Why: Blocks and inodes are separate limits. A filesystem full of tiny files exhausts inodes with space to spare.

Fix:df -h for blocks and df -i for inodes - 4% inode use here against 16% blocks. If inodes are the limit, the fix is deleting files or recreating the filesystem with a different inode ratio.
Averages look fine and users report the machine is slow.

Why: An average over a minute hides a stall that lasted seconds.

Fix:Read /proc/pressure/{cpu,memory,io}, which report the share of time work was actually STALLED. some is any task waiting; full is everything waiting at once.

Official sources

← All LPIC-2 guides