CertGrid CertGrid
Hands-on Lab·PostgreSQL

PostgreSQL Startup Failure Diagnosis

Three failures that all look identical to systemd, and three different places the real reason turns up - one of which is not the PostgreSQL log at all.

Troubleshooting Guide 44 of 47 Intermediate

Written against the versions above. The `pg_ctlcluster` wrapper and `/etc/postgresql/18/main` layout are Debian and Ubuntu. On RHEL the same failures appear in `journalctl -u postgresql-18` and the data directory holds the configuration.

Every command on this page ran on db-b01.
Server NameIP AddressOSRolesCPURAMHDD
db-b01192.168.0.82Ubuntu 26.04 LTSStandby / Replica / Replica Set Member 22 Core4 GB50 GB

Before you start

  1. Ask for more shared memory than the machine has

    The host has 3 GB usable. shared_buffers = '9GB', and the restart fails.

    systemd's message is useless on purpose: Job for postgresql@18-main.service failed because the service did not take the steps required by its unit configuration. It knows the process did not come up. It has no idea why.

    is-active returns failed with exit code 3 - worth knowing for scripts, which otherwise treat a dead database as a successful command.

    systemctl status is better, because Debian's wrapper prints the postmaster's first error into the journal: FATAL: could not map anonymous shared memory: Cannot allocate memory.

    bash Example session 
    free -g | head -2               total        used        free      shared  buff/cache   availableMem:               3           0           1           0           1           2sudo -u postgres psql -c "ALTER SYSTEM SET shared_buffers = '9GB'"ALTER SYSTEMsudo systemctl restart postgresql@18-mainJob for postgresql@18-main.service failed because the service did not take the steps required by its unit configuration.See "systemctl status postgresql@18-main.service" and "journalctl -xeu postgresql@18-main.service" for details.[exit 1]systemctl is-active postgresql@18-mainfailed[exit 3]sudo systemctl status postgresql@18-main --no-pager -l | head -12× postgresql@18-main.service - PostgreSQL Cluster 18-main     Loaded: loaded (/usr/lib/systemd/system/postgresql@.service; enabled-runtime; preset: enabled)     Active: failed (Result: protocol) since Thu 2026-08-27 16:34:32 UTC; 540ms ago   Duration: 26.758s Invocation: 3b38edb483ff4b159ca0799f104dfdfa    Process: 20054 ExecStart=/usr/bin/pg_ctlcluster --skip-systemctl-redirect 18-main start (code=exited, status=1/FAILURE)   Mem peak: 8.9M        CPU: 30ms Aug 27 16:34:32 db-b01 systemd[1]: Starting postgresql@18-main.service - PostgreSQL Cluster 18-main...Aug 27 16:34:32 db-b01 postgresql@18-main[20054]: Error: /usr/lib/postgresql/18/bin/pg_ctl /usr/lib/postgresql/18/bin/pg_ctl start -D /var/lib/postgresql/18/main -l /var/log/postgresql/postgresql-18-main.log -s -o  -c config_file="/etc/postgresql/18/main/postgresql.conf"  exited with status 1:Aug 27 16:34:32 db-b01 postgresql@18-main[20054]: 2026-08-27 16:34:32.769 UTC [20059] FATAL:  could not map anonymous shared memory: Cannot allocate memory

    Expected resultA failed unit and could not map anonymous shared memory in the status.

    Success conditionYou know systemd's message is never the answer.

  2. Read the log, which includes the arithmetic

    The log has the error and a HINT that does the work for you:

    This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 9948741632 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing "shared_buffers" or "max_connections".

    9,948,741,632 bytes on a 3 GB machine. The request is stated in bytes, so there is nothing to work out.

    pg_ctl: could not start server / Examine the log output closes the file. That line is pg_ctl telling you it has already given up.

    bash Example session 
    sudo tail -6 /var/log/postgresql/postgresql-18-main.log2026-08-27 16:34:32.691 UTC [19510] LOG:  database system is shut down2026-08-27 16:34:32.769 UTC [20059] FATAL:  could not map anonymous shared memory: Cannot allocate memory2026-08-27 16:34:32.769 UTC [20059] HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 9948741632 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing "shared_buffers" or "max_connections".2026-08-27 16:34:32.769 UTC [20059] LOG:  database system is shut downpg_ctl: could not start serverExamine the log output.

    Expected resultFATAL: could not map anonymous shared memory and the byte count.

    Success conditionYou read the log file before anything else.

  3. Fix a setting you cannot use ALTER SYSTEM to fix

    The obvious repair is ALTER SYSTEM RESET shared_buffers - and it fails:

    connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory

    ALTER SYSTEM is SQL. It needs a running server. The setting that stops the server starting cannot be repaired with the tool that set it.

    So edit postgresql.auto.conf directly. Its header says *do not edit this file manually* and that is good advice while the server is up; with the server down it is the only route in. grep -n finds it on line 8, sed puts back a sane value, and the cluster starts.

    Deleting the line works equally well - the setting then falls back to the configuration file or the built-in default.

    bash Example session 
    sudo -u postgres psql -c "ALTER SYSTEM RESET shared_buffers"psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory	Is the server running locally and accepting connections on that socket?[exit 2]sudo grep -n shared_buffers /var/lib/postgresql/18/main/postgresql.auto.conf8:shared_buffers = '9GB'sudo sed -i "s/^shared_buffers = '9GB'/shared_buffers = '256MB'/" /var/lib/postgresql/18/main/postgresql.auto.confsudo systemctl start postgresql@18-mainsudo -u postgres psql -c "SHOW shared_buffers" shared_buffers---------------- 256MB(1 row)

    Expected resultALTER SYSTEM refused, then a clean start after editing the file.

    Success conditionYou can repair a cluster that is too broken to accept SQL.

  4. A value that never reaches PostgreSQL at all

    port = notanumber appended to postgresql.auto.conf. The restart fails the same way - and this time the log has nothing. grep -c notanumber returns 0.

    journalctl has it:

    postgresql@18-main[20632]: port_running: invalid port notanumber at /usr/share/perl5/PgCommon.pm line 647.

    That is not PostgreSQL. It is Debian's pg_ctlcluster wrapper, written in Perl, which parses the configuration to work out where to connect *before* starting anything. postgres never ran, so it never opened the log, so the log is not merely unhelpful - it is untouched, still ending with the previous clean shutdown.

    On a packaged install, always check journald as well as the log file. A silent log after a failed start usually means the failure happened above PostgreSQL.

    bash Example session 
    sudo bash -c 'echo "port = notanumber" >> /var/lib/postgresql/18/main/postgresql.auto.conf'sudo systemctl restart postgresql@18-mainJob for postgresql@18-main.service failed because the service did not take the steps required by its unit configuration.See "systemctl status postgresql@18-main.service" and "journalctl -xeu postgresql@18-main.service" for details.[exit 1]sudo grep -c notanumber /var/log/postgresql/postgresql-18-main.log0[exit 1]sudo journalctl -u postgresql@18-main --no-pager -n 6Aug 27 16:34:40 db-b01 systemd[1]: Stopped postgresql@18-main.service - PostgreSQL Cluster 18-main.Aug 27 16:34:40 db-b01 systemd[1]: Starting postgresql@18-main.service - PostgreSQL Cluster 18-main...Aug 27 16:34:40 db-b01 postgresql@18-main[20632]: port_running: invalid port notanumber at /usr/share/perl5/PgCommon.pm line 647.Aug 27 16:34:40 db-b01 systemd[1]: postgresql@18-main.service: Can't open PID file '/run/postgresql/18-main.pid' (yet?) after start: No such file or directoryAug 27 16:34:40 db-b01 systemd[1]: postgresql@18-main.service: Failed with result 'protocol'.Aug 27 16:34:40 db-b01 systemd[1]: Failed to start postgresql@18-main.service - PostgreSQL Cluster 18-main.sudo sed -i '/^port = notanumber/d' /var/lib/postgresql/18/main/postgresql.auto.confsudo systemctl start postgresql@18-mainsystemctl is-active postgresql@18-mainactive

    Expected resultZero matches in the log; the real error only in journald.

    Success conditionYou know the log file can be empty and the cause still findable.

  5. Permissions on the data directory

    chmod 0777 on the data directory, and the start fails again:

    FATAL: data directory "/var/lib/postgresql/18/main" has invalid permissions

    PostgreSQL requires 0700 or 0750 and refuses to start otherwise. That is a deliberate refusal, not a bug: the data directory holds every byte of every table, and world-readable is not a state it will run in.

    This one turns up after a restore, a cp -r that did not preserve modes, or a well-meant chmod -R aimed at something else. chmod 0700 and it starts.

    bash Example session 
    sudo systemctl stop postgresql@18-mainsudo chmod 0777 /var/lib/postgresql/18/mainsudo systemctl start postgresql@18-mainJob for postgresql@18-main.service failed because the service did not take the steps required by its unit configuration.See "systemctl status postgresql@18-main.service" and "journalctl -xeu postgresql@18-main.service" for details.[exit 1]sudo grep -n "invalid permissions" /var/log/postgresql/postgresql-18-main.log | tail -21102:2026-08-27 16:33:59.736 UTC [19344] FATAL:  data directory "/var/lib/postgresql/18/main" has invalid permissions1240:2026-08-27 16:34:48.606 UTC [21113] FATAL:  data directory "/var/lib/postgresql/18/main" has invalid permissionssudo chmod 0700 /var/lib/postgresql/18/mainsudo systemctl start postgresql@18-mainsudo -u postgres psql -c "SELECT 'up' AS status" status-------- up(1 row)

    Expected resulthas invalid permissions, then a clean start at 0700.

    Success conditionYou recognise a permissions refusal from the log line.

  6. Read a setting without starting the server

    postgres -C prints the value the server *would* use and exits. It needs -c config_file= on Debian, because the configuration is under /etc rather than in the data directory.

    shared_buffers comes back as 32768 - the raw internal value in 8 kB blocks, which is 256 MB. SHOW converts units; -C does not.

    This is how you verify a repair *before* attempting a start, which on a production cluster is worth doing. pg_lsclusters gives the other half - version, port, status, data directory and log path for every cluster on the machine, which is the fastest way to find out that you have been reading the wrong cluster's log all along.

    bash Example session 
    sudo -u postgres /usr/lib/postgresql/18/bin/postgres -D /var/lib/postgresql/18/main -c config_file=/etc/postgresql/18/main/postgresql.conf -C shared_buffers32768sudo -u postgres /usr/lib/postgresql/18/bin/postgres -D /var/lib/postgresql/18/main -c config_file=/etc/postgresql/18/main/postgresql.conf -C data_directory/var/lib/postgresql/18/mainsudo pg_lsclustersVer Cluster Port Status Owner    Data directory              Log file18  main    5432 online postgres /var/lib/postgresql/18/main /var/log/postgresql/postgresql-18-main.log

    Expected result32768, the data directory, and one online cluster.

    Success conditionYou can check configuration on a stopped server.

Troubleshooting

Official sources

← All PostgreSQL guides