Pipes, filters and redirection

Essential Commands / Guide 7 of 38 / Intermediate

• OSUbuntu 26.04 LTS
• LVM2.03.31 (Ubuntu) / 2.03.36 (AlmaLinux)
• nftables1.1.6 (Ubuntu) / 1.1.5 (AlmaLinux)
• TimeAbout 18 min

&> and |&, set -o pipefail and PIPESTATUS are bash features - the login shell here is zsh, so the examples run under an explicit bash -c. POSIX sh has none of them.

• Firewallufw 0.36.2 enabled but reporting inactive / firewalld active
• Network confignetplan + systemd-networkd / NetworkManager 1.56.0

This guide includes

Use this for the two traps that catch everybody. This matters because the order of 2>&1 and > is not a style choice - swapped, the errors end up somewhere you did not intend.

• following the three streams, and where each one goes
• finding that the order of the redirections decides the result
• appending, truncating and discarding, and using tee for both at once
• seeing a pipeline hide every exit status but the last
• using xargs on a filename with a space, and feeding a command from a heredoc

Before you start

• guide 6 - the filters going in the pipe.

1. 1

   Three streams, and where each one goes

   One command that writes to both, redirected four ways.

   $mkdir -p ~/ess/p && cd ~/ess/p && bash -c 'echo "to stdout"; echo "to stderr" >&2' ; echo "--- both reached the terminal above"to stdoutto stderr--- both reached the terminal above$cd ~/ess/p && bash -c 'echo "to stdout"; echo "to stderr" >&2' > out.txt; echo "stdout captured, stderr still on screen"; echo "--- out.txt holds:"; cat out.txtto stderrstdout captured, stderr still on screen--- out.txt holds:to stdout$cd ~/ess/p && bash -c 'echo "to stdout"; echo "to stderr" >&2' 2> err.txt; echo "--- err.txt holds:"; cat err.txtto stdout--- err.txt holds:to stderr$cd ~/ess/p && bash -c 'echo "to stdout"; echo "to stderr" >&2' > both.txt 2>&1; echo "--- both.txt holds:"; cat both.txt--- both.txt holds:to stdoutto stderr

   Expected result> out.txt capturing only stdout and leaving stderr on screen; 2> err.txt the reverse; > both.txt 2>&1 capturing both.

   Success conditionYou can send each stream where you want it.

2. 2

   And the order of the redirections is not a style choice

   The same two operators, swapped.

   $cd ~/ess/p && bash -c 'ls /nonexistent 2>&1 > wrong.txt'; echo "--- wrong.txt holds:"; cat wrong.txt; echo "(empty - stderr went to the OLD stdout, the terminal)"ls: cannot access '/nonexistent': No such file or directory--- wrong.txt holds:(empty - stderr went to the OLD stdout, the terminal)$cd ~/ess/p && bash -c 'ls /nonexistent > right.txt 2>&1'; echo "--- right.txt holds:"; cat right.txt--- right.txt holds:ls: cannot access '/nonexistent': No such file or directory

   Expected result2>&1 > wrong.txt leaving wrong.txt empty with the error on screen; > right.txt 2>&1 capturing the error.

   Success conditionYou will not write the broken form again.

3. 3

   Appending, truncating, and discarding

   > against >>, and /dev/null.

   $cd ~/ess/p && printf 'first\n' > trunc.txt && printf 'second\n' > trunc.txt && cat trunc.txt; echo "--- > truncated it"second--- > truncated it$cd ~/ess/p && printf 'first\n' > app.txt && printf 'second\n' >> app.txt && cat app.txt; echo "--- >> appended"firstsecond--- >> appended$cd ~/ess/p && ls /nonexistent 2>/dev/null; echo "exit $? and no message"; ls /nonexistent >/dev/null 2>&1; echo "exit $? and nothing at all"exit 2 and no messageexit 2 and nothing at all

   Expected result> leaving only second; >> leaving both lines; and 2>/dev/null hiding the message while exit 2 survives.

   Success conditionYou can choose between replacing and adding.

4. 4

   tee, for when you want both

   Down the pipe and into a file at the same time - including a file you cannot open.

   $cd ~/ess/p && printf 'a\nb\nc\n' | tee copy.txt | wc -l; echo "--- and copy.txt holds:"; cat copy.txt3--- and copy.txt holds:abc$cd ~/ess/p && printf 'd\n' | tee -a copy.txt >/dev/null && cat copy.txtabcd$cd ~/ess/p && echo "written as root" | sudo tee root-owned.txt >/dev/null && ls -l root-owned.txt; echo "--- which is how you write to a file you cannot open yourself"-rw-r--r-- 1 root root 16 Aug 29 09:42 root-owned.txt--- which is how you write to a file you cannot open yourself

   Expected result3 from wc -l with copy.txt also written; -a appending a fourth line; and a root-owned file created by sudo tee.

   Success conditionYou can capture and pass through in one step.

5. 5

   A pipeline hides every exit status but the last

   false | true, three ways of asking what happened.

   $cd ~/ess/p && bash -c 'false | true; echo "plain pipeline exit: $?"'plain pipeline exit: 0$cd ~/ess/p && bash -c 'set -o pipefail; false | true; echo "with pipefail:     $?"'with pipefail:     1$cd ~/ess/p && bash -c 'false | true; echo "PIPESTATUS: ${PIPESTATUS[*]}"'PIPESTATUS: 1 0

   Expected resultplain pipeline exit: 0, with pipefail: 1, and PIPESTATUS: 1 0.

   Success conditionYou can detect a failure in the middle of a pipeline.

6. 6

   xargs, and the filename with a space in it

   Turning a list of names into arguments - and the version that survives real filenames.

   $cd ~/ess/p && touch f1.log f2.log f3.log && ls *.log | xargs -n1 echo "would process"would process f1.logwould process f2.logwould process f3.log$cd ~/ess/p && printf 'one file.log\n' > 'one file.log' && ls *.log | xargs ls -l 2>&1 | tail -3; echo "--- names with spaces break a plain xargs"-rw-rw-r-- 1 sysadmin sysadmin 0 Aug 29 09:42 f1.log-rw-rw-r-- 1 sysadmin sysadmin 0 Aug 29 09:42 f2.log-rw-rw-r-- 1 sysadmin sysadmin 0 Aug 29 09:42 f3.log--- names with spaces break a plain xargs$cd ~/ess/p && find . -maxdepth 1 -name '*.log' -print0 | xargs -0 ls -l | wc -l; echo "--- -print0 and -0 make it safe"4--- -print0 and -0 make it safe$cd ~/ess/p && find . -maxdepth 1 -name '*.log' -exec ls -l {} + | wc -l; echo "--- or skip xargs entirely with -exec ... +"4--- or skip xargs entirely with -exec ... +

   Expected resultxargs -n1 echoing one per line; a plain ls | xargs failing on one file.log; and both -print0 | xargs -0 and -exec ... + handling it.

   Success conditionYou can act on a list of files safely.

7. 7

   A heredoc and a here-string

   Feeding a command several lines, or one, without a file.

   $cd ~/ess/h && bash unquoted.shan unquoted heredoc expands /home/sysadmin and lfcs-a01$cd ~/ess/h && bash quoted.sha quoted heredoc leaves $HOME and $(hostname) alone$cd ~/ess/h && printf 'cat <<-EOF\n\tthis line was indented with a TAB\nEOF\n' > dash.sh && cat -A dash.sh | head -3; echo "--- and run:"; bash dash.shcat <<-EOF$^Ithis line was indented with a TAB$EOF$--- and run:this line was indented with a TAB$cd ~/ess/h && bash -c 'tr a-z A-Z <<< "a here-string is one line"'; bash -c 'wc -w <<< "three words here"'A HERE-STRING IS ONE LINE3

   Expected resultThe unquoted heredoc expanding to /home/sysadmin and lfcs-a01; the quoted one printing $HOME and $(hostname) literally; cat -A showing the ^I tab that <<- strips.

   Success conditionYou can supply input inline.