Linux Pipes and Redirection Explained: |, >, >> and 2>&1

The real power of Linux comes from joining small tools together. Each command does one job, and you connect them.

Three streams#

Every command has three streams:

Redirection sends them somewhere else.

Write output to a file#

> replaces a file, and >> adds to the end:

echo "first" > log.txt
echo "second" > log.txt
cat log.txt
echo "third" >> log.txt
cat log.txt

second
second
third

Read input from a file#

< feeds a file to a command's input:

sort < log.txt

second
third

Pipes#

| sends the output of one command to the input of the next:

ls /etc | wc -l
ls /etc | grep '^host'

234
host.conf
hostname
hosts
hosts.allow
hosts.deny

You can keep adding steps. This shows the three largest entries in /usr/bin:

ls -lS /usr/bin | tail -n +2 | head -n 3 | awk '{print $5, $9}'

tail: error writing 'standard output': Broken pipe
98450160 kubectl-convert
93831120 kubectl
83666424 dockerd
ls: write error: Broken pipe

Errors are separate#

Error messages use stderr, so > does not capture them:

ls /etc/hostname /nothing > out.txt
echo "--- out.txt:"
cat out.txt

ls: cannot access '/nothing': No such file or directory
--- out.txt:
/etc/hostname

Redirect errors with 2>:

ls /etc/hostname /nothing > out.txt 2> err.txt
cat err.txt

ls: cannot access '/nothing': No such file or directory

Send both to the same place with 2>&1, which means "send stream 2 to wherever stream 1 is going":

ls /etc/hostname /nothing > all.txt 2>&1
wc -l all.txt

2 all.txt

The order matters: 2>&1 must come after > all.txt.

Throw output away#

/dev/null is a special file that discards everything:

ls /nothing 2> /dev/null
echo "Done, and no error shown"

Done, and no error shown

This is common in scripts and cron jobs.

tee: see it and save it#

tee writes to a file and to the screen at the same time:

echo "Build finished" | tee build.log
cat build.log

Build finished
Build finished

To write to a protected file, use echo text | sudo tee /etc/file, because sudo echo text > /etc/file fails: the redirection is done by your shell, which is not root.

Command substitution#

$(command) runs a command and inserts its output:

echo "This folder has $(ls | wc -l) files"
today=$(date +%A)
echo "${#today} letters in the name of today"

This folder has 5 files
8 letters in the name of today

xargs#

Some commands take names as arguments, not from a pipe. xargs turns input lines into arguments:

touch a.tmp b.tmp
ls *.tmp | xargs rm
ls *.tmp 2> /dev/null | wc -l

0

Common mistakes#

Practice#

ls -d /etc/*.conf 2> /dev/null > conf.txt
wc -l < conf.txt

34

Key takeaway#

Next: Searching Text with grep.