Standard streams

Three numbered streams: input, results and errors

any process0 stdinkeyboard, or a file or pipe1 stdoutresults: the terminal2 stderrerrors: also the terminalkept apart, so errors still show when results go to a fileEvery process starts with three numbered streams: 0 reads input, 1 writes results, 2 writes errors.A program does not know or care where they lead. The shell decides, before the program starts: the keyboard and screen bydefault, or a file, or another program. Keeping errors on their own stream is what lets you save results to a file and stillsee what went wrong on the screen. The numbers are file descriptors; a program opens more for each file it uses.

Every process has stdin (0), stdout (1) and stderr (2); the shell decides where each one leads.

Redirection

Rewire the streams before the program starts, left to right

You writeIt meanscmd > out.txtstdout to a file, emptied firstcmd >> out.txtstdout appended to the filecmd < in.txtstdin read from a filecmd 2> err.txtstderr only to a filecmd > all.txt 2>&1both to the file: 2 goes where 1 goes nowcmd &> all.txtthe same, shorter (bash)cmd 2>/dev/nullthrow errors awayOrder matters: redirections apply left to right$ ls /nope /etc/hostname > out.txt 2>&1$ cat out.txtls: cannot access '/nope': .../etc/hostname$ ls /nope /etc/hostname 2>&1 > out.txtls: cannot access '/nope': ...$ cat out.txt/etc/hostnameThe shell rewires the streams before the program runs, one redirection at a time, from left to right.2>&1 means “make 2 point where 1 points right now”. Written after > out.txt, both go to the file. Written before it, 2 copies theterminal and stays there. A single > empties the file before the command even starts, so sort data.txt > data.txt leaves anempty file. set -o noclobber makes > refuse to overwrite; >| overrides it.

Redirection sends a stream to or from a file; 2>&1 copies wherever stream 1 points at that moment.

Pipes

One program’s output becomes the next one’s input, as both run

grep 404writes to fd 1pipe: a buffer in the kernel64 KiB by defaultsortreads from fd 0uniq -cAll three start at once and run side by side.If the buffer fills, the writer waits; if it is empty, the reader waits. Nothing is written to disk.A pipe connects one program’s stdout to the next one’s stdin, through memory, while both run.Each program does one job well, and the pipe composes them, so ten small tools solve problems none of them was written for.Only stdout enters the pipe; use |& or 2>&1 | to send errors along too. A pipeline’s exit status is the last command’s;set -o pipefail in scripts makes any failure count. sort is the exception that must read everything before writing.

A pipe is a kernel buffer joining stdout of one command to stdin of the next; all commands in a pipeline run at once.

grep

Out of 8,000 lines, how do you see only the ones that matter?

$ grep -c ' install ' /var/log/dpkg.log1836$ grep ' install ' /var/log/dpkg.log | tail -222026-10-27 20:02:40 install xbitmaps:all <none> 1.1.1-2.22026-10-27 20:02:40 install xterm:amd64 <none> 390-1ubuntu3$ grep -n 'PASS_MAX' /etc/login.defs3161:# PASS_MAX_DAYS Maximum number of days a password may be ...165:PASS_MAX_DAYS 99999373:#PASS_MAX_LEN$ grep -vc '^#' /etc/login.defs4791-c counts the matching lines836 of 8,0762Without -c, it prints themand a pipe can trim the list3-n adds line numbersso you can go straight there4-v keeps lines that do NOT matchhere: lines that are not commentsgrep prints the lines that contain a pattern. It is the first tool to reach for in any log or settings file.Put the pattern in single quotes so the shell leaves it alone. -i ignores upper and lower case, and -r searches every fileunder a directory: grep -r PermitRootLogin /etc/ssh. The pattern can be far more than a fixed word, which is theregular expressions page later in this lesson.

grep keeps the lines that match a pattern: -c counts them, -n numbers them, -i ignores case and -v inverts the match.

Text tools

Small tools, one job each, joined by pipes

ToolDoesExamplecut -d: -f1,7pick fields by a separatorcut -d: -f1 /etc/passwdsort -k2 -n -r -uorder lines by a key, numbers, reverse, uniquesort -t: -k3 -n /etc/passwduniq -ccount adjacent duplicatessort | uniq -cwc -lcount lines, words, bytesgrep -c is quicker for matcheshead -n 20, tail -n 20first or last linestail -f /var/log/syslog follows ittr a-z A-Ztranslate or delete characterstr -d '\r' fixes Windows filestee filewrite to a file and pass alongcmd | sudo tee /etc/x.confxargsturn lines into argumentsfind ... | xargs rmEach tool does one job on lines of text, so any of them can sit at any point in a pipe.The one trap: sudo applies to a command, not to a redirection. sudo echo x > /etc/file fails because your own shell opens the file;echo x | sudo tee /etc/file works because tee, running as root, opens it. less pages through output; q quits and / searches.

cut, sort, uniq, wc, head, tail, tr, tee and xargs each transform lines of text and fit anywhere in a pipe.

Here-documents · Going deeper

Feed a block of lines to a command from inside a script

$ cat <<EOF > /tmp/motd> Maintenance tonight 22:00 UTC> Host: $(hostname)> EOF$ cat /tmp/motdMaintenance tonight 22:00 UTCHost: web011<<EOF ... EOFthe lines between are stdin2$(...) still expandswrite <<'EOF' to stop it3<<< "text"a here-string: one line as stdin4<(cmd)process substitution: a command as a filediff <(sort a.txt) <(sort b.txt)A here-document feeds the lines of a script to a command as if they were typed.It is how scripts write configuration files and send several lines to tools such as ssh or psql. Quoting the end word turns offexpansion inside it. Process substitution goes the other way: it hands a command’s output to a program that wants a file name.

A here-document supplies lines as stdin; process substitution supplies a command’s output as a file.

Pipelines

Six stations turn 2,000 log lines into one answer

access.log2,000 lines203.0.113.7 - - [14/Nov/2026:10:00:00 +0000] "GET /login HTTP/1.1" 200 8979192.0.2.14 - - [...] "GET /favicon.ico HTTP/1.1" 404 2953grep ' 404 '174 lines192.0.2.14 - - [...] "GET /favicon.ico HTTP/1.1" 404 2953203.0.113.50 - - [...] "GET /old-page HTTP/1.1" 404 8334awk '{print $7}'174 lines/favicon.ico/old-pagesort174 lines/favicon.ico/favicon.icouniq -c3 lines 74 /favicon.ico 58 /old-pagesort -rn | head -33 lines 74 /favicon.ico 58 /old-page 42 /wp-login.phpWhich missing pages are hit most? Six stations, each narrowing the text, answer it in one line.Read a pipeline as a series of stations and check the text after each one: build it a stage at a time and press Enter after each.uniq only merges duplicate lines that are next to each other, which is why sort always comes before it. The counts are real:this is the full answer from a 2,000-line access log, in under a tenth of a second.

Build a pipeline one station at a time, checking the text after each; sort before uniq.

Regular expressions

Describe the shape of a line, and grep finds every one

grep -E '^[^ ]+ .*"(GET|POST) /api/v[12]/[a-z]+' access.logPieceMatchesFamily^ $start, end of linebasic. [abc] [^ ]any one character, one of a set, not a spacebasic*the previous item, zero or more timesbasic+ ? {2,4}one or more, zero or one, two to fourextended (-E)(a|b)a group, either a or bextended (-E)\.a real dotbasicUseful flags-i ignore case-v lines that do not match-c count-n line numbers-r search a tree-o only the match-w whole wordsA regular expression describes a shape of text; grep prints every line containing that shape.Most characters match themselves; a few are operators. grep uses basic expressions, where + ? | ( ) { } are plain text unlessescaped; grep -E switches on extended ones and is easier to read. Always quote the pattern in single quotes, or the shell willtry to expand the * and the brackets before grep sees them.

A regular expression uses anchors, sets and repetition; grep -E enables + ? | and groups without backslashes.

sed

Edit a stream line by line, without opening an editor

sed -E 's/(pass)=[^ ]+/\1=***/g' app.envssubstitute(pass)=[^ ]+find: pass= and the value\1=***replace: group 1, then =***gevery match on the lineindb_user=admin db_pass=hunter2 port=5432outdb_user=admin db_pass=*** port=5432sed -n '10,20p' fileprint lines 10 to 20sed '/^#/d' filedelete comment linessed -i.bak 's/a/b/' fileedit in place, keep a copysed edits a stream line by line: read a line, apply the commands, print it, repeat.It never loads the whole file, so it works on gigabytes and in the middle of pipelines. The substitute command is most of what youwill use: s/find/replace/flags, with \1 recalling a bracketed group. -i edits the file itself; give it a suffix such as -i.bak onproduction systems so a wrong expression can be undone. The delimiter can be any character: s|/usr/local|/opt|.

sed applies commands to each line as it passes; s/find/replace/g is the one you will use most.

awk

Split each line into fields, then act on the lines that match

awk -F: '$3 >= 1000 && $3 < 65534 {print $1, $6}' /etc/passwdana$1x$21001$31001$4Ana Lopez$5/home/ana$6/bin/bash$7name · password placeholder · UID · GID · comment · home · shell: split on -F:pattern$3 >= 1000 && $3 < 65534action{print $1, $6}ubuntu /home/ubuntuana /home/anathe human accounts: UIDs from 1000, without nobody (65534)awk splits each line into numbered fields and runs an action on the lines that match a pattern.By default fields are separated by runs of spaces; -F sets another separator. $0 is the whole line, NF the number of fields andNR the line number, so awk '{sum += $10} END {print sum}' adds up a column. It is a full language, but these two ideas, fields andpattern-action pairs, cover nearly every one-liner you will need.

awk splits lines into fields $1, $2 and so on, and runs {action} on lines matching a pattern.

Lab

Answer questions about a web log with nothing but pipes

$ awk '{print $1}' access.log | sort | uniq -c | sort -rn1 426 198.51.100.23 405 192.0.2.14 404 198.51.100.9 394 203.0.113.7 371 203.0.113.50$ awk '$9 == 500' access.log | wc -l250$ grep -cE '"(GET|POST) /api/' access.log3617$ ls /nope /etc/hostname > out.txt 2>&1; wc -l out.txt42 out.txt1Requests per clientthe classic count-and-rank2Server errorsfield 9 is the status code3API callsan extended expression4Both streams to one fileand in the right orderLab: answer questions about a log with pipes.Download the course’s practice log or use any web server log you have. Press the down arrow for the steps.

Count, rank, total, match and mask with awk, sort, uniq, grep and sed, then redirect both streams correctly.

Lab · Going deeper

Answer five questions about a web server log

Use any access log in the common Apache or nginx format: /var/log/nginx/access.log on a server, or the practice log from this course. Build each pipeline one station at a time.

Steps

  1. See the shape. head -3 access.log; wc -l access.log. Number the fields of one line by hand, counting from $1.
  2. Count by client. awk '{print $1}' access.log | sort | uniq -c | sort -rn | head. Remove the first sort and explain why the counts change.
  3. Find missing pages. grep ' 404 ' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -3.
  4. Total the bytes. awk '{sum += $10} END {print sum/1024/1024 " MiB"}' access.log.
  5. Match with a pattern. grep -E '"(GET|POST) /api/' access.log | wc -l, then the same with -v to count everything else.
  6. Mask a secret. printf 'user=a pass=hunter2\n' > app.env; sed -E -i.bak 's/(pass)=[^ ]+/\1=***/' app.env; diff app.env.bak app.env.
  7. Redirect both ways round. ls /nope ~ > a.txt 2>&1 and ls /nope ~ 2>&1 > b.txt. Explain what reached the screen each time.
  8. Write as root through a pipe. echo '# test' | sudo tee /etc/tee-test.conf; then try sudo echo x > /etc/tee-test.conf and read the error. Remove the file.
Check: Your top client and top missing page match what you get by counting with a second method, such as grep -c for one IP.

Lesson complete

You finished “Pipes and text tools”

Nicely done.

Open in a new tab