Standard streams
Three numbered streams: input, results and errors
Every process has stdin (0), stdout (1) and stderr (2); the shell decides where each one leads.
Every program starts with three streams already open, each with a number. Zero is standard input, normally the keyboard. One is standard output, where results go. Two is standard error, for problems. Both go to the terminal by default, but they are separate, and that separation is the point: you can send results to a file and still see errors on the screen.
Redirection
Rewire the streams before the program starts, left to right
Redirection sends a stream to or from a file; 2>&1 copies wherever stream 1 points at that moment.
Redirection rewires those streams. A single greater-than sends output to a file, emptying it first. Two of them append. Less-than reads input from a file, and two greater-than sends only errors. The subtle one is two greater-than ampersand one. It means: point stream two wherever stream one points right now. Put it after the file redirection and both streams land in the file. Put it before, and errors still go to the screen. The shell works through redirections from left to right.
Pipes
One program’s output becomes the next one’s input, as both run
A pipe is a kernel buffer joining stdout of one command to stdin of the next; all commands in a pipeline run at once.
The vertical bar is a pipe. grep writes its output as usual, but it goes into a small buffer in the kernel instead of the screen, and sort reads from that buffer as its input. Both programs run at the same time. When the buffer is full, the writer waits; when it is empty, the reader waits. Nothing touches the disk. This is the Unix idea: small tools that each do one job, joined into something new.
grep
Out of 8,000 lines, how do you see only the ones that matter?
grep keeps the lines that match a pattern: -c counts them, -n numbers them, -i ignores case and -v inverts the match.
The package log from the last lesson has eight thousand lines. How do you see only the ones you care about? grep reads lines and prints those that contain a pattern. With dash c it only counts them. Without, it prints them, and because it writes to standard output, a pipe can pass the result to another command, here tail. Dash n adds the line number in the file. And dash v turns the test round: keep the lines that do not match.
Text tools
Small tools, one job each, joined by pipes
cut, sort, uniq, wc, head, tail, tr, tee and xargs each transform lines of text and fit anywhere in a pipe.
A few more tools complete the kit. cut picks fields by a separator. sort orders lines, by a chosen key, numerically or in reverse. wc counts. head and tail take the first or last lines, and tail dash f follows a log as it grows. tr swaps characters. tee writes to a file and passes the text on, which is also how you write a root-owned file from a pipe. And xargs turns lines into arguments for another command.
Here-documents · Going deeper
Feed a block of lines to a command from inside a script
A here-document supplies lines as stdin; process substitution supplies a command’s output as a file.
Pipelines
Six stations turn 2,000 log lines into one answer
Build a pipeline one station at a time, checking the text after each; sort before uniq.
Here is a real question: which missing pages are requested most? Start with two thousand lines of web log. grep keeps the one hundred and seventy-four with status four oh four. awk keeps only the seventh field, the path. sort puts identical paths next to each other. uniq dash c collapses each run into one line with a count. And sort dash r n, then head, puts the biggest first. Favicon, seventy-four times. Each station is simple. Together they are an analysis tool.
Regular expressions
Describe the shape of a line, and grep finds every one
A regular expression uses anchors, sets and repetition; grep -E enables + ? | and groups without backslashes.
grep prints lines that match a pattern, and the pattern is a regular expression. This one finds API requests made with GET or POST. Most characters match themselves. A caret anchors to the start of a line, a dot matches any one character, square brackets match one of a set, and a star repeats the item before it. With dash capital E, you also get plus, question mark, braces, groups and the bar for either-or. A handful of flags complete it: ignore case, invert, count, recurse.
sed
Edit a stream line by line, without opening an editor
sed applies commands to each line as it passes; s/find/replace/g is the one you will use most.
sed is a stream editor. The substitute command has four parts: s, what to find, what to put instead, and flags, where g means every match on the line. Brackets capture part of the match, and backslash one puts it back. Here it masks a password in a configuration file. sed also prints ranges of lines, deletes lines that match, and with dash i, edits a file in place. Give dash i a suffix so you keep a backup.
awk
Split each line into fields, then act on the lines that match
awk splits lines into fields $1, $2 and so on, and runs {action} on lines matching a pattern.
awk is the tool for columns. This command lists the real people on a system. Dash F colon tells awk to split each line of the password file at the colons, so dollar one is the name, dollar three the user ID and dollar six the home directory. The pattern keeps lines whose user ID is a thousand or more, but not sixty-five thousand five hundred and thirty-four, the nobody account. The action prints two fields. The result is the human accounts.
Lab
Answer questions about a web log with nothing but pipes
Count, rank, total, match and mask with awk, sort, uniq, grep and sed, then redirect both streams correctly.
In this lab you take a real web server log and answer questions about it: who sends the most requests, which pages are missing, how many server errors there were, and how much data went out. Press the down arrow for the steps.
Lab · Going deeper
Answer five questions about a web server log
Use any access log in the common Apache or nginx format: /var/log/nginx/access.log on a server, or the practice log from this course. Build each pipeline one station at a time.
Use any access log in the common Apache or nginx format: /var/log/nginx/access.log on a server, or the practice log from this course. Build each pipeline one station at a time. Steps. See the shape. head -3 access.log; wc -l access.log. Number the fields of one line by hand, counting from $1. Count by client. awk '{print $1}' access.log | sort | uniq -c | sort -rn | head. Remove the first sort and explain why the counts change. Find missing pages. grep ' 404 ' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -3. Total the bytes. awk '{sum += $10} END {print sum/1024/1024 " MiB"}' access.log. Match with a pattern. grep -E '"(GET|POST) /api/' access.log | wc -l, then the same with -v to count everything else. Mask a secret. printf 'user=a pass=hunter2\n' > app.env; sed -E -i.bak 's/(pass)=[^ ]+/\1=***/' app.env; diff app.env.bak app.env. Redirect both ways round. ls /nope ~ > a.txt 2>&1 and ls /nope ~ 2>&1 > b.txt. Explain what reached the screen each time. Write as root through a pipe. echo '# test' | sudo tee /etc/tee-test.conf; then try sudo echo x > /etc/tee-test.conf and read the error. Remove the file. Check: Your top client and top missing page match what you get by counting with a second method, such as grep -c for one IP.
Steps
- See the shape. head -3 access.log; wc -l access.log. Number the fields of one line by hand, counting from $1.
- Count by client. awk '{print $1}' access.log | sort | uniq -c | sort -rn | head. Remove the first sort and explain why the counts change.
- Find missing pages. grep ' 404 ' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -3.
- Total the bytes. awk '{sum += $10} END {print sum/1024/1024 " MiB"}' access.log.
- Match with a pattern. grep -E '"(GET|POST) /api/' access.log | wc -l, then the same with -v to count everything else.
- Mask a secret. printf 'user=a pass=hunter2\n' > app.env; sed -E -i.bak 's/(pass)=[^ ]+/\1=***/' app.env; diff app.env.bak app.env.
- Redirect both ways round. ls /nope ~ > a.txt 2>&1 and ls /nope ~ 2>&1 > b.txt. Explain what reached the screen each time.
- Write as root through a pipe. echo '# test' | sudo tee /etc/tee-test.conf; then try sudo echo x > /etc/tee-test.conf and read the error. Remove the file.
Lesson complete
You finished “Pipes and text tools”
Nicely done.