awk + sed Mastery

awk is best for column-based parsing and conditional reporting. sed is best for quick line editing, replacements and stream transformations. This page now uses short one-line explanations for fast learning.

awk Essentials

CommandWhat it doesExample
awk '{print $1}' access.logPrints the first column from each line.awk '{print $1}' access.log
awk '{print $1, $4, $9}' access.logPrints selected columns from each line.awk '{print $1, $4, $9}' access.log
awk -F',' '{print $1, $3}' users.csvUses comma as separator and prints chosen CSV columns.awk -F',' '{print $1, $3}' users.csv
awk '$9 >= 500 {print $0}' access.logPrints only lines where column 9 is 500 or more.awk '$9 >= 500 {print $0}' access.log
awk '/ERROR|WARN/' app.logPrints only lines matching ERROR or WARN.awk '/ERROR|WARN/' app.log
awk 'NR==1,NR==20 {print}' app.logPrints a specific line range.awk 'NR==1,NR==20 {print}' app.log
awk '{count[$1]++} END {for (k in count) print k, count[k]}' access.logCounts repeated values in the first column.awk '{count[$1]++} END {for (k in count) print k, count[k]}' access.log

sed Essentials

CommandWhat it doesExample
sed 's/localhost/127.0.0.1/g' config.ymlReplaces text in output.sed 's/localhost/127.0.0.1/g' config.yml
sed -n '1,20p' file.txtPrints only the selected line range.sed -n '1,20p' file.txt
sed '/DEBUG/d' app.logDeletes matching lines from output.sed '/DEBUG/d' app.log
sed -n '/BEGIN/,/END/p' file.txtPrints lines between two patterns.sed -n '/BEGIN/,/END/p' file.txt
sed 's/[[:space:]]\+$//' file.txtRemoves trailing spaces from each line.sed 's/[[:space:]]\+$//' file.txt
sed -i 's/old/new/g' file.txtEdits the file in place and replaces all matches.sed -i 's/old/new/g' file.txt

Practical Combinations

CommandWhat it doesExample
grep ERROR app.log | awk '{print $1, $2, $NF}'Finds error lines and prints selected fields.grep ERROR app.log | awk '{print $1, $2, $NF}'
ps aux | awk '{print $2, $3, $4, $11}' | headShows PID, CPU, memory and command columns.ps aux | awk '{print $2, $3, $4, $11}' | head
df -h | awk 'NR==1 || $5+0 > 80'Shows only filesystems over 80% usage plus header.df -h | awk 'NR==1 || $5+0 > 80'
sed -n '1000,1050p' app.log | grep ExceptionPrints a line range and filters exceptions.sed -n '1000,1050p' app.log | grep Exception

When to Use Which

NeedBetter Tool
Pick columns from structured textawk
Filter by numeric conditionsawk
Replace text quicklysed
Delete or print line rangessed
Summarize countsawk