awk - jasper-zanjani/dotfiles GitHub Wiki

            f                               v        
            F                                        

ARGC ARGV FILENAME FNR FS NF NR OFS ORS RS

gsub() sub() index() length() match() printf() split() substr() tolower() toupper()

"The basic function of awk is to search files for lines that contain certain patterns." (GEAP: 17

Pattern-scanning utility and processing language, one of the two primary commands which accept regular expressions in Unix systems.

awk programs can be defined inline or in a program-file (PGL: 528)

• inline: awk options 'program' input-files
• program-file, also "source-file" (GEAP:18): awk options -f program-file input-files

awk programs can be run without defining input-files, in which case awk will accept input from STDIN

awk programs are the equivalent of sed "instructions", and similarly combine patterns and actions (PGL: 530, GEAP: 17)

Patterns can be:

• regular expressions or fixed strings
• line numbers using builtin variable NR
• predefined patterns BEGIN or END, whose actions are executed before and after processing any lines of the data file, respectively

change ":" to newlines in PATH variable; equivalent to echo $PATH \| tr ":" "\n"

echo $PATH | awk 'BEGIN {RS=":"} {print}'

print the first field of all files in the current directory, taking semicolon ; as the field separator, outputting filename, line number, and first field of matches, with colon : between the filename and line number

awk 'BEGIN {FS=";"} /enable/ {print FILENAME ":" FNR,$1}' *

search for string MA in all files, outputting filename, line, and line number for matches

awk '/MA/ {OFS=" " print FILENAME OFS FNR OFS $0} *

change field separator (FS) to a colon (:) and run awkscr

awk -F: -f awkscr /etc/passwd

flag also works for awk

awk -f script files` `-f

print the first field of each line in the input file

awk '{ print $1 }' list

equivalent to grep MA * ({print} is implied)

awk '/MA/' * | awk '/MA/ {print}' *

-F flag is followed by field separator

awk -F, '/MA/ { print $1 }' list

pipe output of free to awk to get free memory and total memory

free -h | awk '/^Mem|/ {print $3 "/" $2}

pipe output of sensors to awk to get CPU temperature

sensors | awk '/^temp1/ {print $2}

replace initial "fake." with "real;" in file fake_isbn

awk 'sub(^fake.,"real;")' fake_isbn

print all lines

awk '1 { print }' file

remove file header

awk 'NR>1' file

remove file header

awk 'NR>1 { print } file

print lines in a range

awk 'NR>1 && NR < 4' file

remove whitespace-only lines

awk 'NF' file

remove all blank lines

awk '1' RS='' file

extract fields

awk '{ print $1, $3}' FS=, OFS=, file

perform column-wise calculations

awk '{ SUM=SUM+$1 } END { print SUM }' FS=, OFS=, file

count the number of nonempty lines

awk '/./ { COUNT+=1 } END { print COUNT }' file

count the number of nonempty lines

awk 'NF { COUNT+=1 } END { print COUNT }' file

count the number of nonempty lines

awk '+$1 { COUNT+=1 } END { print COUNT }' file

Arrays

awk '+$1 { CREDITS[$3]+=$1 } END { for (NAME in CREDITS) print NAME, CREDITS[NAME] }' FS=, file

Identify duplicate lines

awk 'a[$0]++' file

Remove duplicate lines

awk '!a[$0]++' file

Remove multiple spaces

awk '$1=$1' file

Join lines

awk '{ print $3 }' FS=, ORS=' ' file; echo

awk '+$1 { SUM+=$1; NUM+=1 } END { printf("AVG=%f",SUM/NUM); }' FS=, file` | format 

awk '+$1 { SUM+=$1; NUM+=1 } END { printf("AVG=%6.1f",SUM/NUM); }' FS=, file

Convert to uppercase

awk '$3 { print toupper($0); }' file

Change part of a string

awk '{ $3 = toupper(substr($3,1,1)) substr($3,2) } $3' FS=, OFS=, file

Split the second field ("EXPDATE") by spaces, storing the result into the array DATE; then print credits ($1) and username ($3) as well as the month (DATE[2]) and year (DATE[3])

awk '+$1 { split($2, DATE, " "); print $1,$3, DATE[2], DATE[3] }' FS=, OFS=, file

awk '+$1 { split($4, GRP, ":"); print $3, GRP[1], GRP[2] }' FS=, file

awk '+$1 { split($4, GRP, /:+/); print $3, GRP[1], GRP[2] }' FS=, file

Search and replace with comma

awk '+$1 { gsub(/ +/, "-", $2); print }' FS=, file

Adding date

awk 'BEGIN { printf("UPDATED: "); system("date") } /^UPDATED:/ { next } 1' file

Modify a field externally

awk '+$1 { CMD | getline $5; close(CMD); print }' CMD="uuid -v4" FS=, OFS=, file

Invoke dynamically generated command

awk '+$1 { cmd = sprintf(FMT, $2); cmd | getline $2; close(cmd); print }' FMT='date -I -d "%s"'  FS=, file

Join data

awk '+$1 { CMD | getline $5; print }' CMD='od -vAn -w4 -t x /dev/urandom' FS=, file

Add up all first records to {sum}, then print that number out at the end

awk '{sum += $1} END {print sum}' file