awk - nurmi-labs/blogger GitHub Wiki

awk

Remove a BOM from a UTF-8 encoded DOS format .csv file.

bash-4.3$ awk '{sub(/^\xef\xbb\xbf/,""); printf "%s\r\n", $0}' infile.csv > outfile.csv
bash-4.3$ 

ASCII's integer values 0-31 are C0 control codes, SP is 32 and DEL is 127; 128-255 are "extended" ASCII.

Here are dumped US layout keyboard printable characters.

bash-5.1$ awk 'BEGIN{for(i=32;i<=127;i++)printf "%c",i}';echo
 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
bash-5.1$