bash sed - ghdrako/doc_snipets GitHub Wiki

sed

When invoke the sed command, an execution cycle refers to various options that are specified and executed until the end of the file/input is reached. Specifically, an execution cycle performs the following steps:

  • Reads an entire line from stdin/file.
  • Removes any trailing newline.
  • Places the line in its pattern buffer.
  • Modifies the pattern buffer according to the supplied commands.
  • Prints the pattern buffer to stdout.
sed –n /3/p numbers.txt

The –n option suppresses all output, and the p option prints the matching line. If you omit the –n option, then every line is printed, and the p option causes the matching line to be printed again.

sed –n ̎/123/,/five/p̎ numbers.txt # match two patterns and print everything between the lines that match

SUBSTITUTING STRING PATTERNS

s/old/new/

x=abc
echo $x |sed s/abc/def/

$ echo abcdefabc |sed s/abc//
defabc
$ echo abcdefabc |sed s/abc//g
def

Note that we are operating directly on the main stream with this command, as we are not using the -n tag. You can also suppress the main stream with -n and print the substitution, achieving the same output if you use the terminal p (print) instruction:

$ echo abcdefabc |sed -n s/abc//gp

Addresses for Selective Editing

sed '/WARNING/ s/required/urgent/' data.txt
  • The Address Part: /WARNING/ limits the substitution only to lines containing the word ‘WARNING’.
  • Output: Other lines are now passed through unmodified

Addresses with Multiple Matches:

sed '/ERROR/,/WARNING/ s/required/urgent/' data.txt

Deleting Lines

sed '/^INFO/d' data.txt
  • d is for delete: The address /^INFO/ matches lines starting with ‘INFO’
  • Output: Our ‘INFO’ line vanishes; error messages remain.

Line Numbers:

sed '3d' data.txt # Deletes the 3rd line

Ranges:

sed '1,5 s/\[.*\]//g' data.txt # Removes [Code 127]-like parts from lines 1 to 5

‘g’ Flag for Global Replace:

sed 's/:/;/g' data.txt # Replaces *all* colons with semicolons in each line

Numbered Replacements:

sed 's/\([0-9]\{3\}\)-\([0-9]\{4\}\)/(\2) \1/' phone_data.txt
  • Uses capture groups () and backreferences (\1, \2) to reformat phone numbers!

The ‘y’ Command (Translation):

sed 'y/abc/XYZ/' data.txt # Replaces each 'a' -> 'X', 'b' -> 'Y', etc.

Saving Changes - The ‘-i’ Option

  • Use with Caution: sed -i 's/foo/bar/g' my_file.txt edits the file directly
  • Backups First: sed -i.bak 's/foo/bar/g' my_file.txt is safer (creates my_file.txt.bak)

Multi-Command ‘sed’ Scripts

sed -e 's/ERROR/CRITICAL/g' \
-e '/Code/d' data.txt
  • -e is for specifying multiple commands
  • Each line is basically a mini ‘sed’ program on its own. Order matters!

Command Syntax

/abc/p      # Print all with "abc"
/abc/!p     # Print all without "abc"
/abc/d      # Delete all with "abc"
/abc/!d     # Delete all without "abc"

/start/,/end/!d     # Select a block

s/abc/def/
s/abc\(...\)ghi/\1/ # Back references
            # (see below for correct Shell quoting)

/abc/{s/def/ghi)}   # Conditional replace

Appending lines

aHallo                   # Append 'Hallo' after each line
5 aHallo                 # Append 'Hallo' after line #5
$ aHallo                 # Append 'Hallo' to end of file

Prepending lines

sed -i '1s;^;new line 1\nanother new line 2\n;' <file>
Remove Double Quote from a String

Remove quotes from start and end of the string

sed -e 's/^"//' -e 's/"$//' <<<"$var1"

Remove Double Quote and Store Output in variable

var2=`sed -e 's/^"//' -e 's/"$//' <<<"$var1"`    #Save in another variable
var1=`sed -e 's/^"//' -e 's/"$//' <<<"$var1"`    #Save in same variable

Remove Double Quote and Store Output in file

sed -e 's/^"//' -e 's/"$//' <<<"$var1" > out_var.txt

Advanced use of sed

In-place Editing

To edit file use the -i option this safely changes the file contents without any output redirection needed.

sed -i 's/abc/ABC/' myfile.txt
sed -i '/deleteme/d' *

Drop grep

Often grep and sed are used together. In all those cases grep can be dropped. For example

grep "pattern" file | sed "s/abc/def/"

can be written as

sed -n "/pattern/p; s/abc/def/"

Grouping with sed

Always use single quotes!

sed 's/^.*\(pattern\).*/\1/'

Single Quoting Single Quotes

If you want to do extraction and need a pattern based on single quotes use \x27 instead of trying to insert a single quote. For example:

sed 's/.*var=\x27\([^\x27]*\)\x27.*/\1/'

to extract “some string” from “var=‘some string’”. Or if you don’t know about the quoting, but know there are quotes

sed 's/.*var=.\([^"\x27]*\)..*/\1/'

Conditional Replace with sed

sed '/conditional pattern/{s/pattern/replacement/g}'

Prefix files with a boilerplate using sed

sed -i '1s/^/# DO NOT TOUCH THIS FILE!\n\n/' *

Removing Newlines with sed

The only way to remove new line is this:

sed ':a;N;$!ba;s/\n//g' file

Check out this explanation if you want to know why.

Selecting Blocks

sed '/first line/,/last line/!d' file

Print a specific line from a file

sed -n 10p /path/to/file

Remove a specific line from a file

sed -i 10d /path/to/file
# alternative (BSD): sed -i'' 10d /path/to/file

Remove a range of lines from a file

sed -i <file> -re '<start>,<end>d'

Replace newline(s) with a space

sed ':a;N;$!ba;s/\n/ /g' /path/to/file

# cross-platform compatible syntax:
sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g' /path/to/file
  • :a create a label a
  • N append the next line to the pattern space
  • $! if not the last line, ba branch (go to) label a
  • s substitute, /\n/ regex for new line, / / by a space, /g global match (as many times as it can)

Alternatives:

# perl version (sed-like speed):
perl -p -e 's/\n/ /' /path/to/file

# bash version (slow):
while read line ; do printf "%s" "$line " ; done < file

Delete string +N next lines

sed '/start/,+4d' /path/to/file
⚠️ **GitHub.com Fallback** ⚠️