bash sed - ghdrako/doc_snipets GitHub Wiki

info about sed

info sed

Command	Description
`:`	label
`=`	line_number
`a`	append_text_to_stdout_after_flush
`b`	branch_unconditional
`c`	range_change
`d`	pattern_delete_top/cycle
`D`	pattern_ltrunc(line+nl)_top/cycle
`g`	pattern=hold
`G`	pattern+=nl+hold
`h`	hold=pattern
`H`	hold+=nl+pattern
`i`	insert_text_to_stdout_now
`l`	pattern_list
`n`	pattern_flush=nextline_continue
`N`	pattern+=nl+nextline
`p`	pattern_print
`P`	pattern_first_line_print
`q`	flush_quit
`r`	append_file_to_stdout_after_flush
`s`	substitute
`t`	branch_on_substitute
`w`	append_pattern_to_file_now
`x`	swap_pattern_and_hold
`y`	transform_chars

sed

When invoke the sed command, an execution cycle refers to various options that are specified and executed until the end of the file/input is reached. Specifically, an execution cycle performs the following steps:

Reads an entire line from stdin/file.
Removes any trailing newline.
Places the line in its pattern buffer.
Modifies the pattern buffer according to the supplied commands.
Prints the pattern buffer to stdout.

sed –n /3/p numbers.txt

The –n option suppresses all output, and the p option prints the matching line. If you omit the –n option, then every line is printed, and the p option causes the matching line to be printed again.

sed –n '/123/,/five/p' numbers.txt # match two patterns and print everything between the lines that match

SUBSTITUTING STRING PATTERNS

s/old/new/

x=abc
echo $x |sed s/abc/def/

g - substitute EVERY occurence /../../g
w file - write to separate output file
- okresla dokladnie ktore wystapienie ma zostac podmienione np /.../.../2 - tylko drugie wystopienie w lini bedzie podmienionw

‘g’ Flag for Global Replace:

sed 's/:/;/g' data.txt # Replaces *all* colons with semicolons in each line

$ echo abcdefabc |sed s/abc//
defabc
$ echo abcdefabc |sed s/abc//g
def

`p` - PRINT

p - PRINT, typically used in conjunction with -n(don't print)

Note that we are operating directly on the main stream with this command, as we are not using the -n tag. You can also suppress the main stream with -n and print the substitution, achieving the same output if you use the terminal p (print) instruction:

$ echo abcdefabc |sed -n s/abc//gp

Wyswietlamy tylko linie w ktorych cos zostalo podmienione

sed -n 's/good/bad; s/sentence/paragraph/p' sedFile.txt

sed -n '3,4/p' file.txt # show line 3 to 4

sed -n '/diamond/p' file.txt # like grep show line contain word diamond

Escape special characters

sed 's/\/bin\/bash/\/bin\/zsh/' /etc/passwd
sed 's!/bin/bash!/bin/zsh!' /etc/passwd

Sed Addresses

Line Numbers:

sed '2s/what/WHO/' file.txt  # substritute only in line 2

sed '3d' data.txt # Deletes the 3rd line

Ranges:

sed '2,3s/what/WHO/' file.txt  # substritute only in line in reange 2 to 3

sed '2,$s/what/WHO/' file.txt  # substritute only in line in reange 2 to end of file

sed '1,5 s/\[.*\]//g' data.txt # Removes [Code 127]-like parts from lines 1 to 5

Addresses for Selective Editing

substitute only with line contains WARNING string

sed '/WARNING/ s/required/urgent/' data.txt

The Address Part: /WARNING/ limits the substitution only to lines containing the word ‘WARNING’.
Output: Other lines are now passed through unmodified

Addresses with Multiple Matches:

sed '/ERROR/,/WARNING/ s/required/urgent/' data.txt

Deleting Lines

sed 'd' data.txt  # delete all lines in output
sed '4d' data.txt  # delete line number 4
sed '4,$d' data.txt  # delete line number 4
sed '/world/d' data.txt  # delete lines with word "world"
sed '/^INFO/d' data.txt
sed '/^$/d' data.txt  # remove empty lines

d is for delete: The address /^INFO/ matches lines starting with ‘INFO’
Output: Our ‘INFO’ line vanishes; error messages remain.

Grouping command

sed '2{
s/wonder/PONDER/
s/what/WHO/
}' file.txt

Inserting - use backslash !!!

sed 'i\Line inserted at begining` file.txt  
sed '1i\Line inserted at beggining (before line nr 1)` file.txt  
sed '4i\Line inserted before line number 4` file.txt

Multi-line insert

sed '4i\
Line A\
Line B' file.txt

Appending

sed 'a\Line append at end` file.txt
sed '4a\Line apppend after line number 4` file.txt
sed '/diamond/a\Line apppend after line with word diamond` file.txt

Multi-line append

sed '4a\
Line A\
Line B' file.txt

`r` Read From Extra File and Append

In example append

extra.txt:
Line to add

file.txt
Line1
Line2
Line3
Line4

sed '3r extra.txt' file.txt  # append after line 3
Line1
Line2
Line3
Line to add
Line4

Changing

sed '4c\Line replace line number 4` file.txt

Transform characters - yanke

sed 'y/abc/zyz/' file.txt  # a->x,b->y,c->z

Add line number

sed '=' longfile.txt

Numbering and showing only lines contains ulocked word

sed -n '/unlocked/{=;p}longfile.txt

Numbered Replacements:

sed 's/\([0-9]\{3\}\)-\([0-9]\{4\}\)/(\2) \1/' phone_data.txt

Uses capture groups () and backreferences (\1, \2) to reformat phone numbers!

The ‘y’ Command (Translation):

sed 'y/abc/XYZ/' data.txt # Replaces each 'a' -> 'X', 'b' -> 'Y', etc.

Saving Changes - The ‘-i’ Option

Use with Caution: sed -i 's/foo/bar/g' my_file.txt edits the file directly
Backups First: sed -i.bak 's/foo/bar/g' my_file.txt is safer (creates my_file.txt.bak)

Multi-Command ‘sed’ Scripts

sed -e 's/good/bad; s/sentence/paragraph' sedFile.txt

or prefered because more readable

sed -e '
s/good/bad 
s/sentence/paragraph
' sedFile.txt

sed -e 's/ERROR/CRITICAL/g' \
-e '/Code/d' data.txt

-e is for specifying multiple commands
Each line is basically a mini ‘sed’ program on its own. Order matters!

or using script file

sedScript.sed:
s/good/bad 
s/sentence/paragraph


sed -f sedScript.sed  sedFile.txt

Command Syntax

/abc/p      # Print all with "abc"
/abc/!p     # Print all without "abc"
/abc/d      # Delete all with "abc"
/abc/!d     # Delete all without "abc"

/start/,/end/!d     # Select a block

s/abc/def/
s/abc\(...\)ghi/\1/ # Back references
            # (see below for correct Shell quoting)

/abc/{s/def/ghi)}   # Conditional replace

Appending lines

aHallo                   # Append 'Hallo' after each line
5 aHallo                 # Append 'Hallo' after line #5
$ aHallo                 # Append 'Hallo' to end of file

Prepending lines

sed -i '1s;^;new line 1\nanother new line 2\n;' <file>

Remove Double Quote from a String

Remove quotes from start and end of the string

sed -e 's/^"//' -e 's/"$//' <<<"$var1"

Remove Double Quote and Store Output in variable

var2=`sed -e 's/^"//' -e 's/"$//' <<<"$var1"`    #Save in another variable
var1=`sed -e 's/^"//' -e 's/"$//' <<<"$var1"`    #Save in same variable

Remove Double Quote and Store Output in file

sed -e 's/^"//' -e 's/"$//' <<<"$var1" > out_var.txt

Advanced use of sed

In-place Editing

To edit file use the -i option this safely changes the file contents without any output redirection needed.

sed -i 's/abc/ABC/' myfile.txt
sed -i '/deleteme/d' *

Drop grep

Often grep and sed are used together. In all those cases grep can be dropped. For example

grep "pattern" file | sed "s/abc/def/"

can be written as

sed -n "/pattern/p; s/abc/def/"

Grouping with sed

Always use single quotes!

sed 's/^.*\(pattern\).*/\1/'

Single Quoting Single Quotes

If you want to do extraction and need a pattern based on single quotes use \x27 instead of trying to insert a single quote. For example:

sed 's/.*var=\x27\([^\x27]*\)\x27.*/\1/'

to extract “some string” from “var=‘some string’”. Or if you don’t know about the quoting, but know there are quotes

sed 's/.*var=.\([^"\x27]*\)..*/\1/'

Conditional Replace with sed

sed '/conditional pattern/{s/pattern/replacement/g}'

Prefix files with a boilerplate using sed

sed -i '1s/^/# DO NOT TOUCH THIS FILE!\n\n/' *

Removing Newlines with sed

The only way to remove new line is this:

sed ':a;N;$!ba;s/\n//g' file

Check out this explanation if you want to know why.

Selecting Blocks

sed '/first line/,/last line/!d' file

Print a specific line from a file

sed -n 10p /path/to/file

Remove a specific line from a file

sed -i 10d /path/to/file
# alternative (BSD): sed -i'' 10d /path/to/file

Remove a range of lines from a file

sed -i <file> -re '<start>,<end>d'

Replace newline(s) with a space

sed ':a;N;$!ba;s/\n/ /g' /path/to/file

# cross-platform compatible syntax:
sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g' /path/to/file

:a create a label a
N append the next line to the pattern space
$! if not the last line, ba branch (go to) label a
s substitute, /\n/ regex for new line, / / by a space, /g global match (as many times as it can)

Alternatives:

# perl version (sed-like speed):
perl -p -e 's/\n/ /' /path/to/file

# bash version (slow):
while read line ; do printf "%s" "$line " ; done < file

Delete string +N next lines

sed '/start/,+4d' /path/to/file

Multiline command

Multiple lines can be processed as one buffer using the D,G,H,N,P. They are similar to their lowercase counterparts (d,g, h,n,p), except that these commands append or subtract data while respecting embedded newlines - allowing adding and removing lines from the pattern and hold spaces

By default, when sed reads a line in the pattern space, it discards the terminating newline (\n) character. obraz

If our record is spread on 3 lines we need resize the pattern space to have whole record to match

N;
N;

obraz Then we can search and replace using the substitution (s) command: With new lines available in the pattern space, we can use the \n character as part of the regex.

s/(.*\n)(.*: ).*(\n.*: 500)/\1\2Finance\3/;
p;

obraz

Finally, we can slide the pattern space window by using the d command at the end of the current read-cycle, followed by two invocations of the N command in the next read-cycle:

d;