Linux Bash Scripting Tutorial


SED-Stream Editor

A Stream Editor that takes its commands from a file
Sed is noninteractive stream editor. normally editing files with vi, requires files should be opened, sed is nondescriptive editor. sed makes changes to the file and displays the content on the screen. if u want to save the changed file, then redirect sed output to file. sed often used to find-and-replace actions on lines containg patterns.

sed options:
  • -n --silent --quite
  • -e script add the script to the commands to be executed
  • -f scriptfile add the content of the script file to the commands to be executed
  • -E -r use extended regular expressions in the script
  • -z --null-data seperate lines by null character
  • --version displays sed version information
sed command works on 2 modes
  • Address
  • Regular Expressions

In this tutorial Following table used as Input to SED command

Rank 	Gold	Silver	Bronze	Total  	NOC
1 	39	41	33	113     United States (USA)	
2 	38	32	18	88      China (CHN)		
3 	27	14	17	58      Japan (JPN)*		
4 	22	20	22	64      Great Britain (GBR)	
5 	20	28	23	71      ROC			
6 	17	7	22	46      Australia (AUS)		
7 	10	12	14	36      Netherlands (NED)	
8 	10	12	11	33      France (FRA)		
9 	10	11	16	37      Germany (GER)		
10 	10	10	20	40      Italy (ITA)		
11-93 	137	150	206	493     Remaining Countries
Totals  340	337	402	1079    (93 NOCs)
SED SYNTAXExplanationExample
sed -n /Regular_Expression/p' filenameThis will print matching lines
sed -n s/Regular_Expression/replacement_text' filenameThis will replace the pattern matched string with repleacement text
sed 'n,md' filenameDeletes line from n to m
sed '/Regular Expression/dThis will delete all the lines matching Regular Expression
sed '/Regular Expression/!dThis will deletall all lines except the line(s) contaning Regular Expression

Sed printing with p command

sed '/GBR/p' olympics.txt
all the lines containing GBR will be printed

Note: GBR is Regular Expression Pattern , 'p' is a command for printing.In this example, telling sed command look for word 'GBR'(capitals), if found display matching lines.

In above example, GBR is a single word, below example searches for multi word, 'Great Britain'
$sed -n '/Great Britain/p' olympics.txt

Above 2 examples are case sensitive,below one is case-insensitive. i.e G or B can be small case letters

$sed -n '/[gG]reat [bB]ritain/p' olympics.txt

above 3 conditions displays following output.
4	 Great Britain (GBR)	22	20	22	64
	

display countries based on Medals

$sed -n '/[0-9][0-9]/p' olympics.txt
prints all lines except header.because every line has 2-digit numbers.

$sed -n '/[0-9][0-9][0-9]/p' olympics.txt
print 3 digit medal countries

First line has 113 3-digit number
second and third lines also has 3-digit numbers

1	 United States (USA)	39	41	33	113
11–93	 Remaining	137	150	206	493
Totals   (93 NOCs)	340	337	402	1079
	

sed -n '/[0-9]\{4\}/p ' olympics.txt

Display 4-digit medal count

Totals   (93 NOCs)	340	337	402	1079

above example range with Quantifier used. i.e [0-9]{4} here [0-9] range of numbers followed by quantifier surrounded by curly braces. {4},exactly 4-digits, In bash regular expressions, Curly braces should be escaped with forward slash(\)

Last line has 1079, these are the total medals in olympics.

Display Multiple Countries/Multiple Matches using -e expression

   Multiple Options can be specified using -e expr option

$sed -n -e '/China/p' -e '/GBR/p' olympics.txt
  Displays Lines containing 'China' or 'GBR'

	
2	 China (CHN)	38	32	18	88
4	 Great Britain (GBR)	22	20	22	64
	
above sed command can be rewritten using single Regular Expression

$sed -n '/\(Japan\|ROC\|Germany\)/p' olympics.txt

3	 Japan(JPN)*		27	14	17	58
5	 ROC			20	28	23	71
9	 Germany(GER)		10	11	16	37
	

Above sed command can be rewritten with Extented Regular Expression i.e -E option.

$sed -n -E '/(Japan|ROC|Germany)/p' olympics.txt

Gives same results.

Combining 'awk' and 'sed' command together

    use 'awk' command to Filter columns based on some condition

display country details which got less than 10 Silver Medals.

$awk '$3<10 {print $0}' olympics.txt | sed -n 'p'

	6 	17	7	22	46      Australia (AUS)
	

display country details which got gold medals between 20 and 30

$awk '$2>=20 && $2 <=30 {print $0}' olympics.txt | sed -n 'p'

3 	27	14	17	58      Japan (JPN)*		
4 	22	20	22	64      Great Britain (GBR)	
5 	20	28	23	71      ROC	

display country details which got more gold medals Bronze Medals

$awk '$2>=$4 {print $0}' olympics.txt | sed -n 'p'

Rank 	Gold	Silver	Bronze	Total  	NOC
1 	39	41	33	113     United States (USA)	
2 	38	32	18	88      China (CHN)		
3 	27	14	17	58      Japan (JPN)*

How to modify a file using sed

sed '1,3d' filename > newfile
mv newfile filename

Appending – the a command | Inserting – the i command | Changing – the c command

    a command appends after the address or pattern match

    i command Inserts before the address or pattern match

    c command changes based on Address or Pattern Match

Create a file called tennis.txt, add grand slam details to this file.
 $(touch tennis.txt;echo " ">tennis.txt; sed -i '1i Australian Open' tennis.txt)
 
creates a file called tennis.txt using touch command. File has zero length, 
Inorder to add some text, add empty space to tennis.txt using echo " "> tennis.txt, 

 -i  option  updates the file in-place. i.e command output will not be redirected to Standard Output device, 
 In other words command output will be directly written into file itself.

	$ sed -i '1i Australia Open' tennis.txt
	$ sed -i '1a, French Open' tennis.txt
	$ sed -i '/French*/a Wimbledon' tennis.txt 
	$ sed -i '/Wim*/a US Open' tennis.txt
Print tennis.txt file content using 'cat' command
  $cat  tennis.txt
Australia Open
French Open
Wimbledon
US Open
Now we will see how to change file content using c command. All these events are held different dates in a Calendar Year. In this we will update/change the file content using c command. While using c command, '/patten/c newText' means it looks for pattern in the file, if it finds , replaces entire line with newText.This is the drawback with c command.

Below example, pattern is '/Austra*/' it looks for pattern 'Austra', so we have Australian Open ,it finds a match and replaces entire line with 'Australia Open in January', Note: don't put 'in January' only, if so, it replaces entire line with 'in January' . Inorder to get complete result, newText should be 'Australia Open in January', similarly for all other cases. In brief: c command replaces matched lines with newText
 $ sed -i '/Austra*/c Australia Open in January'  tennis.txt
 $ sed -i '/French*/c French Open in May'  tennis.txt
 $ sed -i '/US*/c US Open in August'  tennis.txt
 $ sed -i '/Wim*/c Wimbledon in June'  tennis.txt
 $ cat tennis.txt 
Australia Open in January
French Open in May
Wimbledon in June
US Open in August
Now replace Wimbledon in June , country UK
$ sed '/Wim*/c country UK'  tennis.txt
Australia Open in January
French Open in May
country UK
US Open in August
above example, it replaced Wimbledon in June with 'Country UK', to avoid this, user should provide full text i.e 'Wimbledon in June, Country UK'
$ sed '/Wim*/c Wimbledon in June, Country UK'  tennis.txt
Australia Open in January
French Open in May
Wimbledon in June, Country UK
US Open in August
Note: Use s command for word(s) replacement/substitution

Sed deleting with d command

sed delete SyntaxExplanationExample
sed '3d' filenamedeletes third line
sed '3,$d' filenamedeletes from third line to last line
$ in the address indicates last line
, is called range operator
sed '$d' filenamedeletes the last line
sed '/RE/d' filenameThe line(s) containing RE pattern deleted
sed '/RE/!d' filenameThe line(s) not conating RE patterns deleted
sed '4,+5d' filenameThis will delete from line 4 to next 5 lines
sed '1,5!d' filenameThis will keep lines from 1 to 5,deletes all other lines
sed '1~3d' filenameThis will delete lines 1,4,7,10 and so on... ~ is step increment
sed '2~2d'This will delete every other line starting with line 2 to be deleted.

Sed Substitution with s command

sed 's/Cashew/Almonds/g' shopping.txt

sed 's/[0–9][0–9]$/&.5/' shopping.txt
this will replace 2 digit numbers at the end of the line .5 appended to them

Sed Range of selected lines: the comma

Multiple edits – the e command

Multiple edits – the e command

Reading from files – the r command

Writing to files – the w command

Transform – the y command

Quit – the q command

    Quit q command is used for Quitting the sed processing without proceeding to the next line.

TopCities.txt
Tokyo,Japan
New York, USA
London, UK
Paris, France
Istanbul,Turkey
Beijing, China
Los Angeleses, USA
Moscow, Russia
$ sed '1q'  citities.txt   # prints  first city
$ sed '3q'  citities.txt   #prints first 3 cities
$ sed '$q'  cities.txt     #prints all lines from cities file.

Holding and getting – the h and g commands

Holding and exchanging – the h and x commands

SED examples

Remove Empty Lines from a Text File(s)

Following command looks for line containing any alphanumeric characters, if it finds, takes those lines into consideration,i.e ignores empty lines.finally updates the file
$ sed -i -n '/^[0-9a-zA-Z]/p' file.txt

Remove Blank Lines from all files

same as above, but sed command takes 2 input files file1.txt file2.txt, removes/filters blank lines.
$ sed -i -n '/[0-9a-zA-Z]/p' file1.txt file2.txt

find e-mail address using sed command

$ sed -n -E '/[a-zA-Z0-9+_.-]+@[a-zA-Z0-9.-]+.[a-z]{2,3}/p' url_email.txt

ADS