Shell腳本中的正則表達(dá)式

博文目錄
一、正則表達(dá)式的定義
二、擴(kuò)展正則表達(dá)式元字符
三、文本處理器

一、正則表達(dá)式的定義

正則表達(dá)式又稱正規(guī)表達(dá)式、常規(guī)表達(dá)式。在代碼中常簡寫為regex、regexp或RE。正則表達(dá)式是使用單個(gè)字符串來描述，匹配一系列符合某個(gè)句法規(guī)則的字符串，簡單來說，是一種匹配字符串的方法，通過一些特殊符號(hào)，實(shí)現(xiàn)快速查找、刪除、替換某個(gè)特定字符串。
正則表達(dá)式是由普通字符與元字符組成的文字模式。模式用于描述在搜索文本時(shí)要匹配的一個(gè)或多個(gè)字符串。正則表達(dá)式作為一個(gè)模板，將某個(gè)字符模式與所搜索的字符串進(jìn)行匹配。其中普通字符包括大小寫字母、數(shù)字、標(biāo)點(diǎn)符號(hào)及一些其他符號(hào)，元字符則是指那些在正則表達(dá)式中具有特殊意義的專用字符，可以用來規(guī)定其前導(dǎo)字符（即位于元字符前面的字符）在目標(biāo)對象中的出現(xiàn)模式。

1、基礎(chǔ)正則表達(dá)式

正則表達(dá)式的字符串表達(dá)方法根據(jù)不同的嚴(yán)謹(jǐn)程度與功能分為基本正則表達(dá)式與擴(kuò)展正則表達(dá)式?；A(chǔ)正則表達(dá)式是常用的正則表達(dá)式的最基礎(chǔ)的部分。在Linux系統(tǒng)中常見的文件處理工具中g(shù)rep與sed支持基礎(chǔ)正則表達(dá)式，而egrep與awk支持?jǐn)U展正則表達(dá)式。

提前準(zhǔn)備一個(gè)名為test.txt的測試文件，文件具體內(nèi)容如下：

[root@centos01 ~]# vim test.txt
he was short and fat.
He was wearing a blue polo shirt with black pants.
The home of Football on BBC Sport online.
the tongue is boneless but it breaks bones.12!
google is the best tools for search keyword.
The year ahead will test our political establishment to the limit.
PI=3.14148223023840-2382924893980--2383892948
a wood cross!
Actions speak louder than words

#wooood #
#woooood #
AxyzxyzxyzxyzxyzC
I bet this place is really spooky late at night!
Misfortunes never come alone/single.
I shouldn't have lett so tast.

1）基礎(chǔ)正則表達(dá)式示例：

[root@centos01 ~]# grep -n 'the' test.txt         <!--查找特定字符，-n顯示行號(hào)-->
4:the tongue is boneless but it breaks bones.12!
5:google is the best tools for search keyword.
6:The year ahead will test our political establishment to the limit.
[root@centos01 ~]# grep -in 'the' test.txt    <!--查找特定字符，-in顯示行號(hào)不區(qū)分大小寫-->
3:The home of Football on BBC Sport online.
4:the tongue is boneless but it breaks bones.12!
5:google is the best tools for search keyword.
6:The year ahead will test our political establishment to the limit.
[root@centos01 ~]# grep -vn 'the' test.txt    <!--查找不包括特定字符的行，-vn選項(xiàng)實(shí)現(xiàn)-->
1:he was short and fat.
2:He was wearing a blue polo shirt with black pants.
3:The home of Football on BBC Sport online.
7:PI=3.14148223023840-2382924893980--2383892948
8:a wood cross!
9:Actions speak louder than words
10:
11:
12:#wooood #
13:#woooood #
14:AxyzxyzxyzxyzxyzC
15:I bet this place is really spooky late at night!
16:Misfortunes never come alone/single.
17:I shouldn't have lett so tast.

2）grep利用中括號(hào)“[]”來查找集合字符

[root@centos01 ~]# grep -n 'sh[io]rt' test.txt      <!--中括號(hào)來查找集合字符，
“[]”中無論有幾個(gè)字符，都僅代表一個(gè)字符，
也就是說“[io]”表示匹配“i”或者“o”-->
1:he was short and fat.
2:He was wearing a blue polo shirt with black pants.
[root@centos01 ~]# grep -n 'oo' test.txt     <!--查找重復(fù)單個(gè)字符-->
3:The home of Football on BBC Sport online.
5:google is the best tools for search keyword.
8:a wood cross!
12:#wooood #
13:#woooood #
15:I bet this place is really spooky late at night!
[root@centos01 ~]# grep -n '[^w]oo' test.txt   <!--查找“oo”前面不是“w”的字符串，
使用“[^]”選項(xiàng)實(shí)現(xiàn)-->
3:The home of Football on BBC Sport online.
5:google is the best tools for search keyword.
12:#wooood #
13:#woooood #
15:I bet this place is really spooky late at night!
[root@centos01 ~]# grep -n '[^a-z]oo' test.txt        <!--查找“oo”前面不存在小寫字母-->
3:The home of Football on BBC Sport online.
[root@centos01 ~]# grep -n '[0-9]' test.txt        <!--查找包含數(shù)字的行-->
4:the tongue is boneless but it breaks bones.12!
7:PI=3.14148223023840-2382924893980--2383892948

3)grep查找行首“^”與行尾字符“$”

[root@centos01 ~]# grep -n '^the' test.txt      <!--查找以“the”字符串為行首的行-->
4:the tongue is boneless but it breaks bones.12!
[root@centos01 ~]# grep -n '^[a-z]' test.txt      <!--查找以小寫字母為行首的行 -->
1:he was short and fat.
4:the tongue is boneless but it breaks bones.12!
5:google is the best tools for search keyword.
8:a wood cross!
[root@centos01 ~]# grep -n '^[A-Z]' test.txt       <!--查找以大寫字母為行首的行-->
2:He was wearing a blue polo shirt with black pants.
3:The home of Football on BBC Sport online.
6:The year ahead will test our political establishment to the limit.
7:PI=3.14148223023840-2382924893980--2383892948
9:Actions speak louder than words
14:AxyzxyzxyzxyzxyzC
15:I bet this place is really spooky late at night!
16:Misfortunes never come alone/single.
17:I shouldn't have lett so tast.
[root@centos01 ~]# grep -n '^[^a-zA-Z]' test.txt    <!--查找不以字母開頭的行-->
12:#wooood #
13:#woooood #
[root@centos01 ~]# grep -n 'w..d' test.txt      <!--查找任意一個(gè)字符“.”與重復(fù)字符“*”-->
5:google is the best tools for search keyword.
8:a wood cross!
9:Actions speak louder than words
[root@centos01 ~]# grep -n 'ooo*' test.txt     <!--查看包含至少兩個(gè)o以上的字符串-->
3:The home of Football on BBC Sport online.
5:google is the best tools for search keyword.
8:a wood cross!
11:#woood #
13:#woooooood #
19:I bet this place is really spooky late at night!
[root@centos01 ~]# grep -n 'woo*d' test.txt      <!--查詢w開頭d結(jié)尾，中間至少包含一個(gè)o的字符串-->
8:a wood cross!
11:#woood #
13:#woooooood #
[root@centos01 ~]# grep -n '[0-9][0-9]*' test.txt   <!--查詢?nèi)我鈹?shù)字所在行-->
4:the tongue is boneless but it breaks bones.12!
7:PI=3.141592653589793238462643383249901429
[root@centos01 ~]# grep -n 'o\{2\}' test.txt       <!--查找連續(xù)兩個(gè)o的字符“{}”-->
3:The home of Football on BBC Sport online.
5:google is the best tools for search keyword.
8:a wood cross!
11:#woood #
13:#woooooood #
19:I bet this place is really spooky late at night!

2、元字符總結(jié)

二、擴(kuò)展正則表達(dá)式元字符

三、文本處理器

在Linux/UNIX系統(tǒng)中包含很多種文本處理器或文本編輯器，其中包括VIM編輯器與grep等。而grep，sed，awk更是shell編程中經(jīng)常用到的文本處理工具，被稱為shell編程三劍客。

1、sed工具

sed（Stream EDitor）是一個(gè)強(qiáng)大而簡單的文本解析轉(zhuǎn)換工具，可以讀取文本，并根據(jù)指定的條件對文本內(nèi)容進(jìn)行編輯（刪除、
替換、添加、移動(dòng)等），最后輸出所有行或者僅輸出處理的某些行。sed也可以在無交互的情況下實(shí)現(xiàn)相當(dāng)復(fù)雜的文本處理操作，被廣泛應(yīng)用于shell腳本中，用以完成各種自動(dòng)化處理任務(wù)。

sed的工作流程主要包括讀取、執(zhí)行和顯示三個(gè)過程：

• 讀取：sed從輸入流（文件、管道、標(biāo)準(zhǔn)輸入）中讀取一行內(nèi)容并存儲(chǔ)到臨時(shí)的緩沖區(qū)中（又稱模式空間，patterm space）。
• 執(zhí)行：默認(rèn)情況下，所有的sed命令都在模式空間中順序地執(zhí)行，除非指定了行的地址，否則sed命令將會(huì)在所有的行上依次執(zhí)行。
• 顯示：發(fā)送修改后的內(nèi)容到輸出流。再發(fā)送數(shù)據(jù)后，模式空間將會(huì)被清空。在所有的文件內(nèi)容都被處理完成之前，上述過程將重復(fù)執(zhí)行，直到所有內(nèi)容被處理完。

2、sed命令常見的用法

sed[選項(xiàng)] '操作'  參數(shù)
sed [選項(xiàng)] -f scriptfile 參數(shù)

常見的sed命令選項(xiàng)主要包含以下幾種：

• -e或--expression=：表示用指定命令或者腳本來處理輸入的文本文件。
• -f或--file=：表示用指定的腳本文件來處理輸入的文本文件。
• -h或--help：顯示幫助。
• -n、--quiet或silent：表示僅顯示處理后的結(jié)果。
• -i：直接編輯文本文件。
  “操作”用于指定對文件操作的動(dòng)作行為，也就是sed的命令。通常情況下是采用的“[n1[,n2]]”操作參數(shù)的格式。n1、n2是可選的，不一定會(huì)存在，代表選擇進(jìn)行操作的行數(shù)，如操作需要在5~20行之間進(jìn)行，則表示為“5,20動(dòng)作行為”。常見的操作包括以下幾種：
• a：增加，在當(dāng)前行下面增加一行指定內(nèi)容。
• c：替換，將選定行替換為指定內(nèi)容。
• d：刪除，刪除選定的行。
• i：插入，在選定行上面插入一行指定內(nèi)容。
• p：打印，如果同時(shí)指定行，表示打印指定行；如果不指定行，則表示打印所有內(nèi)容；如果有非打印字符，則以ASCII碼輸出。其通常與“-n”選項(xiàng)一起使用。
• s：替換，替換指定字符。
• y：字符轉(zhuǎn)換。

3、用法示例

1）輸出符號(hào)條件的文本（p表示正常輸出）

[root@centos01 ~]# sed -n '3p' test.txt        <!--輸出第三行-->
The home of Football on BBC Sport online.
[root@centos01 ~]# sed -n '3,5p' test.txt  <!--輸出第三行到第五行-->
The home of Football on BBC Sport online.
the tongue is boneless but it breaks bones.12!
google is the best tools for search keyword.
[root@centos01 ~]# sed -n 'p;n' test.txt       <!--輸出所有奇數(shù)行-->
he was short and fat.
The home of Football on BBC Sport online.
google is the best tools for search keyword.
PI=3.141592653589793238462643383249901429
Actions speak louder than words
#woood #
#woooooood #

I bet this place is really spooky late at night!
I shouldn't have lett so tast.
[root@centos01 ~]# sed -n 'p;n' test.txt     <!--輸出所有偶數(shù)行-->
he was short and fat.
The home of Football on BBC Sport online.
google is the best tools for search keyword.
PI=3.141592653589793238462643383249901429
Actions speak louder than words
#woood #
#woooooood #

I bet this place is really spooky late at night!
I shouldn't have lett so tast.
[root@centos01 ~]# sed -n '1,5{p;n}' test.txt <!--輸出第一行到第五行之間的奇數(shù)行 -->
he was short and fat.
The home of Football on BBC Sport online.
google is the best tools for search keyword.

[root@centos01 ~]# sed -n '10,${n;p}' test.txt       <!--輸出第10行至文件尾之間的偶數(shù)行-->
#woood #
#woooooood #

I bet this place is really spooky late at night!
I shouldn't have lett so tast.

2）Sed命令結(jié)合正則表達(dá)式

[root@centos01 ~]# sed -n '/the/p' test.txt <!--輸出包含the的行-->
the tongue is boneless but it breaks bones.12!
google is the best tools for search keyword.
The year ahead will test our political establishment to the limit.
[root@centos01 ~]# sed -n '4,/the/p' test.txt<!--輸出從第4行至第一個(gè)包含the的行-->
the tongue is boneless but it breaks bones.12!
google is the best tools for search keyword.
[root@centos01 ~]# sed -n '/the/=' test.txt       <!--輸出包含the的行所在的行號(hào)，
等號(hào)（=）用來輸出行號(hào)-->
4
5
6
[root@centos01 ~]# sed -n '/^PI/p' test.txt <!--輸出以PI開頭的行-->
PI=3.141592653589793238462643383249901429
[root@centos01 ~]# sed -n '/\<wood\>/p' test.txt  <!--輸出包含單詞wood的行，
\<、\>代表單詞邊界-->
a wood cross!

3）刪除符合條件的文件（d）

[root@centos01 ~]# nl test.txt | sed '3d'    <!--刪除第3行-->
     1  he was short and fat.
     2  He was wearing a blue polo shirt with black pants.
     4  the tongue is boneless but it breaks bones.12!
     5  google is the best tools for search keyword.
     6  The year ahead will test our political establishment to the limit.
     7  PI=3.141592653589793238462643383249901429
     8  a wood cross!
     9  Actions speak louder than words
    10  
    11  #woood #
    12  
    13  #woooooood #
    14  
    15  
    16  AxyzxyzxyzxyzC
    17  
    18  
    19  I bet this place is really spooky late at night!
    20  Misfortunes never come alone/single.
    21  I shouldn't have lett so tast.
[root@centos01 ~]# nl test.txt | sed '3,5d'   <!--刪除第3~5行-->
     1  he was short and fat.
     2  He was wearing a blue polo shirt with black pants.
     6  The year ahead will test our political establishment to the limit.
     7  PI=3.141592653589793238462643383249901429
     8  a wood cross!
     9  Actions speak louder than words
    10  
    11  #woood #
    12  
    13  #woooooood #
    14  
    15  
    16  AxyzxyzxyzxyzC
    17  
    18  
    19  I bet this place is really spooky late at night!
    20  Misfortunes never come alone/single.
    21  I shouldn't have lett so tast.
[root@centos01 ~]# sed '/^[a-z]/d' test.txt <!--刪除以小寫字母開頭的行-->
He was wearing a blue polo shirt with black pants.
The home of Football on BBC Sport online.
The year ahead will test our political establishment to the limit.
PI=3.141592653589793238462643383249901429
Actions speak louder than words

#woood #

#woooooood #

AxyzxyzxyzxyzC

I bet this place is really spooky late at night!
Misfortunes never come alone/single.
I shouldn't have lett so tast.

4）替換符合條件的文本

[root@centos01 ~]# sed 's/the/THE/' test.txt <!--將每行中的第一個(gè)the替換為THE-->
he was short and fat.
He was wearing a blue polo shirt with black pants.
The home of Football on BBC Sport online.
THE tongue is boneless but it breaks bones.12!
google is THE best tools for search keyword.
The year ahead will test our political establishment to THE limit.
PI=3.141592653589793238462643383249901429
a wood cross!
Actions speak louder than words

#woood #

#woooooood #

AxyzxyzxyzxyzC

I bet this place is really spooky late at night!
Misfortunes never come alone/single.
I shouldn't have lett so tast.
[root@centos01 ~]# sed 's/l/L/2' test.txt <!--將每行中的第三個(gè)l替換為L-->
he was short and fat.
He was wearing a blue poLo shirt with black pants.
The home of FootbalL on BBC Sport online.
the tongue is boneless but it breaks bones.12!
google is the best tooLs for search keyword.
The year ahead wilL test our political establishment to the limit.
PI=3.141592653589793238462643383249901429
a wood cross!
Actions speak louder than words

#woood #

#woooooood #

AxyzxyzxyzxyzC

I bet this place is reaLly spooky late at night!
Misfortunes never come alone/singLe.
I shouldn't have Lett so tast.
[root@centos01 ~]# sed 's/^/#/' test.txt  <!--在每行行首插入#號(hào)-->
#he was short and fat.
#He was wearing a blue polo shirt with black pants.
#The home of Football on BBC Sport online.
#the tongue is boneless but it breaks bones.12!
#google is the best tools for search keyword.
#The year ahead will test our political establishment to the limit.
#PI=3.141592653589793238462643383249901429
#a wood cross!
#Actions speak louder than words
#
##woood #
#
##woooooood #
#
#
#AxyzxyzxyzxyzC
#
#
#I bet this place is really spooky late at night!
#Misfortunes never come alone/single.
#I shouldn't have lett so tast.
[root@centos01 ~]# sed '/the/s/o/0/g' test.txt  <!--將包含the的所有行中的o都替換為0-->
he was short and fat.
He was wearing a blue polo shirt with black pants.
The home of Football on BBC Sport online.
the t0ngue is b0neless but it breaks b0nes.12!
g00gle is the best t00ls f0r search keyw0rd.
The year ahead will test 0ur p0litical establishment t0 the limit.
PI=3.141592653589793238462643383249901429
a wood cross!
Actions speak louder than words

#woood #

#woooooood #

AxyzxyzxyzxyzC

I bet this place is really spooky late at night!
Misfortunes never come alone/single.
I shouldn't have lett so tast.

3、awk工具

在Linux/UNIX系統(tǒng)中，awk是一個(gè)功能強(qiáng)大的編輯工具，逐行讀取輸入文本，并根據(jù)指定的匹配模式進(jìn)行查找，對符合條件的內(nèi)容進(jìn)行格式化輸出或者過濾處理，可以在無交互的情況下實(shí)現(xiàn)相當(dāng)復(fù)雜的文本操作，被廣泛應(yīng)用于Shell腳本，完成各種自動(dòng)化配置任務(wù)。

1）awk常見用法

通常情況下awk所使用的命令格式如下所示，其中，單引號(hào)加上大括號(hào)“{}”用于設(shè)置對數(shù)據(jù)進(jìn)行的處理動(dòng)作。awk可以直接處理目標(biāo)文件也可以通過“-f”讀取腳本對目標(biāo)文件進(jìn)行處理。

awk 選項(xiàng)  '模式或條件 {編輯指令}' 文件1 文件2 ......
awk -f 腳本文件 文件1 文件2 ...

awk包含幾個(gè)特殊的內(nèi)建變量（可直接用）如下所示：

• NF：當(dāng)前處理的行的字段個(gè)數(shù)。
• FS：指定每行文本的字段分隔符，默認(rèn)為空格或制表位。
• NR：當(dāng)前處理的行的字段個(gè)數(shù)。
• $0：當(dāng)前處理的行的整行內(nèi)容。
• FILENAME：被處理的文件名。
• RS：數(shù)據(jù)記錄分隔，默認(rèn)為\n，即每行為一條記錄。

2）用法示例

[root@centos01 ~]# awk '{print}' test.txt  <!--輸出所有內(nèi)容-->
he was short and fat.
He was wearing a blue polo shirt with black pants.
The home of Football on BBC Sport online.
the tongue is boneless but it breaks bones.12!
google is the best tools for search keyword.
The year ahead will test our political establishment to the limit.
PI=3.141592653589793238462643383249901429
a wood cross!
Actions speak louder than words

#woood #

#woooooood #

AxyzxyzxyzxyzC

I bet this place is really spooky late at night!
Misfortunes never come alone/single.
I shouldn't have lett so tast.
[root@centos01 ~]# awk 'NR==1,NR==3{print}' test.txt <!--輸出1~3行內(nèi)容-->
he was short and fat.
He was wearing a blue polo shirt with black pants.
The home of Football on BBC Sport online.
[root@centos01 ~]# awk '(NR%2)==1{print}' test.txt   <!--輸出所有奇數(shù)行的內(nèi)容-->
he was short and fat.
The home of Football on BBC Sport online.
google is the best tools for search keyword.
PI=3.141592653589793238462643383249901429
Actions speak louder than words
#woood #
#woooooood #

I bet this place is really spooky late at night!
I shouldn't have lett so tast.
[root@centos01 ~]# awk '(NR%2)==0{print}' test.txt   <!--輸出所有偶數(shù)行內(nèi)容-->
He was wearing a blue polo shirt with black pants.
the tongue is boneless but it breaks bones.12!
The year ahead will test our political establishment to the limit.
a wood cross!

AxyzxyzxyzxyzC

Misfortunes never come alone/single.
[root@centos01 ~]# awk '/^root/{print}' /etc/passwd  <!--輸出以root開頭的行-->
root:x:0:0:root:/root:/bin/bash
[root@centos01 ~]# awk '{print $1 $3}' test.txt <!--輸出每行中的第1、3個(gè)字段-->
heshort
Hewearing
Theof
theis
googlethe
Theahead
PI=3.141592653589793238462643383249901429
across!
Actionslouder

#woood

#woooooood

AxyzxyzxyzxyzC

Ithis
Misfortunescome
Ihave

—————— 本文至此結(jié)束，感謝閱讀 ——————