Shell編程之正則表達式（二）

文本處理器

在 Linux/UNIX 系統(tǒng)中包含很多種文本處理器或文本編輯器,其中包括我們之前學(xué)習(xí)過的VIM 編輯器與 grep 等。而 grep、sed、awk 更是 shell 編程中經(jīng)常用到的文本處理工具，被稱之為 Shell 編程三劍客。

sed 工具

sed（Stream EDitor）是一個強大而簡單的文本解析轉(zhuǎn)換工具，可以讀取文本，并根據(jù)指定的條件對文本內(nèi)容進行編輯（刪除、替換、添加、移動等），最后輸出所有行或者僅輸出處理的某些行。sed 也可以在無交互的情況下實現(xiàn)相復(fù)雜的文本處理操作，被廣泛應(yīng)用于 Shell腳本中，用以完成各種自動化處理任務(wù)。

• sed 的工作流程主要包括讀取、執(zhí)行和顯示三個過程

  • 讀取：sed 從輸入流（文件、管道、標準輸入）中讀取一行內(nèi)容并存儲到臨時的緩 沖區(qū)中（又稱模式空間，pattern space）
  • 執(zhí)行：默認情況下，所有的 sed 命令都在模式空間中順序地執(zhí)行，除非指定了行的地址，否則 sed命令將會在所有的行上依次執(zhí)行
  • 顯示：發(fā)送修改后的內(nèi)容到輸出流。再發(fā)送數(shù)據(jù)后，模式空間將會被清空
  • 在所有的文件內(nèi)容都被處理完成之前，上述過程將重復(fù)執(zhí)行，直至所有內(nèi)容被處理完
• 默認情況下,所有的 sed 命令都是在模式空間內(nèi)執(zhí)行的，因此輸入的文件并不會發(fā)生任何變化，除非是用重定向存儲輸出

sed 命令常見用法

• sed 命令有兩種格式：

  sed [選項] '操作' 參數(shù)        //“參數(shù)”是指操作的目標文件，當(dāng)存在多個操作對象時用，文件之間用逗號“，”分隔
  或
  sed [選項] -f scriptfile 參數(shù)   // scriptfile 表示腳本文件，需要用“-f”選項指定

• 常見sed命令選項

  • -e 或--expression=：表示用指定命令或者腳本來處理輸入的文本文件
  • -f 或--file=：表示用指定的腳本文件來處理輸入的文本文件
  • -h 或--help：顯示幫助
  • -n、--quiet 或silent：表示僅顯示處理后的結(jié)果
  • -i：直接編輯文本文件
• “操作”用于指定對文件操作的動作行為，也就是sed 的命令。通常情況下是采用的“[n1[,n2]]”操作參數(shù)的格式。n1、n2 是可選的，不一定會存在，代表選擇進行操作的行數(shù)，如操作需要在 5～20 行之間進行，則表示為“5，20 動作行為”。常見的操作包括以下幾種
  • a：增加，在當(dāng)前行下面增加一行指定內(nèi)容
  • c：替換，將選定行替換為指定內(nèi)容
  • d：刪除，刪除選定的行
  • i：插入，在選定行上面插入一行指定內(nèi)容
  • p：打印，如果同時指定行，表示打印指定行；如果不指定行，則表示打印所有內(nèi)容；如果有非打印字符，則以 ASCII 碼輸出。其通常與“-n”選項一起使用
  • s：替換，替換指定字符
  • y：字符轉(zhuǎn)換

示例

1）輸出符合條件的文本（p 表示正常輸出）

[root@localhost opt]# sed -n 'p' httpd.txt     //輸出文件所有內(nèi)容，等同 cat httpd.txt
#
# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# In particular, see 
# <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so 'log/access_log'
# with ServerRoot set to '/www' will be interpreted by the
# server as '/www/log/access_log', where as '/log/access_log' will be
...//省略部分內(nèi)容....

[root@localhost opt]# sed -n '3p' httpd.txt     //輸出第3行內(nèi)容
# configuration directives that give the server its instructions.

[root@localhost opt]# sed -n '3,5p' httpd.txt    //輸出3~5行內(nèi)容
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# In particular, see 

[root@localhost opt]# sed -n 'p;n' httpd.txt      //輸出所有奇數(shù)行內(nèi)容，n表示讀入下一行資料
#
# configuration directives that give the server its instructions.
# In particular, see 
# for a discussion of each configuration directive.
# Do NOT simply read the instructions in here without understanding
# consult the online docs. You have been warned.  
# Configuration and logfile names: If the filenames you specify for many
# server will use that explicit path.  If the filenames do *not* begin
# with ServerRoot set to '/www' will be interpreted by the
# interpreted as '/log/access_log'.
...//省略部分內(nèi)容....

[root@localhost opt]# sed -n 'n;p' httpd.txt       //輸出所有偶數(shù)行
# This is the main Apache HTTP server configuration file.  It contains the
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
#
# what they do.  They're here only as hints or reminders.  If you are unsure
#
# of the server's control files begin with "/" (or "drive:/" for Win32), the
...//省略部分內(nèi)容....

[root@localhost opt]# sed -n '1,5{p;n}' httpd.txt    //輸出1~5行之間的奇數(shù)行（1、3、5行） 
#
# configuration directives that give the server its instructions.
# In particular, see 

[root@localhost opt]# sed -n '350,${n;p}' httpd.txt   //輸出第350行至文件尾之間的偶數(shù)行
#
IncludeOptional conf.d/*.conf
short
wod
woooood
AxyzxyzC
//在執(zhí)行此命令時，讀取的第1行時第350行，讀取的第二行是第351行，依次類推，所以輸出的偶數(shù)行是文件的第351行、第353行直至文件結(jié)尾，其中包括空行

• sed 命令結(jié)合正則表達式時，格式略有不同，正則表達式以“/”包圍

[root@localhost opt]# sed -n '/the/p' httpd.txt        //輸出包含the的行
# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so 'log/access_log'
# with ServerRoot set to '/www' will be interpreted by the

[root@localhost opt]# sed -n '4,/the/p' httpd.txt     //輸出從第4行至第一個包含the的行
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# In particular, see 
# <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding

[root@localhost opt]# sed -n '/the/=' httpd.txt     //輸出包含the的所在行的行號，等號（=）用來輸出行號
2
3
9
10
11
13
14
...//省略部分內(nèi)容...

[root@localhost opt]# sed -n '/^sh/p' httpd.txt    //輸出以sh開頭的行
shirt
short

[root@localhost opt]# sed -n '/[0-9]$/p' httpd.txt     //輸出以數(shù)字結(jié)尾的行
#Listen 12.34.56.78:80
Listen 80
#ServerName www.example.com:80
AddDefaultCharset UTF-8

[root@localhost opt]# sed -n '/\<wood\>/p' httpd.txt      //輸出包含的單詞wood的行，\<\>代表單詞邊界
wood

2）刪除符合條件的文本（d）

• 下面命令中 nl 命令用于計算文件的行數(shù)，結(jié)合該命令可以更加直觀地查看到命令執(zhí)行的結(jié)果。

[root@localhost opt]# nl httpd.txt | sed '3d'      //刪除第3行
     1  u
     2  # This is the main Apache HTTP server configuration file.  It contains the
     4  # See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
     5  # In particular, see 
     6  # <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
     7  # for a discussion of each configuration directive.
...//省略部分內(nèi)容...

[root@localhost opt]# nl httpd.txt | sed '3,5d'     //刪除3~5行
     1  u
     2  # This is the main Apache HTTP server configuration file.  It contains the
     6  # <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
     7  # for a discussion of each configuration directive.
     ...//省略部分內(nèi)容...

[root@localhost opt]# nl httpd.txt | sed '/wood/d'     //刪除包含wood的行，原第321行被刪除
     1  u                                     //刪除不包含cross 的行,用!符號表示取反操作,如'/cross/！d'
     2  # This is the main Apache HTTP server configuration file.  It contains the
     3  # configuration directives that give the server its instructions.
     ...//省略部分內(nèi)容...
   318  short
   319  wd
   320  wod
   322  woooood
   323  AxyzC
   324  AxyzxyzC 

[root@localhost opt]# sed '/^[a-z]/d' httpd.txt     //刪除以小寫字母開頭的行
# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
...//省略部分內(nèi)容...
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf
AxyzC
AxyzxyzC

[root@localhost opt]# sed '/\.$/d' httpd.txt    //刪除以“.”結(jié)尾的行
u
# This is the main Apache HTTP server configuration file.  It contains the
# In particular, see 
# <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
...//省略部分內(nèi)容...

[root@localhost opt]# sed '/^$/d' httpd.txt      //刪除所有空行
u
# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# In particular, see 
# <URL:http://httpd.apache.org/docs/2.4/mod/directives.html>
# for a discussion of each configuration directive.
//注意： 若是刪除重復(fù)的空行，即連續(xù)的空行只保留一個， 執(zhí)行“ sed –e ‘/^$/{n;/^$/d}’ httpd.txt”命令即可實現(xiàn)。其效果與“cat -s test.txt”相同，n 表示讀下一行數(shù)據(jù)。

3）替換符合條件的文本

• 在使用 sed 命令進行替換操作時需要用到 s（字符串替換）、c（整行/整塊替換）、y（字符轉(zhuǎn)換）命令選項

[root@localhost opt]# sed 's/the/THE/' httpd.txt     //將每行中第一個the替換為THE
u
# This is THE main Apache HTTP server configuration file.  It contains the
# configuration directives that give THE server its instructions.
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# In particular, see 
...//省略部分內(nèi)容...

[root@localhost opt]# vim aaa.txt       //編輯一個新文件
llllll
llllll
llllll                                  //編輯內(nèi)容
llllll
llllll
llllll
~                                                                             
~                                                                             
:wq                                     //保存退出
[root@localhost opt]# sed 's/l/L/3' aaa.txt     //將每行中第3個l替換為L
llLlll
llLlll                                           //顯示替換后內(nèi)容
llLlll
llLlll
llLlll
llLlll

[root@localhost opt]# sed 's/the/THE/g' httpd.txt    //將文件中多有的the替換為THE
u
# This is THE main Apache HTTP server configuration file.  It contains THE
# configuration directives that give THE server its instructions.
# See <URL:http://httpd.apache.org/docs/2.4/> for detailed information.
# In particular, see 
...//省略部分內(nèi)容...

[root@localhost opt]# sed 's/o//g' httpd.txt      //將文件中所有的o刪除（替換為空串）
u
# This is the main Apache HTTP server cnfiguratin file.  It cntains the
# cnfiguratin directives that give the server its instructins.
...//省略部分內(nèi)容...
shirt
shrt
wd
wd
wd
wd
AxyzC
AxyzxyzC

[root@localhost opt]# sed 's/^/#/' aaa.txt     //在每行行首插入#號
#llllll
#llllll
#llllll
#llllll
#llllll
#llllll

[root@localhost opt]# sed 's/$/eof/' aaa.txt     //在每行行尾插入字符串eof
lllllleof
lllllleof
lllllleof
lllllleof
lllllleof
lllllleof

[root@localhost opt]# vim aaa.txt        //編輯文件
llllll
llllll
llllll
llllll
llllll
llllll
aaaaaa
aaaaaa                        //添加內(nèi)容
aaaaaa
aaaaaa
aaaaaa
~                                                                             
~                                                                             
:wq                            //保存退出
[root@localhost opt]# sed '/a/s/^/#/' aaa.txt         //在包含 a 的每行行首插入#號
llllll
llllll
llllll
llllll
llllll
llllll
#aaaaaa
#aaaaaa
#aaaaaa
#aaaaaa
#aaaaaa

[root@localhost opt]# sed '3,5s/l/L/g' aaa.txt    //將第 3~5 行中的所有 l 替換為 L
llllll
llllll
LLLLLL
LLLLLL
LLLLLL
llllll
aaaaaa
aaaaaa
aaaaaa
aaaaaa
aaaaaa

[root@localhost opt]# vim bbb.txt    //編輯一個新文件
this is
the wood
wood
wod                             //編輯內(nèi)容
the wood
this is test
~                                                                             
~                                                                             
:wq                               //保存退出
[root@localhost opt]# sed '/the/s/o/O/g' bbb.txt    //將包含 the 的所有行中的 o 都替換為 O
this is
the wOOd
wood
wod
the wOOd
this is test

4） 遷移符合條件的文本

• H，復(fù)制到剪貼板；g、G，將剪貼板中的數(shù)據(jù)覆蓋/追加至指定行；w，保存為文件；r，讀取指定文件；a，追加指定內(nèi)容。

[root@localhost opt]# sed '/the/{H;d};$G' bbb.txt    //將包含the 的行遷移至文件末尾,{;}用于多個操作
this is
wood
wod
this is test

the wood
the wood

[root@localhost opt]# sed '1,3{H;d};8G' aaa.txt     //將1~3行內(nèi)容遷移到8行之后
llllll
llllll
llllll
aaaaaa
aaaaaa

llllll
llllll
llllll
aaaaaa
aaaaaa
aaaaaa

[root@localhost opt]# sed '/the/w ccc.txt' bbb.txt     //將包含the 的行另存為文件ccc.txt
this is
the wood
wood
wod
the wood
this is test
[root@localhost opt]# cat ccc.txt                  //查看保存的文件內(nèi)容
the wood
the wood

[root@localhost opt]# sed '/the/r /etc/hostname' bbb.txt   
this is                                  //將文件/etc/hostname 的內(nèi)容添加到包含the 的每行以后
the wood
localhost.localdomain
wood
wod
the wood
localhost.localdomain
this is test

[root@localhost opt]# sed '3aNEW' bbb.txt   //在第 3 行后插入一個新行,內(nèi)容為 NEW
this is
the wood
wood
NEW
wod
the wood
this is test

[root@localhost opt]# sed '/the/aNEW' bbb.txt     //在包含the 的每行后插入一個新行,內(nèi)容為 NEW
this is
the wood
NEW
wood
wod
the wood
NEW
this is test

[root@localhost opt]# sed '3aNEW\nNEW2' bbb.txt  //在第 3 行后插入多行內(nèi)容,中間的\n 表示換行
this is
the wood
wood
NEW
NEW2
wod
the wood
this is test

5） 使用腳本編輯文件

• 使用 sed 腳本，將多個編輯指令存放到文件中（每行一條編輯指令），通過“-f”選項來調(diào)用。

sed '1,3{H;d};6G' bbb.txt    //將1~3行的內(nèi)容遷移到6行之后

[root@localhost opt]# vim test
1,3H
1,3d
6G
~                                                                            
:wq 
[root@localhost opt]# sed -f test bbb.txt 
wod
the wood
this is test

this is
the wood
wood

6） sed 直接操作文件示例

• 編寫一個腳本，用來調(diào)整 vsftpd 服務(wù)配置：禁止匿名用戶，但允許本地用戶（也允許寫入）。

[root@localhost ~]# **vim local_only_ftp.sh** 
#!/bin/bash
#指定樣本文件路徑、配置文件路徑
SAMPLE="/usr/share/doc/vsftpd-3.0.2/EXAMPLE/INTERNET_SITE/vsftpd.conf " CONFIG="/etc/vsftpd/vsftpd.conf"
#備份原來的配置文件,檢測文件名為/etc/vsftpd/vsftpd.conf.bak 備份文件是否存在, 若不存在則使用 cp 命令進行文件備份
[ ! -e "$CONFIG.bak" ] && cp $CONFIG $CONFIG.bak      //基于樣本配置進行調(diào)整,覆蓋現(xiàn)有文件
sed -e '/^anonymous_enable/s/YES/NO/g' $SAMPLE > $CONFIG
sed -i -e '/^local_enable/s/NO/YES/g' -e '/^write_enable/s/NO/YES/g' $CONFIG grep "listen" $CONFIG || sed -i '$alisten=YES' $CONFIG
# 啟動vsftpd 服務(wù),并設(shè)為開機后自動運行systemctl restart vsftpd
systemctl enable vsftpd
[root@localhost ~]# **chmod +x local_only_ftp.sh