Nginx+Keepalived主備高可用怎么安裝與配置

本篇內(nèi)容介紹了“Nginx+Keepalived主備高可用怎么安裝與配置”的有關(guān)知識，在實(shí)際案例的操作過程中，不少人都會遇到這樣的困境，接下來就讓小編帶領(lǐng)大家學(xué)習(xí)一下如何處理這些情況吧！希望大家仔細(xì)閱讀，能夠?qū)W有所成！

環(huán)境說明：
操作系統(tǒng)：CentOS6.7 x86_64
Nginx版本：nginx-1.9.7
Keepalived版本：keepalived-1.2.24 
主nginx + Keepalived ：10.219.24.26
備nginx + Keepalived ：10.219.24.23
虛擬IP：10.219.24.100
后端tomcat_1：10.219.24.21:8080
后端tomcat_2：10.219.24.21:8081

架構(gòu)與原理： 前端雙 Nginx + Keepalived ，Nginx反向代理到后端tomcat集群實(shí)現(xiàn)負(fù)載均衡，Keepalived實(shí)現(xiàn)集群高可用，
主nginx故障后虛擬IP自動漂移到備nginx。

一、 Nginx + Keepalived 安裝

一、1 nginx-1.9.7 編譯安裝

下載地址: http://nginx.org/download/nginx-1.9.7.tar.gz

補(bǔ)充: 安裝 nginx之前，需要先安裝一些依賴包：gcc、pcre、zlib
a、nginx gzip模塊需要zlib庫
b、nginx rewrite模塊需要pcre庫
c、nginx ssl模塊需要openssl庫

1、安裝必要依賴包
[root@mysql03 ~]# yum install -y pcre pcre-devel

centos 6.7 配置 yum 本地源 鏈接參考： http://blog.csdn.NET/zhang123456456/article/details/56690945 
2、 Nginx安裝
[root@mysql03 ~]# ll nginx-1.9.7.tar.gz 
-rw-r--r--. 1 root root 885562 Jun 14 21:46 nginx-1.9.7.tar.gz
[root@mysql03 ~]# tar zxvf nginx-1.9.7.tar.gz
[root@mysql03 ~]# cd nginx-1.9.7
-- 配置nginx安裝選項(xiàng)
[root@mysql03 nginx-1.9.7]# ./configure --prefix=/usr/local/nginx
說明： 配置完畢后可以看到一個(gè)配置概要，概要中的5項(xiàng)必須都有了相應(yīng)的庫支持
Configuration summary
+ using system PCRE library
+ OpenSSL library is not used 
##如果想要安裝openssl模塊，安裝時(shí)需指定 ./configure --prefix=/usr/local/nginx --with-openssl=/root/openssl-1.0.2d ## 
+ md5: using system crypto library
+ sha1: using system crypto library
+ using system zlib library
-- 安裝nginx
[root@mysql03 nginx-1.9.7]# make && make install
3、 檢查安裝是否正常
[root@mysql03 nginx-1.9.7]# cd /usr/local/nginx
[root@mysql03 nginx]# ll
total 16
drwxr-xr-x. 2 root root 4096 Jun 14 22:14 conf
drwxr-xr-x. 2 root root 4096 Jun 14 22:14 html
drwxr-xr-x. 2 root root 4096 Jun 14 22:14 logs
drwxr-xr-x. 2 root root 4096 Jun 14 22:14 sbin
-- 啟動
[root@mysql03 nginx]# ./sbin/nginx #如果不能正常啟動，可能是端口占用 
[root@mysql03 nginx]# ps -ef|grep nginx
root 5212 1 0 22:17 ? 00:00:00 nginx: master process ./sbin/nginx
nobody 5213 5212 0 22:17 ? 00:00:00 nginx: worker process
root 5228 2359 0 22:20 pts/0 00:00:00 grep nginx
-- 訪問
瀏覽器輸入： http://10.219.24.26/ #ip換成自己的ip 
看到以下頁面內(nèi)容，一切正常。
Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

-- 關(guān)閉
[root@mysql03 nginx]# ./sbin/nginx -s stop
[root@mysql03 nginx]# ps -ef|grep nginx
root 5241 2359 0 22:25 pts/0 00:00:00 grep nginx

一、2 安裝 keepalived-1.2.24

一、2.1 安裝 LVS

前提：已經(jīng)提前配置好本地 Yum 源 配置過程可參考> http://blog.csdn.NET/zhang123456456/article/details/56690945
1、 檢查一下系統(tǒng)內(nèi)核
安裝之前，首先檢查一下系統(tǒng)內(nèi)核是否支持LVS的IPVS模塊；自Linux 2.6開始，系統(tǒng)內(nèi)核完全內(nèi)置了LVS的各個(gè)模塊。
[root@mysql03 ~]# modprobe -l|grep ipvs
kernel/net/netfilter/ipvs/ip_vs.ko
kernel/net/netfilter/ipvs/ip_vs_rr.ko
kernel/net/netfilter/ipvs/ip_vs_wrr.ko
kernel/net/netfilter/ipvs/ip_vs_lc.ko
....
kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko
看到如上輸出信息，則表明系統(tǒng)內(nèi)核默認(rèn)支持IPVS模塊。

2、安裝依賴包
由于是源碼包安裝，所以編譯時(shí)需要打上如下依賴包。
# yum -y install kernel-headers glibc-headers glibc-devel libgomp gcc libstdc++-devel gcc-c++

3、 yum安裝 ipvsadm-1.26-4 
[root@mysql03 ~]# yum -y install kernel-devel ipvsadm
===================================================================================
Package Arch Version Repository Size
===================================================================================
Installing:
ipvsadm x86_64 1.26-4.el6 c6-media 42 k

Transaction Summary
===================================================================================
Install 1 Package(s)

Total download size: 42 k
Installed size: 78 k
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : ipvsadm-1.26-4.el6.x86_64 1/1 
Verifying : ipvsadm-1.26-4.el6.x86_64 1/1

Installed:
ipvsadm.x86_64 0:1.26-4.el6

Complete!
4、 做個(gè) ln
[root@mysql03 ~]# ln -sv /usr/src/kernels/2.6.32-573.el6.x86_64/ /usr/src/linux <--請自行修改 kernels 版本號

5、 查看 ipvsadm 版本信息
[root@mysql03 ~]# ipvsadm -version
ipvsadm v1.26 2008/5/15 (compiled with popt and IPVS v1.2.1) #> 能看到幫助信息，則說明IPVS安裝成功。

一、2.2 安裝 keepalived-1.2.24

1、官網(wǎng)下載地址：
[root@mysql03 ~]# wget http://www.keepalived.org/software/keepalived-1.2.24.tar.gz
2、解壓安裝包：
[root@mysql03 ~]# ll keepalived-1.2.24.tar.gz 
-rw-r--r--. 1 root root 601873 Jun 16 23:49 keepalived-1.2.24.tar.gz
[root@mysql03 ~]# tar -zxvf keepalived-1.2.24.tar.gz
3、進(jìn)入安裝目錄：
[root@mysql03 ~]# cd keepalived-1.2.24

4、安裝keepalived:
[root@mysql03 keepalived-1.2.24]# ./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/kernels/2.6.32-573.el6.x86_64/

說明：--with-kernel-dri ：指定使用內(nèi)核源碼中的頭文件，即include目錄，該參數(shù)很重要，只有使用LVS時(shí)才用此參數(shù)。
主要輸出：
Keepalived configuration
------------------------
Keepalived version : 1.2.24
Compiler : gcc
Preprocessor flags : 
Compiler flags : -Wall -Wunused -Wstrict-prototypes
Linker flags : 
Extra Lib : -ldl -lssl -lcrypto 
Use IPVS Framework : Yes
IPVS use libnl : No
IPVS syncd attributes : No
IPVS 64 bit stats : No
fwmark socket support : Yes
Use VRRP Framework : Yes
Use VRRP VMAC : Yes
Use VRRP authentication : Yes
With ip rules/routes : Yes
SNMP keepalived support : No
SNMP checker support : No
SNMP RFCv2 support : No
SNMP RFCv3 support : No
SHA1 support : No
Use Debug flags : No
Stacktrace support : No
Memory alloc check : No
libnl version : None
Use IPv4 devconf : No
Use libiptc : No
Use libipset : No
Build genhash : Yes
Build documentation : No
[root@mysql03 keepalived-1.2.24]# make && make install

5、復(fù)制腳本和文件：

5.1 拷貝執(zhí)行文件
[root@mysql03 keepalived-1.2.24]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
5.2 將init.d文件拷貝到etc下,加入開機(jī)啟動項(xiàng)
[root@node6 keepalived-1.2.16]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
5.3 將keepalived文件拷貝到etc下，加入網(wǎng)卡配置
[root@node6 keepalived-1.2.16]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
5.4 創(chuàng)建keepalived文件夾
[root@node6 keepalived-1.2.16]# mkdir /etc/keepalived
5.5 將keepalived配置文件拷貝到etc下
[root@node6 keepalived-1.2.16]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
5.6 復(fù)制啟動腳本
[root@node6 keepalived-1.2.16]# cp /usr/local/keepalived/sbin/keepalived /etc/keepalived/

6、加入開機(jī)啟動項(xiàng)
[root@mysql03 keepalived-1.2.24]# chkconfig --add keepalived #添加時(shí)必須保證/etc/init.d/keepalived存在
7、 啟動
[root@mysql03 keepalived-1.2.24]# service keepalived start
Starting keepalived: [ OK ]
8、 關(guān)閉
[root@mysql03 keepalived-1.2.24]# service keepalived stop
Stopping keepalived: [ OK ] > OK，成功！

說明： 安裝完 主nginx + Keepalived 后， 同理安裝 備nginx + Keepalived 。 

二、 Nginx + Keepalived 配置

1. 主 nginx /usr/local/nginx/conf/nginx.conf 配置
[root@mysql03 ~]# vi /usr/local/nginx/conf/nginx.conf
[root@mysql03 nginx]# cat /usr/local/nginx/conf/nginx.conf
user root root;
worker_processes 1;

events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;

upstream web_app {
server 10.219.24.21:8080 weight=1 max_fails=2 fail_timeout=30s;
server 10.219.24.21:8081 weight=1 max_fails=2 fail_timeout=30s; 
}

server {
listen 80;
server_name aa.com;
charset utf8;
location / {
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_pass http://web_app; 
proxy_redirect default; 
}
}
}

說明： 配置完 主 nginx后 ， 同理 ， 配置 備 nginx

2. 主 nginx 的keepalived 配置
[root@mysql03 nginx]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
router_id nginx-ha1
}

vrrp_script check_nginx {
# 檢查nginx狀態(tài)的腳本
script "/data/script/check_nginx.sh"
interval 2
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 55
priority 100
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.219.24.100
}
track_script {
check_nginx
}
}

3. 備 nginx 的 keepalived 配置
[root@redis01 ~]# vi /etc/keepalived/keepalived.conf
[root@redis01 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
router_id nginx-ha2
}

vrrp_script check_nginx {
script "/data/script/check_nginx.sh"
interval 2
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 55
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.219.24.100
}
track_script {
check_nginx
}
}

補(bǔ)充: 如果開了防火墻，則還需進(jìn)行防火墻設(shè)置，如果沒有防火墻設(shè)置，則可忽略
# iptables放行組播地址流量:
iptables -I INPUT -d 224.0.0.18 -j ACCEPT
service iptables save

4. 部署nginx狀態(tài)檢查腳本check_nginx.sh

4.1 檢查nc是否安裝 ， 沒有安裝nc包 ，要進(jìn)行 yum -y install nc 安裝 ， 不然腳本 check_nginx.sh 中 nc命令用不了。
[root@mysql03 ~]# rpm -q nc 
nc-1.84-24.el6.x86_64

4.2 創(chuàng)建腳本目錄
[root@mysql03 ~]# mkdir -p /data/script/
[root@mysql03 ~]# touch /data/script/check_nginx.sh
[root@mysql03 ~]# cat /data/script/check_nginx.sh
#!/bin/bash
# check nginx server status
# nginx端口，如果有兩個(gè)端口，只需要設(shè)置為 PORTS="80 81",則下面也應(yīng)改為"$ret1" != 11
PORTS="80"

function check_ports {
for port in $PORTS;do
nc -z 127.0.0.1 $port | grep -q succeeded
[ "${PIPESTATUS[1]}" -eq 0 ] && mark=${mark}1
done
# 如果mark值為空說明端口都不通。
# 如果mark等于1，說明有端口是通的。
echo $mark
}

ret1=$(check_ports)
# 如果nginx端口不通，會嘗試重啟一次nginx
if [ "$ret1" != 1 ];then
/sbin/service nginx stop
/sbin/service nginx start
sleep 1
ret2=$(check_ports)
# 如果還是有端口不通，表示nginx服務(wù)不正常，則停掉keepalived，使VIP發(fā)生切換
[ "$ret2" != 1 ] && /etc/init.d/keepalived stop
fi

5. 給腳本設(shè)置可執(zhí)行權(quán)限：
[root@mysql03 ~]# chmod +x /data/script/check_nginx.sh

說明： 4. 5.操作都在主、備上都操作

補(bǔ)充一點(diǎn)：
如果nginx恢復(fù)正常后，keepalived不能自動啟動，需要編寫一個(gè)腳本完成這項(xiàng)工作：判斷nginx正常后，拉起keepalived。
腳本放到cron里每分鐘執(zhí)行。

6. 開啟keepalived的日志 （keepalived默認(rèn)的日志位置為 /var/log/messages ，最好我們將它單獨(dú)放置）
6.1 編輯 /etc/sysconfig/keepalived 將 KEEPALIVED_OPTIONS="-D" 改為 KEEPALIVED_OPTIONS="-D -d -S 0"
[root@mysql03 ~]# vi /etc/sysconfig/keepalived 
KEEPALIVED_OPTIONS="-D -d -S 0"
6.2 編輯 /etc/rsyslog.conf
[root@mysql03 ~]# vi /etc/rsyslog.conf (# 配置文件最后面加上下面一行)
local0.* /var/log/keepalived.log
6.3 重啟rsyslog：
[root@mysql03 ~]# service rsyslog restart
說明：按上面配置后，keepalived會把日志記錄到/var/log/keepalived.log。

說明： 6. 操作在主、備上都操作

7. 測試
7.1 啟動所有的相關(guān)服務(wù)
7.2 查看 master nginx 上的vip ，查看漂浮的vip要用 ip a l ，直接ifconfig是看不到的。
[root@mysql03 ~]# ip a l 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host 
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:79:f4:02 brd ff:ff:ff:ff:ff:ff
inet 10.219.24.26/8 brd 10.255.255.255 scope global eth0
inet 10.219.24.100/32 scope global eth0 ##### VIP #####
inet6 fe80::20c:29ff:fe79:f402/64 scope link 
valid_lft forever preferred_lft forever
7.3 tcpdump 抓包， 發(fā)現(xiàn)只有 master ip 10.219.24.26 多播包 ，這也符合 VRRP 工作機(jī)制
[root@mysql03 ~]# tcpdump -n 'host 224.0.0.18'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:07:46.796186 IP 10.219.24.26 > 224.0.0.18: VRRPv2, Advertisement, vrid 55, prio 100, authtype simple, intvl 1s, length 20
23:07:47.796910 IP 10.219.24.26 > 224.0.0.18: VRRPv2, Advertisement, vrid 55, prio 100, authtype simple, intvl 1s, length 20
23:07:48.798713 IP 10.219.24.26 > 224.0.0.18: VRRPv2, Advertisement, vrid 55, prio 100, authtype simple, intvl 1s, length 20

7.4 master nginx 中 keepalived 關(guān)閉 ， 發(fā)現(xiàn) vip 漂到了 slave nginx 上 , 前端訪問未中斷 
-- master nginx
[root@mysql03 ~]# service keepalived stop
Stopping keepalived: [ OK ]
[root@mysql03 ~]# ip a l 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host 
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:79:f4:02 brd ff:ff:ff:ff:ff:ff
inet 10.219.24.26/8 brd 10.255.255.255 scope global eth0
inet6 fe80::20c:29ff:fe79:f402/64 scope link 
valid_lft forever preferred_lft forever

-- slave nginx
[root@redis01 ~]# ip a l 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host 
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:52:56:60 brd ff:ff:ff:ff:ff:ff
inet 10.219.24.23/8 brd 10.255.255.255 scope global eth0
inet 10.219.24.100/32 scope global eth0
inet6 fe80::20c:29ff:fe52:5660/64 scope link 
valid_lft forever preferred_lft forever

7.5 再將之前關(guān)閉的 keepalived 開啟，發(fā)現(xiàn) vip 未漂回來， 正好符合主備為 非搶占式。
[root@mysql03 ~]# service keepalived start
Starting keepalived: [ OK ]
[root@mysql03 ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host 
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:79:f4:02 brd ff:ff:ff:ff:ff:ff
inet 10.219.24.26/8 brd 10.255.255.255 scope global eth0
inet6 fe80::20c:29ff:fe79:f402/64 scope link 
valid_lft forever preferred_lft forever

7.6 將 slave nginx 上的 nginx 關(guān)閉， 發(fā)現(xiàn) slave nginx 上的 keepalived 進(jìn)程也被腳本 kill 了， 
順利的完成了 vip 漂回 master nginx 上 ，且 前端訪問不受影響。
-- slave nginx
[root@redis01 ~]# /usr/local/nginx/sbin/nginx -s stop
[root@redis01 ~]# ps -ef|grep keepalived
root 4907 2191 0 23:29 pts/0 00:00:00 grep keepalived

-- master nginx
[root@mysql03 ~]# ip a l 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host 
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:79:f4:02 brd ff:ff:ff:ff:ff:ff
inet 10.219.24.26/8 brd 10.255.255.255 scope global eth0
inet 10.219.24.100/32 scope global eth0
inet6 fe80::20c:29ff:fe79:f402/64 scope link 
valid_lft forever preferred_lft forever

“Nginx+Keepalived主備高可用怎么安裝與配置”的內(nèi)容就介紹到這里了，感謝大家的閱讀。如果想了解更多行業(yè)相關(guān)的知識可以關(guān)注億速云網(wǎng)站，小編將為大家輸出更多高質(zhì)量的實(shí)用文章！