說說Keepalived的腦裂

1. 工作場景

Keepalived提供了Loadbalancing和High-Availability的功能, 本文說的是其為2個Mycat節(jié)點提供HA功能的場景.

2. 關(guān)鍵配置如下, 為主備非搶占模式.

! mycat01, 192.168.4.196

global_defs {

    # 一個keepalived.conf, 對應(yīng)一個router_id

    router_id mycat01

}

vrrp_instance VI_1 {

    state BACKUP

    nopreempt

    interface eth0

    # 一個組播內(nèi), 各節(jié)點該值相同

    virtual_router_id 196

    priority 200

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass Zf4aqy

    }

    virtual_ipaddress {

        192.168.4.200

    }

}    

! mycat02, 192.168.4.195

global_defs {

    router_id mycat02

}

vrrp_instance VI_1 {

    state BACKUP

    nopreempt

    interface eth0

    virtual_router_id 196

    priority 200

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass Zf4aqy

    }

    virtual_ipaddress {

        192.168.4.200

    }

}    

3. Keepalived提供HA的工作原理

Keepalived的HA功能, 是通過VRRP(Virtual Router Redundancy Protocol, 虛擬路由冗余協(xié)議)來實現(xiàn)的. 其用IP組播的方式(默認組播地址: 224.0.0.18), 實現(xiàn)服務(wù)節(jié)點間的通信, 通過一種競選機制來將路由任務(wù)交給某臺VRRP路由器. 工作時主節(jié)點發(fā)送VRRP協(xié)議報文, 備節(jié)點接收報文, 若一段時間(默認3個報文發(fā)送時間)備節(jié)點接收不到主節(jié)點發(fā)送的報文, 就會啟動接管程序接管主節(jié)點的資源. 備節(jié)點可以有多個, 通過優(yōu)先級競選.

4. Keepalived發(fā)生腦裂的情況

4.1 根據(jù)上述工作原理知道, 備節(jié)點接收不到報文時, 如兩者間的網(wǎng)絡(luò)不通了, 備節(jié)點就會啟動接管程序接管主節(jié)點的資源, 對外提供服務(wù), 表現(xiàn)形式就是備節(jié)點上出現(xiàn)了虛擬IP, 此時主節(jié)點也是持有虛擬IP的.

問題重現(xiàn)如下, 此時mycat01為主節(jié)點, mycat02為備節(jié)點, 在mycat02上drop掉組播數(shù)據(jù)包.

# iptables -A INPUT -m pkttype --pkt-type multicast -j DROP

然后觀察message日志, 可見其也變成了主節(jié)點.

# tail -f /var/log/messages

Mar 17 00:13:58 mycat02 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team

Mar 17 00:14:00 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Mar 17 00:14:01 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Mar 17 00:14:01 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Mar 17 00:14:01 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.4.200

Mar 17 00:14:01 mycat02 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.4.200 added

Mar 17 00:14:06 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.4.200

在mycat01上抓包, 看到mycat02也在向組播地址發(fā)送數(shù)據(jù)包.

# tcpdump -nnei eth0 | grep -i 'vrid 196'

00:13:58.735738 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

00:13:59.736268 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

00:14:00.736800 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

00:14:00.955581 fa:16:3e:5d:23:10 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.195 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

00:14:01.956340 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

當把iptables停掉后, mycat01和mycat02扔分別持有虛擬IP 192.168.4.200, 這也說明了主備非搶占模式的一個弊端, 主備搶占模式是不會這樣的, 下面是其主要配置.

! mycat01, 192.168.4.196

global_defs {

    router_id mycat01

}

vrrp_instance VI_1 {

    state MASTER

    virtual_router_id 196

    priority 200

}    

! mycat02, 192.168.4.195

global_defs {

    router_id mycat02

}

vrrp_instance VI_1 {

    state BACKUP

    virtual_router_id 196

    priority 150

}    

4.2 同一個組播內(nèi), virtual_router_id配置不相同, 也會發(fā)生腦裂現(xiàn)象, 這里就不在贅述了.

說到最后, 如何避免腦裂現(xiàn)象呢, 其實可以看到, 只要有虛擬IP存在, 就不可能完全規(guī)避這個問題. 也就是沒有虛擬IP了, 就沒有腦裂了, 那么節(jié)點又如何向外提供服務(wù)呢? 引入Zookeeper或Consul的服務(wù)發(fā)現(xiàn)機制, 這些又是另外的事情了...

若感興趣可關(guān)注訂閱號”數(shù)據(jù)庫最佳實踐”(DBBestPractice).