溫馨提示×

您好，登錄后才能下訂單哦！

密碼登錄×
登錄注冊(cè)×
其他方式登錄
點(diǎn)擊 登錄注冊(cè) 即表示同意《億速云用戶服務(wù)條款》

HA（高可用）Cluster實(shí)現(xiàn)

發(fā)布時(shí)間：2020-08-14 04:47:46 來(lái)源：網(wǎng)絡(luò) 閱讀：1298 作者：buyinqi123 欄目：建站服務(wù)器

HA Cluster的目的，為了防止重要的服務(wù)器在提供服務(wù)時(shí)，出現(xiàn)不可抗力的因素，例如硬件故障

自然災(zāi)害，斷電，軟件bug，操作系統(tǒng)bug等，導(dǎo)致提供服務(wù)的主機(jī)出現(xiàn)宕機(jī)，死機(jī)，從而影響正

常業(yè)務(wù)，因此采用高可用的方案，實(shí)現(xiàn)持續(xù)性穩(wěn)定的提供主機(jī)服務(wù)的方案稱為HA

A=可用

MTBF=平均無(wú)故障時(shí)間

MTTR=平均修復(fù)時(shí)長(zhǎng)

換算公式

A=MTBF/(MTBF+MTTR)

注意提升A值的方法：

1.分子足夠大，但是不劃算，成本太高

2.降低分母，即降低平均修復(fù)時(shí)長(zhǎng)，成本較低效果出色

如何降低平均修復(fù)時(shí)長(zhǎng)？

建立備用服務(wù)器，實(shí)現(xiàn)Failover功能。

原理：在主服務(wù)器發(fā)生故障時(shí)，快速轉(zhuǎn)移IP地址（floating ip），以及快速轉(zhuǎn)移應(yīng)用程序至備用服務(wù)器

需要相關(guān)軟件轉(zhuǎn)移IP（轉(zhuǎn)移IP即實(shí)現(xiàn)ip地址從新配置），轉(zhuǎn)移服務(wù)（即在備用主機(jī)上啟動(dòng)相關(guān)服務(wù)應(yīng)用程序）

總體來(lái)說(shuō)HA 即為在主服務(wù)器宕機(jī)時(shí)，實(shí)現(xiàn)備用服務(wù)器的快速切換

關(guān)鍵點(diǎn)：IP地址轉(zhuǎn)移，數(shù)據(jù)共享

HA Cluster （ip，nginx）ip與nginx進(jìn)程通常稱為HA資源


備用服務(wù)器使用“心跳”檢測(cè)，向主服務(wù)器發(fā)起udp報(bào)文（UDP報(bào)文不需要三次握手等),

根據(jù)主服務(wù)器的響應(yīng)來(lái)判斷主服務(wù)器是否正常工作

關(guān)鍵點(diǎn)：響應(yīng)時(shí)間，資源爭(zhēng)用共享存儲(chǔ)

假如鏈接主從服務(wù)器的網(wǎng)線連接中斷，則會(huì)導(dǎo)致，ip爭(zhēng)奪，而且最主要的是存儲(chǔ)的混亂（同一文件

一邊在增加，另一邊再刪除）

會(huì)導(dǎo)致源數(shù)據(jù)的損壞，損害很大，解決方法，使用爆頭設(shè)備（STONITH），在補(bǔ)刀，將未徹底斷電的還在運(yùn)行設(shè)備斷電

當(dāng)處理完主服務(wù)器的故障后，還要將服務(wù)器上線，即為Failback

Failover<------->Failback


HA Cluster實(shí)現(xiàn)方案

1、vrrp協(xié)議的實(shí)現(xiàn)

keepalived

2、ais（available Interface standard）：可用接口標(biāo)準(zhǔn)，完備HA集群

RHCS（cmam）

heartbeat

corosync


Keepalived：

vrrp協(xié)議：Virtual Redundant Protocol

術(shù)語(yǔ):

虛擬路由：virtual router

虛擬路由器標(biāo)識(shí)：VRID（0-255）

物理路由：

master：主設(shè)備

back：備用設(shè)備

priority：優(yōu)先級(jí)

VIP：virtual Ip

VMAC:Virtual MAC (00-00-5e-00-01-VRID)

GraciousARP(免費(fèi)arp)

通告：心跳，優(yōu)先級(jí)等；周期性；

搶占式，非搶占式；

安全工作：

   認(rèn)證：

無(wú)認(rèn)證

簡(jiǎn)單字符認(rèn)證

MD5

工作模式

主/備：?jiǎn)翁摂M路由器；

主/主：主/備（虛擬路由器1），備/主（虛擬路徑器2）

特點(diǎn)：

vrrp協(xié)議的軟件實(shí)現(xiàn)，原生設(shè)計(jì)的目的為了高可用ipvs服務(wù)：

vrrp協(xié)議完成地址流動(dòng)；

為vip地址所在的節(jié)點(diǎn)生成ipvs規(guī)則（在配置文件中預(yù)先定義）；

為ipvs集群的各RS做健康狀態(tài)檢測(cè)；

基于腳本調(diào)用接口通過(guò)執(zhí)行腳本完成腳本中定義的功能，進(jìn)而影響集群事務(wù)；

組件：

核心組件：

vrrp stack

ipvs wrapper

checkers

控制組件：配置文件分析器

IO復(fù)用器

內(nèi)存管理組件

HA Cluster的配置前提：

（1）各節(jié)點(diǎn)時(shí)間必須同步

（2）確保iptables及selinux不會(huì)成為阻礙；

（3）各節(jié)點(diǎn)之間可通過(guò)主機(jī)名互相通信(對(duì)KA并非必須)；

建議使用/etc/hosts文件實(shí)現(xiàn)；

（4）各節(jié)點(diǎn)之間的root用戶可以基于密鑰認(rèn)證的ssh服務(wù)完成互相通信（并非必須）

keepalived安裝配置：

CentOS 6.4+隨base倉(cāng)庫(kù)提供；

1、同步時(shí)間

配置chronyd服務(wù)器172.18。200.100

yum安裝chrony，并啟動(dòng)服務(wù)

[root@localhost ~]# service chronyd start

Starting chronyd:                                          [  OK  ]


使用ntpdate命令，同步172.18.10.10以及172.18.10.11的時(shí)間

[root@localhost ~]# ntpdate 172.18.200.100

2、清空iptables和selinux

iptables -F

setenforce 0

3、配置hosts文件（非必須）

4、安裝keepalived

[root@localhost ~]# yum install keepalived

[root@localhost ~]# cd /etc/keepalived/

[root@localhost keepalived]# ls

keepalived.conf

[root@localhost keepalived]# cp keepalived.conf keepalived.conf.bak

[root@localhost keepalived]# ls

keepalived.conf  keepalived.conf.bak

[root@localhost keepalived]# vim keepalived.conf

主配置文件：/etc/keepalived/keepalived.conf

配置文件組成部分及相關(guān)選項(xiàng)解釋

TOP HIERACHY

GLOBAL CONFIGURATION

Global definitions

Static routes/addresses

VRRPD CONFIGURATION

VRRP synchronization group(s)：vrrp同步組；

VRRP instance(s)：每個(gè)vrrp instance即一個(gè)vrrp路由器；

LVS CONFIGURATION

Virtual server group(s)

Virtual server(s)：ipvs集群的vs和rs；

global_defs {    ###全局定義

   notification_email {

     acassen@firewall.loc

     failover@firewall.loc  ####定義出現(xiàn)問(wèn)題后發(fā)送郵箱的地址

     sysadmin@firewall.loc

   }

   notification_email_from Alexandre.Cassen@firewall.loc   ##從哪里發(fā)過(guò)來(lái)

   smtp_server 192.168.200.1  ###郵件服務(wù)器地址

   smtp_connect_timeout 30#####超時(shí)時(shí)間

   router_id LVS_DEVEL###路由器IP

   vrrp_mcast_group4  224.0.100.5###ipv4多播地址


}


vrrp_instance VI_1 {  ##vrrp配置段

    state MASTER###表示是主還是從這里顯示主，另一個(gè)則為從

    interface eth0###表明工作從哪個(gè)網(wǎng)卡發(fā)出 “多波心跳信息”

    virtual_router_id 51###虛擬路由ID

    priority 100###主的優(yōu)先級(jí)

    advert_int 1       ##通告時(shí)間間隔

    authentication###認(rèn)證

        auth_type PASS####認(rèn)證類型：簡(jiǎn)單密鑰認(rèn)證

        auth_pass 1111#####認(rèn)證密碼：最多不能超過(guò)8位

    }

    virtual_ipaddress {##虛擬IP地址配在哪個(gè)網(wǎng)卡上

        192.168.200.16/24 dev eth0   ##定義配置在哪個(gè)網(wǎng)卡的別名上

        192.168.200.17

        192.168.200.18

    }

}

track_interface {   ##配置要監(jiān)控的網(wǎng)絡(luò)接口，一旦接口出現(xiàn)故障，則轉(zhuǎn)為FAULT狀態(tài)；即接口跟蹤

eth0

eth2

...

}

nopreempt：定義工作模式為非搶占模式；

preempt_delay 300：搶占式模式下，節(jié)點(diǎn)上線后觸發(fā)新選舉操作的延遲時(shí)長(zhǎng)；

5、修改配置文件

[root@localhost keepalived]# vim keepalived.conf

global_defs {

   notification_email {

        root@localhost

   }

   notification_email_from keepalived@localhost

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id node1

   vrrp_mcast_group4  224.0.100.50

}


vrrp_instance myroute {

    state MASTER

    interface eth2

    virtual_router_id 50

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123456

    }

    virtual_ipaddress {

        172.18.50.50/16 dev eth2

    }

}

6、將配置文件發(fā)送到另一臺(tái)機(jī)器10上

[root@localhost keepalived]# scp keepalived.conf 172.18.10.10:/etc/keepalived/

修改配置文件

[root@localhost keepalived]# vim keepalived.conf

global_defs {

   notification_email {

        root@localhost

   }

   notification_email_from keepalived@localhost

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id node2

   vrrp_mcast_group4  224.0.100.50

}


vrrp_instance myroute {

    state BACKUP

    interface eth2

    virtual_router_id 50

    priority 98

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123456

    }

    virtual_ipaddress {

        172.18.50.50/16 dev eth2

    }

}


7、啟動(dòng)服務(wù)

啟動(dòng)備用服務(wù)器11

[root@localhost ~]# service keepalived start

查看地址

[root@localhost ~]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:07:27:ff brd ff:ff:ff:ff:ff:ff

    inet 172.18.10.10/16 brd 172.18.255.255 scope global eth2

    inet 172.18.50.50/16 scope global secondary eth2

    inet6 fe80::20c:29ff:fe07:27ff/64 scope link 

       valid_lft forever preferred_lft forever

發(fā)現(xiàn)地址已經(jīng)添加，這是若開(kāi)啟主服務(wù)器，由于沒(méi)有設(shè)置搶斷延遲，則會(huì)立刻搶斷

8、啟動(dòng)主服務(wù)器

[root@localhost keepalived]# service keepalived start

Starting keepalived:                                       [  OK  ]

[root@localhost keepalived]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:99:76:84 brd ff:ff:ff:ff:ff:ff

    inet 172.18.10.11/16 brd 172.18.255.255 scope global eth2

    inet 172.18.50.50/16 scope global secondary eth2

    inet6 fe80::20c:29ff:fe99:7684/64 scope link 

       valid_lft forever preferred_lft forever

發(fā)現(xiàn)地址已經(jīng)添加

而從服務(wù)器11上

[root@localhost ~]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:07:27:ff brd ff:ff:ff:ff:ff:ff

    inet 172.18.10.10/16 brd 172.18.255.255 scope global eth2

    inet6 fe80::20c:29ff:fe07:27ff/64 scope link 

       valid_lft forever preferred_lft forever

IP地址已經(jīng)刪除

9、使用tcpdump抓包工具查看主從服務(wù)器的相應(yīng)心跳測(cè)試

[root@localhost keepalived]# tcpdump -i eth2 host 224.0.100.50   ###在主服務(wù)器端抓包

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes

16:39:33.357307 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:34.358905 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:35.360605 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:36.362301 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:37.363904 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:38.365658 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:39.367266 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:40.368921 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:41.370599 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

[root@localhost ~]#  tcpdump -i  eth2  -nn host 224.0.100.50   ###在從服務(wù)器端抓包

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes

16:39:40.367044 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:41.368741 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:42.370289 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:43.371983 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:44.373750 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:45.375413 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:46.377092 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

16:39:47.378760 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

分析說(shuō)明。實(shí)現(xiàn)簡(jiǎn)單的vrrp

即從服務(wù)器每隔一秒向主服務(wù)器發(fā)送1個(gè)通報(bào)報(bào)文。探測(cè)主服務(wù)器是否存活，實(shí)現(xiàn)具體實(shí)施軟件keepalived

###############################################################################################################################

雙主模型


1、我們?cè)?72.18.10.11上配置了主服務(wù)器配置，雙主服務(wù)可在配置文件下面繼續(xù)添如下內(nèi)容，配置如下

[root@localhost keepalived]# vim keepalived.conf

vrrp_instance myroute2 {

    state BACKUP

    interface eth2

    virtual_router_id 51

    priority 98

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123457

    }

    virtual_ipaddress {

        172.18.51.51/16 dev eth2

    }

}

2、將內(nèi)容服務(wù)至粘貼至172.18.10.10服務(wù)器的keepalived.conf配置文件中,然后需要在state和priority上進(jìn)行相應(yīng)修改

vrrp_instance myroute2 {

    state MASTER

    interface eth2

    virtual_router_id 51

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123457

    }

    virtual_ipaddress {

        172.18.51.51/16 dev eth2

    }

}

保存并退出，實(shí)現(xiàn)雙主模型的設(shè)置


3、從啟服務(wù)并測(cè)試

service keepalived restart

Stopping keepalived:                                       [  OK  ]

Starting keepalived:                                       [  OK  ]

使用tcpdump抓包，結(jié)果如下

172.18.10.11端

[root@localhost keepalived]#  tcpdump -i  eth2  -nn host 224.0.100.50

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes

00:50:20.150330 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:50:20.521639 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

00:50:21.151175 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:50:21.522539 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

00:50:22.152517 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:50:22.523232 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

00:50:23.154334 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:50:23.524046 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

172.18.10.10端

[root@localhost keepalived]# tcpdump -i eth2 host 224.0.100.50

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes

00:54:01.436075 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:54:01.437266 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

00:54:02.437295 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:54:02.438831 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

00:54:03.438695 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:54:03.439205 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20


分析每次都會(huì)收到兩次信息，一次發(fā)送，一次接收

使用iptable設(shè)置規(guī)則，拒絕172.18.10.11向224.0.100.50發(fā)送通知報(bào)文

[root@localhost keepalived]# iptables -A OUTPUT -s 172.18.10.11 -d 224.0.100.50 -j REJECT

在172.18.10.10端使用tcpdump抓包

[root@localhost keepalived]#  tcpdump -i  eth2  -nn host 224.0.100.50

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes

00:50:20.150330 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:50:20.521639 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

00:50:21.151175 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:50:21.522539 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

00:50:22.152517 IP 172.18.10.10 > 224.0.100.50: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20

00:50:22.523232 IP 172.18.10.11 > 224.0.100.50: VRRPv2, Advertisement, vrid 50, prio 100, authtype simple, intvl 1s, length 20

分析發(fā)現(xiàn)發(fā)送兩條通告，因?yàn)?72.18.10.11不通告，便認(rèn)為172.18.10.11掛掉了，因此搶斷，讓自己變?yōu)橹鳈C(jī)。即別人不通告則認(rèn)為對(duì)方掛掉了

可以使用ip a l 查看相應(yīng)的ip地址獲?。?/p>

[root@localhost keepalived]# ip a l

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:07:27:ff brd ff:ff:ff:ff:ff:ff

    inet 172.18.10.10/16 brd 172.18.255.255 scope global eth2

    inet 172.18.51.51/16 scope global secondary eth2

    inet 172.18.50.50/16 scope global secondary eth2

    inet6 fe80::20c:29ff:fe07:27ff/64 scope link 

       valid_lft forever preferred_lft forever

再次在172.18.10.11服務(wù)器上，清空iptables規(guī)則

[root@localhost keepalived]# iptables -F

再回到172.18.10.10服務(wù)器上使用ip a l 查詢

[root@localhost keepalived]# ip a l

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:07:27:ff brd ff:ff:ff:ff:ff:ff

    inet 172.18.10.10/16 brd 172.18.255.255 scope global eth2

    inet 172.18.51.51/16 scope global secondary eth2

    inet6 fe80::20c:29ff:fe07:27ff/64 scope link 

       valid_lft forever preferred_lft forever

發(fā)現(xiàn)地址已經(jīng)立馬被奪回，是因?yàn)楣ぷ髟趽屨寄Ｊ较隆](méi)有設(shè)置preempt_delay 300搶占延遲時(shí)間，

結(jié)論：實(shí)現(xiàn)雙主模型實(shí)驗(yàn)


##################################################################################################################


如何實(shí)現(xiàn)自定義通知腳本


一、在172.18.10.11服務(wù)器上添加腳本，實(shí)現(xiàn)自動(dòng)發(fā)郵件

1.編寫(xiě)郵件腳本

vim notify.sh

#!/bin/bash

#

contact='root@localhost'


notify() {

        mailsubject="vrrp： $(hostname) to be $1"

        mailbody="$(hostname) to be $1,vrrp transition, $(date)."

        echo "$mailbody" | mail -s "$mailsubject" $contact


}


case $1 in

master)

        notify master ;;

backup)

        notify backup ;;

fault)

        notify fault ;;

*)

        echo "Usage: $(basename $0 ) master|backup|fault"

        exit 1

        ;;

esac

      

2、測(cè)試腳本

語(yǔ)法檢測(cè)

[root@localhost keepalived]# bash -n notify.sh

運(yùn)行腳本測(cè)試

[root@localhost keepalived]# bash -x notify.sh master

+ contact=root@localhost

+ case $1 in

+ notify master

++ hostname

+ mailsubject='localhost.localdomain to be master'

++ hostname

++ date

+ mailbody='localhost.localdomain to be master,vrrp transition, Mon May 15 01:36:33 CST 2017.'

+ echo 'localhost.localdomain to be master,vrrp transition, Mon May 15 01:36:33 CST 2017.'

+ mail -s mailsubject root@localhost

[root@localhost keepalived]# vim notify.sh

You have mail in /var/spool/mail/root


3、查看收到的郵件


[root@localhost keepalived]# mail

Heirloom Mail version 12.4 7/29/08.  Type ? for help.

"/var/spool/mail/root": 1 message 1 new

>N  1 root                  Mon May 15 01:36  18/696   "mailsubject"

&     

Message  1:

From root@localhost.localdomain  Mon May 15 01:36:34 2017

Return-Path: <root@localhost.localdomain>

X-Original-To: root@localhost

Delivered-To: root@localhost.localdomain

Date: Mon, 15 May 2017 01:36:33 +0800

To: root@localhost.localdomain

Subject: mailsubject

User-Agent: Heirloom mailx 12.4 7/29/08

Content-Type: text/plain; charset=us-ascii

From: root@localhost.localdomain (root)

Status: R


localhost.localdomain to be master,vrrp transition, Mon May 15 01:36:33 CST 2017.


& 

4、將腳本發(fā)送至172.18.10.10端


[root@localhost keepalived]# scp notify.sh 172.18.10.10:/etc/keepalived/

root@172.18.10.10's password: 

notify.sh                                                                       100%  367     0.4KB/s   00:00


5、調(diào)用腳本


[root@localhost keepalived]# vim keepalived.conf

在172.18.10.11上的vrrp_instance myrouter1下面添加如下內(nèi)容,注意是放在vrrp_instance myrouter1上下文中調(diào)用


        notify_master "/etc/keepalived/notify.sh master"

        notify_backup "/etc/keepalived/notify.sh backup"

        notify_fault "/etc/keepalived/notify.sh fault"

在172.18.10.10上的vrrp_instance myrouter2下面添加如下內(nèi)容


        notify_master "/etc/keepalived/notify.sh master"

        notify_backup "/etc/keepalived/notify.sh backup"

        notify_fault "/etc/keepalived/notify.sh fault"


6，為了實(shí)現(xiàn)測(cè)試效果，將之前定義的雙主模型刪除，并停止服務(wù)（在10.10和10.11上做同樣的操作）

：.,$d  表示從當(dāng)前行都最后一行全部刪除


[root@localhost keepalived]# service keepalived stop

Stopping keepalived:                                       [  OK  ]


7、給之前編寫(xiě)的腳本加上執(zhí)行權(quán)限

[root@localhost keepalived]# chmod +x  notify.sh 

[root@localhost keepalived]# ll

total 8

-rw-r--r-- 1 root root 658 May 15 02:01 keepalived.conf

-rwxr-xr-x 1 root root 367 May 15 01:41 notify.sh


8、啟動(dòng)服務(wù)

在172.18.10.11端

[root@localhost keepalived]# service keepalived start

Starting keepalived:                                       [  OK  ]

[root@localhost keepalived]# ip a l

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:99:76:84 brd ff:ff:ff:ff:ff:ff

    inet 172.18.10.11/16 brd 172.18.255.255 scope global eth2

    inet 172.18.50.50/16 scope global secondary eth2

    inet6 fe80::20c:29ff:fe99:7684/64 scope link 

       valid_lft forever preferred_lft forever

[root@localhost keepalived]# mail

Heirloom Mail version 12.4 7/29/08.  Type ? for help.

"/var/spool/mail/root": 3 messages 2 unread

    1 root                  Mon May 15 01:36  19/707   "mailsubject"

>U  2 root                  Mon May 15 11:03  19/735   "vrrp: localhost.localdomain to be master"

& 

9、啟動(dòng)172.18.10.10端的keepalived，并且再次到172.18.10.11端查看郵件

[root@localhost ~]# mail

Heirloom Mail version 12.4 7/29/08.  Type ? for help.

"/var/spool/mail/root": 7 messages 5 new 7 unread

 U  1 root                  Mon May 15 11:09  19/735   "vrrp: localhost.localdomain to be backup"

 U  2 root                  Mon May 15 11:11  19/735   "vrrp: localhost.localdomain to be backup"

>N  3 root                  Mon May 15 11:11  18/725   "vrrp: localhost.localdomain to be master"

 N  4 root                  Mon May 15 11:11  18/725   "vrrp: localhost.localdomain to be backup"

 N  5 root                  Mon May 15 11:26  18/725   "vrrp: localhost.localdomain to be backup"

 N  6 root                  Mon May 15 11:26  18/725   "vrrp: localhost.localdomain to be master"

 N  7 root                  Mon May 15 11:26  18/725   "vrrp: localhost.localdomain to be backup"

& 

結(jié)論：通知腳本功能實(shí)現(xiàn)

######################################################################################################


如何實(shí)現(xiàn) keepalived 高可用LVS （重點(diǎn)）


實(shí)驗(yàn)準(zhǔn)備：4臺(tái)虛擬主機(jī)

其中172.18.10.10和172.18.10.11做為VS端分別為VS2和VS1

172.18.200.100和172.18.249.57做為RS分別為RS1和RS2

首先分別再RS1和RS2端安裝httpd


1、進(jìn)行如下配置


[root@localhost ~]# cat /var/www/html/index.html

<h2>RS1:172.18.200.100</h2>

[root@localhost ~]# cat /var/www/html/index.html 

<h2>RS2:172.18.249.57</h2>


2、編寫(xiě)VIP配置腳本

vim setparam.sh

#!/bin/bash

#

vip='172.18.50.50'

netmask='255.255.255.255'

iface='lo:0'


case $1 in

start)

        echo 1 > /pro/sys/net/ipv4/conf/all/arp_ignore

        echo 1 > /pro/sys/net/ipv4/conf/lo/arp_ignore

        echo 2 > /pro/sys/net/ipv4/conf/all/arp_ignore

        echo 2 > /pro/sys/net/ipv4/conf/lo/arp_ignore

        ifconfig $iface $vip netmask $netmask broadcast $vip up

        route add -host $vip dev $iface

        ;;

stop)

        ifconfig $iface down

        echo 0 > /pro/sys/net/ipv4/conf/all/arp_ignore

        echo 0 > /pro/sys/net/ipv4/conf/lo/arp_ignore

        echo 0 > /pro/sys/net/ipv4/conf/all/arp_ignore

        echo 0 > /pro/sys/net/ipv4/conf/lo/arp_ignore

        ;;

esac


3、測(cè)試腳本

[root@localhost ~]# bash -n setparam.sh 

[root@localhost ~]# bash -x setparam.sh start

+ vip=172.18.50.50

+ netmask=255.255.255.255

+ iface=lo:0

+ case $1 in

+ echo 1

setparam.sh: line 9: /pro/sys/net/ipv4/conf/all/arp_ignore: No such file or directory

+ echo 1

setparam.sh: line 10: /pro/sys/net/ipv4/conf/lo/arp_ignore: No such file or directory

+ echo 2

setparam.sh: line 11: /pro/sys/net/ipv4/conf/all/arp_announce: No such file or directory

+ echo 2

setparam.sh: line 12: /pro/sys/net/ipv4/conf/lo/arp_announce: No such file or directory

+ ifconfig lo:0 172.18.50.50 netmask 255.255.255.255 broadcast 172.18.50.50 up

+ route add -host 172.18.50.50 dev lo:0


4、使用scp將腳本分發(fā)至RS2

[root@localhost ~]# scp setparam.sh 172.18.249.57:/root

root@172.18.249.57's password: 

setparam.sh                                                                                  100%  610     0.6KB/s   00:00 


5、在RS2端執(zhí)行腳本，并查看是否生成VIP

[root@localhost ~]# bash -x setparam.sh start

+ vip=172.18.50.50

+ netmask=255.255.255.255

+ iface=lo:0

+ case $1 in

+ echo 1

setparam.sh: line 9: /pro/sys/net/ipv4/conf/all/arp_ignore: No such file or directory

+ echo 1

setparam.sh: line 10: /pro/sys/net/ipv4/conf/lo/arp_ignore: No such file or directory

+ echo 2

setparam.sh: line 11: /pro/sys/net/ipv4/conf/all/arp_announce: No such file or directory

+ echo 2

setparam.sh: line 12: /pro/sys/net/ipv4/conf/lo/arp_announce: No such file or directory

+ ifconfig lo:0 172.18.50.50 netmask 255.255.255.255 broadcast 172.18.50.50 up

+ route add -host 172.18.50.50 dev lo:0

[root@localhost ~]# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet 172.18.50.50/32 brd 172.18.50.50 scope global lo:0

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:b2:ca:ea brd ff:ff:ff:ff:ff:ff

    inet 172.18.249.57/16 brd 172.18.255.255 scope global eth0

    inet6 fe80::20c:29ff:feb2:caea/64 scope link 

       valid_lft forever preferred_lft forever


6、啟動(dòng)RS1和RS2的httpd服務(wù)，并查看端口，兩端都要查看，這里只演示一端的

[root@localhost ~]# service httpd start

[root@localhost ~]# ss -tnl

State       Recv-Q Send-Q                               Local Address:Port                                 Peer Address:Port 

LISTEN      0      128                                             :::80                                             :::*     

LISTEN      0      128                                             :::22                                             :::*     

LISTEN      0      128                                              *:22                                              *:*     

LISTEN      0      100                                            ::1:25                                             :::*     

LISTEN      0      100                                      127.0.0.1:25       


7、在兩個(gè)前段節(jié)點(diǎn)生成ipvs規(guī)則

在VS2端

停止keepalived服務(wù)

配置sorry server頁(yè)面

vim /var/www/html/index.html

Director2 sorry server2


啟動(dòng)httpd服務(wù)

[root@localhost ~]# service httpd start


在VS1端

首先停止keepalived服務(wù)

[root@localhost ~]#  service keepalived stop  

vim /var/www/html/index.html

Director1


啟動(dòng)httpd服務(wù)

[root@localhost ~]# service httpd start


在VS1端編輯keepalived配置文件，添加如下內(nèi)容：

virtual_server 172.18.50.50 80 {

    delay_loop 6

    lb_algo wrr

    lb_kind DR

    persistence_timeout 0

    protocol TCP


    real_server 172.18.10.11 80 {

        weight 1

        HTTP_GET {

    persistence_timeout 0

    protocol TCP

sorry_server 127.0.0.1 80


    real_server 172.18.10.11 80 {

        weight 1

        HTTP_GET {

            url {

              path /

                status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

    real_server 172.18.10.10 80 {

        weight 1

        HTTP_GET {

            url {

              path /

                status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

}


在VS2端，同樣編輯keepalived.conf文件，添加如下內(nèi)容

virtual_server 172.18.50.50 80 {

    delay_loop 6

    lb_algo wrr

    lb_kind DR

    persistence_timeout 0

    protocol TCP

sorry_server 127.0.0.1 80

    real_server 172.18.10.11 80 {

        weight 1

        HTTP_GET {

    persistence_timeout 0

    protocol TCP


    real_server 172.18.10.11 80 {

        weight 1

        HTTP_GET {

            url {

              path /

                status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

    real_server 172.18.10.10 80 {

        weight 1

        HTTP_GET {

            url {

              path /

                status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

}

在VS2上啟動(dòng)keepalived服務(wù)

[root@localhost ~]# service keepalived start

Starting keepalived:                                       [  OK  ]

[root@localhost ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  172.18.50.50:80 wrr

  -> 172.18.200.100:80            Route   1      0          0         

  -> 172.18.249.57:80             Route   1      0          0   


在客戶端使用curl進(jìn)行訪問(wèn)測(cè)試（配置完有一定延遲，稍等片刻在訪問(wèn)）

[root@localhost ~]# curl http://172.18.50.50

<h2>RS2:172.18.249.57</h2>

[root@localhost ~]# curl http://172.18.50.50

<h2>RS1:172.18.200.100</h2>

[root@localhost ~]# curl http://172.18.50.50

<h2>RS2:172.18.249.57</h2>

[root@localhost ~]# curl http://172.18.50.50

<h2>RS1:172.18.200.100</h2>

[root@localhost ~]# curl http://172.18.50.50

<h2>RS2:172.18.249.57</h2>

[root@localhost ~]# curl http://172.18.50.50

<h2>RS1:172.18.200.100</h2>

[root@localhost ~]# curl http://172.18.50.50

<h2>RS2:172.18.249.57</h2>

[root@localhost ~]# curl http://172.18.50.50

<h2>RS1:172.18.200.100</h2>


在172.18.200.100端停止httpd服務(wù)

[root@localhost ~]# service httpd stop

Stopping httpd:                                            [  OK  ]


在VS2端使用ipvsadm觀察

[root@localhost keepalived]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  172.18.50.50:80 wrr

  -> 172.18.249.57:80             Route   1      0          2  


在172.18.200.100端停止httpd服務(wù)

[root@localhost ~]# service httpd start


在VS2端使用ipvsadm觀察

[root@localhost keepalived]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  172.18.50.50:80 wrr

  -> 172.18.200.100:80            Route   1      0          0         

  -> 172.18.249.57:80             Route   1      0          0    


啟動(dòng)VS1上的keepalived服務(wù)，并且關(guān)閉VS2，客戶端使用curl測(cè)試發(fā)現(xiàn)，仍然能夠訪問(wèn)

[root@localhost keepalived]# curl http://172.18.50.50

<h2>RS2:172.18.249.57</h2>

[root@localhost keepalived]# curl http://172.18.50.50

<h2>RS1:172.18.200.100</h2>

[root@localhost keepalived]# curl http://172.18.50.50

<h2>RS2:172.18.249.57</h2>

[root@localhost keepalived]# curl http://172.18.50.50

<h2>RS1:172.18.200.100</h2>


更改配置文件，將之前刪除的雙主內(nèi)容添加進(jìn)去

VS1端

vrrp_instance myroute2 {

    state BACKUP

    interface eth2

    virtual_router_id 51

    priority 98

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123457

    }

    virtual_ipaddress {

172.18.51.51/16 dev eth2

    }

}

VS2端

vrrp_instance myroute2 {

    state MASTER

    interface eth2

    virtual_router_id 51

    priority 98

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123457

    }

    virtual_ipaddress {

172.18.51.51/16 dev eth2

    }

}



重啟keepalived服務(wù),相當(dāng)重要，，，，，不重啟不會(huì)有效果，這就是個(gè)坑




總結(jié)VS端

VS2端keepalived配置

! Configuration File for keepalived


global_defs {

   notification_email {

root@localhost

   }

   notification_email_from keepalived@localhost

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id node2

   vrrp_mcast_group4  224.0.100.50

}


vrrp_instance myroute1 {

    state BACKUP

    interface eth2

    virtual_router_id 50

    priority 98

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123456

    }

    virtual_ipaddress {

172.18.50.50/16 dev eth2

    }


notify_master "/etc/keepalived/notify.sh master"

notify_backup "/etc/keepalived/notify.sh backup"

notify_fault "/etc/keepalived/notify.sh fault"

}


vrrp_instance myroute2 {

    state MASTER

    interface eth2

    virtual_router_id 51

    priority 98

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123457

    }

    virtual_ipaddress {

172.18.51.51/16 dev eth2

    }

}


virtual_server 172.18.50.50 80 {

    delay_loop 6

    lb_algo wrr

    lb_kind DR

    persistence_timeout 0

    protocol TCP

    sorry_server 127.0.0.1 80


    real_server 172.18.200.100 80 {

        weight 1

        HTTP_GET {

            url {

              path /

                status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

    real_server 172.18.249.57 80 {

        weight 1

        HTTP_GET {

            url {

              path /

                status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

}


virtual_server 172.18.51.51 80 {

    delay_loop 6

    lb_algo wrr

    lb_kind DR

    persistence_timeout 0

    protocol TCP

    sorry_server 127.0.0.1 80


    real_server 172.18.200.100 80 {

        weight 1

        HTTP_GET {

            url {

              path /

                status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

    real_server 172.18.249.57 80 {

        weight 1

        HTTP_GET {

            url {

              path /

                status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

}


VS1端keepalived配置

! Configuration File for keepalived


global_defs {

   notification_email {

root@localhost

   }

   notification_email_from keepalived@localhost

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id node1

   vrrp_mcast_group4  224.0.100.50

}


vrrp_instance myroute1 {

    state MASTER

    interface eth2

    virtual_router_id 50

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123456

    }

    virtual_ipaddress {

172.18.50.50/16 dev eth2

    }


notify_master "/etc/keepalived/notify.sh master"

        notify_backup "/etc/keepalived/notify.sh backup"

        notify_fault "/etc/keepalived/notify.sh fault"

}


vrrp_instance myroute2 {

    state BACKUP

    interface eth2

    virtual_router_id 51

    priority 98

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 123457

    }

    virtual_ipaddress {

172.18.51.51/16 dev eth2

    }

}



virtual_server 172.18.50.50 80 {

    delay_loop 6

    lb_algo wrr

    lb_kind DR

    persistence_timeout 0

    protocol TCP

sorry_server 127.0.0.1 80


    real_server 172.18.200.100 80 {

        weight 1

        HTTP_GET {

            url {

              path /

status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

    real_server 172.18.249.57 80 {

        weight 1

        HTTP_GET {

            url {

              path /

status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

}


virtual_server 172.18.51.51 80 {

    delay_loop 6

    lb_algo wrr

    lb_kind DR

    persistence_timeout 0

    protocol TCP

sorry_server 127.0.0.1 80


    real_server 172.18.200.100 80 {

        weight 1

        HTTP_GET {

            url {

              path /

status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

    real_server 172.18.249.57 80 {

        weight 1

        HTTP_GET {

            url {

              path /

status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

}


VIP配置腳本 （由用戶是雙主模型因此VIP有兩個(gè)）

#!/bin/bash

#

vip='172.18.50.50'

vip2='172.18.51.51'

netmask='255.255.255.255'

iface='lo:0'

iface2='lo:1'


case $1 in

start)

echo 1 > /pro/sys/net/ipv4/conf/all/arp_ignore

echo 1 > /pro/sys/net/ipv4/conf/lo/arp_ignore

echo 2 > /pro/sys/net/ipv4/conf/all/arp_announce

echo 2 > /pro/sys/net/ipv4/conf/lo/arp_announce

ifconfig $iface $vip netmask $netmask broadcast $vip up

ifconfig $iface2 $vip2 netmask $netmask broadcast $vip2 up

route add -host $vip dev $iface

;;

stop)

ifconfig $iface down

ifconfig $iface2 down

echo 0 > /pro/sys/net/ipv4/conf/all/arp_ignore

echo 0 > /pro/sys/net/ipv4/conf/lo/arp_ignore

echo 0 > /pro/sys/net/ipv4/conf/all/arp_announce

echo 0 > /pro/sys/net/ipv4/conf/lo/arp_announce

;;

esac


實(shí)驗(yàn)結(jié)論：實(shí)現(xiàn)keepalived 高可用lvs負(fù)載均衡





















向AI問(wèn)一下細(xì)節(jié)

免責(zé)聲明：本站發(fā)布的內(nèi)容（圖片、視頻和文字）以原創(chuàng)、轉(zhuǎn)載和分享為主，文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱：is@yisu.com進(jìn)行舉報(bào)，并提供相關(guān)證據(jù)，一經(jīng)查實(shí)，將立刻刪除涉嫌侵權(quán)內(nèi)容。

AI
助
手