ceph與openstack的對(duì)接集成

實(shí)驗(yàn)環(huán)境
三節(jié)點(diǎn)的社區(qū)版Openstack環(huán)境
controller:172.16.1.100
computer:172.16.1.131
cinder:172.16.1.132
這里三個(gè)節(jié)點(diǎn)使用的IP地址為Openstack的管理地址
ceph2:172.16.1.128
ceph3:172.16.1.129
ceph4:172.16.1.130
我這里使用的Openstack云平臺(tái)跟ceph分布式集群均為手動(dòng)搭建的測(cè)試環(huán)境
使用rbd方式為云平臺(tái)提供存儲(chǔ)
（1）image:保存glance中的映像文件
（2）volume存儲(chǔ)：保存cinder的volume;保存創(chuàng)建虛擬機(jī)時(shí)選擇創(chuàng)建新卷；
（3）vms存儲(chǔ)：保存創(chuàng)建虛擬機(jī)時(shí)不選擇創(chuàng)新卷；

首先在云平臺(tái)安裝rbd相關(guān)軟件（云平臺(tái)三個(gè)節(jié)點(diǎn)上全部安裝）
yum install -y python-rbd
yum install -y ceph-common
#在ceph的控制節(jié)節(jié)點(diǎn)創(chuàng)建三個(gè)POOL
ceph osd pool create volumes 128
ceph osd pool create images 128
ceph osd pool create vms 128
#初始化POOL
rbd pool init volumes
rbd pool init images
rbd pool init vms

將ceph的配置文件導(dǎo)入到云平臺(tái)的三個(gè)節(jié)點(diǎn)

在ceph集群的控制節(jié)點(diǎn)上執(zhí)行
sudo ssh root@172.16.1.100  tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf
sudo ssh root@172.16.1.131  tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf
sudo ssh root@172.16.1.132  tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf

創(chuàng)建ceph用戶(hù)和密鑰

查詢(xún)用戶(hù)，寫(xiě)入文件
#ceph auth get-or-create client.glance | ssh root@172.16.1.100  tee /etc/ceph/ceph.client.glance.keyring
#ssh root@172.16.1.100 chown glance:glance /etc/ceph/ceph.client.glance.keyring
#ceph auth get-or-create client.cinder | ssh root@172.16.1.132  tee /etc/ceph/ceph.client.cinder.keyring
#ssh root@172.16.1.132  chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
#ceph auth get-or-create client.cinder-backup    |     ssh root@172.16.1.132  tee  /etc/ceph/ceph.client.cinder-backup.keyring
#ssh root@172.16.1.132  chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring
#ceph auth get-key client.cinder | ssh root@172.16.1.132 tee client.cinder.key

接下來(lái)與云平臺(tái)進(jìn)行對(duì)接
以下操作在openstack的控制節(jié)點(diǎn)上執(zhí)行
ceph與glance對(duì)接

#更改glance默認(rèn)存儲(chǔ)為ceph
#修改/etc/glance/glance-api.conf文件，添加一下內(nèi)容
[DEFAULT]
default_store = rbddefault_store = rbd
show_image_direct_url = True
show_image_direct_url = True
[glance_store]
#stores = file,http
#default_store = file
#filesystem_store_datadir = /var/lib/glance/images/
stores = rbd
default_store = rbd
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8
#重啟服務(wù)
systemctl restart openstack-glance-api openstack-glance-registry

ceph與nova對(duì)接，在computer節(jié)點(diǎn)上操作
使用uuidgen獲取密鑰信息

編輯一個(gè)secret.xml文檔 
<secret ephemeral='no' private='no'>
  <uuid>545ec73d-06b8-462e-8494-cbf1317dbc1a</uuid>
  <usage type='ceph'>
    <name>client.cinder secret</name>
  </usage>
</secret> 
virsh secret-define --file secret.xml
virsh secret-set-value --secret 545ec73d-06b8-462e-8494-cbf1317dbc1a --base64 AQAVkeJcqR2/GxAAtuPMq1MqF5wUBYTeDuDuiw==
注意--base64后面要跟的參數(shù)為ceph.client.cinder.keyring文件中的密鑰信息，在cinder節(jié)點(diǎn)中將信息復(fù)制到此處

如果openstack有多個(gè)計(jì)算節(jié)點(diǎn)，所有的計(jì)算節(jié)點(diǎn)節(jié)點(diǎn)使用同一個(gè)UUID來(lái)進(jìn)行標(biāo)識(shí)
virsh secret-list 使用命令進(jìn)行查看

在computer節(jié)點(diǎn)的ceph配置文件里添加一下內(nèi)容

[client]
rbd cache = true
rbd cache writethrough until flush = true
admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/qemu/qemu-guest-$pid.log
rbd concurrent management ops = 20
[client.cinder]
keyring = /etc/ceph/ceph.client.cinder.keyring

#重啟服務(wù)
systemctl restart libvirtd.service openstack-nova-compute.service

ceph與cinder對(duì)接

#修改/etc/cinder/cinder.conf配置文件，注意：將之前的lvm的配置都刪掉
[DEFAULT]
#enabled_backends = lvm
enabled_backends = ceph

[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder
rbd_secret_uuid = ff2e1190-30f5-4849-9c1c-886b1e1ee181
 #重啟服務(wù)   
systemctl restart openstack-cinder-volume.service

在執(zhí)行命令時(shí)如果出現(xiàn)下面的命令則說(shuō)明認(rèn)證信息出現(xiàn)錯(cuò)誤，需要把ceph.client.admin.keyring文件拷貝到控制節(jié)點(diǎn)下。

創(chuàng)建一個(gè)新的虛擬機(jī)映像
glance image-create --name cirros_image --disk-format=qcow2 --container-format=bare < cirros-0.3.1-x86_64-disk.img

創(chuàng)建一個(gè)2G大小的cinder卷組
cinder create --display-name ceph-volume01 --display-description "cinder volume on CEPH storage" 2

創(chuàng)建完之后可以在ceph集群的控制臺(tái)看到Pools選項(xiàng)下面出現(xiàn)volumes以及images池

然后登錄云平臺(tái)進(jìn)行虛擬機(jī)創(chuàng)建