Mongodb分片原理詳解及架構(gòu)部署

分片技術(shù)解決的需求痛點

（1）高數(shù)據(jù)量和吞吐量的數(shù)據(jù)庫應(yīng)用會對單機(jī)的性能造成較大壓力；

（2）大的查詢量會將單機(jī)的CPU耗盡；

（3）大的數(shù)據(jù)量對單機(jī)的存儲壓力較大，最終會耗盡系統(tǒng)的內(nèi)存而將壓力轉(zhuǎn)移到鍵盤IO上

何為分片？

MongoDB分片技術(shù)是使用多個服務(wù)器存儲數(shù)據(jù)的方法，以支持巨大的數(shù)據(jù)存儲和對數(shù)據(jù)的操作。當(dāng)MongoDB存儲海量的數(shù)據(jù)時，一臺機(jī)器可能不足以存儲數(shù)據(jù)，也可能不足以提供可接受的讀寫吞吐量。這時，我們就可以通過在多臺機(jī)器上分割數(shù)據(jù)，使得數(shù)據(jù)庫系統(tǒng)能存儲和處理更多的數(shù)據(jù)。

MongoDB分片結(jié)構(gòu)

下圖展示了MongoDB分片集群結(jié)構(gòu)組成分布：

上圖中主要有如下所述三個主要組件：

• Shard:分片服務(wù)器，

 用于存儲實際的數(shù)據(jù)塊，實際生產(chǎn)環(huán)境中一個shard server角色可由幾臺機(jī)器組個一個replica set承擔(dān)，防止主機(jī)單點故障

• Config Server:配置服務(wù)器，

 mongod實例，存儲了整個 分片群集的配置信息，其中包括 chunk信息。

• Query Routers:前端路由，

 客戶端由此接入，且讓整個集群看上去像單一數(shù)據(jù)庫，前端應(yīng)用可以透明使用

MongoDB分片優(yōu)勢：

• 減少單個分片需要處理的請求數(shù)，提高群集的存儲容量和吞吐量

比如，當(dāng)插入一條數(shù)據(jù)時，應(yīng)用只需要訪問存儲這條數(shù)據(jù)的分片

• 減少單分片存儲的數(shù)據(jù)，提高數(shù)據(jù)可用性，提高大型數(shù)據(jù)庫查詢服務(wù)的性能。

當(dāng)MongoDB單點數(shù)據(jù)庫服務(wù)器存儲和性能成為瓶頸，或者需要部署大型應(yīng)用以充分利用內(nèi)存時，可以使用分片技術(shù)

分片群集簡單部署：

下面介紹如何在一臺CentOS7系統(tǒng)的物理服務(wù)器上部署一個簡單結(jié)構(gòu)的MongoDB分片集群 。

• 1臺路由實例（端口 27017）

• 1臺配置實例（端口 37017）

• 2臺分片實例（端口 47017， 47018）

本次實驗使用手工編譯安裝mongodb，

百度網(wǎng)盤免費(fèi)提供安裝包：

https://pan.baidu.com/s/1bZXJDBrn9qxBF6r6IF_aLQ

具體步驟如下：

• 安裝支持的軟件，解壓準(zhǔn)備好的MongoDB軟件包，注意：本次安裝是在64位Linux系統(tǒng)上

yum install openssl-devel -y

tar zxvf mongodb-linux-x86_64-3.2.1.tgz -C /opt/

mv mongodb-linux-x86_64-3.2.1/ /usr/local/mongodb

• 創(chuàng)建MongoDB四個實例的數(shù)據(jù)存儲目錄，日志存儲目錄/data/logs，日志文件，并修改權(quán)限

mkdir -p /data/mongodb/mongodb{1,2,3,4}

 

mkdir /data/logs

touch /data/logs/mongodb{1,2,3,4}.log

chmod -R 777 /data/logs/*.log

• 針對MongoDB服務(wù)進(jìn)行Linux系統(tǒng)內(nèi)核調(diào)優(yōu)

ulimit -n 25000  #最大文件 數(shù)

ulimit -u 25000  #最大進(jìn)程數(shù)

• 某節(jié)點內(nèi)存不足時，從其他節(jié)點分配內(nèi)存

sysctl -w vm.zone_reclaim_mode=0             #永久設(shè)置
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

ln -s /usr/local/mongodb/bin/*  /usr/bin/           #方便服務(wù)管理

部署配置服務(wù)器

• 編輯mongodb1.conf配置文件，端口號為37017，設(shè)置configsvr=true，啟動配置服務(wù)器

  
  cd /usr/local/mongodb/bin/

vim mongodb1.conf

port=37017      #指定服務(wù)端口
dbpath=/data/mongodb/mongodb1    #數(shù)據(jù)存儲目錄
logpath=/data/logs/mongodb1.log        #日志文件
logappend=true           #使用追加方式寫日志
fork=true                       #后臺運(yùn)行
maxConns=5000               #設(shè)定最大同時連接數(shù)，默認(rèn)為2000
storageEngine=mmapv1        #指定存儲引擎為內(nèi)存映射文件
configsvr=true        #配置服務(wù)

• 啟動配置服務(wù)，并進(jìn)入，可看到配置服務(wù)前綴為”configsvr>“，

[root@localhost bin]# mongod -f mongodb1.conf          #啟動配置服務(wù)器
about to fork child process, waiting until server is ready for connections.
forked process: 12594
child process started successfully, parent exiting
[root@localhost bin]# mongo --port 37017          #進(jìn)入服務(wù)
MongoDB shell version: 3.2.1
connecting to: 127.0.0.1:37017/test
Server has startup warnings:
2018-09-15T04:39:24.967+0800 I STORAGE  [initandlisten]
2018-09-15T04:39:24.967+0800 I STORAGE  [initandlisten] ** WARNING: Readahead for /data/mongodb1 is set to 4096KB
2018-09-15T04:39:24.967+0800 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
2018-09-15T04:39:24.967+0800 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
2018-09-15T04:39:25.828+0800 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-09-15T04:39:25.828+0800 I CONTROL  [initandlisten]
2018-09-15T04:39:25.829+0800 I CONTROL  [initandlisten]
2018-09-15T04:39:25.829+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-09-15T04:39:25.829+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-09-15T04:39:25.829+0800 I CONTROL  [initandlisten]
2018-09-15T04:39:25.829+0800 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-09-15T04:39:25.829+0800 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-09-15T04:39:25.829+0800 I CONTROL  [initandlisten]
configsvr>

部署分片服務(wù)器

• 編輯mongodb2.conf配置文件，端口號為47017，設(shè)置shardsvr=true，

vim mongodb2.conf

port=47017
dbpath=/data/mongodb/mongodb2
logpath=/data/logs/mongodb2.log
logappend=true
fork=true
maxConns=5000
storageEngine=mmapv1
shardsvr=true

• 同樣編輯mongodb3.conf配置文件，端口號為47018，

vim mongodb3.conf

port=47018
dbpath=/data/mongodb/mongodb3
logpath=/data/logs/mongodb3.log
logappend=true
fork=true
maxConns=5000
storageEngine=mmapv1
shardsvr=true

• 啟動兩個分片服務(wù)器

[root@localhost bin]# mongod -f mongodb2.conf
about to fork child process, waiting until server is ready for connections.
forked process: 12810
child process started successfully, parent exiting
[root@localhost bin]# mongod -f mongodb3.conf
about to fork child process, waiting until server is ready for connections.
forked process: 12825
child process started successfully, parent exiting

啟動路由服務(wù)器

• 通過./mongos --help命令可以查看啟動路由相關(guān)參數(shù)信息。chunkSize為數(shù)據(jù)塊大小，默認(rèn)為200MB，為了便于測試這里將值設(shè)置為1

[root@localhost bin]# ./mongos --port 27017 --fork --logpath=/usr/local/mongodb/bin/route.log --configdb 192.168.30.55:37017 --chunkSize 1
2018-09-15T05:03:06.217+0800 W SHARDING [main] Running a sharded cluster with fewer than 3 config servers should only be done for testing purposes and is not recommended for production.
about to fork child process, waiting until server is ready for connections.
forked process: 12934
child process started successfully, parent exiting

啟用分片服務(wù)器

• 連接到路由實例后，可以通過sh.status()命令查看分片狀態(tài)信息

[root@localhost bin]# mongo            #進(jìn)入路由實例
MongoDB shell version: 3.2.1
connecting to: test
Server has startup warnings:
2018-09-15T16:00:19.746+0800 I CONTROL  [main] ** WARNING: You are running this process as the root user, which is not recommended.
2018-09-15T16:00:19.746+0800 I CONTROL  [main]
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5b9cbc14b4c77895df796bac")
}
  shards:                        #shards下為空，沒有分片服務(wù)器
  active mongoses:
    "3.2.1" : 1
  balancer:
    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours:
        No recent migrations
  databases:

• 通過sh.addShard命令添加兩個分片服務(wù)器，

mongos> sh.addShard("192.168.30.55:47017")
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> sh.addShard("192.168.30.55:47018")
{ "shardAdded" : "shard0001", "ok" : 1 }

• 再次查看分片信息，可以看到shards：選項下已經(jīng)顯示剛剛添加的分片服務(wù)器

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5b9cbc14b4c77895df796bac")
}
  shards:
    {  "_id" : "shard0000",  "host" : "192.168.30.55:47017" }
    {  "_id" : "shard0001",  "host" : "192.168.30.55:47018" }
  active mongoses:
    "3.2.1" : 1
  balancer:
    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours:
        No recent migrations
  databases:

實現(xiàn)分片功能

• 添加兩個分片服務(wù)器后，數(shù)據(jù)庫和集合還未啟用分片

mongos> show dbs
config  0.031GB
mongos> use school          #進(jìn)入并創(chuàng)建數(shù)據(jù)庫school
switched to db school
mongos> for (var i=1;i<=50000;i++)db.info.insert({"id":i,"name":"tom"+i})           #創(chuàng)建集合info，并使用循環(huán)插入50000條數(shù)據(jù)

WriteResult({ "nInserted" : 1 })

• 使用sh.enableSharding("school")命令啟用school數(shù)據(jù)庫分片

mongos> sh.enableSharding("school")           
{ "ok" : 1 }

• 針對info集合創(chuàng)建索引

mongos> db.info.createIndex({"id":1})
{
    "raw" : {
        "192.168.30.55:47017" : {
            "createdCollectionAutomatically" : false,
            "numIndexesBefore" : 1,
            "numIndexesAfter" : 2,
            "ok" : 1
        }
    },
    "ok" : 1
}

• 使用sh.shardCollection("school.info",{"id":1})命令對集合info進(jìn)行分片

mongos> sh.shardCollection("school.info",{"id":1})
{ "collectionsharded" : "school.info", "ok" : 1 }

• 查看分片信息

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5b9cbc14b4c77895df796bac")
}
  shards:                             
    {  "_id" : "shard0000",  "host" : "192.168.30.55:47017" }              #兩個分片服務(wù)器信息
    {  "_id" : "shard0001",  "host" : "192.168.30.55:47018" }
  active mongoses:
    "3.2.1" : 1
  balancer:
    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours:
        5 : Success
  databases:
    {  "_id" : "school",  "primary" : "shard0000",  "partitioned" : true }             #數(shù)據(jù)庫school的分片信息
        school.info
            shard key: { "id" : 1 }
            unique: false
            balancing: true
            chunks:                               #可以看到chunks均勻分布到兩個分片上
                shard0000    6
                shard0001    5
            { "id" : { "$minKey" : 1 } } -->> { "id" : 4682 } on : shard0001 Timestamp(2, 0)
            { "id" : 4682 } -->> { "id" : 9364 } on : shard0001 Timestamp(3, 0)
            { "id" : 9364 } -->> { "id" : 14046 } on : shard0001 Timestamp(4, 0)
            { "id" : 14046 } -->> { "id" : 18728 } on : shard0001 Timestamp(5, 0)
            { "id" : 18728 } -->> { "id" : 23410 } on : shard0001 Timestamp(6, 0)
            { "id" : 23410 } -->> { "id" : 28092 } on : shard0000 Timestamp(6, 1)
            { "id" : 28092 } -->> { "id" : 32774 } on : shard0000 Timestamp(1, 6)
            { "id" : 32774 } -->> { "id" : 37456 } on : shard0000 Timestamp(1, 7)
            { "id" : 37456 } -->> { "id" : 42138 } on : shard0000 Timestamp(1, 8)
            { "id" : 42138 } -->> { "id" : 46820 } on : shard0000 Timestamp(1, 9)
            { "id" : 46820 } -->> { "id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 10)

• 添加標(biāo)簽

mongos> sh.addShardTag("shard0000","abc01")
mongos> sh.addShardTag("shard0001","abc02")

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5b9cbc14b4c77895df796bac")
}
  shards:
    {  "_id" : "shard0000",  "host" : "192.168.30.55:47017",  "tags" : [ "abc01" ] }
    {  "_id" : "shard0001",  "host" : "192.168.30.55:47018",  "tags" : [ "abc02" ] }

 

MongoDB分片服務(wù)器管理

• 根據(jù)需求可以添加或刪除sharding server。

[root@localhost bin]# cp mongodb3.conf mongodb4.conf
[root@localhost bin]# vim mongodb4.conf

port=47019
logpath=/data/logs/mongodb4.log
dbpath=/data/mongodb4
fork=true
logappend=true
maxConns=5000
storageEngine=mmapv1
shardsvr=true

• 啟動實例4，進(jìn)入路由實例添加新的分片服務(wù)器

[root@localhost bin]# mongod -f mongodb4.conf
about to fork child process, waiting until server is ready for connections.
forked process: 52634
child process started successfully, parent exiting
[root@localhost bin]# mongo
MongoDB shell version: 3.2.1
connecting to: test
Server has startup warnings:
2018-09-15T16:00:19.746+0800 I CONTROL  [main] ** WARNING: You are running this process as the root user, which is not recommended.
2018-09-15T16:00:19.746+0800 I CONTROL  [main]

• 使用sh.addShard("192.168.30.55:47019") 命令添加一個新的分片服務(wù)器，再次查看分片信息，發(fā)現(xiàn)分片均勻地分布到3個分片服務(wù)器上

mongos> sh.addShard("192.168.30.55:47019")          #添加分片服務(wù)器
{ "shardAdded" : "shard0002", "ok" : 1 }
mongos> sh.status()                   #查看分片服務(wù)信息
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5b9cbc14b4c77895df796bac")
}
  shards:
    {  "_id" : "shard0000",  "host" : "192.168.30.55:47017" }
    {  "_id" : "shard0001",  "host" : "192.168.30.55:47018" }
   {  "_id" : "shard0002",  "host" : "192.168.30.55:47019" }
  active mongoses:
    "3.2.1" : 1
  balancer:
    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours:
        8 : Success
  databases:
    {  "_id" : "school",  "primary" : "shard0000",  "partitioned" : true }
        school.info
            shard key: { "id" : 1 }
            unique: false
            balancing: true
            chunks:
                shard0000    4
                shard0001    4
                shard0002    3
            { "id" : { "$minKey" : 1 } } -->> { "id" : 4682 } on : shard0002 Timestamp(9, 0)
            { "id" : 4682 } -->> { "id" : 9364 } on : shard0001 Timestamp(9, 1)
            { "id" : 9364 } -->> { "id" : 14046 } on : shard0001 Timestamp(4, 0)
            { "id" : 14046 } -->> { "id" : 18728 } on : shard0001 Timestamp(5, 0)
            { "id" : 18728 } -->> { "id" : 23410 } on : shard0001 Timestamp(6, 0)
            { "id" : 23410 } -->> { "id" : 28092 } on : shard0002 Timestamp(7, 0)
            { "id" : 28092 } -->> { "id" : 32774 } on : shard0002 Timestamp(8, 0)
            { "id" : 32774 } -->> { "id" : 37456 } on : shard0000 Timestamp(8, 1)
            { "id" : 37456 } -->> { "id" : 42138 } on : shard0000 Timestamp(1, 8)
            { "id" : 42138 } -->> { "id" : 46820 } on : shard0000 Timestamp(1, 9)
            { "id" : 46820 } -->> { "id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 10)

• 使用db.runCommand({"removeshard":"192.168.30.55:47019"})命令可以刪除新添加的分片服務(wù)器

mongos> use admin           # 注：在admin db下執(zhí)行命令。
switched to db admin
mongos> db.runCommand({"removeshard":"192.168.30.55:47019"})
{
    "msg" : "draining started successfully",
    "state" : "started",
    "shard" : "shard0002",
    "note" : "you need to drop or movePrimary these databases",
    "dbsToMove" : [ ],
    "ok" : 1
}
mongos> db.runCommand({"removeshard":"192.168.30.55:47019"})
{
    "msg" : "removeshard completed successfully",
    "state" : "completed",
    "shard" : "shard0002",
    "ok" : 1
}

“注意：該命令至少執(zhí)行兩次才能成功刪除，執(zhí)行到state為completed才真正刪除，否則就是沒有刪除成功，該分片處于"draining" : true狀態(tài)，該狀態(tài)下不但該分片沒有刪除成功，而且還影響接下來刪除其他分片操作，遇到該狀態(tài)再執(zhí)行一次removeshard即可，最好就是刪除分片時一直重復(fù)執(zhí)行刪除命令，直到state為completed

• 再次查看分片信息，可以看到分片已刪除

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5b9cbc14b4c77895df796bac")
}
  shards:
    {  "_id" : "shard0000",  "host" : "192.168.30.55:47017" }
    {  "_id" : "shard0001",  "host" : "192.168.30.55:47018" }
  active mongoses:
    "3.2.1" : 1
  balancer:
    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours:
        11 : Success
  databases:
    {  "_id" : "school",  "primary" : "shard0000",  "partitioned" : true }
        school.info
            shard key: { "id" : 1 }
            unique: false
            balancing: true
            chunks:
                shard0000    6
                shard0001    5
            { "id" : { "$minKey" : 1 } } -->> { "id" : 4682 } on : shard0000 Timestamp(10, 0)
            { "id" : 4682 } -->> { "id" : 9364 } on : shard0001 Timestamp(9, 1)
            { "id" : 9364 } -->> { "id" : 14046 } on : shard0001 Timestamp(4, 0)
            { "id" : 14046 } -->> { "id" : 18728 } on : shard0001 Timestamp(5, 0)
            { "id" : 18728 } -->> { "id" : 23410 } on : shard0001 Timestamp(6, 0)
            { "id" : 23410 } -->> { "id" : 28092 } on : shard0001 Timestamp(11, 0)
            { "id" : 28092 } -->> { "id" : 32774 } on : shard0000 Timestamp(12, 0)
            { "id" : 32774 } -->> { "id" : 37456 } on : shard0000 Timestamp(8, 1)
            { "id" : 37456 } -->> { "id" : 42138 } on : shard0000 Timestamp(1, 8)
            { "id" : 42138 } -->> { "id" : 46820 } on : shard0000 Timestamp(1, 9)
            { "id" : 46820 } -->> { "id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 10)

連接配置服務(wù)器查看信息

• 配置服務(wù)器存儲了MongoDB數(shù)據(jù)庫集合分片的詳細(xì)信息，可以通過以下命令查看

mongo --port 37017

configsvr> use config

configsvr> show collections  

....
collections
chunks
databases   
....

configsvr> db.chunks.findOne()

configsvr> db.collections.find()

configsvr> db.databases.find()