如何查看HDFS文件系統(tǒng)空間使用情況

這篇文章主要介紹如何查看HDFS文件系統(tǒng)空間使用情況，文中介紹的非常詳細，具有一定的參考價值，感興趣的小伙伴們一定要看完！

1、查看集群的空間使用狀態(tài)

[hduser@master1 bin]$ ./hdfs dfsadmin -report

Configured Capacity: 8453768839168 (7.69 TB)

Present Capacity: 7959274510394 (7.24 TB)

DFS Remaining: 4128882869718 (3.76 TB)

DFS Used: 3830391640676 (3.48 TB)

DFS Used%: 48.12%

Under replicated blocks: 18

Blocks with corrupt replicas: 0

Missing blocks: 0

Missing blocks (with replication factor 1): 0

-------------------------------------------------

Live datanodes (5):

Name: 192.168.11.137:50010 (slave5)

Hostname: slave5

Decommission Status : Normal

Configured Capacity: 1056617381888 (984.05 GB)

DFS Used: 736041209958 (685.49 GB)

Non DFS Used: 66876043079 (62.28 GB)

DFS Remaining: 253700128851 (236.28 GB)

DFS Used%: 69.66%

DFS Remaining%: 24.01%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 30

Last contact: Tue Jul 03 15:06:51 CST 2018

Name: 192.168.11.134:50010 (slave2)

Hostname: slave2

Decommission Status : Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 749551308902 (698.07 GB)

Non DFS Used: 120793546401 (112.50 GB)

DFS Remaining: 1243166503161 (1.13 TB)

DFS Used%: 35.46%

DFS Remaining%: 58.82%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 23

Last contact: Tue Jul 03 15:06:50 CST 2018

Name: 192.168.11.133:50010 (slave1)

Hostname: slave1

Decommission Status : Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 792363753676 (737.95 GB)

Non DFS Used: 118847225403 (110.69 GB)

DFS Remaining: 1202300379385 (1.09 TB)

DFS Used%: 37.49%

DFS Remaining%: 56.89%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 21

Last contact: Tue Jul 03 15:06:50 CST 2018

Name: 192.168.11.135:50010 (slave3)

Hostname: slave3

Decommission Status : Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 752023101542 (700.38 GB)

Non DFS Used: 121481747963 (113.14 GB)

DFS Remaining: 1240006508959 (1.13 TB)

DFS Used%: 35.58%

DFS Remaining%: 58.67%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 26

Last contact: Tue Jul 03 15:06:50 CST 2018

Name: 192.168.11.136:50010 (slave4)

Hostname: slave4

Decommission Status : Normal

Configured Capacity: 1056617381888 (984.05 GB)

DFS Used: 800412266598 (745.44 GB)

Non DFS Used: 66495765928 (61.93 GB)

DFS Remaining: 189709349362 (176.68 GB)

DFS Used%: 75.75%

DFS Remaining%: 17.95%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 30

2、分析192.168.11.133

$ df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LVroot

                       17G   12G  4.9G  70% /

tmpfs                 7.8G     0  7.8G   0% /dev/shm

/dev/sda1             190M  125M   56M  70% /boot

/dev/mapper/VolGroup00-LVhome

                      477M  2.4M  449M   1% /home

/dev/mapper/VolGroup00-LVcloud

                      2.9G  247M  2.5G   9% /opt/cloud

/dev/mapper/VolGroup00-LVtmp

                      7.8G   19M  7.4G   1% /tmp

/dev/mapper/VolGroup00-LVvar

                       25G  786M   23G   4% /var

/dev/mapper/VolGroup1-LVdata1

                      985G  384G  551G  42% /data

/dev/mapper/VolGroup2-LVdata2

                      985G  365G  570G  40% /data2

數(shù)據(jù)統(tǒng)計：

/data和/data2為數(shù)據(jù)存放目錄

總的容量 = 985G + 985G = 1970G = 1.92T

已使用容量 = 384G + 365G = 749G

未使用容量 = 551G + 570G = 1121G

未顯示容量 = 1970G - （749G + 1121G）= 100G

/data目錄中的hadoop和hbase的解壓文件占用10G左右

dfs文件系統(tǒng)分析：

Name: 192.168.11.133:50010 (slave1)

Hostname: slave1

Decommission Status : Normal

Configured Capacity: 2113511358464 (1.92 TB)

DFS Used: 792363753676 (737.95 GB)

Non DFS Used: 118847225403 (110.69 GB)

DFS Remaining: 1202300379385 (1.09 TB)

DFS Used%: 37.49%

DFS Remaining%: 56.89%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 21

Last contact: Tue Jul 03 15:06:50 CST 2018

說明解析：

Configured Capacity:配置容量

DFS Used:DFS已使用的容量

Non DFS Used:非DFS使用的容量

DFS Remaining:DFS剩余的容量

數(shù)據(jù)統(tǒng)計：

Configured Capacity = 1.92 TB = 1970 GB

DFS Used = 737.95 GB

Non DFS Used = 110.69 GB

DFS Remaining = 1.09 TB = 1116.16 GB

DFS Used + Non DFS Used + DFS Remaining = 737.95 GB +  110.69 GB +  1116.16 GB = 1964.8 GB = Configured Capacity

所以得出結(jié)果：

du -h和./hdfs dfsadmin -report顯示的容量對應(yīng)關(guān)系為：

總的容量 ---> Configured Capacity

已使用容量 ---> DFS Used

未使用容量 ---> DFS Remaining

未顯示容量 + /data下非dfs文件 ---> Non DFS Used

非dfs文件指的是不是同過dfs文件系統(tǒng)上傳的文件，通過mv、cp等方式傳輸?shù)奈募?/p>

統(tǒng)計192.168.11.136和192.168.11.133類似。

3、尋找消失的100G

Non DFS Used占用了110.69GB，/data下非dfs文件有10GB左右，那么剩下的100GB去哪了？

查看保留空間

# tune2fs -l /dev/mapper/VolGroup1-LVdata1

Reserved block count:     13107148

13107148 * 4 / 1024 /1024 = 49.99

# tune2fs -l /dev/mapper/VolGroup2-LVdata2

Reserved block count:     13107148

13107148 * 4 / 1024 /1024 = 49.99

消失的100G就是兩個分區(qū)的保留空間之和

以上是“如何查看HDFS文件系統(tǒng)空間使用情況”這篇文章的所有內(nèi)容，感謝各位的閱讀！希望分享的內(nèi)容對大家有幫助，更多相關(guān)知識，歡迎關(guān)注億速云行業(yè)資訊頻道！