您好,登錄后才能下訂單哦!
py3nvml實(shí)現(xiàn)GPU相關(guān)信息讀取的示例分析,很多新手對(duì)此不是很清楚,為了幫助大家解決這個(gè)難題,下面小編將為大家詳細(xì)講解,有這方面需求的人可以來(lái)學(xué)習(xí)下,希望你能有所收獲。
在深度學(xué)習(xí)或者其他類型的GPU運(yùn)算過(guò)程中,對(duì)于GPU信息的監(jiān)測(cè)也是一個(gè)非常常用的功能。如果僅僅是使用系統(tǒng)級(jí)的GPU監(jiān)測(cè)工具,就沒(méi)辦法非常細(xì)致的去跟蹤每一步的顯存和使用率的變化。如果是用profiler,又顯得過(guò)于細(xì)致,而且環(huán)境配置、信息輸出和篩選并不是很方便。此時(shí)就可以考慮使用py3nvml這樣的工具,針對(duì)于GPU任務(wù)執(zhí)行的過(guò)程進(jìn)行細(xì)化的分析,有助于提升GPU的利用率和程序執(zhí)行的性能。
隨著模型運(yùn)算量的增長(zhǎng)和硬件技術(shù)的發(fā)展,使用GPU來(lái)完成各種任務(wù)的計(jì)算已經(jīng)漸漸成為算法實(shí)現(xiàn)的主流手段。而對(duì)于運(yùn)行期間的一些GPU的占用,比如每一步的顯存使用率等諸如此類的信息,就需要一些比較細(xì)致的GPU信息讀取的工具,這里我們重點(diǎn)推薦使用py3nvml來(lái)對(duì)python代碼運(yùn)行的一個(gè)過(guò)程進(jìn)行監(jiān)控。
一般大家比較常用的就是nvidia-smi
這個(gè)指令,來(lái)讀取GPU的使用率和顯存占用、驅(qū)動(dòng)版本等信息:
$ nvidia-smi Wed Jan 12 15:52:04 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.42.01 Driver Version: 470.42.01 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro RTX 4000 On | 00000000:03:00.0 On | N/A | | 30% 39C P8 20W / 125W | 538MiB / 7979MiB | 16% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Quadro RTX 4000 On | 00000000:A6:00.0 Off | N/A | | 30% 32C P8 7W / 125W | 6MiB / 7982MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 1643 G /usr/lib/xorg/Xorg 412MiB | | 0 N/A N/A 2940 G /usr/bin/gnome-shell 76MiB | | 0 N/A N/A 47102 G ...AAAAAAAAA= --shared-files 35MiB | | 0 N/A N/A 172424 G ...AAAAAAAAA= --shared-files 11MiB | | 1 N/A N/A 1643 G /usr/lib/xorg/Xorg 4MiB | +-----------------------------------------------------------------------------+
但是如果不使用profile僅僅使用nvidia-smi
這個(gè)指令的輸出的話,是沒(méi)有辦法非常細(xì)致的分析程序運(yùn)行過(guò)程中的變化的。這里順便推薦一個(gè)比較精致的跟nvidia-smi
用法非常類似的小工具:gpustat。這個(gè)工具可以直接使用pip進(jìn)行安裝和管理:
$ python3 -m pip install gpustat Collecting gpustat Downloading gpustat-0.6.0.tar.gz (78 kB) |████████████████████████████████| 78 kB 686 kB/s Requirement already satisfied: six>=1.7 in /home/dechin/.local/lib/python3.8/site-packages (from gpustat) (1.16.0) Collecting nvidia-ml-py3>=7.352.0 Downloading nvidia-ml-py3-7.352.0.tar.gz (19 kB) Requirement already satisfied: psutil in /home/dechin/.local/lib/python3.8/site-packages (from gpustat) (5.8.0) Collecting blessings>=1.6 Downloading blessings-1.7-py3-none-any.whl (18 kB) Building wheels for collected packages: gpustat, nvidia-ml-py3 Building wheel for gpustat (setup.py) ... done Created wheel for gpustat: filename=gpustat-0.6.0-py3-none-any.whl size=12617 sha256=4158e741b609c7a1bc6db07d76224db51cd7656a6f2e146e0b81185ce4e960ba Stored in directory: /home/dechin/.cache/pip/wheels/0d/d9/80/b6cbcdc9946c7b50ce35441cc9e7d8c5a9d066469ba99bae44 Building wheel for nvidia-ml-py3 (setup.py) ... done Created wheel for nvidia-ml-py3: filename=nvidia_ml_py3-7.352.0-py3-none-any.whl size=19191 sha256=70cd8ffc92286944ad9f5dc4053709af76fc0e79928dc61b98a9819a719f1e31 Stored in directory: /home/dechin/.cache/pip/wheels/b9/b1/68/cb4feab29709d4155310d29a421389665dcab9eb3b679b527b Successfully built gpustat nvidia-ml-py3 Installing collected packages: nvidia-ml-py3, blessings, gpustat Successfully installed blessings-1.7 gpustat-0.6.0 nvidia-ml-py3-7.352.0
使用的時(shí)候也是跟nvidia-smi非常類似的操作:
$ watch --color -n1 gpustat -cpu
返回結(jié)果如下所示:
Every 1.0s: gpustat -cpu ubuntu2004: Wed Jan 12 15:58:59 2022
ubuntu2004 Wed Jan 12 15:58:59 2022 470.42.01
[0] Quadro RTX 4000 | 39'C, 3 % | 537 / 7979 MB | root:Xorg/1643(412M) de
chin:gnome-shell/2940(75M) dechin:slack/47102(35M) dechin:chrome/172424(11M)
[1] Quadro RTX 4000 | 32'C, 0 % | 6 / 7982 MB | root:Xorg/1643(4M)
通過(guò)gpustat
返回的結(jié)果,包含了GPU的型號(hào)、使用率和顯存使用大小和GPU當(dāng)前的溫度等常規(guī)信息。
接下來(lái)正式看下py3nvml的安裝和使用方法,這是一個(gè)可以在python中實(shí)時(shí)查看和監(jiān)測(cè)GPU信息的一個(gè)庫(kù),可以通過(guò)pip來(lái)安裝和管理:
$ python3 -m pip install py3nvml Collecting py3nvml Downloading py3nvml-0.2.7-py3-none-any.whl (55 kB) |████████████████████████████████| 55 kB 650 kB/s Requirement already satisfied: xmltodict in /home/dechin/anaconda3/lib/python3.8/site-packages (from py3nvml) (0.12.0) Installing collected packages: py3nvml Successfully installed py3nvml-0.2.7
有一些框架為了性能的最大化,在初始化的時(shí)候就會(huì)默認(rèn)去使用到整個(gè)資源池里面的所有GPU卡,比如如下使用Jax來(lái)演示的一個(gè)案例:
In [1]: import py3nvml In [2]: from jax import numpy as jnp In [3]: x = jnp.ones(1000000000) In [4]: !nvidia-smi Wed Jan 12 16:08:32 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.42.01 Driver Version: 470.42.01 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro RTX 4000 On | 00000000:03:00.0 On | N/A | | 30% 41C P0 38W / 125W | 7245MiB / 7979MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Quadro RTX 4000 On | 00000000:A6:00.0 Off | N/A | | 30% 35C P0 35W / 125W | 101MiB / 7982MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 1643 G /usr/lib/xorg/Xorg 412MiB | | 0 N/A N/A 2940 G /usr/bin/gnome-shell 75MiB | | 0 N/A N/A 47102 G ...AAAAAAAAA= --shared-files 35MiB | | 0 N/A N/A 172424 G ...AAAAAAAAA= --shared-files 11MiB | | 0 N/A N/A 812125 C /usr/local/bin/python 6705MiB | | 1 N/A N/A 1643 G /usr/lib/xorg/Xorg 4MiB | | 1 N/A N/A 812125 C /usr/local/bin/python 93MiB | +-----------------------------------------------------------------------------+
在這個(gè)案例中我們只是在顯存中分配了一塊空間用于存儲(chǔ)一個(gè)向量,但是Jax在初始化之后,自動(dòng)占據(jù)了本地的2張GPU卡。根據(jù)Jax官方提供的方法,我們可以使用如下的操作配置環(huán)境變量,使得Jax只能看到其中的1張卡,這樣就不會(huì)擴(kuò)張:
In [1]: import os In [2]: os.environ["CUDA_VISIBLE_DEVICES"] = "1" In [3]: from jax import numpy as jnp In [4]: x = jnp.ones(1000000000) In [5]: !nvidia-smi Wed Jan 12 16:10:36 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.42.01 Driver Version: 470.42.01 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro RTX 4000 On | 00000000:03:00.0 On | N/A | | 30% 40C P8 19W / 125W | 537MiB / 7979MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Quadro RTX 4000 On | 00000000:A6:00.0 Off | N/A | | 30% 35C P0 35W / 125W | 7195MiB / 7982MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 1643 G /usr/lib/xorg/Xorg 412MiB | | 0 N/A N/A 2940 G /usr/bin/gnome-shell 75MiB | | 0 N/A N/A 47102 G ...AAAAAAAAA= --shared-files 35MiB | | 0 N/A N/A 172424 G ...AAAAAAAAA= --shared-files 11MiB | | 1 N/A N/A 1643 G /usr/lib/xorg/Xorg 4MiB | | 1 N/A N/A 813030 C /usr/local/bin/python 7187MiB | +-----------------------------------------------------------------------------+
可以看到結(jié)果中已經(jīng)是只使用了1張GPU卡,達(dá)到了我們的目的,但是這種通過(guò)配置環(huán)境變量來(lái)實(shí)現(xiàn)的功能還是著實(shí)不夠pythonic,因此py3nvml中也提供了這樣的功能,可以指定某一系列的GPU卡用于執(zhí)行任務(wù):
In [1]: import py3nvml In [2]: from jax import numpy as jnp In [3]: py3nvml.grab_gpus(num_gpus=1,gpu_select=[1]) Out[3]: 1 In [4]: x = jnp.ones(1000000000) In [5]: !nvidia-smi Wed Jan 12 16:12:37 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.42.01 Driver Version: 470.42.01 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Quadro RTX 4000 On | 00000000:03:00.0 On | N/A | | 30% 40C P8 20W / 125W | 537MiB / 7979MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 Quadro RTX 4000 On | 00000000:A6:00.0 Off | N/A | | 30% 36C P0 35W / 125W | 7195MiB / 7982MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 1643 G /usr/lib/xorg/Xorg 412MiB | | 0 N/A N/A 2940 G /usr/bin/gnome-shell 75MiB | | 0 N/A N/A 47102 G ...AAAAAAAAA= --shared-files 35MiB | | 0 N/A N/A 172424 G ...AAAAAAAAA= --shared-files 11MiB | | 1 N/A N/A 1643 G /usr/lib/xorg/Xorg 4MiB | | 1 N/A N/A 814673 C /usr/local/bin/python 7187MiB | +-----------------------------------------------------------------------------+
可以看到結(jié)果中也是只使用了1張GPU卡,達(dá)到了跟上一步的操作一樣的效果。
對(duì)于環(huán)境中可用的GPU,py3nvml的判斷標(biāo)準(zhǔn)就是在這個(gè)GPU上已經(jīng)沒(méi)有任何的進(jìn)程,那么這個(gè)就是一張可用的GPU卡:
In [1]: import py3nvml In [2]: free_gpus = py3nvml.get_free_gpus() In [3]: free_gpus Out[3]: [True, True]
當(dāng)然這里需要說(shuō)明的是,系統(tǒng)應(yīng)用在這里不會(huì)被識(shí)別,應(yīng)該是會(huì)判斷守護(hù)進(jìn)程。
跟nvidia-smi
非常類似的,py3nvml也可以在命令行中通過(guò)調(diào)用py3smi
來(lái)使用。值得一提的是,如果需要用nvidia-smi
來(lái)實(shí)時(shí)的監(jiān)測(cè)GPU的使用信息,往往是需要配合watch -n
來(lái)使用的,但是如果是py3smi
則不需要,直接用py3smi -l
就可以實(shí)現(xiàn)類似的功能。
$ py3smi -l 5 Wed Jan 12 16:17:37 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI Driver Version: 470.42.01 | +---------------------------------+---------------------+---------------------+ | GPU Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | +=================================+=====================+=====================+ | 0 30% 39C 8 19W / 125W | 537MiB / 7979MiB | 0% Default | | 1 30% 33C 8 7W / 125W | 6MiB / 7982MiB | 0% Default | +---------------------------------+---------------------+---------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU Owner PID Uptime Process Name Usage | +=============================================================================+ +-----------------------------------------------------------------------------+
可以看到略有區(qū)別的是,這里并不像nvidia-smi
列出來(lái)的進(jìn)程那么多,應(yīng)該是自動(dòng)忽略了系統(tǒng)進(jìn)程。
在py3nvml中把查看驅(qū)動(dòng)和型號(hào)的功能單獨(dú)列了出來(lái):
In [1]: from py3nvml.py3nvml import * In [2]: nvmlInit() Out[2]: <CDLL 'libnvidia-ml.so.1', handle 560ad4d07a60 at 0x7fd13aa52340> In [3]: print("Driver Version: {}".format(nvmlSystemGetDriverVersion())) Driver Version: 470.42.01 In [4]: deviceCount = nvmlDeviceGetCount() ...: for i in range(deviceCount): ...: handle = nvmlDeviceGetHandleByIndex(i) ...: print("Device {}: {}".format(i, nvmlDeviceGetName(handle))) ...: Device 0: Quadro RTX 4000 Device 1: Quadro RTX 4000 In [5]: nvmlShutdown()
這樣也不需要我們自己再去逐個(gè)的篩選,從靈活性和可擴(kuò)展性上來(lái)說(shuō)還是比較方便的。
這里同樣的也是把顯存的使用信息單獨(dú)列了出來(lái),不需要用戶再去單獨(dú)篩選這個(gè)信息,相對(duì)而言比較細(xì)致:
In [1]: from py3nvml.py3nvml import * In [2]: nvmlInit() Out[2]: <CDLL 'libnvidia-ml.so.1', handle 55ae42aadd90 at 0x7f39c700e040> In [3]: handle = nvmlDeviceGetHandleByIndex(0) In [4]: info = nvmlDeviceGetMemoryInfo(handle) In [5]: print("Total memory: {}MiB".format(info.total >> 20)) Total memory: 7979MiB In [6]: print("Free memory: {}MiB".format(info.free >> 20)) Free memory: 7441MiB In [7]: print("Used memory: {}MiB".format(info.used >> 20)) Used memory: 537MiB
如果把這些代碼插入到程序中,就可以獲悉每一步所占用的顯存的變化。
看完上述內(nèi)容是否對(duì)您有幫助呢?如果還想對(duì)相關(guān)知識(shí)有進(jìn)一步的了解或閱讀更多相關(guān)文章,請(qǐng)關(guān)注億速云行業(yè)資訊頻道,感謝您對(duì)億速云的支持。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。