您好,登錄后才能下訂單哦!
溫馨提示:要看高清無(wú)碼套圖,請(qǐng)使用手機(jī)打開(kāi)并單擊圖片放大查看。
1.文檔編寫(xiě)目的
Anaconda是一個(gè)用于科學(xué)計(jì)算的Python發(fā)行版,支持 Linux, Mac, Windows系統(tǒng),提供了包管理與環(huán)境管理的功能,可以很方便地解決多版本python并存、切換以及各種第三方包安裝問(wèn)題。
在企業(yè)集群無(wú)外網(wǎng)環(huán)境下,通過(guò)搭建企業(yè)私有的Python源,可以使用pip工具方便的安裝,以解決Anaconda環(huán)境中沒(méi)有的依賴包的問(wèn)題。
本文檔講述如何在CDH集群離線安裝Anaconda、搭建Python私有源及pip工具使用。
1.如何在CDH集群上安裝Anaconda
2.如何搭建python私有源
3.pip工具的使用
1.操作系統(tǒng):CentOS6.5
2.CM和CDH版本為5.12.1
3.采用root用戶操作
1.CDH集群運(yùn)行正常
2.Apache服務(wù)正常運(yùn)行
2.CDH集群安裝Anaconda
使用Parcel包的方式在CDH集群安裝Anaconda,下載地址:
https://repo.continuum.io/pkgs/misc/parcels/
根據(jù)自己操作系統(tǒng)版本選擇Parcel包,本文檔是在CentOS6.5版本下操作
2.1部署Anaconda的Parcel包http訪問(wèn)
1.下載Anaconda的Parcel包,下載地址如下
https://repo.continuum.io/pkgs/misc/parcels/Anaconda-4.2.0-el6.parcel
https://repo.continuum.io/pkgs/misc/parcels/Anaconda-4.2.0-el6.parcel.sha
https://repo.continuum.io/pkgs/misc/parcels/manifest.json
2.將上述3個(gè)文件下載至HTTP所在服務(wù)器的/var/×××w/html/anaconda4.2.0目錄下
[root@ip-172-31-6-148 html]# mkdir -p anaconda4.2.0/
[root@ip-172-31-6-148 html]# cd anaconda4.2.0/
[root@ip-172-31-6-148 anaconda4.2.0]# ll
total 541416
-rw-r--r-- 1 root root 554396033 Jun 19 17:47 Anaconda-4.2.0-el6.parcel
-rw-r--r-- 1 root root 41 Jul 7 19:43 Anaconda-4.2.0-el6.parcel.sha
-rw-r--r-- 1 root root 3890 Jul 7 19:43 manifest.json
[root@ip-172-31-6-148 anaconda4.2.0]#
3.驗(yàn)證http是否正常訪問(wèn)
2.2安裝Anaconda
1.配置Anaconda的parcels包訪問(wèn)地址
2.點(diǎn)擊下載、分配并激活
Anaconda默認(rèn)安裝在/opt/cloudera/parcels/Anaconda目錄,在集群所有節(jié)點(diǎn)配置Anaconda的環(huán)境變量,操作如下:
export ANACONDA_HOME=/opt/cloudera/parcels/Anaconda
export PATH=$ANACONDA_HOME/bin:$PATH
2.執(zhí)行source命令,使環(huán)境變量立即生效
[root@ip-172-31-6-148 anaconda4.2.0]# source /etc/profile
3.驗(yàn)證環(huán)境變量是否配置成功
[root@ip-172-31-6-148 anaconda4.2.0]# python
Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:42:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>>
3.搭建Pyton私有源
1.下載pip2pi安裝包,下載地址:
https://codeload.github.com/wolever/pip2pi/zip/master
2.下載Python源碼依賴包(tar.gz/zip/whl等格式安裝包)
3.1安裝pip2pi工具
[root@ip-172-31-6-148 ~]# unzip master
2.使用Python命令安裝pip2pi
[root@ip-172-31-6-148 python]# cd pip2pi-master/
[root@ip-172-31-6-148 pip2pi-master]# ll
total 44
-rw-r--r-- 1 root root 2797 Mar 19 15:51 CHANGELOG.txt
drwxr-xr-x 2 root root 4096 Mar 19 15:51 libpip2pi
-rw-r--r-- 1 root root 1649 Mar 19 15:51 LICENSE.txt
-rw-r--r-- 1 root root 19 Mar 19 15:51 MANIFEST.in
-rw-r--r-- 1 root root 5080 Mar 19 15:51 README.rst
-rwxr-xr-x 1 root root 84 Mar 19 15:51 run-tests
-rw-r--r-- 1 root root 20 Mar 19 15:51 setup.cfg
-rw-r--r-- 1 root root 1422 Mar 19 15:51 setup.py
drwxr-xr-x 5 root root 4096 Mar 19 15:51 tests
-rw-r--r-- 1 root root 194 Mar 19 15:51 tox.ini
[root@ip-172-31-6-148 pip2pi-master]# python setup.py install
3.測(cè)試pip2pi和dir2pi命令是否安裝成功
3.2部署依賴包
[root@ip-172-31-6-148 html]# mkdir python-packages
[root@ip-172-31-6-148 html]# pwd
/var/×××w/html
[root@ip-172-31-6-148 html]# ll
total 32
drwxr-xr-x 2 root root 4096 Sep 11 15:52 python-packages
[root@ip-172-31-6-148 html]#
2.將本地下載的Python依賴包上傳至服務(wù)器的/var/×××w/html/python-packages目錄
[root@ip-172-31-6-148 python-packages]# ll
total 1780
-rw-r--r-- 1 root root 2421 Sep 24 2016 bit_array-0.1.0.tar.gz
-rw-r--r-- 1 root root 1254 Sep 12 07:18 enum-compat-0.0.2.tar.gz
-rw-r--r-- 1 root root 339017 Sep 12 07:27 pymongo-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 368294 Sep 12 07:25 pymongo-3.5.1-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 799601 Sep 12 07:04 Routes-1.12.3.tar.gz
-rw-r--r-- 1 root root 30409 Apr 23 2016 sasl-0.2.1.tar.gz
-rw-r--r-- 1 root root 29630 Apr 21 2016 six-1.10.0.tar.gz
-rw-r--r-- 1 root root 87832 Jan 14 2017 thrift-0.10.0.zip
-rw-r--r-- 1 root root 3885 Jun 9 2016 thrift_sasl-0.2.1.tar.gz
-rw-r--r-- 1 root root 140861 Dec 3 2016 ym-impyla-0.14.0.tar.gz
3.使用dir2pi命令建立包索引
[root@ip-172-31-6-148 python-packages]# dir2pi /var/×××w/html/python-packages/
[root@ip-172-31-6-148 python-packages]# ll
total 1784
-rw-r--r-- 1 root root 2421 Sep 24 2016 bit_array-0.1.0.tar.gz
-rw-r--r-- 1 root root 1254 Sep 12 07:18 enum-compat-0.0.2.tar.gz
-rw-r--r-- 1 root root 339017 Sep 12 07:27 pymongo-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 368294 Sep 12 07:25 pymongo-3.5.1-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 799601 Sep 12 07:04 Routes-1.12.3.tar.gz
-rw-r--r-- 1 root root 30409 Apr 23 2016 sasl-0.2.1.tar.gz
drwxr-xr-x 11 root root 4096 Sep 12 08:06 simple
-rw-r--r-- 1 root root 29630 Apr 21 2016 six-1.10.0.tar.gz
-rw-r--r-- 1 root root 87832 Jan 14 2017 thrift-0.10.0.zip
-rw-r--r-- 1 root root 3885 Jun 9 2016 thrift_sasl-0.2.1.tar.gz
-rw-r--r-- 1 root root 140861 Dec 3 2016 ym-impyla-0.14.0.tar.gz
[root@ip-172-31-6-148 python-packages]#
4.驗(yàn)證Http是否正常訪問(wèn),訪問(wèn)地址
http://ip-172-31-6-148.fayson.com/python-packages/simple/
至此Python的私有源服務(wù)器搭建完成。
3.3配置pip命令使用私有源
1.在當(dāng)前用戶的home目錄(即“~”)目錄,創(chuàng)建.pip目錄
[root@ip-172-31-6-148 ~]# mkdir -p .pip
[root@ip-172-31-6-148 ~]# pwd
/root
[root@ip-172-31-6-148 ~]#
2.在~/.pip目錄下新建pip.conf文件,文件內(nèi)容如下
[root@ip-172-31-6-148 ~]# cd .pip/
[root@ip-172-31-6-148 .pip]# pwd
/root/.pip
[root@ip-172-31-6-148 .pip]# vim pip.conf
[global]
index-url = http://ip-172-31-6-148.fayson.com/python-packages/simple
[install]
trusted-host = ip-172-31-6-148.fayson.com
3.測(cè)試pip命令是否從私有源下載依賴包
[root@ip-172-31-6-148 .pip]# pip install bit-array
Collecting bit-array
Downloading http://ip-172-31-6-148.fayson.com/python-packages/simple/bit-array/bit-array-0.1.0.tar.gz
Building wheels for collected packages: bit-array
Running setup.py bdist_wheel for bit-array ... done
Stored in directory: /root/.cache/pip/wheels/7a/7f/04/3bdc776730b06ce5bedd46f9023d263a32f82941e1365a054f
Successfully built bit-array
Installing collected packages: bit-array
Successfully installed bit-array-0.1.0
[root@ip-172-31-6-148 .pip]#
可以看到上圖標(biāo)注部分顯示從剛搭建好的源下載依賴包。
4.pip使用
1.指定版本安裝package
[root@ip-172-31-6-148 ~] pip install SomePackage # 最新版本
[root@ip-172-31-6-148 ~] pip install SomePackage==1.0.4 # 具體的版本
[root@ip-172-31-6-148 ~] pip install 'SomePackage>=1.0.4' # 最低版本
2.批量安裝packages
創(chuàng)建文本文件requirements.txt,內(nèi)容如下
[root@ip-172-31-6-148 ~]# vim requirements.txt
six
bit_array
thrift==0.10.0
使用pip批量安裝
[root@ip-172-31-6-148 ~]# pip install -r requirements.txt
Collecting six (from -r requirements.txt (line 1))
Collecting bit_array (from -r requirements.txt (line 2))
Collecting thrift==0.10.0 (from -r requirements.txt (line 3))
Downloading http://ip-172-31-6-148.fayson.com/python-packages/simple/thrift/thrift-0.10.0.zip (87kB)
100% |████████████████████████████████| 92kB 58.1MB/s
Building wheels for collected packages: thrift
Running setup.py bdist_wheel for thrift ... done
Stored in directory: /root/.cache/pip/wheels/21/53/2d/121a953df877103ba4f2fe9041be0d28d6979438c6dd7b2b15
Successfully built thrift
Installing collected packages: six, bit-array, thrift
Successfully installed bit-array-0.1.0 six-1.10.0 thrift-0.10.0
[root@ip-172-31-6-148 ~]#
3.輸出當(dāng)前python環(huán)境安裝的所有package及版本到指定文件
[root@ip-172-31-6-148 ~]# pip freeze > requirements.txt
4.卸載package
[root@ip-172-31-6-148 ~]# pip uninstall six
5.列出當(dāng)前Python環(huán)境安裝的所有package
[root@ip-172-31-6-148 ~]# pip list --format=columns
Package Version
---------------------------------- -----------------------
alabaster 0.7.9
anaconda-clean 1.0
anaconda-client 1.5.1
anaconda-navigator 1.3.1
argcomplete 1.0.0
astroid 1.4.7
astropy 1.2.1
Babel 2.3.4
backports-abc 0.4
6.查看安裝的package詳細(xì)信息
[root@ip-172-31-6-148 ~]# pip show six
Name: six
Version: 1.10.0
Summary: Python 2 and 3 compatibility utilities
Home-page: http://pypi.python.org/pypi/six/
Author: Benjamin Peterson
Author-email: benjamin@python.org
License: MIT
Location: /opt/cloudera/parcels/Anaconda-4.2.0/lib/python2.7/site-packages
Requires:
[root@ip-172-31-6-148 ~]#
5.備注
package下載方式
在本地安裝pip2pi工具,可以通過(guò)pip2tgz命令批量下載package
six
bit_array
thrift==0.10.0
由于pip2tgz是通過(guò)將參數(shù)傳遞給pip調(diào)用來(lái)完成包的下載,所以可以指定任何pip可以識(shí)別的參數(shù)格式。
2.使用pip2tgz命令進(jìn)行批量下載
[root@ip-172-31-6-148 .pip]# pip2tgz /root/python/ -r /root/requirements.txt
注意:pip命令默認(rèn)使用的是國(guó)外的源下載package,如果國(guó)外源不穩(wěn)定則可以參考3.3章節(jié)配置國(guó)內(nèi)源。
6.常見(jiàn)問(wèn)題
[root@ip-172-31-6-148 .pip]# pip install six
Collecting six
The repository located at ip-172-31-6-148.fayson.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host ip-172-31-6-148.fayson.com'.
Could not find a version that satisfies the requirement six (from versions: )
No matching distribution found for six
[root@ip-172-31-6-148 .pip]#
解決方法:
在pip.conf文件中增加如下配置:
trusted-host = ip-172-31-6-148.fayson.com
醉酒鞭名馬,少年多浮夸! 嶺南浣溪沙,嘔吐酒肆下!摯友不肯放,數(shù)據(jù)玩的花!
溫馨提示:要看高清無(wú)碼套圖,請(qǐng)使用手機(jī)打開(kāi)并單擊圖片放大查看
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。