您好,登錄后才能下訂單哦!
本篇內(nèi)容介紹了“Multipart Upload的presign怎么使用”的有關(guān)知識(shí),在實(shí)際案例的操作過程中,不少人都會(huì)遇到這樣的困境,接下來就讓小編帶領(lǐng)大家學(xué)習(xí)一下如何處理這些情況吧!希望大家仔細(xì)閱讀,能夠?qū)W有所成!
有項(xiàng)目需要在客戶端上傳數(shù)據(jù),但是出于安全考慮不能在客戶端本地任何位置存儲(chǔ)secret-key的信息,同時(shí)普通的presign上傳請(qǐng)求也有局限性,特別是當(dāng)文件很大的時(shí)候整個(gè)上傳效率非常低。因此需要調(diào)研Multipart Upload是否支持presign機(jī)制。(Multipart Upload是否支持presign,AWS及ceph官方都沒有明確說明)
整個(gè)Multipart Upload大致分為3個(gè)階段
1. initiate_multipart_upload:使用HTTP POST請(qǐng)求構(gòu)建初始化數(shù)據(jù),object的metadata、content-type這些都是在這個(gè)階段設(shè)置,請(qǐng)求成功以后的response里面回返回uploadID,后續(xù)使用這個(gè)uploadID進(jìn)行數(shù)據(jù)上傳操作。
2. upload_part:使用HTTP PUT請(qǐng)求構(gòu)建數(shù)據(jù)上傳操作,客戶端本地將大文件拆分成多個(gè)part文件,每個(gè)part使用相同的uploadID,按uploadNumber順序依次上傳數(shù)據(jù)(可以并行)。每個(gè)part文件上傳成功會(huì)返回對(duì)應(yīng)的etag來完成每個(gè)part文件的一致性校驗(yàn)。
3. compelte_upload:使用HTTP PUT請(qǐng)求構(gòu)建最后的合并請(qǐng)求操作,完成各個(gè)分塊的邏輯合并,注意返回的etag不具備一致性校驗(yàn)功能。
Multipart Upload特性總結(jié):
1.拆分大文件到多個(gè)part文件,實(shí)現(xiàn)超大文件的上傳。
2.拆分的part文件可以并行上傳,極大地提高了上傳效率。
3.Multipart Upload實(shí)現(xiàn)的是過程一致性校驗(yàn),而非最終一致性校驗(yàn)。
4.metadata和content-type一類的信息必須在initiate階段提前設(shè)置。
流程介紹:和普通Multipart Upload一樣先在client端實(shí)現(xiàn)文件的拆分,之后對(duì)每個(gè)part文件生成單獨(dú)的PreSignURL,最后使用生成的URL完成各個(gè)part文件的上傳(不再需要secret-key)。
分為兩部分,server_demo.py主要實(shí)現(xiàn)最關(guān)鍵的S3Sign,代碼如下
# -*- coding: utf-8 -*- import time import hmac from hashlib import sha1 as sha py3k = False try: from urlparse import urlparse, unquote from base64 import encodestring except: py3k = True from urllib.parse import urlparse, unquote from base64 import encodebytes as encodestring class S3PreSign(): def __init__(self, access_key, secret_key, service_url, bucket_name, object_name, upload_ID, expires): self.service_url = str(service_url) self.access_key = str(access_key) self.secret_key = str(secret_key) self.bucket_name = str(bucket_name) self.object_name = str(object_name) self.upload_ID = str(upload_ID) self.Expires = int(time.time()) + int(expires) def get_signature_str(self, sign_str): if py3k: key = self.secret_key.encode('utf-8') msg = sign_str.encode('utf-8') else: key = self.secret_key msg = sign_str h = hmac.new(key, msg, digestmod=sha) return (encodestring(h.digest()).strip()).replace('+', '%2b') def build_url(self, partNumber, Signature): url_ = "http://{bucket_name}.{service_url}/{object_name}?uploadId={uploadId}&partNumber={partNumber}&Expires={Expires}&AWSAccessKeyId={AWSAccessKeyId}&Signature={Signature}".format( bucket_name=self.bucket_name, service_url=self.service_url, object_name=self.object_name, uploadId=self.upload_ID, partNumber=partNumber, Expires=self.Expires, AWSAccessKeyId=self.access_key, Signature=Signature ) return url_ def build_url_with_partid(self, partMd5, partNumber): sign_str = "PUT\n{partMd5}\n\n{Expires}\n/{bucket_name}/{object_name}?partNumber={partNumber}&uploadId={uploadId}".format( partMd5=partMd5, Expires=self.Expires, bucket_name=self.bucket_name, object_name=self.object_name, partNumber=partNumber, uploadId=self.upload_ID) Signature_ = self.get_signature_str(sign_str) return self.build_url(partNumber, Signature_)
最后在client_demo.py中實(shí)現(xiàn)對(duì)應(yīng)的并行上傳用例,這里只是簡(jiǎn)單的說明過程,如果需要隱藏secret-key這些信息,需要將這個(gè)生成presignURL的過程封裝成一個(gè)獨(dú)立的服務(wù)接口,客戶端通過請(qǐng)求去獲取這些生成的presignURL。
# -*- coding: utf-8 -*- from server_demo import S3PreSign import requests from base64 import encodestring from hashlib import md5 import os from multiprocessing import Pool class S3client(): def __init__(self, part_num, uploadfile_path): self.part_num = part_num self.uploadfile_path = uploadfile_path def split_file(self): filelist = [] statinfo = os.stat(self.uploadfile_path) chunksize = statinfo.st_size / self.part_num print "File size: %d(MB)" % (statinfo.st_size / (1024 * 1024)) print self.uploadfile_path,chunksize with open(self.uploadfile_path, "rb") as f: index = 1 while True: chunk = f.read(chunksize) if (chunk): fn = "%s.part.%d" % (self.uploadfile_path, index) print "creating", fn with open(fn, "wb") as fw: fw.write(chunk) partMD5 = self.compute_hash(fn) tmp_ = {} tmp_[fn] = str(partMD5) filelist.append(tmp_) index = index + 1 else: break return filelist def compute_hash(self, filepath, buf_size=8192, size=None, hash_algorithm=md5): hash_obj = hash_algorithm() with open(filepath) as fp: spos = fp.tell() if size and size < buf_size: s = fp.read(size) else: s = fp.read(buf_size) while s: if not isinstance(s, bytes): s = s.encode('utf-8') hash_obj.update(s) if size: size -= len(s) if size <= 0: break if size and size < buf_size: s = fp.read(size) else: s = fp.read(buf_size) base64_digest = encodestring(hash_obj.digest()).decode('utf-8') if base64_digest[-1] == '\n': base64_digest = base64_digest[0:-1] return base64_digest def make_upload_list(self,S3Sign): upload_file_list = self.split_file() for f in upload_file_list: part_path = f.keys()[0] partMD5 = f.values()[0] partnum_ = f.keys()[0].split(".")[-1] url_ = S3Sign.build_url_with_partid(partMD5, partnum_) # print "Ready to upload {part_path} with URL={url_} MD5={partMD5}".format(part_path=part_path,url_=url_,partMD5=partMD5) yield (url_, part_path, partMD5) def multipart_upload_with_part(url_, part_file_path, partMD5): headers = {} headers["Content-MD5"] = partMD5 with open(part_file_path,'r') as fh: response = requests.put(url_, headers=headers, data=fh.read()) if response.status_code == 200: print "{} upload Sucessful !".format(part_file_path) if __name__ == "__main__": endpoint = 's3.ceph.work' access_key = '' secret_key = '' key_name = 'cosben-0.4.2.c4.zip' part_num = 6 #大文件切分?jǐn)?shù)量 expires = 300 #生成的presignURL有效時(shí)長(zhǎng) bucket_name = 'multi-upload' file_path = '/tmp/cosbench-0.4.2.c4.zip' upload_ID = '2~EAEhzt0luJqhV4KkZDGQH3CmegO00FX' #uploadID使用boto或者其他方法生成 processes_num = 2 #并行上傳進(jìn)程數(shù) s3sign = S3PreSign(access_key=access_key, secret_key=secret_key, service_url=endpoint, bucket_name=bucket_name, object_name=key_name, upload_ID=upload_ID, expires=expires) s3client = S3client(part_num,file_path) # s3client.split_file() upload_file_list = s3client.make_upload_list(s3sign) p = Pool(processes=processes_num) for i in upload_file_list: # print i p.apply_async(multipart_upload_with_part, (i[0], i[1], i[2],)) print 'Waiting for all subprocesses done...' p.close() p.join() print 'All subprocesses done.'
“Multipart Upload的presign怎么使用”的內(nèi)容就介紹到這里了,感謝大家的閱讀。如果想了解更多行業(yè)相關(guān)的知識(shí)可以關(guān)注億速云網(wǎng)站,小編將為大家輸出更多高質(zhì)量的實(shí)用文章!
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。