您好,登錄后才能下訂單哦!
這篇文章主要介紹了python如何實現(xiàn)每5分鐘從kafka中提取數(shù)據(jù)功能,具有一定借鑒價值,感興趣的朋友可以參考下,希望大家閱讀完這篇文章之后大有收獲,下面讓小編帶著大家一起了解一下。
代碼
import sys sys.path.append("..") from datetime import datetime from utils.kafka2file import KafkaDownloader import os """ 實現(xiàn)取kafka數(shù)據(jù),文件按照取數(shù)據(jù)的間隔命名 如每5分鐘從kafka取數(shù)據(jù)寫入文件中,文件名為當(dāng)前時間加5 """ TOPIC = "rtz_queue" HOSTS = "ip:9092,ip:9092" GROUP = "2001" def get_end_time(hour,minute,time_step): if (minute+time_step)%60<60: if (minute+time_step)%60<10: return str(hour+int((minute+time_step)/60))+":"+"0"+str((minute+time_step)%60) else: return str(hour+int((minute+time_step)/60))+":"+str((minute+time_step)%60) else: pass def kafkawritefile(time_step,time_num): start = datetime.now() downloader = KafkaDownloader(HOSTS, TOPIC, GROUP) i = 1 while(i<=time_num): end_time = get_end_time(start.hour, start.minute,i*time_step) end_time_file = end_time.replace(':', '_') outfile_path = "/data/tmp/" + end_time_file + ".csv" if os.path.exists(outfile_path): os.remove(outfile_path) writefile = open(outfile_path, 'a+', encoding='utf-8') for msg in downloader.message(): curr_time = datetime.now() curr_time = str(curr_time) split_curr_time = curr_time.split(' ') curr_time_str = split_curr_time[1][0:5] if curr_time_str >= str(end_time): break i += 1 if __name__=='__main__': time_step = 15 time_num = 1 kafkawritefile(time_step,time_num)
感謝你能夠認真閱讀完這篇文章,希望小編分享的“python如何實現(xiàn)每5分鐘從kafka中提取數(shù)據(jù)功能”這篇文章對大家有幫助,同時也希望大家多多支持億速云,關(guān)注億速云行業(yè)資訊頻道,更多相關(guān)知識等著你來學(xué)習(xí)!
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進行舉報,并提供相關(guān)證據(jù),一經(jīng)查實,將立刻刪除涉嫌侵權(quán)內(nèi)容。