您好,登錄后才能下訂單哦!
1. 場景描述
一直做java,因項(xiàng)目原因,需要封裝一些經(jīng)典的算法到平臺(tái)上去,就一邊學(xué)習(xí)python,一邊網(wǎng)上尋找經(jīng)典算法代碼,今天介紹下經(jīng)典的K-means聚類算法,算法原理就不介紹了,只從代碼層面進(jìn)行介紹,包含:rest接口、連接mpp數(shù)據(jù)庫、回傳json數(shù)據(jù)、下載圖片及數(shù)據(jù)。
2. 解決方案
2.1 項(xiàng)目套路
(1)python經(jīng)典算法是單獨(dú)的服務(wù)器部署,提供rest接口出來,供java平臺(tái)調(diào)用,交互的方式是http+json;
(2)數(shù)據(jù)從mpp數(shù)據(jù)庫-Greenplum中獲取;
(3)返回的數(shù)據(jù)包括三個(gè):1是生成聚類圖片的地址;2是聚類項(xiàng)目完整數(shù)據(jù)地址;3是返回給前端的200條json預(yù)覽數(shù)據(jù)。
2.2 restapi類
分兩個(gè)類,第一個(gè)是restapi類,封裝rest接口類,其他的經(jīng)典算法在這里都有對(duì)應(yīng)的方法,是個(gè)公共類。
完整代碼:
# -*- coding: utf-8 -*- from flask import Flask, request, send_from_directory from k_means import exec import logging app = Flask(__name__) #1.服務(wù)器上更改為服務(wù)器地址,用于存放數(shù)據(jù) dirpath = 'E:\\ruanjianlaowang' #2. 測(cè)試連通性,軟件老王 @app.route('/') def index(): return "Hello, World!" #3. k-means算法 軟件老王 @app.route('/getKmeansInfoByLaowang', methods=['POST']) def getKmeansInfoByLaowang(): try: result = exec(request.get_json(), dirpath) except IndexError as e: logging.error(str(e)) return 'exception:' + str(e) except KeyError as e: logging.error(str(e)) return 'exception:' + str(e) except ValueError as e: logging.error(str(e)) return 'exception:' + str(e) except Exception as e: logging.error(str(e)) return 'exception:' + str(e) else: return result #4.文件下載(圖片及csv) @app.route("/<path:filename>") def getImages(filename): return send_from_directory(dirpath, filename, as_attachment=True) #5.啟動(dòng) if __name__ == '__main__': app.run(host="0.0.0.0", port=5000, debug=True)
代碼說明:
使用的是第三方的flask提供的rest服務(wù)
(1)服務(wù)器上更改為服務(wù)器地址,用于存放數(shù)據(jù)
(2)測(cè)試連通性,軟件老王
(3)k-means算法 軟件老王
(4)文件下載(圖片及csv)
(5)啟動(dòng)
2.3 k-means算法類
完整代碼:
import pandas as pd import dbgp as dbgp from pandas.io import json from numpy import * import matplotlib.pyplot as plt import numpy as np plt.switch_backend('agg') import logging # 執(zhí)行 軟件老王 def exec(params, dirpath): #1.獲取參數(shù),軟件老王 sql = params.get("sql") xlines = params.get("xlines") ylines = params.get("ylines") xlinesname = params.get("xlinesname") ylinesname = params.get("ylinesname") grouplinesname = params.get("grouplinesname") times = int(params.get("times")) groupnum = int(params.get("groupnum")) url = params.get("url") name = params.get("name") #2. 校驗(yàn)是否為空,軟件老王 flag = checkparam(sql, xlines, ylines, times, groupnum) if not flag is None and len(flag) != 0: return flag #3. 從數(shù)據(jù)庫獲取數(shù)據(jù),軟件老王 try: data = dbgp.queryGp(sql) except IndexError: return sql except KeyError: return sql except ValueError: return sql except Exception: return sql if data.empty: return "exception:此數(shù)據(jù)集無數(shù)據(jù),請(qǐng)確認(rèn)后重試" #4 調(diào)用第三方sklearn的KMeans聚類算法,軟件老王 # data_zs = 1.0 * (data - data.mean()) / data.std() 數(shù)據(jù)標(biāo)準(zhǔn)化,不需要標(biāo)準(zhǔn)話 from sklearn.cluster import KMeans model = KMeans(n_clusters=groupnum, n_jobs=4, max_iter=times) model.fit(data) # 開始聚類 return export(model, data, data, url, dirpath, name,grouplinesname,xlines, ylines,xlinesname,ylinesname) # 5.生成導(dǎo)出excel 軟件老王 def export(model, data, data_zs, url, dirpath, name,grouplinesname,xlines, ylines,xlinesname,ylinesname): # #詳細(xì)輸出原始數(shù)據(jù)及其類別 detail_data = pd.DataFrame().append(data) if not grouplinesname is None and len(grouplinesname) != 0: detail_data.columns = grouplinesname.split(',') r_detail_new = pd.concat([detail_data, pd.Series(model.labels_, index=detail_data.index)], axis=1) # 詳細(xì)輸出每個(gè)樣本對(duì)應(yīng)的類別 r_detail_new.columns = list(detail_data.columns) + [u'聚類類別'] # 重命名表頭 outputfile = dirpath + name + '.csv' r_detail_new.to_csv(outputfile, encoding='utf_8_sig') # 保存結(jié)果 #重命名表頭 r1 = pd.Series(model.labels_).value_counts() # 統(tǒng)計(jì)各個(gè)類別的數(shù)目 r2 = pd.DataFrame(model.cluster_centers_) # 找出聚類中心 r = pd.concat([r2, r1], axis=1) # 橫向連接(0是縱向),得到聚類中心對(duì)應(yīng)的類別下的數(shù)目 r.columns = list(data.columns) + [u'類別數(shù)目'] # 重命名表頭 return generateimage(r, data_zs, url, dirpath, name,model,xlines, ylines,xlinesname,ylinesname) #6.生成圖片及返回json,軟件老王 def generateimage(r, data_zs, url, dirpath, name,model,xlines, ylines,xlinesname,ylinesname): image = dirpath + name + '.jpg' #6.1 中文處理,軟件老王 plt.rcParams['font.sans-serif'] = ['simhei'] plt.rcParams['font.family'] = 'sans-serif' plt.rcParams['axes.unicode_minus'] = False # 6.2 畫圖,生成圖片,軟件老王 labels = model.labels_ centers = model.cluster_centers_ data_zs['label'] = labels data_zs['label'] = data_zs['label'].astype(np.int) # 圖標(biāo)集合 markers = ['o', 's', '+', 'x', '^', 'v', '<', '>'] colors = ['b', 'c', 'g', 'k', 'm', 'r', 'y'] symbols = [] for m in markers: for c in colors: symbols.append((m, c)) # 畫每個(gè)類別的散點(diǎn)及質(zhì)心 for i in range(0, len(centers)): df_i = data_zs.loc[data_zs['label'] == i] symbol = symbols[i] center = centers[i] x = df_i[xlines].values.tolist() y = df_i[ylines].values.tolist() plt.scatter(x, y, marker=symbol[0], color=symbol[1], s=10) plt.scatter(center[0], center[1], marker='*', color=symbol[1], s=50) plt.title(name) plt.xlabel(xlinesname) plt.ylabel(ylinesname) plt.savefig(image, dpi=150) plt.clf() plt.close(0) # 6.3 返回json數(shù)據(jù)給前端展示,軟件老王 result = {} result['image_url'] = url + '/' + name + '.jpg' result['details_url'] = url + '/' + name + '.csv' result['data'] = r[:200] #顯示200,多的話,相當(dāng)于預(yù)覽 result = json.dumps(result, ensure_ascii=False) result = result.replace('\\', '') return result def checkparam(sql, xlines, ylines, times, groupnum): if sql is None or sql.strip() == '' or len(sql.strip()) == 0: return "數(shù)據(jù)集或聚類數(shù)據(jù)列,不能為空" if xlines is None or xlines.strip() == '' or len(xlines.strip()) == 0: return "X軸,不能為空" if ylines is None or ylines.strip() == '' or len(ylines.strip()) == 0: return "Y軸,不能為空" if times is None or times <= 0: return "聚類個(gè)數(shù),不能為空或小于等于0" if groupnum is None or groupnum <= 0: return "迭代次數(shù),不能為空或小于等于0"
代碼說明:
(1)獲取參數(shù),軟件老王;
(2)校驗(yàn)是否為空,軟件老王;
(3)從數(shù)據(jù)庫獲取數(shù)據(jù),軟件老王;
(4)第三方sklearn的KMeans聚類算法,軟件老王;
(5)生成導(dǎo)出excel 軟件老王
(6)生成圖片及返回json,軟件老王
(6.1) 中文處理,軟件老王
(6.2) 畫圖,生成圖片,軟件老王
(6.3) 返回json數(shù)據(jù)給前端展示,軟件老王
2.4 執(zhí)行效果
2.4.1 json返回
{"image_url":"http://10.192.168.1:5000/ruanjianlaowang_65652.jpg","details_url":"http://10.192.168.1:5000/ruanjianlaowang_65652.csv","data":{"empno":{"0":7747.2,"1":7699.625,"2":7839.0},"mgr":{"0":7729.8,"1":7745.25,"2":7566.0},"sal":{"0":2855.0,"1":1218.75,"2":5000.0},"comm":{"0":29.5110766,"1":117.383964625,"2":31.281453},"deptno":{"0":20.0,"1":25.0,"2":10.0},"類別數(shù)目":{"0":5,"1":8,"2":1}}}
2.4.2 返回圖片
2.4.3 返回的數(shù)據(jù)
另外說明: 目前項(xiàng)目環(huán)境上用的是8核16G的虛擬機(jī),執(zhí)行數(shù)據(jù)量是30萬,運(yùn)行狀況良好。
以上就是本文的全部內(nèi)容,希望對(duì)大家的學(xué)習(xí)有所幫助,也希望大家多多支持億速云。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場,如果涉及侵權(quán)請(qǐng)聯(lián)系站長郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。