您好,登錄后才能下訂單哦!
這篇文章主要為大家展示了“HBase Compaction算法之ExploringCompactionPolicy怎么用”,內(nèi)容簡(jiǎn)而易懂,條理清晰,希望能夠幫助大家解決疑惑,下面讓小編帶領(lǐng)大家一起研究并學(xué)習(xí)一下“HBase Compaction算法之ExploringCompactionPolicy怎么用”這篇文章吧。
在0.98版本中,默認(rèn)的compaction算法換成了ExploringCompactionPolicy,之前是RatioBasedCompactionPolicy
ExploringCompactionPolicy繼承RatioBasedCompactionPolicy,重寫(xiě)了applyCompactionPolicy方法,applyCompactionPolicy是對(duì)minor compaction的選擇文件的策略算法。
applyCompactionPolicy方法內(nèi)容:
public List<StoreFile> applyCompactionPolicy(final List<StoreFile> candidates, boolean mightBeStuck, boolean mayUseOffPeak, int minFiles, int maxFiles) { //此ratio為后面算法使用,可設(shè)置非高峰時(shí)間段的ratio(默認(rèn):5.0)從而合并更多的數(shù)據(jù) final double currentRatio = mayUseOffPeak ? comConf.getCompactionRatioOffPeak() : comConf.getCompactionRatio(); // Start off choosing nothing. List<StoreFile> bestSelection = new ArrayList<StoreFile>(0); List<StoreFile> smallest = mightBeStuck ? new ArrayList<StoreFile>(0) : null; long bestSize = 0; long smallestSize = Long.MAX_VALUE; int opts = 0, optsInRatio = 0, bestStart = -1; // for debug logging // Consider every starting place. for (int start = 0; start < candidates.size(); start++) { // Consider every different sub list permutation in between start and end with min files. for (int currentEnd = start + minFiles - 1; currentEnd < candidates.size(); currentEnd++) { List<StoreFile> potentialMatchFiles = candidates.subList(start, currentEnd + 1); // Sanity checks if (potentialMatchFiles.size() < minFiles) { continue; } if (potentialMatchFiles.size() > maxFiles) { continue; } // Compute the total size of files that will // have to be read if this set of files is compacted. long size = getTotalStoreSize(potentialMatchFiles); // Store the smallest set of files. This stored set of files will be used // if it looks like the algorithm is stuck. if (mightBeStuck && size < smallestSize) { smallest = potentialMatchFiles; smallestSize = size; } if (size > comConf.getMaxCompactSize()) { continue; } ++opts; if (size >= comConf.getMinCompactSize() && !filesInRatio(potentialMatchFiles, currentRatio)) { continue; } ++optsInRatio; if (isBetterSelection(bestSelection, bestSize, potentialMatchFiles, size, mightBeStuck)) { bestSelection = potentialMatchFiles; bestSize = size; bestStart = start; } } } if (bestSelection.size() == 0 && mightBeStuck) { LOG.debug("Exploring compaction algorithm has selected " + smallest.size() + " files of size "+ smallestSize + " because the store might be stuck"); return new ArrayList<StoreFile>(smallest); } LOG.debug("Exploring compaction algorithm has selected " + bestSelection.size() + " files of size " + bestSize + " starting at candidate #" + bestStart + " after considering " + opts + " permutations with " + optsInRatio + " in ratio"); return new ArrayList<StoreFile>(bestSelection);
從代碼得知,主要算法如下:
從頭到尾遍歷文件,判斷所有符合條件的組合
選擇組合的文件數(shù)必須 >= minFiles(默認(rèn)值:3)
選擇組合的文件數(shù)必須 <= maxFiles(默認(rèn)值:10)
計(jì)算組合的文件總大小size,size必須 <= maxCompactSize(通過(guò)hbase.hstore.compaction.max.size配置,默認(rèn)值:LONG.MAX_VALUE,相當(dāng)于沒(méi)起作用,官方文檔里面說(shuō)只有覺(jué)得compaction經(jīng)常發(fā)生并且沒(méi)有多大的用時(shí),可以修改這個(gè)值)
組合的文件大小 < minCompactSize 則是符合要求,如果 >= minCompactSize ,還需要判斷filesInRatio
filesInRatio算法:FileSize(i) <= ( Sum(0,N,FileSize(_)) - FileSize(i) ) * Ratio,也就是說(shuō)組合里面的所有單個(gè)文件大小都必須滿足 singleFileSize <= (totalFileSize - singleFileSize) * currentRatio,此算法的意義是為了限制太大的compaction,選擇出來(lái)的文件不至于有一個(gè)很大的,應(yīng)該盡可能先合并一些小的大小相差不大的文件,代碼如下
private boolean filesInRatio(final List<StoreFile> files, final double currentRatio) { if (files.size() < 2) { return true; } long totalFileSize = getTotalStoreSize(files); for (StoreFile file : files) { long singleFileSize = file.getReader().length(); long sumAllOtherFileSizes = totalFileSize - singleFileSize; if (singleFileSize > sumAllOtherFileSizes * currentRatio) { return false; } } return true; }
尋找最有解,優(yōu)先選擇文件組合文件數(shù)多的,當(dāng)文件數(shù)一樣多時(shí)選擇文件數(shù)小的,此目的是為了盡可能合并更多的文件并且產(chǎn)生的IO越少越好
private boolean isBetterSelection(List<StoreFile> bestSelection, long bestSize, List<StoreFile> selection, long size, boolean mightBeStuck) { if (mightBeStuck && bestSize > 0 && size > 0) { // Keep the selection that removes most files for least size. That penaltizes adding // large files to compaction, but not small files, so we don't become totally inefficient // (might want to tweak that in future). Also, given the current order of looking at // permutations, prefer earlier files and smaller selection if the difference is small. final double REPLACE_IF_BETTER_BY = 1.05; double thresholdQuality = ((double)bestSelection.size() / bestSize) * REPLACE_IF_BETTER_BY; return thresholdQuality < ((double)selection.size() / size); } // Keep if this gets rid of more files. Or the same number of files for less io. return selection.size() > bestSelection.size() || (selection.size() == bestSelection.size() && size < bestSize); }
主要算法至此結(jié)束,下面說(shuō)說(shuō)其他細(xì)節(jié)及其優(yōu)化部分:
步驟6的ratio默認(rèn)值是1.2,但是打開(kāi)了非高峰時(shí)間段的優(yōu)化時(shí),可以有不同的值,非高峰的ratio默認(rèn)值是5.0,此優(yōu)化目的是為了在業(yè)務(wù)低估時(shí)可以合并更多的數(shù)據(jù),目前此優(yōu)化只能是天的小說(shuō)時(shí)間段,還不算靈活。
算法中關(guān)于mightBeStuck的邏輯部分,這個(gè)參數(shù)是用來(lái)表示是否有可能compaction會(huì)被卡住,它的狀態(tài)是 待選文件數(shù) - 正在做compaction的文件數(shù) + futureFiles(默認(rèn)值是0,有正在做compaction的文件時(shí)是1) >= hbase.hstore.blockingStoreFiles (默認(rèn)是10,此配置在flush中也會(huì)用到,以后分析flush的時(shí)候會(huì)補(bǔ)充),如果是true時(shí):
選擇文件算法還會(huì)去尋找一個(gè)最小解。在上文步驟4之前,會(huì)記錄一個(gè)文件大小最小的組合
isBetterSelection部分,算法改為 (bestSelection.size() / bestSize) * 1.05 < selection.size() / size,通過(guò)文件大小和文件數(shù)的比值去選擇一個(gè)合適的解
返回結(jié)果時(shí),沒(méi)有合適的最優(yōu)解或返回一個(gè)最小解。
mightBeStuck的優(yōu)化部分,相當(dāng)于保證在很多的文件數(shù)的情況下,也可以選出一個(gè)最小解去做compaction,而不用再讓文件繼續(xù)增長(zhǎng)下去直到有一個(gè)合適的組合出現(xiàn)。
此算法跟RatioBasedCompactionPolicy的區(qū)別,簡(jiǎn)單的說(shuō)就是RatioBasedCompactionPolicy是簡(jiǎn)單的從頭到尾遍歷StoreFile列表,遇到一個(gè)符合Ratio條件的序列就選定執(zhí)行Compaction。而ExploringCompactionPolicy則是從頭到尾遍歷的同時(shí)記錄下當(dāng)前最優(yōu),然后從中選擇一個(gè)全局最優(yōu)列表。
以上是“HBase Compaction算法之ExploringCompactionPolicy怎么用”這篇文章的所有內(nèi)容,感謝各位的閱讀!相信大家都有了一定的了解,希望分享的內(nèi)容對(duì)大家有所幫助,如果還想學(xué)習(xí)更多知識(shí),歡迎關(guān)注億速云行業(yè)資訊頻道!
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。