您好,登錄后才能下訂單哦!
本篇文章為大家展示了HDFS中reportWrittenBlock函數(shù)的作用是什么,內(nèi)容簡(jiǎn)明扼要并且容易理解,絕對(duì)能使你眼前一亮,通過(guò)這篇文章的詳細(xì)介紹希望你能有所收獲。
/**
* The client can report in a set written blocks that it wrote.
* These blocks are reported via the client instead of the datanode
* to prevent weird heartbeat race conditions.
*/
public void reportWrittenBlock(LocatedBlock lb) throws IOException {
Block b = lb.getBlock();//獲取完成的這個(gè)Block信息
DatanodeInfo targets[] = lb.getLocations();//獲取節(jié)點(diǎn)信息
for (int i = 0; i < targets.length; i++) {
namesystem.blockReceived(b, targets[i].getName());//對(duì)于每個(gè)DataNode來(lái)說(shuō),都要調(diào)用一次此函數(shù)
}
}
C1:2014-12-19 18:26:00 C2:2014-12-19 18:59:00 C3:2014-12-19 19:03:00
=========================
那么,接下來(lái)就是理解 namesystem.blockReceived(b, targets[i].getName());了。
/**
* The given node is reporting that it received a certain block.
*/
public synchronized void blockReceived(Block block, UTF8 name) {
DatanodeInfo node = (DatanodeInfo) datanodeMap.get(name);//獲取對(duì)應(yīng)的datanode
if (node == null) {//為空可不行
throw new IllegalArgumentException("Unexpected exception. Got blockReceived message from node " + name + ", but there is no info for " + name);
}
//
// Modify the blocks->datanode map
//
addStoredBlock(block, node);//下面兩行是來(lái)執(zhí)行block和node的一個(gè)映射。
//
// Supplement node's blockreport
//
node.addBlock(block);//同上
}
C1:2014-12-19 19:11:00 C2:2014-12-19 19:11:00 C3:2014-12-19 19:12:00
===============那么接下來(lái)還有2個(gè)函數(shù)需要攻破,分別是addStoredBlock和node.addBlock(block);
后面一個(gè)函數(shù)非常簡(jiǎn)單,不細(xì)講,所以就剩下最后一個(gè)函數(shù)了!
addStoredBlock(block, node);的執(zhí)行過(guò)程如下:
synchronized void addStoredBlock(Block block, DatanodeInfo node) {
TreeSet containingNodes = (TreeSet) blocksMap.get(block);//獲取當(dāng)前block已經(jīng)存在的datanode信息
if (containingNodes == null) {//這里保證肯定存在datanode集合,不保證一定有節(jié)點(diǎn)在內(nèi)
containingNodes = new TreeSet();
blocksMap.put(block, containingNodes);
}
if (! containingNodes.contains(node)) {//根據(jù)需要決定是否加入此datanode信息
containingNodes.add(node);
} else {
LOG.info("Redundant addStoredBlock request received for block " + block + " on node " + node);
}
//接下來(lái)的邏輯是確定是否需要重新備份
synchronized (neededReplications) {//鎖定neededReplications
if (dir.isValidBlock(block)) {//不懂這一句
if (containingNodes.size() >= this.desiredReplication) {//如果已經(jīng)超過(guò)最大備份個(gè)數(shù)
neededReplications.remove(block);//刪除此block
pendingReplications.remove(block);//刪除此block
} else if (containingNodes.size() < this.desiredReplication) {
if (! neededReplications.contains(block)) {
neededReplications.add(block);//否則表示需要重新備份,這代碼寫(xiě)的真夠差的。。。
}
}
//
// Find how many of the containing nodes are "extra", if any.
// If there are any extras, call chooseExcessReplicates() to
// mark them in the excessReplicateMap.
//
//也有可能一個(gè)block存儲(chǔ)的datanode節(jié)點(diǎn)數(shù)太多了,同樣要?jiǎng)h除這些block
Vector nonExcess = new Vector();//構(gòu)造一個(gè)空的Vector
for (Iterator it = containingNodes.iterator(); it.hasNext(); ) {
DatanodeInfo cur = (DatanodeInfo) it.next();//對(duì)于當(dāng)前節(jié)點(diǎn)來(lái)說(shuō)
TreeSet excessBlocks = (TreeSet) excessReplicateMap.get(cur.getName());//取到當(dāng)前節(jié)點(diǎn)的多余塊信息
if (excessBlocks == null || ! excessBlocks.contains(block)) {//如果之前沒(méi)有標(biāo)志在這個(gè)節(jié)點(diǎn)的多余塊信息里
nonExcess.add(cur);//則表明當(dāng)前節(jié)點(diǎn)存儲(chǔ)了這個(gè)block
}
}
if (nonExcess.size() > this.maxReplication) {//如果超過(guò)了最大備份數(shù)
chooseExcessReplicates(nonExcess, block, this.maxReplication);//選擇若干來(lái)消除塊
}
}
}
}
void chooseExcessReplicates(Vector nonExcess, Block b, int maxReps) {
while (nonExcess.size() - maxReps > 0) {//如果還有需要
int chosenNode = r.nextInt(nonExcess.size());//隨機(jī)選擇一個(gè)節(jié)點(diǎn)
DatanodeInfo cur = (DatanodeInfo) nonExcess.elementAt(chosenNode);
nonExcess.removeElementAt(chosenNode);//獲取這個(gè)節(jié)點(diǎn)
TreeSet excessBlocks = (TreeSet) excessReplicateMap.get(cur.getName());
if (excessBlocks == null) {
excessBlocks = new TreeSet();
excessReplicateMap.put(cur.getName(), excessBlocks);
}
excessBlocks.add(b);//加入此block到excessReplicateMap
//
// The 'excessblocks' tracks blocks until we get confirmation
// that the datanode has deleted them; the only way we remove them
// is when we get a "removeBlock" message.
//
// The 'invalidate' list is used to inform the datanode the block
// should be deleted. Items are removed from the invalidate list
// upon giving instructions to the namenode.
//
Vector invalidateSet = (Vector) recentInvalidateSets.get(cur.getName());
if (invalidateSet == null) {
invalidateSet = new Vector();
recentInvalidateSets.put(cur.getName(), invalidateSet);
}
invalidateSet.add(b);//同樣的,更新recentInvalidateSets,沒(méi)啥好解釋的
}
}
上述內(nèi)容就是HDFS中reportWrittenBlock函數(shù)的作用是什么,你們學(xué)到知識(shí)或技能了嗎?如果還想學(xué)到更多技能或者豐富自己的知識(shí)儲(chǔ)備,歡迎關(guān)注億速云行業(yè)資訊頻道。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。