HDFS平衡，如何平衡hdfs数据？

Question

我们有Hadoop版本- 2.6.4

在datanode机器上，我们可以看到hdfs数据是不平衡的

在一些磁盘上，我们有不同的使用大小，如SDB11G和SDD17G。

/dev/sdd 20G 3.0G 17G 15%   /grid/sdd 
/dev/sdb 20G 11G 9.3G 53%   /grid/sdb <-- WHY DISK DISK NOT BALANCED AS SDD DISK , WHY DISKS ARE DIFF USED SIZE!!!

在谷歌搜索之后，我找到了以下CLI (来自https://community.hortonworks.com/questions/19694/help-with-exception-from-hdfs-balancer.html )

hdfs balancer -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 20 1>/tmp/balancer-out.log 2>/tmp/balancer-debug.log

在我运行它之后，我们得到相同的hdfs大小

/dev/sdd 20G 3.0G 17G 15% /grid/sdd 
/dev/sdb 20G 11G 9.3G 53% /grid/sdb


more /tmp/balancer-out.log Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved The cluster is balanced. Exiting... Mar 7, 2019 5:02:34 PM 0 0 B 0 B 0 B Mar 7, 2019 5:02:34 PM Balancing took 1.453 seconds

因此，实际上我们在hdfs中没有达到平衡。

请告知，如何平衡hdfs数据，使所有磁盘都具有相同的使用大小。

Answer 1

NameNode在选择接收这些块的DataNodes之前会考虑各种参数。其中一些考虑因素是：

1. Policy to keep one of the replicas of a block on the same node as the node that is writing the block.
2. Need to spread different replicas of a block across the racks so that cluster can survive loss of whole rack.
3. One of the replicas is usually placed on the same rack as the node writing to the file so that cross-rack network I/O is reduced.
4. Spread HDFS data uniformly across the DataNodes in the cluster.

所以在你的例子中，可能是上面的一些匹配。

阿帕奇均衡器指挥部。

hdfs balancer [-threshold <threshold>] [-policy <policy>]
 -- threshold *threshold* Percentage of disk capacity. This overwrites the default threshold.
 -- policy *policy* *datanode* (default): Cluster is balanced if each datanode is balanced.  
                    *blockpool*: Cluster is balanced if each block pool in each datanode is balanced.