The Hadoop Distributed Filesystem (HDFS) forms the basis of many large-scale storage systems at Facebook and throughout the world. Our Hadoop clusters include the largest single HDFS cluster that we know of, with more than 100 PB physical disk space in a single
HDFS filesystem. Optimizing HDFS is crucial to ensuring that our systems stay efficient and reliable for users and applications on Facebook.
How the HDFS Namenode works
HDFS clients perform filesystem metadata operations through a single server known as the Namenode, and send and retrieve filesystem data by communicating with a pool of Datanodes. Data is replicated on multiple datanodes, so the loss of a single Datanode should
never be fatal to the cluster or cause data loss.
But the loss of the Namenode cannot be tolerated. All metadata operations go through the Namenode, so if the Namenode is unavailable, no clients can read from or write to HDFS. Clients can still read individual data blocks from Datanodes if the Namenode is
down, but for all intents and purposes, if the Namenode is unavailable, HDFS is down, and users and applications that depend on HDFS won’t be able to function properly.
The HDFS Namenode is a single point of failure (SPOF)
At Facebook, we wanted to know the extent of the “Namenode-as-SPOF” problem and build a system that would allow us to get past the limitations of the Namenode as a SPOF. Keep the picture above in mind--we’ll get back to it. But before we do, let’s talk a bit
about HDFS usage at Facebook, to give you more context about what we’ve been working on and what problems we’ve been facing.
The Data Warehouse use case
At Facebook, one of largest deployments of HDFS is in our Data Warehouse. The Data Warehouse use case is a traditional HadoopMapReduce workload: a small number of very large clusters running MapReduce batch jobs. The load on the Namenode is very heavy because
the clusters are very large, and both clients and Datanodes send a very high amount of metadata traffic to the Namenode. In a Data Warehouse cluster, it is not uncommon for the Namenode to be under heavy CPU, memory, disk and network pressure. When we cataloged
Data Warehouse failures, we found that HDFS caused 41% of incidents.
The HDFS Namenode, while a major component of HDFS, is a small but significant part of our overall Data Warehouse. While only 10% of the overall Warehouse issues and unplanned downtime would have been preventable if we had some kind of a highly-available Namenode,
eliminating the Namenode as a SPOF is still a huge win because it allows us to perform scheduled maintenance on hardware and software. In fact, we estimate that it would eliminate 50% of our planned downtime, time in which the cluster would be unavailable.
So what would a High Availability Namenode look like, and how would it work? Let’s look at a new diagram, showing a highly available Namenode:
In this configuration, clients could talk to either a Primary or Standby Namenode. Likewise, Datanodes would be able to send block reports to either Primary or Standby Namenode. That’s essentially what we’ve done with Avatarnode, our solution to a highly
available Namenode.
Avatarnode: A working solution for Namenode failover
To address the architectural shortcomings of the single Namenode, almost two years ago we began work at Facebook on theAvatarnode.
The Avatarnode, which Facebook has contributed back to the community as open source, offers a highly-available Namenode with hot failover and failback. After countless hours of testing and bug-fixing, the Avatarnode is now in production at Facebook running
our largest Hadoop Data Warehouse clusters thanks in large part toDmytro Molkov.
The Avatarnode is a two-node, highly available Namenode with manual failover. Avatarnode works by wrapping the existing Namenode code in a Zookeeper layer. The fundamental concepts in Avatarnode are:
1) There is a Primary and a Standby Avatarnode. Either Avatarnode can adopt either the Primary or the Standby “Avatar.”
2) The host name of the current master is kept in Zookeeper
3) A modified Datanode sends block reports to both the Primary and the Standby.
4) A modified HDFS client checks Zookeeper before beginning each transaction, and again in the middle of a transaction if one fails. This allows writes to complete even if a Avatarnode failover takes place in the middle of a write.
Avatarnode client view
Avatarnode datanode view
For those curious about the name,Dhruba Borthakur, one of our HDFS developers, came up with it around the time the James Cameron film “Avatar” came out (I
suppose we should be glad it was not 1998, or we could have had “Titanicnode").
The Avatarnode is running our most demanding production workloads inside of Facebook today, and will continue to lead to substantial improvements in reliability and administration of HDFS clusters. Moving forward, we're working to improve Avatarnode further
and integrate it with a general high-availability framework that will permit unattended, automated, and safe failover.
Whether your system has hundreds of nodes or thousands, HDFS is the most scalable and most reliable open-source distributed filesystem available. The failure data we collected is representative of a wide variety of use cases, and we've open sourced Avatarnode
as one of our best solutions to Namenode as SPOF. Because of work done by the Hadoop community, and the companies that support Hadoop contributors, all HDFS users and administrators can benefit from these efforts. You can check out Facebook's release of Hadoop,
which includes Avatarnode, onGitHub.
Andrew Ryan has been working with Hadoop at Facebook since 2009. During that time he’s helped grow our Hadoop and HDFS infrastructure from a single 600TB cluster in one datacenter to over 100 HDFS clusters in many datacenters.
分享到:
相关推荐
学习用terminal管理mac os x的很好的书
Why Python is Slow: Looking Under the Hood # Why Python is Slow Python is slower than Fortran and C for a variety of reasons # Python meta-hacking: Don't take my word for it # Just for fun: a few ...
Que - Windows XP under the Hood.pdf
With such emphasis on these performance requirements the need for using a multiphase synchronous boost converter arises. Selecting a number of phases can increase component counts and costs, which ...
A Look under the Hood of CBO - the 10053 Event
Peek under the hood to understand how React runs in the browser Create application presentation layers by mounting and composing React components Use component trees to manage data and reduce the time...
有关.NET编程的源代码,您可以从此处下载http://www.kutub.info/library/book/16743
Going under the hood of Inbox
robin_hood::unordered_map和robin_hood::unordered_set是std::unordered_map / std::unordered_set独立于平台的替代品,对于实际的用例而言,它既更快,更高效。 安装及使用 直接纳入 将添加到您的C ++项目。 ...
Android实现菜单弹出框Activity带值例子.rar
Babel 核心贡献者Nicolò Ribaudo分享 Babel 都做了什么?将来能做什么?Babel_ Under the Hood--Nicolò Ribaudo .pdf
Under the Hood: Dalvik patch for Facebook for Android Implemention By David Reiss on Monday, March 4, 2013 at 1:59pm Facebook is one of the most feature-rich apps available for Android. With features ...
一个很有用的介绍Puseaudio的文章,帮助理解pulseaudio的框架结构和功能原理。
藏经阁-SparkR Under the Hood -How to.pdf
NET file format - Signatures under the hood, Part 2 of 2
藏经阁-SparkR under the hood How to debug your SparkR code.pdf
Peek under the hood to understand how React runs in the browser Create application presentation layers by mounting and composing React components Use component trees to manage data and reduce the ...
The COFF .LIB File Structure
kubernetes-under-the-hood:本教程是打算安装Kubernetes集群的人员,并且想了解所有内容如何组合在一起
With Cisco SD-Access, you can overcome these challenges and provide your organization with the infrastructure required to meet your business objectives. Come to this session to get a look into the ...