Hadoop Architecture & Administration for Big Data Solutions

Installing the Hadoop Distributed File System (HDFS)

• Defining key design assumptions and architecture
• Configuring and setting up the file system
• Issuing commands from the console
• Reading and writing files

Setting the stage for MapReduce

• Reviewing the MapReduce approach
• Introducing the computing daemons
• Dissecting a MapReduce job

Planning the architecture

• Selecting appropriate hardware
• Designing a scalable cluster

Building the cluster

• Installing Hadoop daemons
• Optimizing the network architecture

Preparing HDFS

• Setting basic configuration parameters
• Configuring block allocation, redundancy and replication

Deploying MapReduce

• Installing and setting up the MapReduce environment
• Delivering redundant load balancing via Rack Awareness

Creating a fault–tolerant file system

• Isolating single points of failure
• Maintaining High Availability
• Triggering manual failover
• Automating failover with Zookeeper

Leveraging NameNode Federation

• Extending HDFS resources
• Managing the namespace volumes

Introducing YARN

• Critiquing the YARN architecture
• Identifying the new daemons

Allocating resources

• Setting quotas to constrain HDFS utilization
• Prioritizing access to MapReduce using schedulers

Maintaining HDFS

• Starting and stopping Hadoop daemons
• Monitoring HDFS status
• Adding and removing data nodes

Administering MapReduce

• Managing MapReduce jobs
• Tracking progress with monitoring tools
• Commissioning and decommissioning compute nodes

Employing the standard built–in tools

• Managing and debugging processes using JVM metrics
• Performing Hadoop status checks

Tuning with supplementary tools

• Assessing performance with Ganglia
• Benchmarking to ensure continued performance

Simplifying information access

• Enabling SQL–like querying with Hive
• Installing Pig to create MapReduce jobs

Integrating additional elements of the ecosystem

• Imposing a tabular view on HDFS with HBase
• Configuring Oozie to schedule workflows

Facilitating generic input/output

• Moving bulk data into and out of Hadoop
• Transmitting HDFS data over HTTP with WebHDFS

Acquiring application–specific data

• Collecting multi–sourced log files with Flume
• Importing and exporting relational information with Sqoop

• Coping with inevitable hardware failures
• Securing your Hadoop cluster