Hadoop Architecture & Administration Training for Big Data Solutions

Hadoop Administration Course Outline

• Introduction to Data Storage and Processing

  Installing the Hadoop Distributed File System (HDFS)

  • Defining key design assumptions and architecture
  • Configuring and setting up the file system
  • Issuing commands from the console
  • Reading and writing files

  Setting the stage for MapReduce

  • Reviewing the MapReduce approach
  • Introducing the computing daemons
  • Dissecting a MapReduce job

• Defining Hadoop Cluster Requirements

  Planning the architecture

  • Selecting appropriate hardware
  • Designing a scalable cluster

  Building the cluster

  • Installing Hadoop daemons
  • Optimising the network architecture

• Configuring a Cluster

  Preparing HDFS

  • Setting basic configuration parameters
  • Configuring block allocation, redundancy and replication

  Deploying MapReduce

  • Installing and setting up the MapReduce environment
  • Delivering redundant load balancing via Rack Awareness

• Maximising HDFS Robustness

  Creating a fault–tolerant file system

  • Isolating single points of failure
  • Maintaining High Availability
  • Triggering manual failover
  • Automating failover with Zookeeper

  Leveraging NameNode Federation

  • Extending HDFS resources
  • Managing the namespace volumes

  Introducing YARN

  • Critiquing the YARN architecture
  • Identifying the new daemons

• Managing Resources and Cluster Health

  Allocating resources

  • Setting quotas to constrain HDFS utilization
  • Prioritising access to MapReduce using schedulers

  Maintaining HDFS

  • Starting and stopping Hadoop daemons
  • Monitoring HDFS status
  • Adding and removing data nodes

  Administering MapReduce

  • Managing MapReduce jobs
  • Tracking progress with monitoring tools
  • Commissioning and decommissioning compute nodes

• Maintaining a Cluster

  Employing the standard built–in tools

  • Managing and debugging processes using JVM metrics
  • Performing Hadoop status checks

  Tuning with supplementary tools

  • Assessing performance with Ganglia
  • Benchmarking to ensure continued performance

• Extending Hadoop

  Simplifying information access

  • Enabling SQL–like querying with Hive
  • Installing Pig to create MapReduce jobs

  Integrating additional elements of the ecosystem

  • Imposing a tabular view on HDFS with HBase
  • Leveraging memory with Spark

• Implementing Data Ingress and Egress

  Facilitating generic input/output

  • Moving bulk data into and out of Hadoop
  • Transmitting HDFS data over HTTP with WebHDFS

  Acquiring application–specific data

  • Collecting multi–sourced log files with Flume
  • Importing and exporting relational information with Sqoop

• Planning for Backup, Recovery and Security

  • Coping with inevitable hardware failures
  • Securing your Hadoop cluster