Hadoop Logging

Logging Overview

Find log4j log levels for daemon service using http://<namenode>:50070/logLevel or http://<resource manager>:8088/logLevel
You can temporarily change log level using the above links. Log levels resets to the log4j.properties after restarts
You can also find log level and temporarily set the log level using shell command

$ hadoop daemonlog -getlevel <resource manager>:8088 <daemon class name as appears in jps -l command as below>

$ sudo jps -l

5826 org.apache.hadoop.yarn.server.nodemanager.NodeManager

5401 org.apache.hadoop.hdfs.server.namenode.NameNode

5537 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode

6115 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager

7454 org.apache.spark.deploy.history.HistoryServer

5725 org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer

5310 org.apache.hadoop.hdfs.qjournal.server.JournalNode

5203 org.apache.hadoop.hdfs.server.datanode.DataNode

Enable HDFS audit logging by setting the env variable in hadoop-env.sh and set log4j log hanger in log4j.properties

export HDFS_AUDIT_LOGGER="INFO, RFAAUDIT"

Job history is maintained for all completed job by job history service
Job history is stored HDFS as specified in mapreduce.jobhistory.done-dir [mapred-site.xml]
Job history records are stored for a week because system deletes them
Job history includes - job, tasks, attempts and stored in json format.
View job history information by http://<job history server>:19888

MapReduce task logs are produced by log4j and stored locally by default
MapReduce task logs can be aggregated to a HDFS location by setting yarn.log-aggregation-enable [yarn-site.xml] property
Task logs are stored for 3 hrs or as configured in yarn.log-aggregation.retain-seconds [yarn-site.xml]
Logging level at the task level can be altered using the following properties
- mapreduce.map.log.level
- mapreduce.reduce.log.level