Hadoop Stress Test

Hadoop contains the suit of stress testing tools in the distribution. 

$ find /usr/lib/hadoop-mapreduce/ -name hadoop-*test*.jar


Find all tests

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar


TestDFSIO is designed to use 1 map task per file. (1:1 mapping from files to map tasks)

Write test

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 100
16/12/26 02:41:27 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/12/26 02:41:27 INFO fs.TestDFSIO:            Date & time: Mon Dec 26 02:41:27 PST 2016
16/12/26 02:41:27 INFO fs.TestDFSIO:        Number of files: 10
16/12/26 02:41:27 INFO fs.TestDFSIO: Total MBytes processed: 1000.0
16/12/26 02:41:27 INFO fs.TestDFSIO:      Throughput mb/sec: 7.658316548856231
16/12/26 02:41:27 INFO fs.TestDFSIO: Average IO rate mb/sec: 10.4342679977417
16/12/26 02:41:27 INFO fs.TestDFSIO:  IO rate std deviation: 7.105071350812148
16/12/26 02:41:27 INFO fs.TestDFSIO:     Test exec time sec: 134.375
16/12/26 02:41:27 INFO fs.TestDFSIO:

Write speed significantly vary depending on the replication factor. Below are some indicative numbers

RF 1: 190 MB/s

RF 2: 25 MB/s

RF 3: 13 MB/s

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 100
16/12/26 02:47:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
16/12/26 02:47:44 INFO fs.TestDFSIO:            Date & time: Mon Dec 26 02:47:44 PST 2016
16/12/26 02:47:44 INFO fs.TestDFSIO:        Number of files: 10
16/12/26 02:47:44 INFO fs.TestDFSIO: Total MBytes processed: 1000.0
16/12/26 02:47:44 INFO fs.TestDFSIO:      Throughput mb/sec: 46.15100609193281
16/12/26 02:47:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 70.32638549804688
16/12/26 02:47:44 INFO fs.TestDFSIO:  IO rate std deviation: 50.15305958356481
16/12/26 02:47:44 INFO fs.TestDFSIO:     Test exec time sec: 72.964
16/12/26 02:47:44 INFO fs.TestDFSIO:

TeraSort - combines testing of hadoop and mapreduce layer

  • Iron Out hadoop configuration
  • Determine whether your mapreduce related parameters are set to proper values

This test is run in combination of three mapreduce jobs

teragen > terasort > teravalidate

Generate test data using teragen. (4 hours on 4 node)

$ find /usr/lib/hadoop-mapreduce/ -name hadoop-*examples*.jar
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen <number of records 10000000000 /user/cloudera/terasort/input

Sort data using terasort. (5 hours on 4 node)

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort /user/cloudera/terasort/input /user/cloudera/terasort/output

Validate terasort output

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teravalidate /user/cloudera/terasort/output /user/cloudera/terasort/validate


LoadTest NameNode

  • Generate HDFS related requests with small payloads
  • Purpose: put a high HDFS management stress on the NameNode
  • Can simulate requests for creating, reading, renaming and deleting files
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar nnbench \
-operation create_file \
-maps 12 \
-reduces 6 \
-blockSize 1 \
-bytesToWrite 0 \
-numberOfFiles 1000 \
-replicationFactorPerFile 3 \
-readFileAfterOpen true \
-baseDir /user/cloudera/NNBnech-`hostname -s`

Loop small Job a number of times

  • Checks whether small job runs are responsive and running efficiently on the cluster
  • Complementary to terasort
  • Focus on mapreduce layer, impact on HDFS is limited

Run 50 small jobs

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar mrbench -baseDir /user/cloudera/MRbench -numRuns 50


Big Bench Test

https://github.com/intel-hadoop/Big-Data-Benchmark-for-Big-Bench

Reference:

  • http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
  • http://www.slideshare.net/BartVandewoestyne/big-data-benchmarking
  • http://sortbenchmark.org/
  • https://github.com/jbenninghoff/cluster-validation