Hadoop Stress Test

Hadoop contains the suit of stress testing tools in the distribution.

$ find /usr/lib/hadoop-mapreduce/ -name hadoop-*test*.jar


Find all tests

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar


TestDFSIO is designed to use 1 map task per file. (1:1 mapping from files to map tasks)

Write test

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 100

16/12/26 02:41:27 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write

16/12/26 02:41:27 INFO fs.TestDFSIO: Date & time: Mon Dec 26 02:41:27 PST 2016

16/12/26 02:41:27 INFO fs.TestDFSIO: Number of files: 10

16/12/26 02:41:27 INFO fs.TestDFSIO: Total MBytes processed: 1000.0

16/12/26 02:41:27 INFO fs.TestDFSIO: Throughput mb/sec: 7.658316548856231

16/12/26 02:41:27 INFO fs.TestDFSIO: Average IO rate mb/sec: 10.4342679977417

16/12/26 02:41:27 INFO fs.TestDFSIO: IO rate std deviation: 7.105071350812148

16/12/26 02:41:27 INFO fs.TestDFSIO: Test exec time sec: 134.375

16/12/26 02:41:27 INFO fs.TestDFSIO:

Write speed significantly vary depending on the replication factor. Below are some indicative numbers

RF 1: 190 MB/s

RF 2: 25 MB/s

RF 3: 13 MB/s

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 100

16/12/26 02:47:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read

16/12/26 02:47:44 INFO fs.TestDFSIO: Date & time: Mon Dec 26 02:47:44 PST 2016

16/12/26 02:47:44 INFO fs.TestDFSIO: Number of files: 10

16/12/26 02:47:44 INFO fs.TestDFSIO: Total MBytes processed: 1000.0

16/12/26 02:47:44 INFO fs.TestDFSIO: Throughput mb/sec: 46.15100609193281

16/12/26 02:47:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 70.32638549804688

16/12/26 02:47:44 INFO fs.TestDFSIO: IO rate std deviation: 50.15305958356481

16/12/26 02:47:44 INFO fs.TestDFSIO: Test exec time sec: 72.964

16/12/26 02:47:44 INFO fs.TestDFSIO:

TeraSort - combines testing of hadoop and mapreduce layer

  • Iron Out hadoop configuration

  • Determine whether your mapreduce related parameters are set to proper values

This test is run in combination of three mapreduce jobs

teragen > terasort > teravalidate

Generate test data using teragen. (4 hours on 4 node)

$ find /usr/lib/hadoop-mapreduce/ -name hadoop-*examples*.jar

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen <number of records 10000000000 /user/cloudera/terasort/input

Sort data using terasort. (5 hours on 4 node)

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort /user/cloudera/terasort/input /user/cloudera/terasort/output

Validate terasort output

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teravalidate /user/cloudera/terasort/output /user/cloudera/terasort/validate


LoadTest NameNode

  • Generate HDFS related requests with small payloads

  • Purpose: put a high HDFS management stress on the NameNode

  • Can simulate requests for creating, reading, renaming and deleting files

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar nnbench \

-operation create_file \

-maps 12 \

-reduces 6 \

-blockSize 1 \

-bytesToWrite 0 \

-numberOfFiles 1000 \

-replicationFactorPerFile 3 \

-readFileAfterOpen true \

-baseDir /user/cloudera/NNBnech-`hostname -s`

Loop small Job a number of times

  • Checks whether small job runs are responsive and running efficiently on the cluster

  • Complementary to terasort

  • Focus on mapreduce layer, impact on HDFS is limited

Run 50 small jobs

$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar mrbench -baseDir /user/cloudera/MRbench -numRuns 50


Big Bench Test

https://github.com/intel-hadoop/Big-Data-Benchmark-for-Big-Bench

Reference:

  • http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/

  • http://www.slideshare.net/BartVandewoestyne/big-data-benchmarking

  • http://sortbenchmark.org/

  • https://github.com/jbenninghoff/cluster-validation