Hadoop Stress Test
Hadoop contains the suit of stress testing tools in the distribution.
$ find /usr/lib/hadoop-mapreduce/ -name hadoop-*test*.jar
Find all tests
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar
TestDFSIO is designed to use 1 map task per file. (1:1 mapping from files to map tasks)
Write test
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 100
16/12/26 02:41:27 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/12/26 02:41:27 INFO fs.TestDFSIO: Date & time: Mon Dec 26 02:41:27 PST 2016
16/12/26 02:41:27 INFO fs.TestDFSIO: Number of files: 10
16/12/26 02:41:27 INFO fs.TestDFSIO: Total MBytes processed: 1000.0
16/12/26 02:41:27 INFO fs.TestDFSIO: Throughput mb/sec: 7.658316548856231
16/12/26 02:41:27 INFO fs.TestDFSIO: Average IO rate mb/sec: 10.4342679977417
16/12/26 02:41:27 INFO fs.TestDFSIO: IO rate std deviation: 7.105071350812148
16/12/26 02:41:27 INFO fs.TestDFSIO: Test exec time sec: 134.375
16/12/26 02:41:27 INFO fs.TestDFSIO:
Write speed significantly vary depending on the replication factor. Below are some indicative numbers
RF 1: 190 MB/s
RF 2: 25 MB/s
RF 3: 13 MB/s
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 100
16/12/26 02:47:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
16/12/26 02:47:44 INFO fs.TestDFSIO: Date & time: Mon Dec 26 02:47:44 PST 2016
16/12/26 02:47:44 INFO fs.TestDFSIO: Number of files: 10
16/12/26 02:47:44 INFO fs.TestDFSIO: Total MBytes processed: 1000.0
16/12/26 02:47:44 INFO fs.TestDFSIO: Throughput mb/sec: 46.15100609193281
16/12/26 02:47:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 70.32638549804688
16/12/26 02:47:44 INFO fs.TestDFSIO: IO rate std deviation: 50.15305958356481
16/12/26 02:47:44 INFO fs.TestDFSIO: Test exec time sec: 72.964
16/12/26 02:47:44 INFO fs.TestDFSIO:
TeraSort - combines testing of hadoop and mapreduce layer
- Iron Out hadoop configuration
- Determine whether your mapreduce related parameters are set to proper values
This test is run in combination of three mapreduce jobs
teragen > terasort > teravalidate
Generate test data using teragen. (4 hours on 4 node)
$ find /usr/lib/hadoop-mapreduce/ -name hadoop-*examples*.jar
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen <number of records 10000000000 /user/cloudera/terasort/input
Sort data using terasort. (5 hours on 4 node)
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort /user/cloudera/terasort/input /user/cloudera/terasort/output
Validate terasort output
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teravalidate /user/cloudera/terasort/output /user/cloudera/terasort/validate
LoadTest NameNode
- Generate HDFS related requests with small payloads
- Purpose: put a high HDFS management stress on the NameNode
- Can simulate requests for creating, reading, renaming and deleting files
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar nnbench \
-operation create_file \
-maps 12 \
-reduces 6 \
-blockSize 1 \
-bytesToWrite 0 \
-numberOfFiles 1000 \
-replicationFactorPerFile 3 \
-readFileAfterOpen true \
-baseDir /user/cloudera/NNBnech-`hostname -s`
Loop small Job a number of times
- Checks whether small job runs are responsive and running efficiently on the cluster
- Complementary to terasort
- Focus on mapreduce layer, impact on HDFS is limited
Run 50 small jobs
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar mrbench -baseDir /user/cloudera/MRbench -numRuns 50
Big Bench Test
https://github.com/intel-hadoop/Big-Data-Benchmark-for-Big-Bench
Reference:
- http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/
- http://www.slideshare.net/BartVandewoestyne/big-data-benchmarking
- http://sortbenchmark.org/
- https://github.com/jbenninghoff/cluster-validation