crail - animeshtrivedi/notes GitHub Wiki

Crail (www.crail.io) specific notes

How to run Hdfs standalone benchmarks

What to do when this happens

~/hadoop-2.7.3$ ./bin/hadoop jar ~/HdfsBench-1.0.jar readRandomHeap $((128*1024*1024)) $((8*32)) crail://flex11-40g0:9060/sql/local-32gb
Picked up JAVA_TOOL_OPTIONS: -XX:+PreserveFramePointer
reading random file in heap mode crail://flex11-40g0:9060/sql/local-32gb
java.lang.IllegalArgumentException: Wrong FS: crail://flex11-40g0:9060/sql/local-32gb, expected: hdfs://flex11-40g0:9000
	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:648)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
	at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
	at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
	at com.github.animeshtrivedi.HdfsIOBenchmark.HdfsIOBenchmark.readRandomHeap(HdfsIOBenchmark.java:281)
	at com.github.animeshtrivedi.HdfsIOBenchmark.HdfsIOBenchmark.run(HdfsIOBenchmark.java:71)
	at com.github.animeshtrivedi.HdfsIOBenchmark.HdfsIOBenchmark.main(HdfsIOBenchmark.java:483)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Solution:

Run either by making crail the default file system in the core-site.xml file or run natively with the classpath as

java -cp /home/demo/HdfsBench-1.0.jar:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/common/lib/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/common/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/hdfs:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/hdfs/lib/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/hdfs/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/yarn/lib/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/yarn/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/mapreduce/lib/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/mapreduce/*:/home/demo/crail-deployment/hadoop//contrib/capacity-scheduler/*.jar:/home/demo/crail-deployment/crail//conf/:/home/demo/crail-deployment/crail//jars/* com.github.animeshtrivedi.HdfsIOBenchmark.HdfsIOBenchmark readRandomHeap 134217728 256 crail://flex11-40g0:9060/sql/local-32gb