demo@flex11:~$ java -cp /home/demo/crail-deployment/hadoop//etc/hadoop/:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/common/lib/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/common/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/hdfs:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/hdfs/lib/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/hdfs/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/yarn/lib/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/yarn/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/mapreduce/lib/*:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/mapreduce/*:/home/demo/crail-deployment/hadoop//contrib/capacity-scheduler/*.jar:/home/demo/crail-deployment/crail//conf:/home/demo/crail-deployment/crail//jars/*:/home/demo/crail-deployment/crail//lib/*:$CRAIL_HOME/../spark/jars/*:/home/demo/parquet-tools-1.0.jar com.ibm.crail.parquet.Main hdfs://flex11-40g0:9000/sql/small.parquet
Picked up JAVA_TOOL_OPTIONS: -XX:+PreserveFramePointer
Hello World!
concat arguments = hdfs://flex11-40g0:9000/sql/small.parquet
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/demo/crail-deployment/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/demo/crail-deployment/crail-1.0/jars-atr/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/demo/crail-deployment/spark-2.1.0-bin-hadoop2.7/jars-atr/v2.1.0-summit/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/06/16 10:51:07 DEBUG Shell: setsid exited with exit code 0
17/06/16 10:51:07 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
17/06/16 10:51:07 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
17/06/16 10:51:07 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
17/06/16 10:51:07 DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
17/06/16 10:51:07 DEBUG KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
17/06/16 10:51:07 DEBUG Groups: Creating new Groups object
17/06/16 10:51:07 DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
17/06/16 10:51:07 DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
17/06/16 10:51:07 DEBUG NativeCodeLoader: java.library.path=/home/demo/crail-deployment/crail//lib/:/home/demo/local/lib/:/usr/local/lib/::/home/jpf/Source/3rd/dpdk/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
17/06/16 10:51:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/06/16 10:51:07 DEBUG PerformanceAdvisory: Falling back to shell based
17/06/16 10:51:07 DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
17/06/16 10:51:07 DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
17/06/16 10:51:07 DEBUG UserGroupInformation: hadoop login
17/06/16 10:51:07 DEBUG UserGroupInformation: hadoop login commit
17/06/16 10:51:07 DEBUG UserGroupInformation: using local user:UnixPrincipal: demo
17/06/16 10:51:07 DEBUG UserGroupInformation: Using user: "UnixPrincipal: demo" with name demo
17/06/16 10:51:07 DEBUG UserGroupInformation: User entry: "demo"
17/06/16 10:51:07 DEBUG UserGroupInformation: UGI loginUser:demo (auth:SIMPLE)
17/06/16 10:51:07 DEBUG BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
17/06/16 10:51:07 DEBUG BlockReaderLocal: dfs.client.read.shortcircuit = false
17/06/16 10:51:07 DEBUG BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
17/06/16 10:51:07 DEBUG BlockReaderLocal: dfs.domain.socket.path =
17/06/16 10:51:07 DEBUG RetryUtils: multipleLinearRandomRetry = null
17/06/16 10:51:07 DEBUG Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@1838ccb8
17/06/16 10:51:07 DEBUG Client: getting client out of cache: org.apache.hadoop.ipc.Client@1583741e
17/06/16 10:51:07 DEBUG PerformanceAdvisory: Both short-circuit local reads and UNIX domain socket are disabled.
17/06/16 10:51:07 DEBUG DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
17/06/16 10:51:07 DEBUG Client: The ping interval is 60000 ms.
17/06/16 10:51:07 DEBUG Client: Connecting to flex11-40g0/10.40.0.11:9000
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo: starting, having connections 1
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #0
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #0
17/06/16 10:51:07 DEBUG ProtobufRpcEngine: Call: getListing took 80ms
17/06/16 10:51:07 INFO ParquetFileReader: Initiating action with parallelism: 5
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #1
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #1
17/06/16 10:51:07 DEBUG ProtobufRpcEngine: Call: getFileInfo took 1ms
17/06/16 10:51:07 INFO ParquetFileReader: reading another 10 footers
17/06/16 10:51:07 INFO ParquetFileReader: Initiating action with parallelism: 5
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #5
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #6
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #2
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #5
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #3
17/06/16 10:51:07 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 1ms
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #6
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #4
17/06/16 10:51:07 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 2ms
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #2
17/06/16 10:51:07 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 2ms
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #3
17/06/16 10:51:07 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 2ms
17/06/16 10:51:07 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #4
17/06/16 10:51:07 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 2ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770096_29272; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.16:50010,DS-6b452b44-11c4-4608-86d5-c778915d5d29,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770096_29272; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.16:50010,DS-6b452b44-11c4-4608-86d5-c778915d5d29,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770097_29273; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.21:50010,DS-589ba9c4-ca96-4f79-ae58-bec019b77995,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770097_29273; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.21:50010,DS-589ba9c4-ca96-4f79-ae58-bec019b77995,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770093_29269; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.22:50010,DS-bb3468dd-3e03-41ac-b1b4-854b353dd7f0,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770093_29269; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.22:50010,DS-bb3468dd-3e03-41ac-b1b4-854b353dd7f0,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770094_29270; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.20:50010,DS-7e06b257-5427-4f5b-bb30-610c5653429c,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770094_29270; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.20:50010,DS-7e06b257-5427-4f5b-bb30-610c5653429c,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770098_29274; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.19:50010,DS-d3e4af75-898b-4733-a54a-18e81a7020af,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770098_29274; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.19:50010,DS-d3e4af75-898b-4733-a54a-18e81a7020af,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.19:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.16:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.20:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.21:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.22:50010
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #7
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #8
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #10
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #7
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #11
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getServerDefaults took 0ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #9
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #8
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getServerDefaults took 0ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #10
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #11
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getServerDefaults took 1ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #9
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getServerDefaults took 1ms
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getServerDefaults took 1ms
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.21, datanodeId = DatanodeInfoWithStorage[10.40.0.21:50010,DS-589ba9c4-ca96-4f79-ae58-bec019b77995,DISK]
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.22, datanodeId = DatanodeInfoWithStorage[10.40.0.22:50010,DS-bb3468dd-3e03-41ac-b1b4-854b353dd7f0,DISK]
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.16, datanodeId = DatanodeInfoWithStorage[10.40.0.16:50010,DS-6b452b44-11c4-4608-86d5-c778915d5d29,DISK]
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.20, datanodeId = DatanodeInfoWithStorage[10.40.0.20:50010,DS-7e06b257-5427-4f5b-bb30-610c5653429c,DISK]
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.19, datanodeId = DatanodeInfoWithStorage[10.40.0.19:50010,DS-d3e4af75-898b-4733-a54a-18e81a7020af,DISK]
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.20:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.22:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.21:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.19:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.16:50010
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[BIT_PACKED, RLE, PLAIN], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:77 B1 DD 74, min:F9 CA 2E 02, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[BIT_PACKED, RLE, PLAIN], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:7E CA EC E3 63 5D 31 42 53 47 DD 32 F0 D0 0F 31 0B DB 9F 08 22 3B 37 BF B0 EF 84 D7 B4 0B 9A 8E BB E0 D4 64 D4 15 DD 0F 71 E2 C5 01 7C FD 31 AF 48 70 DD FD 02 6D FF 50 51 CB 12 74 B1 CD A2 8E BD BF 20 ED 6F EF A6 62 EB E0 62 2B B5 3E 27 E6 A5 BD E7 3A 68 AD E8 FA 7D CA 4E 13 57 9A D5 9B 64 F6 B0 D8 60 9C E6 CB EE 9D EF 4E 8F 71 79 C5 E4 EB 12 4C BB 3F 1C A4 11 3A A1 54 67 23 1D 46..., min:C6 D3 73 F2 C9 6D 71 31 60 86 30 E2 84 0E 5C 6F 11 3E 05 46 18 3D 13 02 DB A5 0E 26 6C 86 A7 36 97 E8 FA 81 29 FD 62 57 13 89 AD 5F 3C 8F EC 69 A1 19 C0 A9 DA 37 C9 D0 54 56 B5 D1 1C D6 D9 0E 38 FD 90 DE 63 03 C6 32 45 F2 8D 77 22 2A 7A 18 E8 58 96 5C CA A4 EE BF 4A 63 41 24 E4 A5 14 4B C5 37 22 93 9C 65 FE 18 29 87 96 23 4E 5E ED 7E B3 D2 3C B1 D9 90 32 C4 EF FE 6E 4A B6 AA 61 28..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[RLE, PLAIN, BIT_PACKED], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:7B 31 41 58, min:FF 6A 05 0A, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[RLE, PLAIN, BIT_PACKED], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:7F FA B8 31 67 B2 DE 78 33 74 3C C1 26 B4 CF E8 A3 22 3A 51 25 4D 5C E6 06 90 20 FF AA C4 9F 8C 2F 03 E1 03 95 B5 86 2E 2E B3 17 A6 65 A6 FC 50 47 4F 09 D4 AA 4D 5B BD 1A 46 CB 7B CF 77 66 AA 36 B5 CC 57 1F C4 22 A1 6A 32 F8 95 6C B2 B7 DB 91 C2 68 1F CB 8F FB 1F B2 DD A2 FA 3D 08 43 04 9D 66 DC 4B 39 4C 40 48 7C 8C 49 C3 79 24 ED 20 ED B6 F0 E7 78 CB 1E 39 93 71 AB 06 7C 43 55 78..., min:E8 19 B7 78 8F 7A 5C BC 70 26 0C F7 78 65 C2 5E DE B5 EF EA 20 29 EB 83 B4 BB 24 EE E9 96 6E C1 B9 94 30 35 33 14 AB E0 AB 03 E0 F7 2E 2E D9 D5 C6 D0 2D DD 9E 46 A5 36 FE 5A F1 9C 71 62 A3 7E 1C 88 60 39 1C 84 6C 75 26 69 58 88 08 5F 9C 93 B8 AA 90 27 A8 66 2E E1 32 15 74 65 E7 92 6B 14 E9 06 AB 79 0E 7D 28 DD C6 82 C9 D5 56 03 4C 4D EE 37 C1 3B BE 3E 38 E1 66 5B CF F3 E3 95 1D AE..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[BIT_PACKED, RLE, PLAIN], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:C2 A3 AB 56, min:C0 32 1A 0C, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[BIT_PACKED, RLE, PLAIN], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:4E 74 9D 11 44 90 D9 44 A2 37 2E 38 6E EB 25 12 56 D2 3A 20 7D 15 64 E2 53 9B 04 CF 87 CA CD 8A DA 8F 35 E2 DA 88 B6 F7 5A BF 96 B3 2A 0F 06 9C BE 02 05 3B 98 19 E0 DE BB 38 C5 F6 6F 0C AF 10 DB EF 17 EF BE 59 6B FE 62 C8 FA 0E 41 57 78 13 37 3B B8 46 39 FC 8E 3C 7A A8 F1 2F 1D 2B CC F2 75 E7 7C D2 48 D1 29 26 FF DA EF AC 19 D7 85 EB 8F 18 98 98 1A A5 57 E5 91 4D 61 2D 27 77 33 EA..., min:87 99 AC 01 0E 16 3E 96 65 D2 53 42 C8 B1 20 23 26 8E 1A EF B2 89 04 D1 42 D7 03 62 BB 1B 9C 05 BB D4 CA 38 35 D4 53 F9 1A 50 77 F0 7E 38 30 4F AE 78 30 53 12 26 D4 6D 19 7A DB 0D 60 37 23 7B 73 BF 74 EF 0F 82 C9 A2 87 D1 97 8C A4 6E B3 08 FB F5 13 5B 9E 6F 5D B7 5E DE 24 E4 86 6E C9 CC 6D 21 75 45 86 EF E6 DD 1B B6 AA FC 80 C0 89 94 15 87 7F 34 C2 96 AA 44 D9 46 25 63 C9 D6 AC 82..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[PLAIN, RLE, BIT_PACKED], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:66 A9 3A 7A, min:E7 4E 85 23, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[PLAIN, RLE, BIT_PACKED], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:53 22 21 25 60 02 9A A7 6E 4A F7 54 69 64 72 9F 76 05 48 CF 45 C6 83 D7 9E FB 7D 6A 26 32 C6 FB 7A 66 0B C0 1D 4A 9D 9D 4E 5A 86 D4 86 15 72 04 BE EE 4B 46 7E AA 8A F9 F2 F9 0E B8 B9 1D 39 3F 6F 1C 9E 8F 70 6D F6 F2 24 F9 60 F0 47 10 76 9C 86 F4 90 D6 50 7F 7C 76 B4 86 82 18 9B 0A DE 66 F3 5F 59 B3 F6 99 4D A8 03 24 A4 D6 C5 37 0D 88 1A A8 B9 3F 1A 7A 7F 7A 90 38 6F DC 7B 9E 02 A7..., min:9F 76 75 25 16 2D 9D 15 DC AD 93 4E D1 E6 B7 B6 21 28 41 3C 9B 4D F2 A0 F2 26 EF 2A B9 CF 71 E4 7B A4 9E B7 BC 71 9E 69 2A 60 03 12 8B 7E 9A 7B C6 1E 3E 12 E6 D7 E5 73 21 C1 FD F0 7E 4F 0B 6B DB AC 90 50 E0 03 A1 B7 CC 34 1C F3 B8 A1 5B 7F 15 CB C6 47 42 F1 AC 75 E0 DC B6 CA E0 10 5E 4B 5B D8 4C 89 27 79 07 1F 74 17 A7 75 F0 33 3F C2 5C CE C4 64 21 DF 5F 5E E3 BA 66 B7 4F F5 D0 34..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[PLAIN, RLE, BIT_PACKED], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:3A A6 D1 6A, min:5C FD AC 24, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[PLAIN, RLE, BIT_PACKED], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:37 73 D2 62 86 EF AA 27 64 79 0D CE BD 38 78 B2 D9 B2 E7 F8 ED 95 DB B1 2F 3A 33 13 B7 72 52 93 11 34 E6 42 C2 C8 09 D3 3E 05 AF 88 CC 6D 21 98 0F 99 86 BE 61 E8 8F 35 8B A0 FC E1 97 C6 7F 66 58 45 60 0A 81 A4 F7 45 FB B0 BA AC 35 2C 61 59 46 50 05 04 96 6F A4 3B F4 B3 19 3D 49 9C FD 4C B3 AA C6 BE 76 44 27 3E 32 7E 48 41 46 DB 65 A1 F1 6B A8 D7 3F A2 56 2A DB 53 29 27 EC AC B0 C6..., min:87 08 73 BD 5F 95 6B A2 BD 51 13 E7 6D B2 3D F0 C4 F0 CE 92 2D 7E 7E 07 3B C4 BE 41 52 96 CA 6C 01 EE 5F E8 E9 9C 3D D3 13 4D 6F 24 43 F9 80 DB 98 D3 7C 25 DC 2C 3E 09 FC 93 62 8C 5F 93 90 7C 19 11 28 99 33 5D C9 69 A9 90 CB 06 3D E6 96 1E A4 B4 59 DA BA C6 E6 EC 3A C7 C4 02 54 32 03 F8 7B BE 36 16 EA F9 71 29 F5 57 C5 76 9D DE F9 D7 D2 99 31 F3 25 69 EC 71 35 D6 2A AD 76 36 14 D5..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 1960685943,
"min" : 36621049,
"maxBytes" : "d7HddA==",
"minBytes" : "+couAg==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 1480667515,
"min" : 168127231,
"maxBytes" : "ezFBWA==",
"minBytes" : "/2oFCg==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 1454089154,
"min" : 203043520,
"maxBytes" : "wqOrVg==",
"minBytes" : "wDIaDA==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 1792124474,
"min" : 615316828,
"maxBytes" : "OqbRag==",
"minBytes" : "XP2sJA==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 2050664806,
"min" : 595939047,
"maxBytes" : "Zqk6eg==",
"minBytes" : "506FIw==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #12
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #15
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #14
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #12
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #13
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 2ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #15
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #16
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 1ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #14
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #13
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 2ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770091_29267; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.15:50010,DS-eea85b06-06f6-470c-925e-43616e9837b7,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770091_29267; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.15:50010,DS-eea85b06-06f6-470c-925e-43616e9837b7,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770092_29268; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.23:50010,DS-affd1c1e-3e1c-4abe-86f2-2918066d12ad,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770092_29268; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.23:50010,DS-affd1c1e-3e1c-4abe-86f2-2918066d12ad,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #16
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 2ms
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 2ms
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770090_29266; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.14:50010,DS-695bf534-8da7-4857-80d8-b996b18b3e49,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770090_29266; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.14:50010,DS-695bf534-8da7-4857-80d8-b996b18b3e49,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.15:50010
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770089_29265; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.13:50010,DS-e0c53985-588a-4578-b388-2ab8e64ea512,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770089_29265; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.13:50010,DS-e0c53985-588a-4578-b388-2ab8e64ea512,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770095_29271; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.18:50010,DS-02976b48-62ba-4785-b182-5037b8db76fb,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770095_29271; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.18:50010,DS-02976b48-62ba-4785-b182-5037b8db76fb,DISK]]}
isLastBlockComplete=true}
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.14:50010
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.23:50010
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.15, datanodeId = DatanodeInfoWithStorage[10.40.0.15:50010,DS-eea85b06-06f6-470c-925e-43616e9837b7,DISK]
17/06/16 10:51:08 DEBUG ParquetFileReader: File length 14963
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.13:50010
17/06/16 10:51:08 DEBUG ParquetFileReader: reading footer index at 14955
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.14, datanodeId = DatanodeInfoWithStorage[10.40.0.14:50010,DS-695bf534-8da7-4857-80d8-b996b18b3e49,DISK]
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.23, datanodeId = DatanodeInfoWithStorage[10.40.0.23:50010,DS-affd1c1e-3e1c-4abe-86f2-2918066d12ad,DISK]
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.18:50010
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.13, datanodeId = DatanodeInfoWithStorage[10.40.0.13:50010,DS-e0c53985-588a-4578-b388-2ab8e64ea512,DISK]
17/06/16 10:51:08 DEBUG SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.40.0.18, datanodeId = DatanodeInfoWithStorage[10.40.0.18:50010,DS-02976b48-62ba-4785-b182-5037b8db76fb,DISK]
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.15:50010
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.14:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.23:50010
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
17/06/16 10:51:08 DEBUG ParquetFileReader: read footer length: 2507, footer index: 12448
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.13:50010
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.18:50010
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[PLAIN, BIT_PACKED, RLE], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:42 19 62 68, min:7C C9 0F 00, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[PLAIN, BIT_PACKED, RLE], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:7F E0 91 6D A8 4F 00 4D AC 81 B5 AB D2 C2 03 52 C7 7A 02 1F 22 F3 7B DD 92 9A 74 C7 3C 8C 31 BF 14 AB 73 37 C0 F7 12 5F 4B BE EF F8 1C 01 95 F2 B3 9D 77 7C 54 12 88 83 80 AB A3 67 99 12 9A B0 BB A2 F4 02 9C 50 D8 81 AA 8B 4D DD 0A 46 8C 02 29 FC 80 6B 93 36 B6 85 D8 E7 1F BB 63 04 BB CE 4C EA 2A 20 02 DD 08 DD AE A5 CB 16 EB EC 4E 00 63 9A 8A 48 5C E5 25 37 B0 50 AD 56 32 1F 15 39..., min:9D 97 13 58 05 59 7A 03 BB EB 30 FA 1B ED 0C 4B 2B 39 46 62 06 9F 4C A7 01 F6 E1 D2 2A 06 6F 96 3F 91 1A A9 05 8A 3C C6 9F 54 F0 E7 6E 84 9B 4A 24 F0 F0 B4 B5 AA 33 E3 73 59 4B 87 02 53 2B 2D 4E 0C 1C 7B B6 ED 7A 9D 98 EE 83 DF 93 36 43 8D D2 71 20 8B F5 47 CF 49 F8 D4 AB 64 67 6A 4A D1 17 F5 36 AE 53 45 EA E5 24 B8 D1 E8 A1 89 FF 7C CD 1C 77 4A 84 5D B2 B1 67 2B 0A 3C C3 48 1C 0F..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[BIT_PACKED, RLE, PLAIN], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:FB 07 D3 61, min:D4 9C 7C 02, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[BIT_PACKED, RLE, PLAIN], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:62 C5 3A 57 E9 03 2D 88 CF 5F 41 16 66 95 AF 8D 08 DC 18 92 F7 9C 90 0C DF F8 D8 1D A0 6E 55 CB 3E 89 01 D3 24 4D FB 10 71 43 C9 C4 71 6B 33 C8 09 34 85 DF 65 A6 59 71 72 3E F0 2A BE 7E 14 B1 2F 2B 40 08 30 D4 C1 CE 57 64 73 9B F4 9D 4A A2 D4 BA 0B 49 52 2A FB 88 61 F3 39 19 F3 9D 5C EB 67 78 78 97 FE 4C EF B6 22 F6 8C 1B 49 6B 80 78 BE E8 46 82 28 46 7D D3 40 D0 90 4A A3 11 84 03..., min:83 89 3C 68 6C 4D 7A 87 D9 CA 41 CF 3B 46 AC 5E A9 C3 24 26 18 63 0B A4 CC A7 DC B4 44 BD 21 F5 69 AA 55 56 60 F3 4C 2C E3 30 E7 B0 9E 89 69 5D CA 1D 88 47 67 68 EB E3 3A FE 4A 92 AA 17 C8 E6 42 2E 8E 9D 76 72 A6 68 4C FD 02 8B 8E 0D 1F 92 9F 89 43 2B 0F 3D B4 19 73 02 79 1D 6C D8 30 D1 2A 52 1D 57 41 61 7A B1 AF 6E 6C F7 D5 58 73 BC 09 C4 BD DF 3E 29 33 CF A6 ED 76 AC 80 62 FC 63..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[BIT_PACKED, RLE, PLAIN], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:2F AE EA 67, min:D1 33 7A 06, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[BIT_PACKED, RLE, PLAIN], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:70 C5 AB A7 F0 48 E0 83 3B 9D C2 CE 5A C6 EB 68 CF B3 68 EC E1 DB 04 5F 6B A1 32 A4 FD D3 33 BB 3E 7E A2 D5 C1 6A D5 71 9F C0 5E C7 14 72 2E 11 EA 88 8D 1C 25 63 20 B2 06 58 49 41 93 C9 03 D2 E5 48 8D 53 DF 8E E9 BA 80 92 23 24 19 71 31 6B 15 A7 38 3A EC 6D 1F 54 B8 86 77 D8 76 33 EF 14 6F 1E 4D 60 90 CA 31 31 77 A2 8D 55 6E 3A 3B BD 65 09 F9 8C BF EB 3A AC 37 F7 6C 01 BC A1 37 9D..., min:81 54 79 5E 2D 66 8E FC 41 1E 04 89 F5 A1 40 8F 06 B1 5B E5 21 D9 B4 73 D2 2E 7C 53 72 49 F0 7C 7A 63 CF 2F 31 C0 BC 09 ED 54 FB C2 33 BC 07 9D 82 80 0A 3B 95 85 17 62 0B EE FD 3D 5C 48 7B 00 AB A2 01 39 58 27 2C BA D1 60 D8 76 FE FE D6 50 BF D7 D8 C0 17 BC 15 A0 84 BC CD C2 75 D0 12 CF 5A 8F E5 C4 08 00 3F B6 BC 5B AC 9B 10 8B 61 53 BC 5E C8 6D 3E 2E BB 82 57 7E 85 FE F7 9F 9E 1A..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[PLAIN, RLE, BIT_PACKED], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:65 DE 04 78, min:AC C4 CC 19, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[PLAIN, RLE, BIT_PACKED], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:71 62 D1 02 D2 E6 81 F5 C4 4C 46 55 28 76 CC 7E 7B 1E 17 87 90 8B BB 96 7F 51 14 45 E4 C0 E2 4C EA AF C8 28 0B B0 9A 59 F0 4A 7B B8 68 ED 42 2A 25 E6 EE B1 E1 1B 76 E2 EC BE 41 6A AA 7A EA 76 10 91 8C AD 36 ED 31 69 D0 74 35 11 FE B7 48 DD E9 7F 65 28 FB 90 5C 5A 6C 42 3E D7 03 41 53 72 54 19 36 AF 72 7B EC 11 3E 34 9C E1 C0 D3 0B B8 B3 E0 43 41 F1 64 02 BC E2 84 E3 82 09 E5 0A FD..., min:98 20 DD D3 8C 07 ED A3 17 DF 82 8A 66 FF 43 99 E1 18 32 D8 A4 9F 9F 5E 4B 96 22 8A 90 EC 7F F3 E6 97 48 43 8C 7F CF CB 44 70 57 EC 2C 81 8C 09 78 53 A0 B7 CB 52 B8 F3 21 93 3C 6F C4 EA 7F 33 FF 8E 44 64 AB BE FF C5 5A 35 2A C4 40 49 C1 AC CF D3 27 EA F8 55 CE EE 36 80 07 67 44 52 4A 2E BF 5A 12 5B 07 85 9D 41 80 A5 B3 25 D7 46 D5 19 FC FA 14 92 F9 B9 29 A8 79 1C FE 83 44 1F 42 DA..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: FileMetaData(version:1, schema:[SchemaElement(name:spark_schema, num_children:2), SchemaElement(type:INT32, repetition_type:OPTIONAL, name:intKey), SchemaElement(type:BYTE_ARRAY, repetition_type:OPTIONAL, name:payload)], num_rows:10, row_groups:[RowGroup(columns:[ColumnChunk(file_offset:4, meta_data:ColumnMetaData(type:INT32, encodings:[PLAIN, RLE, BIT_PACKED], path_in_schema:[intKey], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:79, total_compressed_size:79, data_page_offset:4, statistics:Statistics(max:46 64 B0 7B, min:8A 0C 12 16, null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)])), ColumnChunk(file_offset:83, meta_data:ColumnMetaData(type:BYTE_ARRAY, encodings:[PLAIN, RLE, BIT_PACKED], path_in_schema:[payload], codec:UNCOMPRESSED, num_values:10, total_uncompressed_size:12365, total_compressed_size:12365, data_page_offset:83, statistics:Statistics(max:7F 63 43 31 78 A7 2E C3 CC 24 6F F0 F9 96 A2 86 F4 BD 6C FD 8E FE 92 F8 8C 00 82 62 DB 8A 29 75 93 0B 4C 09 BC 04 0E 79 9F 8D DA D9 80 AF CB F7 F2 68 6F 84 B3 E0 5E 52 6B ED DD E2 54 5B 62 8F 77 02 99 8B 76 C1 45 05 6D CB 41 4F 97 7A FE 31 F9 9A 10 A5 30 8A 61 0A 81 FE 3E 30 51 51 0A F2 A7 58 34 A3 42 E7 1B FC 5F 7A 76 91 83 E8 70 60 29 FC 4E 9F 9C 20 99 E6 68 55 75 B6 9E 9A B2 A3..., min:9D 5C AA 59 1D 91 55 9A 3B B1 BA 93 87 0B 7B B0 BF 8B 8A 8E 65 F1 71 8D 60 09 DB 74 FC 5D 70 FC B8 7A 16 7B 20 98 29 CC B4 A6 EA 1A C6 E8 D0 79 E7 95 57 C8 97 42 61 98 4E 41 BE 2E E7 AC A8 34 C1 CA A5 21 C5 EE A0 A6 AB 42 A0 4C FD 2A 58 36 C0 BD 84 5D 3D 0C AD 97 C1 6E A7 36 C7 58 D2 06 9A EB 49 FD 4B 57 07 48 55 8A A6 B0 E7 EA 85 CE B6 29 67 2E 5A C2 FF 4E 89 92 3F AC E9 F6 EA F2..., null_count:0), encoding_stats:[PageEncodingStats(page_type:DATA_PAGE, encoding:PLAIN, count:1)]))], total_byte_size:12444, num_rows:10)], key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata, value:{"type":"struct","fields":[{"name":"intKey","type":"integer","nullable":true,"metadata":{}},{"name":"payload","type":"binary","nullable":true,"metadata":{}}]})], created_by:parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea))
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 1641220091,
"min" : 41721044,
"maxBytes" : "+wfTYQ==",
"minBytes" : "1Jx8Ag==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 1751259458,
"min" : 1034620,
"maxBytes" : "QhliaA==",
"minBytes" : "fMkPAA==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 1743433263,
"min" : 108671953,
"maxBytes" : "L67qZw==",
"minBytes" : "0TN6Bg==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 2013584997,
"min" : 432850092,
"maxBytes" : "Zd4EeA==",
"minBytes" : "rMTMGQ==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG ParquetMetadataConverter: {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "intKey",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "INT32",
"typeLength" : 0
}, {
"name" : "payload",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"decimalMetadata" : null,
"primitiveTypeName" : "BINARY",
"typeLength" : 0
} ],
"paths" : [ [ "intKey" ], [ "payload" ] ],
"columns" : [ {
"path" : [ "intKey" ],
"type" : "INT32",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
}, {
"path" : [ "payload" ],
"type" : "BINARY",
"typeLength" : 0,
"maxRepetitionLevel" : 0,
"maxDefinitionLevel" : 1
} ],
"fieldCount" : 2,
"primitive" : false
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"intKey\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"payload\",\"type\":\"binary\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr version 1.8.2 (build aa78e929195723e4f9bf2bbad1b39e7e0277f8ea)"
},
"blocks" : [ {
"columns" : [ {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 79,
"totalUncompressedSize" : 79,
"statistics" : {
"max" : 2075157574,
"min" : 370281610,
"maxBytes" : "RmSwew==",
"minBytes" : "igwSFg==",
"numNulls" : 0,
"empty" : false
},
"firstDataPageOffset" : 4,
"codec" : "UNCOMPRESSED",
"startingPos" : 4,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "INT32",
"path" : [ "intKey" ]
}, {
"encodingStats" : {
"dictionaryEncodings" : [ ],
"dataEncodings" : [ "PLAIN" ]
},
"dictionaryPageOffset" : 0,
"valueCount" : 10,
"totalSize" : 12365,
"totalUncompressedSize" : 12365,
"statistics" : {
"max" : null,
"min" : null,
"maxBytes" : null,
"minBytes" : null,
"numNulls" : 0,
"empty" : true
},
"firstDataPageOffset" : 83,
"codec" : "UNCOMPRESSED",
"startingPos" : 83,
"encodings" : [ "RLE", "PLAIN", "BIT_PACKED" ],
"type" : "BINARY",
"path" : [ "payload" ]
} ],
"rowCount" : 10,
"totalByteSize" : 12444,
"path" : null,
"compressedSize" : 12444,
"startingPos" : 4
} ]
}
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #17
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #17
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 1ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #18
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #18
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 1ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770093_29269; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.22:50010,DS-bb3468dd-3e03-41ac-b1b4-854b353dd7f0,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770093_29269; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.22:50010,DS-bb3468dd-3e03-41ac-b1b4-854b353dd7f0,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@470a696f
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.22:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 20 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 1767502105
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@1eba372c
asked to add Int data, 2010470064
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@21ec5d87
asked to add Int data, 1503106924
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@25f9407e
asked to add Int data, 595939047
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@552518c3
asked to add Int data, 1504941844
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@1a69561c
asked to add Int data, 786117779
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@59aa20b3
asked to add Int data, 711350180
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@363f6148
asked to add Int data, 1325738544
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4b21844c
asked to add Int data, 2050664806
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@1b28f282
asked to add Int data, 1029374237
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@138fe6ec
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #19
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #19
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 0ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #20
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #20
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 0ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770097_29273; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.21:50010,DS-589ba9c4-ca96-4f79-ae58-bec019b77995,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770097_29273; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.21:50010,DS-589ba9c4-ca96-4f79-ae58-bec019b77995,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@5e77f0f4
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.21:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 3 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 1083244878
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@455351c4
asked to add Int data, 529869652
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@29876704
asked to add Int data, 213109580
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4816c290
asked to add Int data, 267970347
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4940809c
asked to add Int data, 1469274339
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@16423501
asked to add Int data, 1480667515
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4efcf8a
asked to add Int data, 912032156
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7a138fc5
asked to add Int data, 674911159
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@379ab47b
asked to add Int data, 1042846280
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@307765b4
asked to add Int data, 168127231
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4a9e6faf
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #21
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #21
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 1ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #22
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #22
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 0ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770098_29274; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.19:50010,DS-d3e4af75-898b-4733-a54a-18e81a7020af,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770098_29274; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.19:50010,DS-d3e4af75-898b-4733-a54a-18e81a7020af,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@2c95ac9e
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.19:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 1 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 210195771
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@459f7aa3
asked to add Int data, 1563916745
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7cc586a8
asked to add Int data, 669898722
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7db534f2
asked to add Int data, 483816099
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@44a2b17b
asked to add Int data, 1960685943
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7a56812e
asked to add Int data, 1820529147
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@2a76b80a
asked to add Int data, 1330841900
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7eb01b12
asked to add Int data, 1097704321
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@2f4854d6
asked to add Int data, 36621049
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@61d9efe0
asked to add Int data, 1555628993
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7e70bd39
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #23
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #23
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 1ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #24
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #24
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 1ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770096_29272; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.16:50010,DS-6b452b44-11c4-4608-86d5-c778915d5d29,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770096_29272; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.16:50010,DS-6b452b44-11c4-4608-86d5-c778915d5d29,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@e6516e
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.16:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 1 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 1235745028
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@43ed0ff3
asked to add Int data, 1108253670
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@388ffbc2
asked to add Int data, 1108912330
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@a50b09c
asked to add Int data, 1792124474
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4da855dd
asked to add Int data, 1311872842
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@6691490c
asked to add Int data, 615316828
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@2187fff7
asked to add Int data, 799710642
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@2e5c7f0b
asked to add Int data, 1638233091
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@21d5c1a0
asked to add Int data, 1512662781
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4de025bf
asked to add Int data, 1025333507
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@538613b3
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #25
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #25
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 0ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #26
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #26
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 0ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770094_29270; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.20:50010,DS-7e06b257-5427-4f5b-bb30-610c5653429c,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770094_29270; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.20:50010,DS-7e06b257-5427-4f5b-bb30-610c5653429c,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@1eef9aef
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.20:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 1 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 871137289
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@5db99216
asked to add Int data, 808304854
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@3ec11999
asked to add Int data, 1304350100
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@5c1bd44c
asked to add Int data, 266507506
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@9f46d94
asked to add Int data, 1047909197
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@18cc679e
asked to add Int data, 1409738409
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@2e77b8cf
asked to add Int data, 218865113
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@2c4ca0f9
asked to add Int data, 432456876
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@67ef029
asked to add Int data, 203043520
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7df587ef
asked to add Int data, 1454089154
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@6e57e95e
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #27
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #27
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 0ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #28
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #28
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 0ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770092_29268; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.23:50010,DS-affd1c1e-3e1c-4abe-86f2-2918066d12ad,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770092_29268; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.23:50010,DS-affd1c1e-3e1c-4abe-86f2-2918066d12ad,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@56db847e
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.23:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 1 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 289123495
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@560cbf1a
asked to add Int data, 787355198
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@5fe8b721
asked to add Int data, 1034620
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@551a20d6
asked to add Int data, 542834271
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@578524c3
asked to add Int data, 282078122
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@64c2b546
asked to add Int data, 607593428
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7e094740
asked to add Int data, 1751259458
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7a11c4c7
asked to add Int data, 719497121
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4cc547a
asked to add Int data, 1451731558
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@7555b920
asked to add Int data, 910831270
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4152d38d
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #29
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #29
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 1ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #30
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #30
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 0ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770095_29271; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.18:50010,DS-02976b48-62ba-4785-b182-5037b8db76fb,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770095_29271; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.18:50010,DS-02976b48-62ba-4785-b182-5037b8db76fb,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@3591009c
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.18:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 1 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 132504183
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@b5cc23a
asked to add Int data, 1743433263
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@5cc5b667
asked to add Int data, 1605161162
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@61edc883
asked to add Int data, 1498782600
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@758f4f03
asked to add Int data, 858735428
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@182f1e9a
asked to add Int data, 1215573346
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@6928f576
asked to add Int data, 449349506
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@660e9100
asked to add Int data, 1336233886
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@69f63d95
asked to add Int data, 1399693374
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@9cd25ff
asked to add Int data, 108671953
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@27e0f2f5
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #31
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #31
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 0ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #32
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #32
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 1ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770090_29266; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.14:50010,DS-695bf534-8da7-4857-80d8-b996b18b3e49,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770090_29266; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.14:50010,DS-695bf534-8da7-4857-80d8-b996b18b3e49,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@3574e198
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.14:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 2 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 564487500
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@db44aa2
asked to add Int data, 1876805267
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@2de366bb
asked to add Int data, 1784190633
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@3f093abe
asked to add Int data, 911374161
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@61a002b1
asked to add Int data, 1094937543
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@4eeea57d
asked to add Int data, 1299288867
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@780ec4a5
asked to add Int data, 2075157574
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@e24ddd0
asked to add Int data, 1824648229
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@6f70f32f
asked to add Int data, 1010500958
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@548e76f1
asked to add Int data, 370281610
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@5aabbb29
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #33
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #33
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 1ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #34
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #34
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 0ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770091_29267; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.15:50010,DS-eea85b06-06f6-470c-925e-43616e9837b7,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770091_29267; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.15:50010,DS-eea85b06-06f6-470c-925e-43616e9837b7,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@1ac85b0c
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.15:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 1 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 1641220091
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@3aa3193a
asked to add Int data, 894232958
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@1ee4730
asked to add Int data, 325302008
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@59a67c3a
asked to add Int data, 442239887
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@5003041b
asked to add Int data, 41721044
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@724bade8
asked to add Int data, 305535016
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@16fb356
asked to add Int data, 182981404
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@6bc248ed
asked to add Int data, 579992201
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@23a9ba52
asked to add Int data, 1220629628
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@ca27722
asked to add Int data, 1175630378
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@70ab80e3
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #35
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #35
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getFileInfo took 0ms
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo sending #36
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo got value #36
17/06/16 10:51:08 DEBUG ProtobufRpcEngine: Call: getBlockLocations took 1ms
17/06/16 10:51:08 DEBUG DFSClient: newInfo = LocatedBlocks{
fileLength=14963
underConstruction=false
blocks=[LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770089_29265; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.13:50010,DS-e0c53985-588a-4578-b388-2ab8e64ea512,DISK]]}]
lastLocatedBlock=LocatedBlock{BP-350330489-10.40.0.11-1492009574738:blk_1073770089_29265; getBlockSize()=14963; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.40.0.13:50010,DS-e0c53985-588a-4578-b388-2ab8e64ea512,DISK]]}
isLastBlockComplete=true}
1. init is called with context: org.apache.parquet.hadoop.api.InitContext@9573b3b
2. called prepare for Read with schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
adding convertor for field: intKey
adding convertor for field: payload
17/06/16 10:51:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 10 records.
17/06/16 10:51:08 INFO InternalParquetRecordReader: at row 0. reading next block
17/06/16 10:51:08 DEBUG DFSClient: Connecting to datanode 10.40.0.13:50010
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 46 bytes
17/06/16 10:51:08 DEBUG BytesInput: BytesInput from array of 10286 bytes
17/06/16 10:51:08 INFO InternalParquetRecordReader: block read in memory in 2 ms. row count = 10
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: initializing Record assembly with requested schema message spark_schema {
optional int32 intKey;
optional binary payload;
}
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 46 to byteArray of 46 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 46 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG PlainValuesReader: init from page at offset 6 for length 40
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
17/06/16 10:51:08 DEBUG ColumnReaderImpl: loading page
17/06/16 10:51:08 DEBUG BytesInput: converted 10286 to byteArray of 10286 bytes
17/06/16 10:51:08 DEBUG ColumnReaderImpl: page size 10286 bytes and 0 records
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading repetition levels at 0
17/06/16 10:51:08 DEBUG ByteBitPackingValuesReader: reading 0 bytes for 0 values of size 0 bits.
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading definition levels at 0
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: decoding bitWidth 1
17/06/16 10:51:08 DEBUG ColumnReaderImpl: reading data at 6
17/06/16 10:51:08 DEBUG BinaryPlainValuesReader: init from page at offset 6 for length 10280
17/06/16 10:51:08 DEBUG RunLengthBitPackingHybridDecoder: reading 10 values RLE
asked to add Int data, 817377054
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@78461bc4
asked to add Int data, 1632183837
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@544630b7
asked to add Int data, 880151679
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@64f857e7
asked to add Int data, 1962239826
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@1095f122
asked to add Int data, 2013584997
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@58c540cf
asked to add Int data, 1521291452
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@3d6300e8
asked to add Int data, 1780743047
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@1b822fcc
asked to add Int data, 1315217383
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@24a1c17f
asked to add Int data, 1431159712
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@56102e1c
asked to add Int data, 432850092
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
asked to add binary data, sz 1024
17/06/16 10:51:08 DEBUG ColumnReaderImpl: end reached
17/06/16 10:51:08 DEBUG InternalParquetRecordReader: read value: com.ibm.crail.parquet.AtrRecord@73511076
Read : 100 record, took 175182 usec, roughly 1751829 ns/record
17/06/16 10:51:08 DEBUG Client: stopping client from cache: org.apache.hadoop.ipc.Client@1583741e
17/06/16 10:51:08 DEBUG Client: removing client from cache: org.apache.hadoop.ipc.Client@1583741e
17/06/16 10:51:08 DEBUG Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@1583741e
17/06/16 10:51:08 DEBUG Client: Stopping client
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo: closed
17/06/16 10:51:08 DEBUG Client: IPC Client (1168924571) connection to flex11-40g0/10.40.0.11:9000 from demo: stopped, remaining connections 0
demo@flex11:~$