Notes - USF-HII/snptk GitHub Wiki
Notes
using multiprocessing (just chromosome 1 .d):
Command being timed: "python3 /home/j/jmoreno8/dev/usf-hii/snptk/tmp/test_futures.py --input_dir /home/j/jmoreno8/dev/usf-hii/snptk/tmp/data/grch38_json-03-11-2020/refsnp-chr1.json.bz2.d --outfile /home/j/jmoreno8/dev/usf-hii/snptk/test.out"
User time (seconds): 11169.10
System time (seconds): 66.84
Percent of CPU this job got: 1157%
Elapsed (wall clock) time (h:mm:ss or m:ss): 16:10.49
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9042504
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 18112496
Voluntary context switches: 371116
Involuntary context switches: 1526124
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
using concurrent futures (just chromosome 1 .d):
Command being timed: "python3 /home/j/jmoreno8/dev/usf-hii/snptk/tmp/test_futures.py --input_dir /home/j/jmoreno8/dev/usf-hii/snptk/tmp/data/grch38_json-03-11-2020/refsnp-chr1.json.bz2.d --outfile /home/j/jmoreno8/dev/usf-hii/snptk/test.out"
User time (seconds): 11205.34
System time (seconds): 63.19
Percent of CPU this job got: 1159%
Elapsed (wall clock) time (h:mm:ss or m:ss): 16:11.64
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9042584
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 17869878
Voluntary context switches: 324886
Involuntary context switches: 1486430
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
splitting bz2 file into 32 parts and parsing through 32 files: :
mkdir: created directory ‘/home/j/jmoreno8/dev/usf-hii/snptk/tmp/chr1_split’
Calculating size of /home/j/jmoreno8/dev/usf-hii/snptk/tmp/data/grch38_json-03-11-2020/refsnp-chr1.json.bz2 in bytes...
Splitting file into 32 splits in directory /home/j/jmoreno8/dev/usf-hii/snptk/tmp/chr1_split...
Complete
Command being timed: "/home/j/jmoreno8/dev/usf-hii/snptk/bin/snptk-split /home/j/jmoreno8/dev/usf-hii/snptk/tmp/data/grch38_json-03-11-2020/refsnp-chr1.json.bz2 /home/j/jmoreno8/dev/usf-hii/snptk/tmp/chr1_split 32"
User time (seconds): 34864.93
System time (seconds): 1796.81
Percent of CPU this job got: 141%
Elapsed (wall clock) time (h:mm:ss or m:ss): 7:13:07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4224
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 510347
Voluntary context switches: 144085808
Involuntary context switches: 2909452
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
parsing through 32 files:
Command being timed: "python3 /home/j/jmoreno8/dev/usf-hii/snptk/tmp/test_futures.py --input_dir /home/j/jmoreno8/dev/usf-hii/snptk/tmp/chr1_split --outfile /home/j/jmoreno8/dev/usf-hii/snptk/tmp/test.out"
User time (seconds): 11560.45
System time (seconds): 72.77
Percent of CPU this job got: 805%
Elapsed (wall clock) time (h:mm:ss or m:ss): 24:03.42
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9047072
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 20515686
Voluntary context switches: 382373
Involuntary context switches: 1544888
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
parsing through just bz2 file original:
Command being timed: "python3 /home/j/jmoreno8/dev/usf-hii/snptk/tmp/t2.py /home/j/jmoreno8/dev/usf-hii/snptk/tmp/data/grch38_json-03-11-2020/refsnp-chr1.json.bz2"
User time (seconds): 18821.01
System time (seconds): 6.43
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 5:13:56
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 19904
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1234732
Voluntary context switches: 312
Involuntary context switches: 46279
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Parsing Rsmerge bz2 file original
Command being timed: "bin/snptk-parse-rsmerge-json.py --input_file tmp/data/rsmerge-json-03-11-2020/refsnp-merged.json.bz2 --outfile tmp/data/Rsmerge"
User time (seconds): 1345.16
System time (seconds): 0.42
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 22:25.87
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 25380
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 98943
Voluntary context switches: 67
Involuntary context switches: 3851
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0 ```