Output Files - Golob-Minot/geneshot GitHub Wiki

Once you've run Geneshot, you will want to look at the results. Because Geneshot runs many different types of analyses, you may have to consult this documentation in order to find the element of the results which is most important for your application.

Many of the individual aspects of the results can be found in two formats:

• A compressed text file in the output directory specified by --output
• A table in either the ${output_prefix}.results.hdf5 or ${output_prefix}.details.hdf5 output files. These are both written to the same output directory using the file name prefix specified by --output_prefix

NOTE: The file format HDF5 is used to collect all of the results because of its flexible support for combining multiple tabular datasets into a single data object. We have found that the most convenient way to read and write this data format is with the Python/Pandas library.

Example:

import pandas as pd
manifest_df = pd.read_hdf("geneshot.results.hdf5", "/manifest")

If the --savereads flag is used, all of the preprocessed WGS FASTQ files will be written to the qc/ folder.

The table used as input for the entire analysis will be saved in the results.hdf5 under /manifest.

Each sample can be summarized overall with a number of metrics, including:

• The number of paired-end reads input per sample (2X the number of read pairs, to be explicit)
• The number of paired-end reads which successfully align against the gene catalog
• The estimated total number of protein coding genes in a given sample (using the breakaway algorithm)
• The number of genes detected by assembly
• The number of genes detected by alignment

For example:

/summary/all

All of the contigs, scaffolds, gene annotations, and logs produced during de novo assembly for each specimen will be found in the assembly/<specimen>/ folder.

A CSV with information on every protein-coding sequence assembled from every sample (operationally termed 'alleles'), including the contig, position, sequencing depth, and GC content will be written to assembly/allele.assembly.metrics.csv.gz. This table will also be available in details.hdf5 under /abund/allele/assembly/<specimen> with one table per specimen.

Example:

The DIAMOND reference database for all of the deduplicated gene sequences will be written to ref/genes.dmnd. This file can be converted into FASTA by the user if needed.

A two column CSV describing which genes were grouped into which samples will be written to ref/CAGs.csv.gz. Columns are CAG and gene, with one row per gene. It will also be available in results.hdf5 under /annot/gene/cag.

Example:

A short summary of every CAG will be available in results.hdf5 under /annot/cag/all, containing the:

• Size (number of genes) per CAG
• Prevalence: proportion of samples in which the CAG was observed with a relative abundance > 0
• Mean abundance: Average abundance across all samples
• Std. abundance: Standard deviation of the abundance values across all samples

Example:

Abundance of every CAG in every sample in feather format will be written to abund/CAG.abund.feather. This table will be available in results.hdf5 under /abund/cag/wide. Abundances are calculated as the sum of the gene-level sequencing depth divided by the sum of sequencing depth for all genes in that sample.

Example:

Abundance of every gene in every sample in will be available in results.hdf5 under /abund/gene/long/<specimen>. Here is an example of the information found in that table:

The taxonomic assignment for each gene (DIAMOND output in taxonomic assignment mode) in TSV format will be written to annot/genes.tax.aln.gz. It will also be available in results.hdf5 under /annot/gene/tax.

Example:

The functional assignment for each gene (eggNOG output) in TSV format will be written to annot/genes.emapper.annotations.gz. It will also be available in results.hdf5 under /annot/gene/eggnog.

Example:

The taxonomy used for the taxonomic annotation is provided in the output HDF5 under /ref/taxonomy, e.g.:

For convenience, ordination is performed using the CAG abundance information to help provide a quick measure of sample similarity to the user. This includes both PCA and t-SNE, and the output tables can be found in results.hdf5 under /ordination/pca and /ordination/tsne. Examples of those outputs are:

A CSV with the Corncob results for this dataset will be written to stats/corncob.results.csv. It will also be available in results.hdf5 under /stats/cag/corncob.

Example:

Here's an example of how the output files can be structured. This is what you get when you analyze some of the test data, so depending on how many samples you have, etc., you will see different exact set of files, but in the same pattern.

output
├── [ 160]  abund
│   ├── [ 664]  CAG.abund.feather
│   ├── [ 130]  CAG.readcounts.csv.gz
│   └── [ 224]  details
│       ├── [2.9K]  Mock__11.json.gz
│       ├── [ 404]  Mock__12.json.gz
│       ├── [3.7K]  Mock__13.json.gz
│       ├── [3.1K]  Mock__14.json.gz
│       └── [3.1K]  Mock__15.json.gz
├── [ 224]  assembly
│   ├── [ 224]  Mock__11
│   │   ├── [ 14K]  Mock__11.contigs.fasta.gz
│   │   ├── [1.8K]  Mock__11.csv.gz
│   │   ├── [ 10K]  Mock__11.faa.gz
│   │   ├── [2.9K]  Mock__11.gff.gz
│   │   └── [ 67K]  Mock__11.megahit.log
│   ├── [ 224]  Mock__12
│   │   ├── [1.8K]  Mock__12.contigs.fasta.gz
│   │   ├── [ 399]  Mock__12.csv.gz
│   │   ├── [1.3K]  Mock__12.faa.gz
│   │   ├── [ 653]  Mock__12.gff.gz
│   │   └── [ 51K]  Mock__12.megahit.log
│   ├── [ 224]  Mock__13
│   │   ├── [ 15K]  Mock__13.contigs.fasta.gz
│   │   ├── [3.4K]  Mock__13.csv.gz
│   │   ├── [ 11K]  Mock__13.faa.gz
│   │   ├── [6.8K]  Mock__13.gff.gz
│   │   └── [ 68K]  Mock__13.megahit.log
│   ├── [ 224]  Mock__14
│   │   ├── [ 16K]  Mock__14.contigs.fasta.gz
│   │   ├── [2.1K]  Mock__14.csv.gz
│   │   ├── [ 11K]  Mock__14.faa.gz
│   │   ├── [3.4K]  Mock__14.gff.gz
│   │   └── [ 67K]  Mock__14.megahit.log
│   └── [ 224]  Mock__15
│       ├── [ 16K]  Mock__15.contigs.fasta.gz
│       ├── [2.0K]  Mock__15.csv.gz
│       ├── [ 11K]  Mock__15.faa.gz
│       ├── [3.2K]  Mock__15.gff.gz
│       └── [ 67K]  Mock__15.megahit.log
├── [316K]  geneshot.details.hdf5
├── [372K]  geneshot.results.hdf5
├── [  96]  qc
│   └── [  86]  readcounts.csv
├── [  96]  ref
│   └── [ 21K]  genes.dmnd
└── [ 128]  stats
    ├── [1.4K]  corncob.results.csv
    └── [ 104]  geneshot.breakaway.csv.gz