Prokaryotic Genomes

• sync of Archaea done partially by Pierre and partially by Donovan using different scripts; have settled on the method implemented as ncbi_sync in the GTDB Migration Tk which specifically limits requests per second to minimize errors messages from NCBI

• Bacterial RefSeq genomes (assembly_summary_bacteria_refseq_nolargeproject.txt):

  • 526,771 genomes with 148,760 identified as unchanged
  • 58.5 hours using 16 requests per second (--rate 16)
  • 1 failed genomes (GCF_056491165.1 which is now suppressed at NCBI)

• ncbi_genome_sync: GTDB Migration Tk was modified after the RefSeq genomes were sync'ed to sync all required genomes for a GTDB release:

  • 1,346,119 total genomes: 199,839 new genomes; 83,285 requiring update
  • 43 hours using 20 requests per second (--rate 20)
  • 1 failed genomes (GCA_056491145.1 which is now suppressed at NCBI)

• ncbi_genome_sync: Verification of FTP sync:

  • 1,319 genomes passed verification after fix to GTDB Migration Tk to check the uncompressed_checksums.txt file. Unfortunately, the md5checksums.txt file is out-of-date for a small number of *_fcs_report.txt report files, but the correct MD5 sum is in uncompressed_checksums.txt file.

• update_genomes: Copy genomes to GTDB genome directory

  • copied 1,043,035 genomes (444,299 RefSeq, 598,736 GenBank)
  • 1 failed genomes (GCF_030217685.1 which is now suppressed at NCBI)

• trans_table: Run gTranslate over GTDB genomes

  • ~23 hrs for the ~1.3 million GTDB R237 genomes when run using 96 CPUs on Chifley, Howard, Holt, Hurley, and Page (i.e. effectively using 480 CPUs)
  • 786,144 genome(s) have a translation table from NCBI, and gTranslate disagrees with NCBI on 377 of them (0.05%)

• prodigal: Run Prodigal over GTDB genomes

  • took ~10 hrs for the ~300,000 GTDB R237 genomes without prior Prodigal results when run using 96 CPUs on Chifley, Howard, Holt, Hurley, and Page (i.e. effectively using 480 CPUs)
  • 8 genome(s) had no called genes; this is acceptable as Prodigal does fail to process extremely poor quality genomes (e.g. excessive number of ambiguous bases)

• hmmsearch: Run hmmsearch over GTDB genomes

  • took ~9 hrs for the ~300,000 GTDB R237 genomes without prior Pfam results when run using 96 CPUs on Chifley, Howard, Holt, Hurley, and Page (i.e. effectively using 480 CPUs)
  • took ~15 hrs for the ~300,000 GTDB R237 genomes without prior TIGRfam results when run using 96 CPUs on Chifley, Howard, Holt, Hurley, and Page (i.e. effectively using 480 CPUs)

• genomic_metadata: Calculate common gene and protein metadata attributes

  • took ~9 hrs for the 1,346,118 GTDB R237 genomes using 40 CPUs
  • 3 genomes failed: GCA_056140655.1, GCA_056491145.1, GCF_056491165.1 (all suppressed in RefSeq and GenBank)

• trnascan: Run tRNAscan-SE over GTDB genomes

  • took ~20 hrs for the ~300,000 GTDB R237 genomes without prior tRNA results when run using 96 CPUs on Chifley, Howard, Holt, Hurley, and Page (i.e. effectively using 480 CPUs)

• rna_silva: Identify and classify 5S, 16S, and 23S rRNA genes

  • 16S: ... seems stuck on one genome ...

• checkm: Run CheckM over new/updated GTDB genomes

  • 7 genomes not assess as they either have no protein file (3 genomes) or exceed the 100 Mbp size limit (4 genomes)
  • Important to use pplacer alpha22 use earlier version of pplacer have a corner case bug that causes it to crash on a number of good genomes
  • took ~3 days for the ~300,000 GTDB R237 genomes when run using 96 CPUs on Chifley, Howard, Holt, Hurley, and Page (i.e. effectively using 480 CPUs)

• checkm2: Run CheckM v2 over new/updated GTDB genomes

  • 10 genomes not assess as they either have no protein file (3 genomes) or exceed the 100 Mbp size limit (7 genomes)
    • 3 more genome failed the genome size criterion as this filtering was only added part way through the CheckM v1 execution
  • took ~21 hrs for the ~300,000 GTDB R237 genomes when run using 96 CPUs on Chifley, Howard, Holt, Hurley, and Page (i.e. effectively using 480 CPUs)

Fungal Genomes

• Sync of fungal genomes:

  • 26,328 genomes all download from scratch
  • ~5 hrs with 20 requests per second
  • 230 genomes failed with HTTP 404 errors; manual spot checking confirmed these are genomes without required NCBI data files
  • 17 genomes where uncompressed_checksums.txt was used to validated file MD5 checksum

• Verification of FTP sync:

  • 22 min to verify 26,328 genomes
  • 230 genomes reported as missing a directory as expected

• Copy genomes to GTDB genome directory

  • 26,098 genomes to copy (674 RefSeq, 25,424 GenBank)
  • ~1 hr