Updating GTDB species representatives - Ecogenomics/GTDBNCBI GitHub Wiki

Updating and selecting new GTDB species representatives is done using the GTDB Species Cluster Toolkit. It consists of a number of steps which need to be done in serial and may require manual intervention to resolve challenging cases not currently covered by the implemented methodology. Such cases are indicated as warnings or errors in the output log of each step and may result in a specific output file which can be used to further understand the underlying issue.

Once the GTDB species clusters are established, a new curation tree can be inferred for the species representatives and this tree provided to the curation team for manual curation of taxa above the rank of species. The curation team may also update the name (but not the representative genome) of species clusters based on the latest taxonomic opinion as expressed in published literature. Once manual curation is finished a set of post-curation validation steps have been implemented to identify common issues. It is typical for some issues to arise during validation and these must be resolved before publishing the GTDB.

GTDB currently uses a ledger system (set of files) to indicate exceptions that would otherwise result in incorrect formation of species clusters or naming of taxa. These files are all stored on OneDrive in ./ledgers/. These ledger add a substantial amount of complexity and data ambiguity to the GTDB, and this scheme should be re-examined at some point. A few notes on side effects and issues caused by the ledgers:

• ledgers are filled out as accurately as possible using data available at the time. This means that ledger entries may become invalid as external data changes. For example, a genome is added to the ncbi_untrustworth_sp_assignment ledger, but the species assignment of this genomes is later updated at NCBI. Ledgers should be inspected before each release to ensure they are still valid, but in practice this can be challenging given limited resources and the manual nature of verifying the correctness of ledger entries.
• ledgers effectively change the interpretation of data as it standard in the GTDB and this isn't reflected on the GTDB website. For example, the ncbi_env_bioprojects ledger indicates NCBI BioProjects where all genomes should be considered MAGs. However, the data in this ledger is never propagated to the GTDB database and is not reflected on the GTDB website.

The ledgers are stored on OneDrive in the ledgers directory. Ledger files should be copied from the previous release (e.g. ledgers/r226) to a new directory (e.g. ledgers/r232) and updated as required. These ledger files must also be copied into TSV files for processing by the GTDB Species Cluster Toolkit (note that some ledgers have 2 sheets!). These should be placed in the metadata/<release>/representatives/ledgers directory.

The tree from the prior release should be manually inspected and any genomes on excessively long branches that represent clear issues should be added to the qc_expections ledger with the Include field set to FALSE.

This document uses the following two placeholders to indicate the name of paths and files:

• <cur>: indicates the current release number, e.g. 232
• <prev>: indicates the previous release number, e.g. 226

Metadata, genome path, and domain information files are pulled from the GTDB using the GTDB CLI. This must be updated to point to the latest GTDB database by updating the following fields in Config.py:

• NCBI_PREFIX
• DB_SERVERS
• LATEST_DB

Once updated, the following TSV files can be generated using the GTDB CLI:

• gtdb metadata export --format tab --output gtdb_r<cur>_metadata.tsv.gz
• gtdb power genome_paths --output gtdb_r<cur>_genome_paths.tsv.gz
• gtdb power domain_report --output gtdb_r<cur>_domain_report.tsv.gz

The species clustering relies on nomenclatural information and metadata for genome in the previous release. This is in the ar53_metadata_r<prev>.tar.gz and bac120_metadata_r<prev>.tar.gz files provided with each GTDB release.

The gtdb_inputs.toml file should be copied from the previous release and updated to indicate the current release and root_dir. In addition, all data_files must be updated to properly indicate current and previous release versions and the date of the lpsn_gss_file and ltp_taxonomy_file files updated as required. The Living Tree Project (LTP) taxonomy file and output directory name also need to be updated is a new LTP version has been run across the GTDB genomes (see: Generate Ribosomal Metadata).

Species clusters are updated by running all the commands beginning with a u in the order specified in the CLI help menu for the GTDB Species Cluster Toolkit. This workflow should be performed in the directory /srv/db/gtdb/metadata/release<cur>/representatives/sp_cluster_update as follows:

1. u_new_genomes: identify new and updated genomes in current GTDB release
   • gtdb_species_clusters u_new_genomes gtdb_inputs.toml
   • good sanity checks are to ensure the number of lost GTDB representatives and genomes without GTDB metadata as indicated in the log file are both low (say, <50) and that the number of new genomes corresponds to expectations
2. u_qc_genomes: quality check new and updated genomes
   • gtdb_species_clusters u_qc_genomes gtdb_inputs.toml
   • the log file will report any cases where the GTDB domain report, GTDB taxonomy, and NCBI taxonomy disagree. It is critical that the GTDB taxonomy indicate the correct domain as this will be used to determine which domain level tree the genome belongs in. If there is uncertainty the genome should be added to the qc_exceptions ledger and the Include field for the genome set to FALSE.
3. u_gtdbtk: perform initial classification of new and updated genomes using GTDB-Tk
   • gtdb_species_clusters u_gtdbtk gtdb_inputs.toml -c 64
   • Note: this step is resource intensive and takes a couple days to run even running batches in parallel across several servers.
   • Recommendation: recommend running this step across multiple servers so it takes only a day or two to complete
   • GTDB-Tk results are used as the initial GTDB assignment for new genomes. This helps with manual curation since the majority of new genomes now have correct GTDB classifications. GTDB-Tk must be using reference data from the last GTDB release (e.g. R226 for the R232 update).
4. u_lpsn_rna_types: identify type genomes based on type 16S rRNA sequences indicated at LPSN
   • gtdb_species_clusters u_lpsn_rna_types gtdb_inputs.toml -c 64
   • genomes in the output table lpsn_ssu_type_genomes.tsv where the Is GTDB type genomes is FALSE should be added to the gtdb_type_strains ledger (both on OneDrive and in the ../ledgers directory) unless there is conflicting evidence suggesting the genome is not in fact the type strain of the species despite having a type 16S sequence. I recommend re-running this command to verify there are no missing type genomes (or that any exceptions are understood).
5. u_resolve_types: resolve cases where a species has multiple genomes assembled from the type strain
   • gtdb_species_clusters u_resolve_types gtdb_inputs.toml -c 96
   • the file unresolved_type_strain_genomes.tsv lists species with multiple divergent genomes indicated as being assembled from the type strain and where the correct genome could not be automatically resolved. Ideally, this file would be manually inspected and external resources consulted to determine which of the genomes is actually assembled from the type strain of the species. In practice, this is challenging and the <10 cases that occur have just been left unresolved. This means a random genome will be selected as the GTDB representative which isn't ideal, but is sensible given resource limitations to validate the correct assembly.
   • the key file produced by this step is untrustworthy_type_material.tsv which lists genomes that are annotated as being assembled from the type strain of a species, but where evidence exists indicating the genome is not type material. This occurs for any species where there are multiple, divergent genomes all claiming to be assembled from the type strain and some criteria could be used to establish the correct or at least most likely genome to actually be assembled from the type strain.
   • two types of warnings are commonly produced by this command and can generally be ignored:
     • "Retaining genome {genome_id} from {species} despite being marked as untrustworthy as type or not used as type at NCBI": these occur when there is evidence that these genomes are assembled from the type strain and removing them would result in there being no type strain for the species. The assumption here is that the NCBI assertion is incorrect. It appears NCBI will mark a genome as not used as type if it has any potential issues such as genome length too large or even genus undefined, but our experience is that these assertions are generated automatically and not always correct.
     • "Genome {genome_id} from {species} marked as untrustworthy, but this conflicts with high confidence LTP 16S rRNA assignment; ...": these occur when a genome has a 16S rRNA sequence that does appear to be from the expected NCBI species, but it was still flagged as being untrustworthy as type material. This can occur for legitimate reasons. For example, 3 genomes claiming to be the type strain for a species, with one genome being divergent from the other 2.
6. u_rep_changes: identify lost, updated, reassigned, and unchanged species representatives
   • gtdb_species_clusters u_rep_changes gtdb_inputs.toml
   • warnings should be investigated and appropriate action taken though admittedly some of these have proven extremely hard to understand (e.g. reassignment of G000817735 in GTDB R202). One common warning is Updated GTDB representative XZY reassigned from ... which occurs when the genomic FASTA file from a genome is updated and as a result it now clusters with a new species cluster instead of with its previous version. Such warning can be ignored as they are handled by the next step.
   • the gtdb_disband_cluster.tsv ledger indicates GTDB species clusters in the previous GTDB release that should be explicitly disbanded (i.e. no longer given any priority to form a species cluster in this release). In general, this ledger should be empty. In the past, we have used this ledger to indicate that Shigella species clusters should be disbanded since we decided to make them synonyms of E. coli (see here). This ledger was also needed when we changed the alignment fraction (AF) criteria from 0.65 to 0.5. All clusters that were impacted by this change had to be disbanded so they could be form new clusters that adhered to the new criteria.
7. u_rep_actions: perform initial actions required for modified representatives
   • gtdb_species_clusters u_rep_actions gtdb_inputs.toml -c 96
   • GTDB preferentially select isolate genomes as representative, including the replacement of existing representatives. This can cause issues if a number of genomes are incorrectly considered to be isolates which are actually MAGs. To help mitigate this issue, BioProjects resulting in 2 or more representatives being replaced are indicated with a warning (e.g. BioProject PRJNA625506 responsible for 2 representatives being replaced with presumed isolates.). These BioProjects should be looked up on NCBI to verify they do in fact represent isolates. If a BioProject is found to be a set of MAGs it should be added to the ncbi_env_bioprojects.tsv ledger, ideally brought to the attention of NCBI, and this step rerun so it reflects the genomes being MAGs. Please update this list of BioProjects verified to contain isolate genomes.
   • the most common error produced is in regards to the ambiguity of naming priority (e.g. Species need to be manually resolved in priority ledger). ALL such cases need to be manually resolved, added to the species_priority.tsv ledger, and this step re-run. This is admittedly time consuming, but necessary if the GTDB is to properly reflect naming priority. The list of ambiguous priorities is given in ambiguous_sp_priority.tsv. Resolving the initial list of ambiguous naming priorities often results in additional cases that need to be resolved.
   • multiple reassignment warnings (e.g. Representative G000297375 was reassigned multiple times) can generally be ignored. These are reported as warnings to allow for a quick inspection and to ensure that this remains an unusual case. The most common situation is for the second reassignment to be due to nomenclatural priority which should always trump any other reason for a reassignment.
   • other warnings should be investigated, but generally indicate challenging situations which are resolved adequately
8. u_sel_reps: select representatives for all named species at NCBI
   • gtdb_species_clusters u_sel_reps gtdb_inputs.toml -c 96
   • warnings should be investigated, but generally indicate challenging situations which are resolved adequately. The most common warning is regarding multiple species clusters with the same name (e.g. s__Pseudarthrobacter sulfonivorans is represented by both [('G001484605', False)] and [('G014712225', True)]). These are reported so these species can be sanity checked at the end of the species clustering workflow, but should ultimately be resolved correctly. This situation occurs since at this stage in the pipeline species clusters do not have finalized names so any new introduction of type material can result in the duplicate use of names.
   • this method currently hard codes the fact that Shigella species are considered synonyms of E. coli in GTDB as of release R207 (this should be moved to a ledger)
9. u_cluster_named_reps: cluster genomes to selected GTDB representatives
   • gtdb_species_clusters u_cluster_named_reps gtdb_inputs.toml -c 96
   • warning regarding ANI neigbhours having an ANI slightly >97% (usual threshold for merging) should be monitored (10 cases in R202; 15 cases in R207; 13 cases in R226; 12 cases in R232), but are expected. A "fudge factor" of 0.1% ANI was put in place to stop previous GTDB representatives from being merged when they are extremely close to the 97% boundary in order to account for small changes to genomes and any differences between Skani versions. If the reported ANI value is >97.1% the situation should be investigated and likely the species should be merged at this point.
   • the "Identified N genome pairs meeting ANI radius criteria, but with an AF <0.5" is expected and should be monitored. The AF criterion was changed from 0.65 to 0.5 starting in R207. These represent cases where perhaps genomes should be merged into a single species, but ultimately the criteria is a bit arbitrary. So long as this number remains low (20 in R207; 0 in R232) it isn't critical though a formal exploration of the best AF criteria to use would be interesting.
10. u_cluster_de_novo: infer de novo species clusters and representatives for remaining genomes
    • gtdb_species_clusters u_cluster_de_novo gtdb_inputs.toml -c 96
    • no warnings are expected
11. u_cluster_stats: summary statistics indicating changes to GTDB species cluster membership
    • gtdb_species_clusters u_cluster_stats gtdb_inputs.toml
    • the logging output should be consulted to verify that few genomes have migrated between species clusters (<1%) and that few species representatives have been changed (<5%), lost (<5%), or merged (<1%). It is critical that genomes do not "bounce" between different species clusters with each GTDB release and that the GTDB representatives genomes are largely stable (though, not 100% stable since we replace representatives when suitably higher quality genomes become available). If these values are high it may indicate an issue with the workflow or some major change to the set of genomes at NCBI which needs to be investigated.
12. u_ncbi_erroneous: identify genomes with erroneous NCBI species assignments under the GTDB
    • gtdb_species_clusters u_ncbi_erroneous gtdb_inputs.toml
    • an unrecoverable error will be produced whenever there are multiple effective type strain genomes from the same species in multiple species clusters. In these cases, it is unclear which cluster represents the actual species and thus how erroneous NCBI classifications should be established. These should all be species that couldn't be resolved by u_resolve_types and thus appear in the unresolved_type_strain_genomes.tsv file. These cases need to be resolved at this time by adding all offending genomes to the ncbi_untrustworthy_sp_assignments ledger as these genomes represent multiple species making the NCBI assignment to a single species untrustworthy.
    • a large number of genomes are identified as misclassified at NCBI as the GTDB and NCBI have different species delineating criteria:
      • R220: 35,820 genomes from 2,534 species
      • R226: 43,417 genomes from 3,016 species
      • R232: 67,133 genomes from 3,400 species
13. u_synonyms: determine synonyms for validly or effectively published species
    • gtdb_species_clusters u_synonyms gtdb_inputs.toml
    • no warnings are expected
    • R220:
      • 570 GTDB representatives resulting in 663 type strain synonyms
      • 146 GTDB representatives resulting in 165 majority vote synonyms
    • R226:
      • 652 GTDB representatives resulting in 775 type strain synonyms
      • 144 GTDB representatives resulting in 164 majority vote synonyms
    • R232:
      • 693 GTDB representatives resulting in 804 type strain synonyms
      • 147 GTDB representatives resulting in 166 majority vote synonyms
14. u_curation_trees: produce curation trees highlighting new NCBI taxa; GTDB species cluster with >= 1 genomes assigned to a new NCBI taxa are highlighted

• gtdb_species_clusters u_curation_trees gtdb_inputs.toml
  • no warnings are expected
  • these trees need to be provided to the GTDB curators to help them identify changes between releases

15. u_species_init: produce initial best guess at names for GTDB species clusters
    • gtdb_species_clusters u_species_init gtdb_inputs.toml
    • Important: this method makes use of historical GTDB assignments to refine results and the GTDB taxonomy files for all previous releases need to be in the /srv/projects/gtdb/data/taxonomy_gtdb directory.
    • this is the most questionable part of the GTDB species cluster workflow as correctly assigning names at this stage is challenging since the curators have not yet establish the final genus names. As such, this step is a best guess at the names which are then further refined in the next step which implements a set of post-curation rules. Curators need the majority of species names to be correctly establish to help with curation, but genus curation needs to be done before final species names can be established. A chicken and egg situation, and the current solution works but is not ideal.
    • errors regarding assignment of synonyms should be inspected, but are general the result of a genus transfer resulting in the required use of the synonym name with lower naming priority (e.g. Gemmiger qucibialis vs. Cibiobacter qucibialis)
    • warnings indicating conflicts between GTDB ledgers and NCBI assignments are expected. These largely occur as the result of a species being assigned to a different genus. These conflicts should be monitored to confirm that our ledgers are correct. Masha is the best person to verify that these warnings are acceptable.
    • warnings regarding species name of type strain representatives already being in use are acceptable (6 cases in R232)
    • other warnings are also acceptable and are provided with the idea that intuition may indicate that something is off, but names will be refined in the next step
16. u_pmc_species_names: refine species names using post-manual curation rules
    • gtdb_species_clusters u_pmc_species_names gtdb_inputs.toml manual_sp_file.tsv
    • the manual_sp_file.tsv is used to indicate names that should be set based on manual curation and is used to resolve any warnings/errors produced by this step (see previous GTDB release to see format of this file); this file needs to exist, but can be empty if no manual curation is required at this point
    • all warnings should be understood and deemed acceptable
    • all errors should be resolved. In particular, all GTDB species clusters must have a unique name. Violations of this requirement are indicated by errors of the form Species name <name> assigned to at least 2 GTDB representatives. This can be resolved by adding genomes to the manual_sp_file.tsv. It is often helpful to look at the previous GTDB classification of these genomes in order to resolve these violations.
    • species names do not need to be 100% correct at this point since this is done through the post-manual curation (PMC) workflow that is conducted once GTDB curators have finalized genus names. However, GTDB curators use these initial species names to aid in the manual curation so a reasonable effort should be made at this point.
    • the file gtdb_sp_clusters.ncbi_sp.tsv should be passed along to the curation team
    • the file species_classification_ledger_updates.tsv should be inspected and cases resulting in a name change evaluated to see if they are still valid or is the species classification ledger needs modification

The new GTDB representative genomes and updated species clustering information should be added to the GTDB after completing the above workflow. This can be done using the GTDB Migration Toolkit:

gtdb_migration_tk update_reps_db --hostname <hostname> -u <user> -p <password> -d <database> --final_cluster_file ./10_u_cluster_de_novo/gtdb_clusters_de_novo.tsv

A new metadata file with the updated GTDB species clustering should then be dumped from the database and this metadata file used for any subsequent processing:

gtdb metadata export --format tab --output gtdb_r<cur>_metadata.updated_reps.<date>.tsv

• ToDo: should modify these output files so they follow the new NCBI directory structure format (e.g. GCA/0123/456/789) as otherwise there can be a problematic number of files in a single directory
• Use the GTDB Release Toolkit to create new protein (gtdb_proteins_aa_reps_r<#>.tar.gz) and nucleotide (gtdb_proteins_nt_reps_r226.tar.gz) files and send these to Ben Woodcroft so he can start updating the SingleM results for Sandpiper. Also send Ben the initial (draft) version of the GTDB metadata file with genome size and CheckM v1/v2 estimates.
• Send the species cluster file (gtdb_r<#>_species_clusters.tsv) file to Daan Speth so he can start updating GlobDB. WIP: Daan also needs the genomic FASTA file for all representative genomes (gtdb_genomes_reps_r<#>.tar.gz) which can be pulled together using the GTDB Release Toolkit (may need some work to ensure this can be done at this point in the process).

Manual curation is performed after updating the GTDB species clusters. The following information should be provided to curators by creating a directory in /srv/projects/gtdb/{release}/{domain}/pre_curation/bac120/{date} which contains the following files:

• trees produced by 14_u_curation_trees in the directory ncbi_new_taxa_trees
• bacterial and archaeal snapshot trees and associated files: gtdb_<release>_bac120.decorated.tree, gtdb_<release>_bac120.decorated.tree-table, gtdb_<release>_bac120.decorated.tree-summary, gtdb_<release>_bac120.decorated.tree-taxonomy
• results of phylorank outliers in the directory phylorank_outliers
• ARB database and filter file, e.g.: gtdb_r202_bac120_arb_metadata.txt, gtdb_r202_bac120_arb_filter.ift

Analogous data needs to be put together for Archaea. Inferring the archaea tree typically takes much longer since we use iqtree instead of fasttree. Trees with alternative marker sets (e.g., rp2) also need to be put together, but these are far less critical than the bac120 and ar53 trees since these are the primary trees used for curation.

After manual curation, GTDB species names must be finalized and validated by running all the commands beginning with a pmc in the order specified in the CLI help menu for the GTDB Species Cluster Toolkit. Validation results should be performed in the directory /srv/projects/gtdb/<gtdb_release>/<domain>/post_curation/<date> as follows:

1. gtdb_species_clusters pmc_manual_species gtdb_inputs.toml Bacteria gtdb_r<release>_bac120-scaled-<date>.tree 1_pmc_manual_species
   • no errors or warnings are expected
2. gtdb_species_clusters pmc_replace_generic ./1_pmc_manual_species/manual_species_names.tsv gtdb_r<release>_bac120_<date>.tsv 2_pmc_replace_generic
   • no errors or warnings are expected
3. gtdb_species_clusters pmc_species_names gtdb_inputs.toml Bacteria gtdb_r<release>_bac120-scaled-<date>.tree ./1_pmc_manual_species/manual_species_names.tsv ./2_pmc_replace_generic/taxonomy_updated_sp.tsv pmc_custom_species.tsv 3_pmc_species_names
   • species names that needs to be manually adjusted should be specified in pmc_custom_species.tsv
     • in general, it is a good idea to cross reference the pmc_custom_species.tsv file from the previous release when resolving issues and to see if there are any manual changes made in the previous release that need to be retained
   • going through warnings in ALLCAPS has generally revealed issues:
     • INCONGRUENT_TYPE_STRAIN: often identifies incorrectly modified specific names due to typos or use of inaccurate spellings at NCBI, or due to a species transfer between genera; these should be checked to ensure the species name in final_taxonomy.tsv is as desired and any names requiring a change added to pmc_custom_species.tsv
     • INCONGRUENT_TYPE_SPECIES_GENERIC: these are generally acceptable and are legitimate transfer of type strains between genera
     • ERRONEOUS_SUFFIX_TYPE_STRAIN: these may represent cases where the specific name of a type strain has erroneously been given an alphabetic suffix
   • any reported errors should be manually resolved by adding desired species name to pmc_custom_species.tsv
     • The ERROR: Species name <name> assigned to at least 2 GTDB representatives need to be resolved which should be done by consulting the pmc_custom_species.tsv file for the previous release and prior classifications for the problematic genomes. Generally, such cases require identifing a suitable alphabetic prefix for one of the genomes and adding this name to the pmc_custom_species.tsv file.
   • I also manually inspect all Finalized GTDB species name warnings as this generally results in identifying a few cases of genus transfers where the specific name should be given a suffix.
   • specific_epithet_map.new_cases.tsv should be sent to curation team (Masha) and results added to the specific_epithet_transfer_map ledger
   • the Species classification ledger warnings should also be brought to the attention of the curation team (Masha) to see if the species classification ledger for these cases is still valid
   • this step must be repeated with the updated specific_epithet_transfer_map ledger and after resolving of any issues identified in steps 4, 5, or 6
   • the taxonomy is finalized once this step and steps 4, 5, or 6 pass without any reported errors or all such errors are considered acceptable
4. gtdb_species_clusters pmc_check_type_species gtdb_inputs.toml ./3_pmc_species_names/final_taxonomy.tsv ./1_pmc_manual_species/manual_species_names.tsv pmc_custom_species.tsv 4_pmc_check_type_species
   • amiguous_sp_priority.tsv should be empty or any reported cases manually verified
   • type_species_incongruencies.tsv should be empty or any reported cases manually verified
   • TBD: the species_classification.<date>.tsv ledger should be considered to remove known cases of incongruence between GTDB and NCBI
5. gtdb_species_clusters pmc_check_type_strains gtdb_inputs.toml ./3_pmc_species_names/final_taxonomy.tsv ./1_pmc_manual_species/manual_species_names.tsv pmc_custom_species.tsv 5_pmc_check_type_strains
   • send type_species_incongruencies.tsv from previous step and type_strains_incongruencies.tsv to curation team (Masha) for manual inspection as ideally these files should be empty
   • TBD: the species_classification.<date>.tsv ledger should be considered to remove known cases of incongruence between GTDB and NCBI
6. Taxonomy should be verified by running:
   • gtdb_validation_tk check_file ./3_pmc_species_names/final_taxonomy.tsv --include_species
   • The archaeal and bacterial taxonomy files need to be merged and validated to ensure there are no duplicate names:
     • gtdb_validation_tk check_file ./3_pmc_species_names/combined_final_taxonomy.tsv --include_species
   • gtdb_validation_tk check_generic ./3_pmc_species_names/final_taxonomy.tsv gtdb_r<version>_metadata.updated_reps.tsv /srv/db/gtdb/metadata/release<version>/representatives/ledgers/species_classification.tsv NONE ./3_pmc_species_names/check_generic
     • unsupported_generic_names.tsv indicates GTDB genera where it is unclear where the genus name originates from; ideally we would be able to account for all GTDB genera; however, perhaps there is too much historical issues (e.g. names changing at NCBI; changing GTDB rules) for this check to be useful
   • gtdb_validation_tk check_specific ./3_pmc_species_names/final_taxonomy.tsv gtdb_r<version>_metadata.updated_reps.tsv /srv/db/gtdb/metadata/release<version>/representatives/ledgers/species_classification.tsv /srv/db/gtdb/metadata/release<version>/representatives/sp_cluster_update/13_u_synonyms/synonyms.tsv ./3_pmc_species_names/check_specific
     • Ideally, invalid_specific_names.tsv should be empty or only have a few exceptions due to retired genomes at NCBI, but this is still a work in progress
   • gtdb_species_clusters pmc_validate gtdb_inputs.toml Archaea ./1_pmc_manual_species/manual_species_names.tsv ./3_pmc_species_names/final_taxonomy.tsv gtdb_r<version>_ar53_<date>-scaled.tree pmc_custom_species.tsv ground_truth_test_cases.tsv 6_pmc_validate
     • this runs a lot of different validation tests; ideally, all tests should past, but in practice there are challenging cases that are exceptions so it is more a matter of manually inspecting cases that fail to establish if they are acceptable and improving the code to better handle identified exceptions for the next release
     • modified_specific_names.tsv should be consulted and ALPHANUMERIC -> ALPHANUMERIC, ALPHANUMERIC -> SUFFIXED_LATIN, SUFFIXED_LATIN -> ALPHANUMERIC transitions added to the pmc_custom_species.tsv file as a placeholder name should not replace a placeholder. The SUFFIXED_LATIN -> SUFFIXED_LATIN are harder to verify as a species may be split up or a genome legitimately reassigned to a different species. Steps 3 to 6 should then be repeated to take these new species names into account.
7. The final taxonomy and curation tree for GTDB representatives is ./3_pmc_species_names/final_taxonomy.tsv and ./3_pmc_species_names/curation_tree.final_species.tree. The final GTDB synonyms file is ./3_pmc_species_names/gtdb_synonyms_final.tsv. These files should be renamed with an appropriate prefix, sent to the curation team, and put in the directory final_data. These files are uses as the bases for creating the final data files for the GTDB website (Pierre: where have we documented the next steps?). Files should be renamed as follows:
   • ./3_pmc_species_names/final_taxonomy.tsv to ./final_data/gtdb_r_ar53_curation_taxonomy.tsv
   • ./3_pmc_species_names/curation_tree.final_species.tree to ./final_data/gtdb_r_ar53_curation_scaled.tree
   • ./3_pmc_species_names/gtdb_synonyms_final.tsv to ./final_data/gtdb_r220_ar53_synonyms.tsv
   • ./archaea/pre_curation/ar53//filtered/final_mask.txt ./final_data/ar53_msa_mask_r.txt
   • ./archaea/pre_curation/ar53/20251030/filtered/filtered_msa.faa ./final_data/ar53_msa_reps_r.faa
8. We also need an unscaled tree with the finalized taxonomy which can be obtained by stripping and decorating a tree:
   • gtdb_validation_tk add_dummy NONE --taxonomy_file gtdb_r220_ar53_curation_taxonomy.tsv
   • genometreetk strip ../gtdb_r220_ar53_curation_unscaled_20240326.tree gtdb_r220_ar53_curation_unscaled.stripped.tree
   • phylorank decorate gtdb_r220_ar53_curation_unscaled.stripped.tree gtdb_r220_ar53_curation_taxonomy.dummy_nodes.tsv gtdb_r220_ar53_curation_unscaled.tree --skip_rd_refine
   • Unfortunately, ARB has a tendency to modify branch lengths by a factor of 100 if trees aren't imported and exported correctly. The mean root to leaf branch lengths should be unchanged from the pre-curation, rooted tree. A quick sanity check is to ensure the mean root to leaf branch lengths are in the [~0.5, ~2.0] range using:
   • gtdb_validation_tk branch_len gtdb_r220_ar53_curation_unscaled.tree
9. pmc_cleanup: removes temporary files and compress large files in order to reduce required disk space. This should only be run after GTDB has been released as this will make it difficult to re-run any steps.
   • gtdb_species_cluster pmc_cleanup gtdb_directories.toml
   • The gtdb_directories.toml is a TOML file indicating the output directory of each step of the GTDB species cluster pipeline. See a previous release for the format of this file.

• Send final taxonomy and RED median files to Ben Woodcroft so he can finalize updating the SingleM results for Sandpiper
• Send final taxonomy files to Daan Speth so he can finalize GlobDB

Any manually modified species names which are reported by the pmc_manual_species likely need to be added to an official ledger so these changes can be retained moving forward. The ledger are currently our only mechanism for ensuring a name persists between releases. Otherwise, names are always determined de novo based on NCBI assignments as, in general, we want the GTDB to reflect changing taxonomic opinion as reflected by changes at NCBI.

Final ledgers should be copied to the GTDB OneDrive under GTDB/ledgers/.

• ideally all ledgers would be tables in the GTDB database and we would provide curators with a web interface to modify these ledgers