We aim to update GTDB annually (September: NCBI data download; April: GTDB release). Genome assemblies are mirrored from NCBI's FTP site.

Note: download NCBI, LPSN, and SeqCode metadata files on the same day you sync with NCBI genomes

• <release>: release of GTDB being prepared (e.g. release237 or just 237 depending on context)
• <prev_release>: prior release of GTDB that is being updated (e.g. release 232 or just 232 depending on context)
• <date>: date in YYYYMMDD format

Download the NCBI taxonomy and assembly summary files, and generate standardised 7-rank NCBI taxonomy files for each genome:

gtdb_migration_tk ncbi_metadata_sync -o /srv/db/gtdb/metadata/<release>/ncbi -r 237 --group PROK

gtdb_migration_tk ncbi_metadata_sync -o /srv/db/gtdb_fungi/metadata/<release>/ncbi -r 237 --group FUNGI

1. Determine NCBI genomes that will comprise the next GTDB release:

gtdb_migration_tk select_genomes -n assembly_summary_archaea_genbank.txt.gz assembly_summary_archaea_refseq.txt.gz assembly_summary_bacteria_genbank.txt.gz assembly_summary_bacteria_refseq.txt.gz -o /srv/db/gtdb/metadata/<release>/ncbi/select_genomes

gtdb_migration_tk select_genomes -n assembly_summary_fungi_genbank.txt.gz assembly_summary_fungi_refseq.txt.gz -o /srv/db/gtdb_fungi/metadata/<release>/ncbi/select_genomes

Note: extremely small (--min_genome_size, default = 10 kb) and large (--max_genome_size, default = 100 Mb) are excluded as these can cause problems in downstream processing. The small assemblies are often amplicons erroneously submitted as a genome, while large assemblies are often full metagenomes.

2. Sync NCBI prokaryotic (/srv/db/ncbi_ftp_sync/prok) and fungal (/srv/db/ncbi_ftp_sync/fungi) genomes that have been selected for inclusion in next GTDB release (from the directory):

gtdb_migration_tk ncbi_genome_sync -g /srv/db/gtdb/metadata/<release>/ncbi/select_genomes/gtdb_selected_genomes.tsv.gz --root genomes --log ncbi_genome_sync-r<release>.log

gtdb_migration_tk ncbi_genome_sync -g /srv/db/gtdb_fungi/metadata/<release>/ncbi/select_genomes/gtdb_selected_genomes.tsv.gz --root genomes --log ncbi_genome_sync-r<release>.log

The sync includes removing any genomes that are no longer in GTDB, downloading new genomes, and updating files that have changed for existing genomes.

Note: NCBI retired rsync on 1 June 2026

3. Retry genomes that failed to sync

Genome assemblies that failed to sync are given in the *.fail file. These should be manually inspected. Often these will be genomes that have been suppressed at NCBI and no longer have a complete set of data files on the NCBI FTP site. This will result in a HTTP 404 failure and such failures are acceptable. However, it is possible to have genomes that have failed unexpectedly (e.g. due to a temporary network error or exceeding NCBI's request rate limit). Syncing of such genomes should be retried using the --retry argument, e.g.:

gtdb_migration_tk ncbi_genome_sync --retry ncbi_genome_sync-r<release>.fail --root genomes --log ncbi_genome_sync-r<release>-retry.log

4. Verify integrity and re-fetch any corrupt genomes:

All genomes should sync with no failures reported (see *.fail file). Proper syncing should then be verified using:

gtdb_migration_tk ncbi_genome_sync -g /srv/db/gtdb/metadata/<release>/ncbi/select_genomes/gtdb_selected_genomes.tsv.gz --root genomes --verify-only -l gtdb_selected_genomes-r<release>-verify.log

Any genome with a file that does not match it's checksum will be reported in the *.bad file. This can happen due to bit rot or because a file has been accidently changed. For such genomes, the full genome directory can be deleted and then resynced, e.g.:

gtdb_migration_tk ncbi_genome_sync --retry gtdb_selected_genomes-r<release>-verify.bad --root genomes --verify --delete -l gtdb_selected_genomes-r<release>-fix.log

Running --verify-only will also report any additional files found for a genome assembly in the *.extra file. These are files not required by GTDB and in general should no be present here as the NCBI sync is meant to be a mirror of files required by GTDB. Such files should be manually removed.

1. List all genome assemblies in the GenBank and RefSeq NCBI FTP directory:

gtdb_migration_tk list_genomes -g /srv/db/gtdb/metadata/<release>/ncbi/select_genomes/gtdb_selected_genomes.tsv.gz -d /srv/db/ncbi_ftp_sync/prok/genomes/all -o /srv/db/gtdb/metadata/<release>/ncbi/genome_dirs_ftp.tsv

gtdb_migration_tk list_genomes -g /srv/db/gtdb_fungi/metadata/<release>/ncbi/select_genomes/gtdb_selected_genomes.tsv.gz -d /srv/db/ncbi_ftp_sync/fungi/genomes/all -o /srv/db/gtdb_fungi/metadata/<release>/ncbi/genome_dirs_ftp.tsv

2. Copy genomes comprising the GTDB release:

gtdb_migration_tk update_genomes --ftp_directory /srv/db/ncbi_ftp_sync/prok/genomes --new_directory /srv/db/gtdb/genomes/ncbi/<release> --ftp_genome_dirs_file /srv/db/gtdb/metadata/<release>/ncbi/genome_dirs_ftp.tsv --old_genome_dirs_file /srv/db/gtdb_fungi/genomes/ncbi/<prev_release>/genome_dirs.tsv

gtdb_migration_tk update_genomes --ftp_directory /srv/db/ncbi_ftp_sync/fungi/genomes --new_directory /srv/db/gtdb_fungi/genomes/ncbi/<release> --ftp_genome_dirs_file /srv/db/gtdb_fungi/metadata/<release>/ncbi/genome_dirs_ftp.tsv --old_genome_dirs_file /srv/db/gtdb/genomes/ncbi/<prev_release>/genome_dirs.tsv

3. Resolved genomes requiring manual curation:

In general, this step should not be required. However, if the expected set of files for a genome have become compromised it is possible that the update_genomes step will encounter a situation it can't resolve. Such cases are written to the report.log file (grep to_curate) and must be manually inspected and resolved.

1. Pull LPSN data from website:

gtdb_migration_tk lpsn pull_html -o /srv/db/gtdb/metadata/<release>/lpsn/lpsn_<date>

It may be necessary to run lpsn pull_html multiple times since downloading of LPSN pages can fail. Failed downloads are indicated in the *_failed.lst files. The --skip_taxa_per_letter_dl can be used to speed up additional runs of pull_html. There should be no failed downloads before proceeding with lpsn parse_html.

2. Download the lpsn_gss_file file from the Download section of the LPSN website.

3. Generate metadata files from the LPSN website and GSS file (from the lpsn/lpsn_ directory):

gtdb_migration_tk lpsn parse_html --in_dir . -o parse_html --lpsn_gss_file lpsn_gss_2022-09-19.csv

4. Create date table (from the lpsn/lpsn_ directory):

gtdb_migration_tk strains date_table --lpsn_scraped_species_info ./parse_html/all_ranks/lpsn_species.tsv --lpsn_gss_file lpsn_gss_<date>.csv --output_file ../../strain_table/lpsn_year_table.tsv

Previous parsing disregard strain shorter than 2 characters and strains without any digit. Starting with R237, we include such strain IDs as there are some genomes where these provide a legitimate match. To compare results with these two parsing strategies, there is a boolean in the check_format_strain function in taxon_utils.py.

old_format = False

then

mkdir old_format_with_digits
cd old_format_with_digits
gtdb_migration_tk strains type_table --lpsn_dir ../../lpsn/lpsn_20240917/parse_html/all_ranks/ --year_table ../year_table.tsv --metadata /srv/db/gtdb/metadata/release226/strain_table/metadata_file.tsv --ncbi_names /srv/db/gtdb/metadata/release226/ncbi/taxonomy/taxdump_20240914/names.dmp --ncbi_nodes /srv/db/gtdb/metadata/release226/ncbi/taxonomy/taxdump_20240914/nodes.dmp --cpus 2 --output_dir . --lpsn_gss_file ../../lpsn/lpsn_20240917/lpsn_gss_2024-10-04.csv

Then compare both files with the compare_strain_files.py (local sandbox, its a bit of a hack for now)

gtdb_migration_tk seqcode download -g ../../genomes/ncbi/<release>/genome_dirs.tsv -n ./ncbi/assembly_summary_*.txt.gz -o seqcode -l ./logs/seqcode_download.log

NCBI, LPSN, and SeqCode data are combined to determine type material:

gtdb_migration_tk strains type_table -g ../../genomes/ncbi/<release>/genome_dirs.tsv -n ./ncbi/assembly_summary_*.txt.gz --ncbi_names ./ncbi/taxonomy/taxdump_<ncbi_date>/names.dmp --ncbi_nodes ./ncbi/taxonomy/taxdump_<ncbi_date>/nodes.dmp --lpsn_gss_file ./lpsn/lpsn_<date>/lpsn_gss_<date>.csv --lpsn_dir ./lpsn/lpsn_<date>/parse_html/all_ranks/ --year_table ./strain_table/lpsn_year_table.tsv --seqcode_table ./seqcode/seqcode_table.tsv -o ./strain_table

Next steps of the update are under Generate GTDB metadata.