Importing Chants from a CSV - DDMAL/CantusDB GitHub Wiki

How to bulk-create chant records from a spreadsheet using the import_chants_from_csv management command.

Each row names its own source, so one file can cover several sources at once. The format below was agreed with the DACT project team in May 2026 and is implemented in main_app/management/commands/import_chants_from_csv.py.

Workflow

Two people are normally involved: whoever prepares the spreadsheet, and a system administrator with access to the server, who runs the command and reports back.

Preparing the file

  1. Create the source record(s) in the admin interface, one for every source the file covers. The command only creates chants; it never creates a source.
  2. Copy each source's ID from its URL: https://cantusdatabase.org/source/1000021.
  3. Add a source_id column and put the right ID on every row. Rows belonging to different sources can sit in the same file.
  4. Save as CSV (UTF-8). See Spreadsheet pitfalls; leading zeroes are easy to lose here.
  5. Send the file to a system administrator.

Running the import (system administrator)

  1. Dry run first. This validates the file and reports duplicates without writing anything: 
    python manage.py import_chants_from_csv path/to/file.csv --dry-run
  2. Send the output back to whoever prepared the file, including both errors and warnings quoted with their row numbers. They correct the spreadsheet and resend. Repeat until the dry run is clean.
  3. Import for real: 
    python manage.py import_chants_from_csv path/to/file.csv

The real import runs inside a single transaction: if any row fails, nothing is written.

The template

Download the blank template (right-click, Save Link As). It is a single header row holding every recognised column, ready to fill in.

Column order does not matter, and column names are matched case-insensitively. Every column except source_id is optional: a column you leave out is not set at all, and a cell you leave blank is stored as blank. Delete any columns you have no data for.

The same header row, if you would rather copy and paste it:

source_id,marginalia,folio,content structure,sequence,incipit,feast,office,genre,position,cantus_id,mode,finalis,differentia,fulltext_standardized,fulltext_ms,fulltext_standardized_proofread,fulltext_ms_proofread,volpiano,volpiano_proofread,image_link,melody_id,cao_concordances,addendum,extra,indexing_notes,node_id

A filled-in row looks like this (abridged):

source_id,folio,content structure,sequence,incipit,feast,office,genre,position,cantus_id,mode,differentia,fulltext_standardized,fulltext_ms,image_link,cao_concordances,addendum,node_id
1000533,001r,011r,1,Canite tuba in Sion vocate,Dom. 4 Adventus,M,R,1.1,006265,1,a,Canite tuba in Sion vocate gentes annuntiate populis et dicite ecce deus salvator noster adveniet,CANITE TUBA IN SYON VOCATE gentes annuntiate populis et dicite ecce deus salvator noster adveniet,https://fragmentarium.ms/view/page/F-hbco/845/12671,CGBEMVHRDFSL,B1r,241800

Column reference

Column Goes to Notes
source_id Source (foreign key) Required on every row. An integer matching an existing source. May differ from row to row.
marginalia Marginalia Max 63 characters.
folio Folio Max 255. Use the binding order (001r, 001v).
content structure Content structure Max 64. Additional folio number, used when folio numbers appear on the leaves but differ from the binding order. Note the space in the column name.
sequence Sequence Must be a whole number. Each folio restarts at 1.
incipit Incipit Max 255.
feast Feast (foreign key) Looked up by exact feast name, ignoring case (e.g. Dom. 4 Adventus). Unrecognised names are an error.
office Service (foreign key) Looked up by the service abbreviation: V, C, M, L, P, T, S, N, V2, E, X, D, H, R, MI, CA, ?.
genre Genre (foreign key) Looked up by the genre abbreviation: A, R, V, I, Ag, Al, AlV, AV, BD, Ca, Cap, …
position Position Max 63. Free text (1.1, 01, 2.).
cantus_id Cantus ID Max 255, stored as text. Keep leading zeroes.
mode Mode Max 63. Free text: 1–8, r, etc.
finalis Finalis Max 63.
differentia Differentia Max 63.
fulltext_standardized Full text, standardized spelling Unlimited length.
fulltext_ms Full text as in manuscript Unlimited length.
fulltext_standardized_proofread Standardized-spelling proofread flag 1 = proofread, 0 = not.
fulltext_ms_proofread Manuscript-spelling proofread flag 1 / 0.
volpiano Volpiano Unlimited length.
volpiano_proofread Volpiano proofread flag 1 / 0.
image_link Image link Must be a valid URL, max 200 characters. Whitespace is trimmed and spaces percent-encoded automatically.
melody_id Melody ID Max 63.
cao_concordances CAO concordances Max 63.
addendum Addendum Max 255.
extra Extra Max 63.
indexing_notes Indexing notes Unlimited length.
node_id json_info → {"nid": …} The chant's record ID on the old Cantus site. Only relevant for data exported from old Cantus.
frag_id (ignored) Fragmentarium ID belongs to the source, not the chant. Recognised and skipped, so it can be left in the spreadsheet harmlessly.
siglum Legacy chant siglum Not needed, since the source is identified by source_id. If present it is stored on the chant's legacy siglum field.

Any other column is reported as Unknown and skipped. That is a safety net, not a failure, but read the list, because a typo in a column name silently drops the whole column.

Proofread flags accept only 1 and 0; anything else warns and stores nothing. Leaving the columns out entirely leaves the flags unset for that import.

Spreadsheet pitfalls

  • Leading zeroes. Every field is text, and Excel will offer to "optimize" 006265 down to 6265; decline. Cantus IDs are the usual casualty. As a backstop, the command warns on any Cantus ID shorter than six characters.
  • Trailing blank rows. A visually empty row still has no source_id and fails validation. Delete them before saving.
  • Unnamed trailing columns. Empty columns to the right of the data show up as Unknown. Harmless, but tidier to remove.
  • Encoding. Save as UTF-8; a byte-order mark is handled automatically.
  • Placeholder Cantus IDs. 909999 and 900000 are deliberate placeholders for ambiguous or unknown chants. They are not Cantus Index identifiers and are not rejected, so prefer them to leaving the field blank.

What the dry run checks

Both modes verify that the file is readable, that a source_id column exists, that source_id and sequence are integers, that every source_id matches a real source, and that every feast, office and genre resolves. Any failure aborts with a numbered list of errors and writes nothing.

The dry run adds two checks the real import skips:

  • Field-level validation on each unsaved chant: max lengths, URL format, choice constraints.
  • Duplicate detection, within the file and against the database, keyed on source + folio + sequence: 
    Row 9: already exists in database (source 1000533, folio 001r, seq 4) — chant ID(s): 241803
    These are warnings, not errors, since a repeated slot is sometimes intentional.

Limitation: duplicate detection only compares source + folio + sequence. It cannot tell you that a chant's content already exists under a different source or a different foliation, which is exactly the case when a reconstructed source is being split into individual leaves and re-foliated, as with the Gottschalk Antiphoner. Compare node_id values against existing records if you need that.

Sample output

(Dry-run mode: no database changes will be made.)
Reading /tmp/example.csv...
Found 42 data row(s) and 25 column(s).
  Mapped  : source_id, marginalia, folio, content structure, sequence, incipit, feast,
            office, genre, position, cantus_id, mode, finalis, differentia,
            fulltext_standardized, fulltext_ms, volpiano, image_link, melody_id,
            cao_concordances, addendum, extra, node_id, indexing_notes
  Skipped : frag_id
Validating data...
All 42 row(s) validated successfully.
Checking for duplicates...
No duplicates found.
Dry run complete — 42 row(s) validated, ready to import.

The real run prints the same header and validation lines, then Successfully imported 42 chant(s).

Imported chants are attributed to the admin user. Management commands run outside the request cycle, so the automatic revision tracking that covers edits made through the site does not apply, so imported chants have no initial revision history.

Error reference

Message Cause
CSV file not found: … Wrong path.
CSV is empty (no header) / (no rows) No header row, or header only.
CSV is missing source_id No source_id column. Add one, filled from each source's URL.
Row N: source_id is required but missing or empty Usually a blank trailing row.
Row N: source_id 'x' is not a valid integer Non-numeric value, often a stray siglum or merged cell.
Row N: sequence 'x' is not a valid integer Sequence must be a whole number.
Row N: Source with ID x not found Source not created yet, or wrong ID.
Row N: Feast / Service / Genre 'x' not found Value does not match an existing record. Check the abbreviation lists above.

Warnings that do not stop the import: short Cantus ID, unparseable proofread flag, duplicate folio/sequence slot.