Part of FreeREG-Core-Records.

Models: app/models/freereg1_csv_file.rb, freereg1_csv_entry.rb, batch_error.rb, multiple_witness.rb, physical_file.rb, attic_file.rb · Controllers: freereg1_csv_files_controller.rb, freereg1_csv_entries_controller.rb, physical_files_controller.rb

These are the persistent records of a transcription. The upload-time Csvfile object is discarded at the end of the request (Csvfile-System). After that, the data lives in these documents plus the file on disk.

Record	Granularity	What it's for
PhysicalFile	one per userid + file_name	Tracks the file on disk (Rails.application.config.datafiles/<userid>/<file_name>): uploaded / waiting to be processed / processed. It's also the processor's queue (PhysicalFile.waiting)
Freereg1CsvFile	one per userid + file_name + location + record type	A batch: the records from one file for one county/place/church/register type. Holds header metadata, counters, lock flags, error count
Freereg1CsvEntry	one per data line	A single baptism, marriage or burial as transcribed

One uploaded file normally produces one batch. A file whose lines span several churches or register types produces several, and the processor warns the transcriber when that happens (CSV-Processing-Pipeline).

Location fields: county (actually a Chapman code), chapman_code, place, place_name, church_name, register_type, record_type. The comment in the model says the location fields are "NOT USED" in favour of the Register → Church → Place chain. In practice both are used: the processor looks batches up by these strings (setup_batch_for_processing), and the display uses the chain. Relocation code keeps both in sync.

Header metadata: transcriber_name/_email/_syndicate, credit_name/_email, first_comment/second_comment, transcription_date, modification_date (both strings like "12 Mar 2019", defaulting to '01 Jan 1998'), uploaded_date, lds ('yes'/'no': file includes film/film_number columns), def (Boolean: flexible column order) and order (the column → index map used to write the file back out), characterset, digest (MD5 of the file on disk), software_version, search_record_version.

Counters: records (String), datemin, datemax, daterange (Array of per-decade counts), zero_entries (Boolean: some entries have no year), error (Integer: count of BatchErrors).

State flags:

• processed / processed_date: true once search records have been created. A trainee's "check only" run leaves it false.
• locked_by_transcriber / locked_by_coordinator: see Locking below.

Callbacks:

• before_save: add_lower_case_userid_to_file, add_country_to_file, check_register_type (adds an error if register_type isn't in RegisterType::OPTIONS, but because it runs in before_save rather than as a validation, Mongoid still saves the document; the error only shows up if the caller inspects errors afterwards).
• after_save: recalculate_last_amended (walks the whole place's churches, registers and files) and update_number_of_files (recounts all of the user's files and records). Every save of a batch triggers both. That adds up in loops.
• before_destroy: moves the disk file to the attic and destroy_alls the entries, which in turn destroy their search records. remove_batch deliberately calls delete, not destroy, to skip this (see Removal below).

Colour coding (define_colour, used in batch listings): red = errors & unlocked, orange = not processed, green = clean & unlocked, blue = clean & locked, maroon = errors & locked.

include Mongoid::Attributes::Dynamic, so an entry can carry any field. The declared fields cover the standard and extended layouts for all three record types (see the field lists in FreeregOptionsConstants: ENTRY_ORDER_DEFINITION for the classic layout, ADDITIONAL_*_FIELDS / EXTENDED_*_LAYOUT for the flexible one).

Common fields: county, place, church_name, register_type, record_type, register_entry_number, notes, notes_from_transcriber, image_file_name, file_line_number, line_id ("<userid>.<FILE_NAME>.<line>"), year, film, film_number, record_digest.

Relationships:

• belongs_to :freereg1_csv_file
• embeds_many :multiple_witnesses (up to MAXIMUM_WINESSES = 8). The processor turns witnessN_forename/_surname columns into these for marriages and baptisms.
• embeds_many :embargo_records: history of embargo decisions for this entry; the most recently updated one is the current state. See Embargoes-and-Gaps.
• has_one :search_record, dependent: :restrict_with_error, but also before_destroy { SearchRecord.destroy_all(freereg1_csv_entry_id: id) }, which removes the search record first so the restriction never fires.

Callbacks:

• before_save: sanitize_fields, add_digest (MD5 of a type-specific concatenation of fields → record_digest), captitalize_surnames (every *_surname is upcased on save), check_register_type.
• after_update :update_seach_record_location → SearchRecord#update_location: rewrites location_names/place_id from the batch's current register chain.

Validation: errors_in_fields runs the FreeregValidations checks (clean text, valid dates, valid ages, sex, etc.) per record type. On a CSV line, a failure becomes a BatchError instead of an entry. In the online editor it's a normal form error.

record_digest drives re-upload reconciliation. The processor matches incoming lines to existing entries by digest, not by line number. Any change to cal_digest/create_*_string makes every existing entry look new on its next re-upload, which leads to a full delete-and-recreate of its search records.

enough_name_fields? decides whether an entry gets a search record at all. For example, a baptism needs at least one of person forename, father forename/surname, mother forename/surname. Entries that fail this exist but are unsearchable.

A line that failed validation during processing: error_type, record_number, error_message, record_type, and data_line (the parsed hash, so the fix-it form can be pre-filled). Transcribers correct these online through Freereg1CsvEntriesController#create with session[:error_id] set. That path reuses the error's line number and line id instead of appending a new line.

Two independent Booleans, set on every batch sharing the file's userid + file_name:

• locked_by_transcriber is set when the owner edits online (session[:my_own]) or toggles the lock.
• locked_by_coordinator is set when a coordinator edits, renames or relocates the place above it (Place#propogate_batch_lock), merges batches, or toggles the lock.

A locked batch can't be replaced by re-upload (Csvfile#setup_batch_on_replace) or removed. The reason: the database now differs from the file on disk, and a re-upload would silently throw away the online edits. Downloading clears both locks. Freereg1CsvFilesController#download first regenerates the CSV from the database (backup_file → write_csv_file, previous version moved to the attic), updates digest, then calls force_unlock.

Since Sept 2026 (01594b4cf), an embargo-only edit of an entry doesn't back up or lock the file, because it doesn't change the CSV.

update, create and destroy all follow the same shape:

1. Validate the entry → file link (valid_freereg1_csv_entry? / valid_freereg1_csv_file?).
2. Refuse if the file is waiting in the processor queue (can_we_edit? → no PhysicalFile.waiting for it).
3. Save the entry. For DEF files, check_and_augment_def adds any newly used column to the file's order so a download includes it.
4. update_file_statistics: SearchRecord.update_create_search_record (re-applies embargo rules, re-translates, re-indexes), then calculate_distribution, backup_file (rewrites the CSV on disk), lock_all, set modification_date.
5. Update the place's UCF list for this record (update_place_ucf_list).

create also calls update_other_statistics and destroy calls update_statistics_and_access; both recalculate the Register/Church/Place counters. update doesn't (the update_statistics_and_access call is commented out), so after an edit that changes a date, the contents pages lag until the next freereg:calculate_freereg_content or re-upload.

• lock: toggles the transcriber lock (owner) or the coordinator lock (coordinator; clearing the coordinator lock also clears the transcriber lock).

• download: see Locking. Refused by check_file if any batch of the file has no register/church/place, no entries, or any error > 0. Error lines exist only as BatchErrors, so they'd be lost from the regenerated file.

• remove: remove_batch(max_records). Refused if the batch is locked, or if it has more records than the user's role allows (MAX_RECORDS_COORDINATOR = 5000, MAX_RECORDS_DATA_MANAGER = 15000, MAX_RECORDS_SYSTEM_ADMINISTRATOR = 100000); over the limit, the data manager gets an email instead. Otherwise it:

  1. appends id,userid,file_name to the delete list (config.delete_list),
  2. moves the disk file to the attic,
  3. removes the batch from the place's UCF list,
  4. deletes (not destroys) the batch document, and
  5. deletes the PhysicalFile.

  The entries and search records stay in the database until the delete_file_no_sleep rake task works through the delete list. Until then they're orphans. They don't show in results, because SearchQuery#persist_results walks each hit's entry → batch → register → church → place chain (valid_freereg_entry_ids_for_result_hashes), deletes any search record whose chain is broken, and drops it from the results. They do still count against the 500-result cap, though, and a search can return fewer rows than it matched. See FreeREG-Operations-and-Rake-Tasks.

• merge: merge_batches moves the entries of other batches with the same userid + file_name on the same register into this one and deletes them. It adds to records; the batch's "refresh file information" action (refresh_file_information) recalculates the date range afterwards.

• relocate: moves a batch to a different place/church/register (file_update_location), then rewrites every entry's search record location (propogate_file_location_change).

• change_userid: change_owner_of_file. Moves the disk file into the new user's folder, updates the PhysicalFile, and rewrites userid on every batch and entry of the file.

• zero_year / show_zero_startyear_entries: list entries with no extractable year. These have no search_date, so they only turn up in searches without a date range.

• unique_names: distinct surnames/forenames in the batch.

• Listing variants (my_own, display_my_error_files, display_my_own_files_waiting_to_be_processed, sorted by upload date…) are the transcriber's "My batches" views. Coordinators get the equivalent per county via ManageCountiesController (Coordinator-Admin-Tools) and per syndicate via ManageSyndicatesController (Syndicate-System).

Before a disk file is overwritten or removed, save_to_attic renames it to <file_name>.<unix_time>, moves it to <datafiles>/<userid>/.attic/, and creates an AtticFile record. Nothing prunes the attic automatically.