Csvfile System (Batch Upload)

Part of Transcriber & Coordinator Workflow.

Model: app/models/csvfile.rb · Controller: app/controllers/csvfiles_controller.rb

This is how a transcriber gets a completed CSV batch into the system through the website.

Csvfile is not the permanent record

Csvfile (CarrierWave::Uploader::Base + Mongoid::Document, mount_uploader :csvfile, CsvfileUploader) is a short-lived staging object, not the record of a transcription batch. A request builds one, uses it to validate and physically process the uploaded file, and then deletes it (csvfile.delete) at the end of the same request — win or lose. The actual persistent records it creates/updates are:

• PhysicalFile — "does a file with this name exist for this user" tracking: base/base_uploaded_date (initial upload), change/change_uploaded_date (a replace), file_processed/file_processed_date, waiting_to_be_processed/waiting_date.
• Freereg1CsvFile — the actual per-batch transcription record: county/place/church metadata, records (stringified count), error (Integer error count), processed/processed_date, locked_by_transcriber/locked_by_coordinator.

So when you're chasing a bug about "my upload status", you're looking for state on PhysicalFile/Freereg1CsvFile, not on Csvfile — by the time you could query it, it's already gone.

Upload vs Replace

One endpoint (CsvfilesController#create); which path runs is chosen by a field in the payload:

case params[:csvfile][:action]
when 'Replace' then @csvfile.setup_batch_on_replace(@csvfile.file_name)
when 'Upload'  then @csvfile.setup_batch_on_upload
end

setup_batch_on_upload unconditionally rejects a file if one with that name already exists for the user:

def setup_batch_on_upload
  if PhysicalFile.userid(userid).file_name(file_name).first.present?
    return [false, 'You cannot upload a file with the same name as one you have already uploaded. You must use the REPLACE option.']
  end
  [true, '']
end

setup_batch_on_replace guards

Before allowing a replace, Csvfile#setup_batch_on_replace checks, in order: the new file's name matches the original being replaced; the user actually owns an existing physical file with that name (coordinators replacing someone else's file need to do so under that user's userid); the file isn't transcriber_locked (online edits pending download) or coordinator_locked. Only then does it update/create the PhysicalFile record.

Processing after a successful upload/replace

Csvfile#process_the_batch(user):

1. check_for_existing_file_and_save — moves any existing physical file on disk into a per-user .attic/ directory (with an AtticFile audit record) before the new one lands.
2. Rejects if the file is too small (< 100 bytes) or a batch with the same name is already waiting_to_be_processed.
3. For FreeREG (MyopicVicar::Application.config.template_set == 'freereg'): estimates processing time (size * 0.001); trainees always get a synchronous-feeling rake build:freereg_new_update[...] kicked off via Kernel.spawn with an "individual" check; files under the size/time threshold (PROCESSING_TIME_THRESHOLD = 1000) get queued (waiting_to_be_processed: true) and a background rake spawned; files over threshold are rejected and emailed to the data manager instead of processed. (The same method also has a freecen branch for the FreeCEN skin, not relevant here.)

What the spawned rake then does with the file is documented in CSV-Processing-Pipeline.

Important: Kernel.spawn is fire-and-forget — the HTTP response returns immediately with a "your file is being processed, you'll get an email" message. There is no way to learn the real outcome (parsed OK? how many records? any errors?) from this response — that only shows up later on Freereg1CsvFile#processed/error, or via the emailed notification.

Routes

resources :csvfiles, plus csvfiles/:id/error → #replace (odd naming — it's actually the replace-with-errors-shown view), csvfiles/:id/download → #download.