Emendation System

Part of FreeREG-Core-Records.

Models: app/models/emendation_rule.rb, app/models/emendation_type.rb · Lib: lib/emendor.rb · Controllers: app/controllers/emendation_rules_controller.rb, app/controllers/emendation_types_controller.rb · Rake: lib/tasks/load_emendations.rake, apply_emendations.rake, correct_all_emendations.rake, check_emendation.rake

Parish registers are full of abbreviated forenames: Wm, Jno, Thos, Eliz. Transcribers record them as written. Emendation makes those records findable under the full name: when a search record is built, each forename that matches a rule gets an extra search name with the expansion. A record transcribed as Wm SMITH is then found by a search for William Smith, and the original wm is still searchable too.

Emendation changes only the search index (SearchRecord.search_names). The transcription (Freereg1CsvEntry) and what the record page displays are untouched.

Data model

• EmendationRule: original (the abbreviation, lower-case, e.g. wm), replacement (e.g. william), gender ('m', 'f', or nil for both), belongs_to :emendation_type. The unique index on (original, replacement) means one abbreviation can map to several expansions (al → albert and al → alfred), but each pair only once.
• EmendationType: a named group of rules: name, target_field (stored as a String, exposed as a Symbol; in practice always first_name), origin (who created the group, e.g. 'load_emendations rake task'). A type with rules can't be deleted.

How rules are applied

SearchRecord#transform → emend_all → Emendor.emend(search_names) (step 4 of the pipeline in Search-Record-Indexing, after names are lower-cased and multi-part names are split):

rules = @@emendations[name[:first_name]]          # exact match on the whole forename
rules.each do |rule|
  next if rule.gender && name.gender && name.gender != rule.gender   # (see below for nil gender)
  emended = SearchName.new(name.attributes.except('_id'))
  emended.first_name = rule.replacement
  emended.origin = 'e'                               # SearchRecord::Source::EMENDOR
  emended_names << emended
end

• Matching is on the whole forename after lower-casing and splitting: wm matches; wm. only matches if there's a rule for wm. too (the seed data creates wm, wm., wm:, wm- variants for common abbreviations); wm henry is split first, so wm matches.
• Gender: a gendered rule is skipped when the name's gender is known and different. The gender comes from the name's role (father → m, mother → f, baptised child → person_sex, deceased → guessed from relationship; see Search-Record-Indexing). When the gender is unknown, the rule is applied.
• Emended names keep the original's role, type and surname, and are marked origin: 'e'. That makes it easy to see in the data why a record matched.
• Emendation only applies to first names. Surnames are never emended.
• Soundex codes are created after emendation, so fuzzy searches also match the expansion.

The rule set is cached per process. Emendor.load_emendations reads every rule into the class variable @@emendations the first time it's needed and never reloads. A rule added or edited in the admin screen affects new search records only after the web server and any long-running rake process restart. Existing search records are only changed by re-indexing them (below).

Managing rules

Admin screens (/emendation_rules, /emendation_types):

• EmendationRulesController requires login and session[:role] == 'system_administrator' (require_system_administrator). The index groups rules A–Z by replacement. Create/edit take original, replacement, gender and type.
• /emendation_rules/forename_abbreviations is public (no login): an A–Z list of each full forename with the abbreviations that expand to it. It's useful to point researchers and transcribers at it.

Seed data: rake load_emendations deletes every rule belonging to a type whose origin is 'load_emendations rake task', then recreates that type and its ~2,500 rules from the hard-coded list in lib/tasks/load_emendations.rake. Any rule an admin added to that type through the UI is lost when this task is rerun. Put local additions in a different type.

Re-indexing existing records after a rule change

Task	What it does	Status
rake correct_all_emendations[limit,fix,reset]	For each rule (alphabetically), finds search records with search_names.first_name == original that lack an origin: 'e' name equal to replacement, and with fix=true re-runs emend_all and saves. Resumable: progress is kept in tmp/correct_all_emendations_progress.txt; reset=true starts again. Writes log/correct_all_emendation.txt	Use this one
rake check_emendation[limit,original,replacement,type,fix]	The same check for a single rule	Works
rake apply_emendations[range,verbose,pretend]	Per rule and county, runs Emendor.apply_emendation → full transform + save	Older task; use correct_all_emendations instead

A full reprocess of the affected batches (FreeREG-Operations-and-Rake-Tasks) also re-emends, because it rebuilds the search records from scratch.