Per Document Generator - digitalspace/eagle-demi GitHub Wiki

D - Per-Document Generator — Algorithm and Flow Documentation

This document describes the design, execution pipeline, and algorithmic heuristics used by the AI Classifier Tool (AICT) Generator defined in Per-Document-Generator.py.

Core Pipeline Architecture

The generator operates as a multi-stage deterministic sorting and metadata extraction pipeline:

flowchart TD
    START[Raw PDF Document & Metadata] --> GRAZE{1. Grazing Pre-pass}
    GRAZE -->|Yes: Study, Plan, Appendix| PT[Vein: PT - Project Technical Detail]
    GRAZE -->|No: Administrative File| COARSE{2. Family / Vein Sort}
    
    COARSE --> CP[Vein: CP - Certificate Proper]
    COARSE --> PE[Vein: PE - Public Engagement]
    COARSE --> AT[Vein: AT - Act Transition]
    COARSE --> CM[Vein: CM - Communication]
    COARSE --> OA[Vein: OA - Official EA Artifact]
    
    PT & CP & PE & AT & CM & OA --> ACT_REC{3. Operative Act Recovery}
    ACT_REC --> OUTPUT[Generate Verified JSON Metadata Output]
Loading

Processing Stages

1. The Grazing Pre-pass (Isolate Bulk Content)

  • Objective: Exclude high-volume technical and public submission attachments (~76% of all files) from deep classification.
  • Rule: If a document type or name contains standard bulk keywords (e.g., appendix, chapter, technical memo, baseline report, study, management plan), set Project Content = YES and tag it as PT (Project Technical Detail).
  • Performance Impact: Reduces fine classification evaluation scope by 74%, dramatically preserving processing resources.

2. Functional Family and Vein Sorting

For non-bulk administrative records, the generator categorizes files into functional tracks ("Veins"):

Vein Code Vein Name Precedence Triggers & Identifiers
CP Certificate Proper 1 Contains certificate in title and does not contain amend, transfer, application.
AT Act Transition 2 Contains request to start new, section 10 order, issued process order with transition indicators.
PE Public Engagement 3 Title contains comment, open house, notice of intent, PCP, or advertisement.
OA Official EA Artifacts 4 Title/Type contains order, amendment, exemption, extension, assessment report.
CM Communication 5 Contains letter or meeting notes (lowest-precedence).

3. Operative Act Recovery

  • Rule: Resolves whether a document operates under the 2002 Act or the 2018 Act.
  • Threshold Date: Uses the formal enactment and implementation of the 2018 EAA (e.g., 2021-04-16 Process Order date).
  • Result: Publication dates before the threshold are assigned to the 2002 Act, while newer dates default to the 2018 Act.

Outputs & Naming Conventions

The generator modifies the physical filename suffix to represent the coarse classification state without destructive renaming:

  • Pattern: <original_name> -[VV] -P.json (where VV is the two-letter Vein code, and P is Pending Verification status).
  • Schema structure matches Per-Document-Data-Format.md.
⚠️ **GitHub.com Fallback** ⚠️