Profile Testing Worksheet - nonmodernist/magic-lantern GitHub Wiki
Magic Lantern Profile Testing Worksheet
Profile Information
- Profile Name: ________________________________
- Test Date: ___________________________________
- Films Tested: ________________________________
- Corpus Size: [ ] Test (1) [ ] Small (5) [ ] Medium (10+)
1. Strategy Performance Matrix
Track which strategies actually find results:
| Strategy Type | Films with Results | Avg Results/Film | Best Performing Film | Notes |
|---|---|---|---|---|
| exact_title | ___/___films | ___ | _____________ | |
| author_title | ___/___films | ___ | _____________ | |
| title_no_article | ___/___films | ___ | _____________ | |
| studio_title | ___/___films | ___ | _____________ | |
| director_title | ___/___films | ___ | _____________ | |
| title_box_office | ___/___films | ___ | _____________ | |
| title_exhibitor | ___/___films | ___ | _____________ | |
| source_adaptation | ___/___films | ___ | _____________ | |
| novel_film_title | ___/___films | ___ | _____________ | |
| [custom strategy] | ___/___films | ___ | _____________ |
Red Flags:
- Strategy with 0 results across all films
- Strategy with <20% success rate
- High-weight strategy underperforming
2. Publication Distribution
Track what sources you're actually finding:
Top 10 Publications by Frequency:
- _________________ (___%) Weight: ___
- _________________ (___%) Weight: ___
- _________________ (___%) Weight: ___
- _________________ (___%) Weight: ___
- _________________ (___%) Weight: ___
- _________________ (___%) Weight: ___
- _________________ (___%) Weight: ___
- _________________ (___%) Weight: ___
- _________________ (___%) Weight: ___
- _________________ (___%) Weight: ___
Publications with HIGH weight but LOW results:
- _________________ Weight: ___ Found: ___times
- _________________ Weight: ___ Found: ___times
- _________________ Weight: ___ Found: ___times
Publications with LOW weight but HIGH results:
- _________________ Weight: ___ Found: ___times
- _________________ Weight: ___ Found: ___times
- _________________ Weight: ___ Found: ___times
3. Content Type Analysis
What kinds of content is this profile finding?
Content Types Found (from full text analysis):
- Reviews (___ count) - Quality: High/Med/Low
- Production news (___ count) - Quality: High/Med/Low
- Box office (___ count) - Quality: High/Med/Low
- Advertisements (___ count) - Quality: High/Med/Low
- Photos (___ count) - Quality: High/Med/Low
- Interviews (___ count) - Quality: High/Med/Low
- Labor/strikes (___ count) - Quality: High/Med/Low
- Awards coverage (___ count) - Quality: High/Med/Low
- Other: __________ (___ count)
Does this match profile intent? [ ] Yes [ ] No [ ] Partially
4. Date Range Effectiveness
Results Distribution by Year Offset:
- -3 years: ___% of results
- -2 years: ___% of results
- -1 year: ___% of results
- Film year: ___% of results
- +1 year: ___% of results
- +2 years: ___% of results
- +3 years: ___% of results
Optimal date range appears to be: -___ to +___
Current profile setting: -___ to +___
5. Treasure Analysis
Examining the highest-value finds:
Top 5 Treasures Found:
- Film: __________ Type: ________ Score: ___ Why valuable: ______________
- Film: __________ Type: ________ Score: ___ Why valuable: ______________
- Film: __________ Type: ________ Score: ___ Why valuable: ______________
- Film: __________ Type: ________ Score: ___ Why valuable: ______________
- Film: __________ Type: ________ Score: ___ Why valuable: ______________
Common patterns in treasures:
- Publication: _________________
- Strategy that found them: _________________
- Content type: _________________
- Year offset from release: _________________
6. Problem Identification
Films with disappointingly few results (<5):
- _________________ (___results) Possible reason: ______________
- _________________ (___results) Possible reason: ______________
- _________________ (___results) Possible reason: ______________
Films with overwhelming results (>100):
- _________________ (___results) Needs refinement? Y/N
- _________________ (___results) Needs refinement? Y/N
Search strategies that seem broken:
- _________________ Problem: ______________
- _________________ Problem: ______________
7. Improvement Hypotheses
Based on this test, what changes might improve the profile?
Weight Adjustments Needed:
- Increase ______________ from ___ to ___ (because: ___________)
- Decrease ______________ from ___ to ___ (because: ___________)
- Add new publication: ______________ weight: ___
- Remove publication: ______________ (never found)
Strategy Adjustments:
- Disable strategy: ______________ (0% success rate)
- Boost strategy: ______________ to weight: ___
- Reduce strategy: ______________ to weight: ___
- Add custom strategy for: ______________
Date Range Adjustments:
- Widen to -___ / +___ years
- Narrow to -___ / +___ years
- Different ranges by confidence level
- Special handling for certain years: ______________
Missing Coverage:
- Profile misses this type of content: ______________
- Profile misses this publication: ______________
- Profile misses this time period: ______________
- Profile needs this search pattern: ______________
8. Comparative Analysis
If testing multiple profiles:
Compared to _____________ profile:
- More results? [ ] Yes [ ] No - By how much: ___%
- Better treasures? [ ] Yes [ ] No
- More relevant? [ ] Yes [ ] No
- Faster to complete? [ ] Yes [ ] No
Key differences:
9. Action Items
Changes to implement:
Next test should include:
- Different era films (years: _________)
- Different genre: ______________
- Larger corpus size
- Specific problematic films: ______________
- Films with known good coverage: ______________
10. General Notes
Surprises:
Profile works best for:
Profile struggles with:
Ideas for new profiles based on findings:
Overall assessment:
- Ready to use
- Needs minor adjustments
- Needs major revision
- Fundamentally broken - start over
Date completed: _______________
Time spent on analysis: _______________
Version of Magic Lantern used: _______________