IUCN Red List Integration - jra3/mulm GitHub Wiki

IUCN Red List Integration

This page documents the integration with the IUCN Red List API to provide conservation status information for species in the BAP system.

Overview

The IUCN Red List integration enriches species data with global conservation status information, complementing the existing CARES (Conservation, Awareness, Recognition, Encouragement, and Support) species tracking. This helps members understand the conservation importance of the species they're breeding.

Related:

Issue #179 - Full specification and implementation plan
Species Names System - How species data is structured
Database Schema - Database design

IUCN Red List Categories

The IUCN Red List uses a standardized set of categories to assess extinction risk:

Code	Category	Description	Color
EX	Extinct	No living individuals remain	Red
EW	Extinct in the Wild	Only survives in captivity	Red
CR	Critically Endangered	Extremely high risk of extinction	Red
EN	Endangered	High risk of extinction	Orange
VU	Vulnerable	High risk of endangerment	Yellow
NT	Near Threatened	Close to qualifying for threatened	Blue
LC	Least Concern	Low risk of extinction	Green
DD	Data Deficient	Insufficient data for assessment	Gray
NE	Not Evaluated	Not yet assessed	Gray

Reference: IUCN Red List Categories and Criteria

Implementation Approach

The integration is being implemented in phases to provide incremental value:

Phase 1: Database Foundation (✅ Complete)

Migration: 036-add-iucn-integration.sql

Added four new fields to the species_name_group table:

iucn_redlist_category - IUCN conservation status code (EX, EW, CR, EN, VU, NT, LC, DD, NE)
iucn_redlist_id - IUCN species identifier for API lookups
iucn_last_updated - Timestamp of last sync with IUCN API
iucn_population_trend - Population trend (Increasing, Decreasing, Stable, Unknown)

Created iucn_sync_log table to track all sync attempts:

Records success/failure of each species lookup
Logs error messages for troubleshooting
Supports multiple data sources (API, CSV import)

Indexes:

idx_species_iucn_category - Fast filtering by conservation status
idx_iucn_sync_group - Lookup sync history by species
idx_iucn_sync_date - Find recent sync attempts

Phase 1.5: Quick Win - CSV Data Import (🚧 In Progress)

Script: scripts/import-cares-iucn-data.ts

Before building the full API integration, we're importing existing IUCN data from cares_species.csv:

768 CARES species with conservation status
Maps CARES-prefixed codes (CVU → VU, CEN → EN, etc.) to standard IUCN categories
Provides immediate value while API integration is built
Creates baseline data for testing UI components

Why CSV First?

Immediate value - Data available in ~5 minutes
No API rate limits during import
Test database schema before API client
Validate UI designs with real data

Phase 2: API Client Development (📋 Planned)

File: src/integrations/iucn.ts

Building a robust API client with:

Rate Limiting: 2-second delay between API calls (IUCN requirement)
Retry Logic: Exponential backoff for failed requests
Error Handling: Graceful handling of missing species, API errors
Caching: Minimize redundant API calls
Type Safety: Full TypeScript types for all API responses

Configuration: src/config.json

{
  "iucn": {
    "apiToken": "YOUR_TOKEN_HERE",
    "baseUrl": "https://apiv3.iucnredlist.org/api/v3",
    "rateLimitMs": 2000,
    "enableSync": true,
    "maxRetries": 3,
    "timeoutMs": 10000
  }
}

Key API Endpoints:

/species/{name} - Get species details by scientific name
/species/synonym/{name} - Check if name is a synonym
/species/id/{id} - Get species by IUCN ID

Phase 3: Bulk Sync Script (📋 Planned)

Script: scripts/sync-iucn-data.ts

Command-line tool for synchronizing species data:

# Sync all species (respects rate limits)
npm run script scripts/sync-iucn-data.ts

# Dry run mode (preview without updating)
npm run script scripts/sync-iucn-data.ts -- --dry-run

# Sync only species missing IUCN data
npm run script scripts/sync-iucn-data.ts -- --missing-only

# Sync a specific species by ID
npm run script scripts/sync-iucn-data.ts -- --species-id 123

Features:

Progress tracking (N of M species processed)
Resume capability for interrupted syncs
Detailed logging to iucn_sync_log table
Summary report (success/error counts)
Handles synonyms and name mismatches

Performance:

~10 minutes per 300 species (due to 2-second rate limit)
Full sync of 1000 species: ~35 minutes

Phase 4: Admin Interface (✅ Complete)

Routes:

POST /admin/species/bulk-sync-iucn - Bulk sync IUCN data for selected species
POST /admin/species/sync-all-iucn - Trigger full sync of all species
Admin species list includes IUCN filtering and sync controls

UI Components:

Species list filters by conservation status
IUCN badges on admin species pages
Bulk sync interface for admins
Canonical name recommendations from IUCN

Phase 5: Public Display (✅ Complete)

Species Detail Page:

Conservation status badge (color-coded via iucnBadge mixin)
Population trend indicator
Link to IUCN Red List entry (when available)
Conservation status display

Species Explorer:

Filter by conservation status
Visual indicators for threatened species (CR, EN, VU)

Member Recognition:

Highlight breeding achievements for threatened species
Special recognition in activity feed

Database Schema Details

species_name_group Table (Extended)

CREATE TABLE species_name_group (
  -- Existing fields...

  -- IUCN Integration Fields (added in migration 036)
  iucn_redlist_category TEXT DEFAULT NULL
    CHECK (iucn_redlist_category IN ('EX', 'EW', 'CR', 'EN', 'VU', 'NT', 'LC', 'DD', 'NE')),
  iucn_redlist_id INTEGER DEFAULT NULL,
  iucn_last_updated DATETIME DEFAULT NULL,
  iucn_population_trend TEXT DEFAULT NULL
    CHECK (iucn_population_trend IN ('Increasing', 'Decreasing', 'Stable', 'Unknown'))
);

CREATE INDEX idx_species_iucn_category ON species_name_group (iucn_redlist_category);

iucn_sync_log Table (New)

CREATE TABLE iucn_sync_log (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  group_id INTEGER REFERENCES species_name_group(group_id) ON DELETE CASCADE,
  sync_date DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
  status TEXT NOT NULL
    CHECK (status IN ('success', 'not_found', 'api_error', 'rate_limited', 'csv_import')),
  category_found TEXT,
  error_message TEXT
);

CREATE INDEX idx_iucn_sync_group ON iucn_sync_log (group_id);
CREATE INDEX idx_iucn_sync_date ON iucn_sync_log (sync_date DESC);

Status Values:

success - Species found in IUCN, data updated
not_found - Species not in IUCN database (expected for many aquarium species)
api_error - API request failed (network, server error)
rate_limited - Hit API rate limit (should retry)
csv_import - Data imported from CSV file

API Access & Rate Limits

Obtaining an API Token

Visit: https://apiv3.iucnredlist.org/api/v3/token
Fill out the form (name, email, purpose)
Agree to Terms of Use
Token delivered via email (usually within 24 hours)
Non-commercial use only (BAP qualifies)

Terms of Use

By using the IUCN API, we agree to:

✅ Non-commercial use for educational/conservation purposes
✅ Proper attribution of IUCN as data source
✅ Respect rate limits (2-second delay between calls)
❌ No bulk redistribution of data
❌ No commercial use without permission

Reference: IUCN Terms of Use

Rate Limiting Strategy

IUCN Requirement: 2-second delay between API calls

Our Implementation:

Client-side delay: setTimeout() between requests
Exponential backoff: Retry with increasing delays on errors
Caching: Store responses to avoid redundant calls
Batch processing: Sync during off-peak hours
Monitoring: Track API usage in iucn_sync_log

Example:

async function getSpecies(name: string): Promise<IUCNSpecies | null> {
  // Wait for rate limit
  await delay(config.iucn.rateLimitMs); // 2000ms

  try {
    const response = await fetch(`${baseUrl}/species/${name}?token=${token}`);
    return await response.json();
  } catch (error) {
    // Log to iucn_sync_log
    await recordSyncError(name, error);
    return null;
  }
}

Data Challenges & Solutions

Challenge 1: Name Mismatches

Problem: Species names in BAP may not match IUCN exactly

Different authorities (IUCN vs. aquarium hobby)
Taxonomic revisions
Spelling variations

Solutions:

Try exact match first: Corydoras paleatus
Try synonym endpoint if not found
Log mismatches for manual review
Admin interface to correct mappings

Challenge 2: Many Species Not in IUCN

Problem: Most aquarium species lack IUCN assessments

~20-30% of BAP species may have IUCN data
Ornamental fish often not prioritized for assessment

Solutions:

Accept not_found as valid result
Display absence gracefully in UI ("Not Assessed")
Focus on species that do have data
Don't retry not_found species frequently

Challenge 3: Data Staleness

Problem: IUCN assessments can be outdated

Some assessments 10+ years old
Species status may have changed

Solutions:

Store iucn_last_updated to track data age
Show assessment date on UI
Re-sync periodically (e.g., annually)
Flag assessments >5 years old

Challenge 4: Slow Initial Sync

Problem: Rate limits make bulk sync very slow

1000 species × 2 seconds = 33+ minutes

Solutions:

CSV Import First: Get 600+ species instantly
Background Processing: Run sync during maintenance windows
Resume Capability: Don't lose progress on failures
Prioritization: Sync CARES species first
Manual Triggers: Let admins sync on-demand

Deployment Checklist

Development Environment

Migration 036 applied to dev database
IUCN config added to src/config.json
API token obtained and configured
CSV import script created and tested
Database schema validated

Production Environment

Apply migration 036 to production database
Add IUCN API token to /mnt/basny-data/app/config/config.production.json
Deploy updated application code
Run CSV import script (maintenance window)
Verify health checks pass
Monitor iucn_sync_log for errors

Initial Data Load

CSV Import (Quick - ~5 minutes)

ssh BAP "cd /opt/basny && npm run script scripts/import-cares-iucn-data.ts"

Full API Sync (Slow - ~30-60 minutes)

ssh BAP "cd /opt/basny && npm run script scripts/sync-iucn-data.ts -- --missing-only"

Verification

ssh BAP "cd /opt/basny && sqlite3 /mnt/basny-data/app/database/database.db 'SELECT COUNT(*) FROM species_name_group WHERE iucn_redlist_category IS NOT NULL;'"

Maintenance & Operations

Regular Maintenance

Annual Re-sync: Run full sync once per year to catch updated assessments:

npm run script scripts/sync-iucn-data.ts

Monitor Sync Errors: Check iucn_sync_log for failed syncs:

SELECT * FROM iucn_sync_log
WHERE status IN ('api_error', 'rate_limited')
ORDER BY sync_date DESC
LIMIT 50;

Data Coverage: Track percentage of species with IUCN data:

SELECT
  COUNT(*) FILTER (WHERE iucn_redlist_category IS NOT NULL) * 100.0 / COUNT(*) AS coverage_pct
FROM species_name_group;

Troubleshooting

Problem: API returns 401 Unauthorized

Cause: Invalid or expired API token
Solution: Request new token, update config

Problem: API returns 429 Too Many Requests

Cause: Exceeded rate limit
Solution: Increase rateLimitMs in config, retry later

Problem: Many species return not_found

Cause: Species not in IUCN database (expected)
Solution: Normal - not all species are assessed

Problem: Sync script hangs

Cause: Network timeout, API down
Solution: Kill script, restart with --resume

Future Enhancements

Potential Improvements

Automated Periodic Sync
- Cron job to re-sync stale data (>1 year old)
- Email notifications for species status changes
Enhanced Recognition
- Special BAP awards for breeding CR/EN species
- "Conservation Hero" badge for threatened species
Integration with Other Databases
- FishBase for additional taxonomy data
- CITES for trade restrictions
- Catalog of Fishes for taxonomic authority
Population Trend Tracking
- Historical tracking of status changes
- Alerts when species status worsens
- Dashboard showing conservation impact
Member Education
- Species detail page: "Why is this species threatened?"
- Conservation tips for threatened species
- Links to conservation organizations

References

IUCN Resources

External Projects

Implementation Status

Phase	Status	Completion Date	Notes
Phase 0: Pre-requisites	✅ Complete	2025-10-24	API token obtained
Phase 1: Database Schema	✅ Complete	2025-10-24	Migration 036 applied
Phase 1.5: CSV Import	✅ Complete	2025-10-24	Script ready, CSV-parse installed
Phase 2: API Client	✅ Complete	2025-10-24	Full TypeScript client with rate limiting
Phase 3: Database Functions	✅ Complete	2025-10-24	TDD approach, 15 tests, all passing
Phase 4: Bulk Sync Script	✅ Complete	2025-10-24	Full CLI tool with progress tracking
Phase 5: Admin UI	✅ Complete	2025-10-24	Admin routes and bulk sync interface
Phase 5: Public Display	✅ Complete	2025-10-24	IUCN badges and conservation status display
Phase 6: Testing	✅ Complete	2025-10-24	Integration tests and E2E coverage
Phase 7: Documentation	✅ Complete	2025-10-24	Wiki pages and inline docs
Phase 8: Deployment	✅ Complete	2025-10-24	Deployed to production

Legend: ✅ Complete | 🚧 In Progress | 📋 Planned

Last Updated: 2025-11-04 Maintained By: Development Team Related Issue: #179

IUCN Red List Integration - jra3/mulm GitHub Wiki

IUCN Red List Integration

Overview

IUCN Red List Categories

Implementation Approach

Phase 1: Database Foundation (✅ Complete)

Phase 1.5: Quick Win - CSV Data Import (🚧 In Progress)

Phase 2: API Client Development (📋 Planned)

Phase 3: Bulk Sync Script (📋 Planned)

Phase 4: Admin Interface (✅ Complete)

Phase 5: Public Display (✅ Complete)

Database Schema Details

species_name_group Table (Extended)

iucn_sync_log Table (New)

API Access & Rate Limits

Obtaining an API Token

Terms of Use

Rate Limiting Strategy

Data Challenges & Solutions

Challenge 1: Name Mismatches

Challenge 2: Many Species Not in IUCN

Challenge 3: Data Staleness

Challenge 4: Slow Initial Sync

Deployment Checklist

Development Environment

Production Environment

Initial Data Load

Maintenance & Operations

Regular Maintenance

Troubleshooting

Future Enhancements

Potential Improvements

References

IUCN Resources

Related Documentation

External Projects

Implementation Status