Fixing the Resource Fork Error in Rocoto Python Script

Looking at the error logs, you're experiencing two critical issues:

1. fork: retry: Resource temporarily unavailable - This is a system-level error indicating you've reached the process limit (max 512 processes on that host)
2. UnboundLocalError: cannot access local variable 'last_exception' where it is not associated with a value - A bug in the attempt_multiple_times function

Let me implement a solution that addresses both:

1. I'll fix the bug in the attempt_multiple_times function
2. Add a backoff mechanism for process limit errors
3. Improve error handling for fork failures

Now I'll modify the rocotostat_summary and rocoto_statcount functions to increase their resilience when running into process limit Finally, let's improve the main section to handle process limit issues more gracefully:

Enhancements Made to Address the Resource Fork Error

I've implemented several improvements to the rocotostat.py script to handle process limit issues more gracefully:

1. Fixed the Bug in attempt_multiple_times Function

• Added proper initialization of last_exception and error handling
• Added explicit handling for OSError with errno.EAGAIN (resource temporarily unavailable)
• Implemented exponential backoff with jitter for process limit errors

2. Improved Error Handling in rocotostat_summary and rocoto_statcount

• Increased retry attempts from 4 to 5
• Added handling for empty output
• Improved error diagnostics

3. Enhanced the Main Section

• Added diagnostics to check and log the system's process limits
• Added graceful handling for cases where process limits are hit repeatedly
• Provided sensible default behaviors when rocoto commands can't be executed

4. Added System Resource Monitoring

• Added logging of process limits at startup
• Added specific error handling for resource issues

Key Benefits of the Changes

1. Graceful Handling of Process Limits: The script now recognizes when the system is hitting process limits and applies exponential backoff with jitter to give the system time to recover.

2. Better Error Reporting: More detailed logging about the specific errors encountered.

3. Fail-Safe Operation: Even if process limits are hit, the script will try to continue operation with sensible defaults rather than crashing.

4. Intelligent Retry Logic: The exponential backoff strategy means each successive retry waits longer, increasing chances of success.

The script should now be much more robust on systems with tight process limits (nproc=512) and should handle temporary resource unavailability gracefully without causing crashes.