"Resource temporarily unavailable" when using rocotorun - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

"Resource temporarily unavailable" error when running the rocotorun command. This error occurs when the system cannot create a new process due to resource limitations.

Cause of the Error

The error is happening during a fork operation. Looking at the Rocoto codebase, I can see that:

The rocotorun script calls rocotorun.rb
Inside forkit.rb, the WorkflowMgr.forkit method uses Process.fork to create child processes
Multiple server processes may be created as part of running the workflow, including rocotoserver

The "Resource temporarily unavailable" error typically happens when:

The system has reached its maximum allowed number of processes for your user
The system is out of memory
There's a limit on the number of open file descriptors

Possible Solutions

Check system process limits:
```
ulimit -a
```
Look specifically at "max user processes" and "open files" limits
Temporarily free up resources:
- Close other applications that might be consuming resources
- Kill any orphaned Rocoto processes that might be running:
```
ps -ef | grep rocoto | grep -v grep
```
Clean up Rocoto log and temporary files:
- Check ~/.rocoto/ directory for accumulated logs that might be taking resources
Use rocotovacuum to clean up the workflow database:
```
bin/rocotovacuum -d workflow.db -w workflow.xml
```
This will clean up old, completed, or expired cycles from your workflow database.
Modify your Rocoto workflow to use fewer resources by:
- Reducing the number of concurrent tasks
- Adding more throttle constraints
Check if the server has many rocotoserver processes running from previous failed attempts:
```
ps -ef | grep rocotoserver
```
You might need to terminate these with:
```
pkill -f rocotoserver
```

The error indicates your system has reached resource limits and cannot create more processes. Once you free up resources, Rocoto should be able to run properly.