My theme is crashing in SM-RIO but it works in 5.1/5.0.12/etc

Your theme uses metrics and it is missing metrics for P3/P4. Since SM-RIO is 4 players, those are needed because metrics are loaded for all players instead of only loading when the player is joined.

Error spam after playing routine chart

Routine forcibly joins p3 and p4 so you can play co-op x4 then removes them after. I wish I had a better solution, but for now you need to account for P3/P4 being joined in Evaluation and ScreenSelectMusic, even if it means loading empty actors or breaking a for loop early or changing your for loops to for pn in ivalues({PLAYER_1,PLAYER_2}) do

I just compiled SM-RIO and replaced my stepmania exe/elf, but SM-RIO is crashing immediately.

Clear your cache. Also clear it when switching back.