What is the purpose of Custom States?

Well Psych's modding isnt as... flexible as some people would like. specifically a lot of people would be able to do more if you could change the Menus with a mod. although you could make your own menus with a custom song SubState, it requires a bit more setup then custom states.

Isnt this similar, if not the same concept to another deprecated engine?

Probably.