Splatoon 2 Support

What is done

Note: Under development - Now we focus JA version only

category	scene	items	status	remarks
Lobby	-	-	-
Game	go_sign			Read timer value rather than detecting "Go!" message?
Game	low_ink		done	Detect player's "low ink!" status
Game	dead	code	done
Game	dead	cause of death ML	Need update
Game	finish		done
Game	in_game
Game	kill		done
Game	map		done	Detect map scene ("X" button in the battle)
Game	paint_tracker	code	done
Game	paint_tracker	ML	partial	Need new feature extraction method
Game	inklings_tracker			Need new feature extraction method
Game	ranked_battle_events	Clam Blitz	done	Using old image matching method
Game	ranked_battle_events	Rainmaker	done	Using old image matching method
Game	ranked_battle_events	Splat Zones	done	Using old image matching method
Game	ranked_battle_events	Tower Control	done	Using old image matching methodNo checkpoint detections yet.
Game	ranked_battle_events	Tower Control: Checkpoints		Need samples
Game	special_weapon_activiation	code	done
Game	special_weapon_activiation	ML	Need update
Game	start: stage	code	done
Game	start: stage	ML	partial	Need update
Game	start: rule	code	done
Game	start: rule	ML	partial	Need update
Game	sub_weapon		WIP
Game	timer_icon	Read timer value	done
Game	timer_icon	Read overtime status	done?
result	judge	Win/Lose detection	done
result	scoreboard	numbers classification	done	ML models needs to be updated
result	scoreboard	weapons classification	-	ML models needs to be generated
salmon_run	count	Read the egg count	done
salmon_run	game_over		done	Detect failure of salmon run games
salmon_run	game_start	code	done	Detect start of salmon run games
salmon_run	game_start	ML	Need update
salmon_run	mr_grizz	code	done	Detect Mr. Grizz's directions, and trigger events
salmon_run	mr_grizz	ML	Need update
salmon_run	norma	code	done	Detect how many eggs needed in the wave.
salmon_run	norma	ML	Need update
salmon_run	player_status	code	Partial	Detect each players' status (dead or alive, weapon type)
salmon_run	player_status	ML	Need update
salmon_run	result	code	Partial	Detect scoreboard
salmon_run	result	ML	Needs update
salmon_run	session		done?	Manages salmon_run scenes (enable/disable scenes)
salmon_run	wave_finish	code	done	Detect finish of the wave
salmon_run	wave_finish	ML	done
salmon_run	wave_start	code	done	Detect start of the wave
salmon_run	wave_start	ML	done
salmon_run	weapon_specified	code	Partial
salmon_run	weapon_specified	ML	Need update

Challenges for Splatoon 2

paint_tracker

• In Splatoon 1, the background color of paint counter was always black. it was easier to classify characters.
• In Splatoon 2, the background is transparent. Needs some ML approach to the same feature.

Inklings_tracker

• In Splatoon 1, all inkings shapes are very simple. IkaLog used simple algorithms to detect each status.
• Indicators in Splatoon 2 because more colorful. Needs new approach to detect inklings.

in_game

• In Splatoon 1, there is a timer icon that can be used for detection very easily.
• In Splatoon 2, we don't see any good static icons on the screen. Needs to find something different.

Number classification

• Splatoon 2 fonts are thin. If we use the original approach, confusion of 1/7 happens. Need some updates.

Map View

• In Splatoon 2, players can see the stage by pressing X button, it disturbs image recognition.
• We have an opportunity to detect "when/how long the player activated map view" and log as timeline events.

Console message

• Some messages, such as Friends online and Capture button, disturbs image recognition.

Other Challenges

Looking for new ML approaches

• SVM is the goal? Hmm, maybe it isn't. IkaLog runs 100+ classes classification real-time.
• Neural network? could be, but convolution is not applicable for the cpu-based application.
• IkaMatcher is actually fast enough (run in microseconds) in typical computers with x86 CPUs, if it is applicable and we can tune the parameters.
• OpenCV has SVM and MLP features, but it's python binding lacks (or limited) load/save of trained models.
• Python community has much ML modules but what we need is for ultra-fast, real-time detection.
• Each classifier need to run in sub-millisecond order in typical users' hardware!
• It also should work on non-powerful processors (such as dual-core ARM)