Capturing Frames and Regions - nanpuhaha/SerpentAI GitHub Wiki

One very common task in the workflow of Game Agent creation is capturing game frames or regions of said game frames. In this section we'll cover common use cases and how to tackle them with Serpent.AI.

• Measuring game screen regions.
• Extracting game sprites.
• Testing image processing code.

1. Launch the Game through the serpent launch command. (example: serpent launch SuperHexagon)
2. Get to the section of the game you are interested in manually.
3. Run serpent capture frame <game_name> <interval> (example: serpent capture frame SuperHexagon 1)

The captured game frames will be saved to datasets/collect_frames.

• Building a dataset for game context classification.
• Building a dataset for other machine learning training tasks.

1. Launch the Game through the serpent launch command. (example: serpent launch SuperHexagon)
2. Get to the section of the game you are interested in manually.
3. Run serpent capture context <game_name> <interval> <context_name> (example: serpent capture context SuperHexagon 1 main_menu)

The captured game frames will be saved to datasets/collect_frames_for_context/<context_name>. Since this is mainly meant for image classification machine learning tasks, the captured game frames are downsized to half-width and half-height.

• Capturing rectangular game sprites
• Collecting non-rectangular game sprites in preparation to isolate them from their backgrounds
• Collecting samples with text to establish its OCR preset.

1. Define at least one screen region in your Game plugin.
2. Launch the Game through the serpent launch command. (example: serpent launch SuperHexagon)
3. Get to the section of the game you are interested in manually.
4. Run serpent capture region <game_name> <interval> <screen_region> (example: serpent capture region SuperHexagon 1 MAIN_MENU_OPTIONS)

The captured game frames will be saved to datasets/collect_frames/<screen_region>.