The SerpentAI Hello World - nanpuhaha/SerpentAI GitHub Wiki

Note: Before proceeding with this guide, make sure you have followed the installation guide for your OS, including running 'serpent setup'

In this section, we will create our first, minimal SerpentAI project: A Hello World of sorts. The examples will use the Steam version of Super Hexagon as the target game, but you can swap it with another Steam game if you don't happen to own it.

Game Preparation

Whenever you start working with a new game, there will be few preparation steps to go through:

  1. Install the game
  2. Run the game
  3. Tweak the video settings:
    • switch to Windowed mode
    • select a Resolution (if applicable)
  4. Write down the window identifier:
    • On Linux and Windows: The window title of the game.
    • On macOS: The process name of the game. Since this isn't immediately obvious and can be tricky to get right, a utility ships with the framework to get the proper value. Run serpent window_name and follow the instructions.

Creating a Game Plugin

In Serpent.AI, to add support for a new game, you must create a plugin for it. The serpent executable provides a code generator for that exact task, so creating one is trivial.

Run serpent generate game

You will be greeted by a prompt that looks like this:

Provide the following answers:

  • SuperHexagon
  • steam

You will see some plugin installation output that should end with: SerpentSuperHexagonGamePlugin was installed successfully!

If you go and take a look at your plugins directory, you will see that files were generated for your Super Hexagon game plugin.

Tweaking the Game Plugin

To get Super Hexagon to launch through Serpent.AI, you will need to make a few edits in the game plugin.

Open plugins/SerpentSuperHexagonGamePlugin/files/serpent_SuperHexagon_game.py. The constructor should look similar to this:

    def __init__(self, **kwargs):
        kwargs["platform"] = "steam"

        kwargs["window_name"] = "WINDOW_NAME"

        kwargs["app_id"] = "APP_ID"
        kwargs["app_args"] = None
        

        super().__init__(**kwargs)

        self.api_class = SuperHexagonAPI
        self.api_instance = None
  • Replace WINDOW_NAME with the game window identifier you wrote down earlier (Super Hexagon for Super Hexagon).
  • Replace APP_ID with the Steam APPID of the game (221640 for Super Hexagon).

Pro-tip: You can find Steam APPIDs quickly by searching for game titles on SteamDB

Executable Game

If you are doing the Hello World with an executable game

   def __init__(self, **kwargs):

        kwargs["platform"] = "executable"
        
        kwargs["window_name"] = "WINDOW_NAME"
        
        kwargs["executable_path"] = "EXECUTABLE_PATH"

        super().__init__(**kwargs)

        self.api_class = StreetFighterXMegaManAPI
        self.api_instance = None
  • Replace WINDOW_NAME with the game window identifier you wrote down earlier.
  • Replace EXECUTABLE_PATH with an executable in PATH or the full path to a file you want to launch.

Launching the Game

Run serpent launch SuperHexagon

The game will proceed to launch. You will know that Serpent.AI can find the game window if it repositions it to the top left corner.

Leave the game running.

Creating a Game Agent Plugin

In Serpent.AI, to receive frames from the game and interface with it, you must create a game agent plugin. There is also a code generator for this task to accelerate development.

Run serpent generate game_agent

You will be greeted by a prompt that looks like this:

Provide the following answers:

  • SuperHexagon

You will see some plugin installation output that should end with: SerpentSuperHexagonGameAgentPlugin was installed successfully!

If you go and take a look at your plugins directory, you will see that files were generated for your Super Hexagon game agent plugin.

Hello World

Open plugins/SerpentSuperHexagonGameAgentPlugin/files/serpent_SuperHexagon_game_agent.py.

Change handle_play so it looks like this:

    def handle_play(self, game_frame):
        print("Hello World!")

Run serpent play SuperHexagon SerpentSuperHexagonGameAgent

If all goes well, you see a bunch of "Hello World!" start streaming in your terminal.

Every time you see a new "Hello World!" appear, your game agent was just fed the latest frame from the game. You might have noticed they are not streaming extremely fast. That's because there is a limiter that throttles the maximum amount of frames an agent can see and react to per second. The threshold is configurable (see config/config.plugins.yml) but there is a reason for the default of 2 FPS: Reacting to 2 frames per second is still 120 APM (actions per minute). This is probably faster then you personally react to most games. Unless you are dealing with a game that purely deals with analog controls (racing games, for example), playing at extreme APMs will make the gameplay look unnatural and cause your agents to overreact in most cases.

Something a Little More Visual

Note: This section requires the GUI extra module to be installed. You can safely skip it as it involves having a working Kivy environment which is pretty complicated if the pip install doesn't work out. A new Hello World article will be provided in the future; one that avoids using the extra modules.

Let's say that on top of printing "Hello World!" you would like to visualize the frames your game agent saw. Serpent.AI ships with an application called the Visual Debugger. The Visual Debugger allows you to examine image data while your agent is running.

Open another terminal and run serpent visual_debugger

You should see something like this appear:

Resize it if needed and position it under the game.

Now change handle_play to this:

    def handle_play(self, game_frame):
        print("Hello World!")

        for i, game_frame in enumerate(self.game_frame_buffer.frames):
            self.visual_debugger.store_image_data(
                game_frame.frame,
                game_frame.frame.shape,
                str(i)
            )

Go to the level select of Super Hexagon.

Run serpent play SuperHexagon SerpentSuperHexagonGameAgent

You should see the frames start cycling in the visual debugger, loosely matching the rate at which the "Hello World!" strings are printed.

A game agent automatically maintains a small buffer of the frames it has received which make it possible to send them to the Visual Debugger in this example.

Something a Little More Interactive

As a last step in this Serpent.AI Hello World, let's add in some key presses.

Add the following import to your game agent file:

from serpent.input_controller import KeyboardKey

Change handle_play to this:

    def handle_play(self, game_frame):
        print("Hello World!")

        for i, game_frame in enumerate(self.game_frame_buffer.frames):
            self.visual_debugger.store_image_data(
                game_frame.frame,
                game_frame.frame.shape,
                str(i)
            )

        self.input_controller.tap_key(KeyboardKey.KEY_RIGHT)

Can you guess what is going to happen?

Try it out! Run serpent play SuperHexagon SerpentSuperHexagonGameAgent

This concludes the Serpent.AI Hello World. It was a speck of sand on a beach in terms of what can be accomplished but hopefully it got you interested and you are curious to know more!