Implementing Real‐Time Data Tracking in Games with MQTT and Python Buffers - ECE-180D-WS-2024/Wiki-Knowledge-Base GitHub Wiki

Introduction

Have you ever poured hours into a game, only to lose all your progress when the power goes out? Unfortunately, this was a frequent occurrence in the past. But fear not, intrepid gamer! With advancements like Python and MQTT, we can now create games that remember your every move, ensuring your victories (and defeats) are never lost. In this article, we'll explore how to use data buffers and MQTT to make sure your game remembers you, even if your dog accidentally pulls the plug! Data buffers will be the way to save our game in the form of python data structures and MQTT will be the avenue to which we save it to a local server for peristent memory. MQTT is our lightweight protocol designed for efficient communication betwee devices, such as a computer playing a game and a local server. Buckle up, and let's dive into the world of persistent player data with Python!

On MQTT

MQTT is a lightweight messaging protocol commonly used in internet of things (IoT) applications and game development. In our game, MQTT will act as a communication bridge between players and a central server. Players' data which is buffered temporarily on their devices will be published to MQTT topics, allowing real-time updates and efficient data exchange between players and the server. This is where our Python code and carefully chosen data structures come in - they'll handle the temporary storage and management of data before it's published through MQTT or saved to a persistent storage. If the players system involuntarely disconnects, their game data can be saved through MQTT's on_disconnect() function. This allows the game state to be saved, avoiding unwanted headaches. MQTT will be the bridge that connects our players and their data, acting as a barrier to protect our player's attributes, accolades, scores, and settings. To reiterate, MQTT serves as the crucial communication bridge between players and the central server in our game. Leveraging Python alongside MQTT ensures efficient data exchange and seamless storage of player progress, safeguarding against interruptions like power outages or network instability. Through MQTT's on_disconnect() function, player data is securely saved, guaranteeing a hassle-free gaming experience.

Background on Data Buffers

In computer science, the term "data buffer" or simply "buffer" refers to a chunk of memory used to store data that is moved from one place to another. Usually, a buffer is the connection between two devices or processes that work at different speeds. Storing data in the buffer effectively mitigates this difference, making buffers crucial to ensuring smooth data transistion between the two devices or processes.

Suppose we have a system consisting of a producer and a consumer. The producer is a processes or device that is generating data, while the consumer is a processes or device that is using up the generated data. If the producer generates data faster than the consumer could consume, the excess data could be lost or corrupted before the consumer could process them, leading to sub-optimal or incorrect outcome. By placing a designated chunk of memory as the intermediate data storage between these processes, we can accommodate the speed difference between the producer and the consumer. The producer can now continuously place data into the buffer, and the consumer can take data from the buffer at the speed it can process them. When the buffer is correctly sized, it reduces latency and prevents data loss by ensuring continuous data flow, even when processes have different processing speeds. Additionally, the physical storage area of a buffer is usually implemented in software and allocated in RAM, which is much faster to access and reallocate compared to memory on hard disks, optimizing for speed and felxibility.

Different types of buffers serve various functions within a computing system. Input buffers temporarily hold data coming into the system from an external source, such as user input or data from a peripheral device. Output buffers, on the other hand, temporarily hold data that is to be sent out from the system, such as data being written to a disk or sent to a printer. Double buffering uses two buffers, allowing one to be filled while the other is being emptied, thereby enhancing efficiency and throughput in data processing. An upgrade of double buffering is circular buffering. Circular buffers consist of a chain of smaller buffers that receive data from the previous buffer and give data to the next, until data is retrieved from the last buffer by the consumer. This mechanism is usually used when a large chunk of data is transferred and when a faster transfer speed is needed.

In modern computing, buffers are ubiquitous and found in various applications. Operating systems rely on buffers for managing system calls, file operations, and inter-process communication. For instance, when a user reads a file, the operating system loads the data into a buffer before delivering it to the application. Network devices, such as routers and switches, use buffers to manage data packets, prevent packet loss, and handle network congestion. Buffers in networking equipment are crucial for managing bursts of traffic and ensuring data integrity. Audio and video playback systems depend on buffers to ensure smooth streaming and synchronization, preventing playback interruptions due to variations in data retrieval times. Database management systems utilize buffers to optimize query processing and transaction management. By caching frequently accessed data in buffers, databases can significantly reduce read and write times.

Python Data Buffers

In our Python game, we'll use software-based data structures as temporary buffers to store player data before sending it to a persistent storage or broadcasting it with MQTT. Key structures include lists to hold individual player scores and queues to manage data flow for processing or transmission. These buffers allow us to collect and manage data before permanently saving it or sharing it with other players, ensuring smooth gameplay even if network issues or other events occur. To illustrate how this would be implemented, its best to use a simple game we all know: Rock, Paper, Scissors:

Example Code

To demonstrate the effectiveness of MQTT any pyton data sturctures. We will analyze a simplified coding example which tracks the lifetime wins of a player against the computer. When the player disconnects, the lifetime wins will be saved to a local file which will be opened every time the game is played. The next sections will go into detail analyzing the code.

Implementing the Code

The following sub-sections are labeled with a number which corresponds to a section in the code which is shared in the appendix. Please refer to the provided code sections as needed.

1. Including Libraries

Assuming the paho MQTT library is installed on your machine, the first step to use MQTT is including the library "paho.mqtt.client" in your python script. This libarary will allow us to create a client which will communicate through the MQTT broker. In addition, we also want to include the "json" library for file writing/reading, "random" to randomize the computers moves, and "time" which will be used to create delays in our code.

2. Defining Data Buffers

In this section of code, we are declaring the data structure which will keep track of game data. This can be any data structure that suits your particular game. Good choices are lists, defenitions, and qeueus. If you would like to learn more information on the differnet data structures and how they work; see reference #3 at the bottom of this wiki article.

For this particular game, we use "dictionaries" because we are able to define the player and their lifetime score. In Python, a dictionary is a collection of key-value pairs. It is defined using curly braces {}, and each key-value pair is separated by a colon :. Here we are pairing users (human) to their lifetime score and the computer (AI) to its lifetime score. Initially the script sets the score to zero, but we will update this later if their exists history of passed wins.

# 2.Define data structure to store player data in memory (buffer)
game_data_buffer = {'human_score': 0, 'AI_score': 0}
}

3. Loading Game History (If it exists)

Here we define a function which upon call will locate a file called "player_wins.json" and load it into our data buffer. Here is a quick breakdown of what is happening:

def load_game_data_from_file():
    try:
        with open("game_data.json", "r") as file:
            return json.load(file)
    except FileNotFoundError:
        return {'human_score': 0, 'AI_score': 0}

game_data_buffer = load_game_data_from_file()

try: This block of code is attempting to open and read the contents of a file named "player_wins.json". The file is opened in read mode ("r").
with open(...) as file: The with statement is used for file handling. It ensures that the file is properly closed after reading its contents. The open("player_wins.json", "r") part opens the file in read mode and associates it with the variable file.
json.load(file): The json.load() function reads the JSON data from the file and parses it into a Python dictionary. JSON is a common data format used for storing and exchanging data.
except FileNotFoundError: If the specified file ("player_wins.json") is not found, this block catches the FileNotFoundError exception.
return {}: If an exception occurs (e.g., the file is not found), an empty dictionary ({}) is returned.

4. MQTT Callbacks

MQTT callbacks are specific functions that run under specific circumsatances when you try to connect a client over the MQTT broker. The three main ones are:

on_connect() : This function runs when the client sucessfully connects through MQTT. For simpler programs, this function only confirms connection through a print statement and allows the client to subscribe to a specific topic. However, for our application, we use the on_connect() function to a little bit more. After confimation of connection and subscribing to the topic we additionally search for game data if it exists. This is shown in the line:

    # Load existing game data
    load_game_data_from_file()

    # Recover and display game data
    recover_game_data()


     #Here we are calling two important functions:

    #load_game_data_from_file(): This function first searches for existing game data and loads it into our data buffer if it exists
    #recover_game_data(): This function displays the lifetime scores. If data was recovered, then it will display the lifetime scores,                  otherwise it will diplay the default scores of '0'.

on_disconnect: This function is called when our client cuts connection with MQTT either intentionally or unintentionally. This function will save the game data to a local file in either case which we can then access upon reconnect at a later time. This is how we avoid losing user data. This is achieved using the following line:


        # Save player data to file when the client disconnects
        save_player_data_to_file()

on_message(): This function is called anytime a new new message is received through MQTT. In our case, this is called whenever our user inputs their choice for rock, paper, or scissors. Once the input is received, we publish it and then this function is called. The function will read both the players and computers moves, and afterwards call the determine_winner() function which handles the game logic. The important thing to notice is that afterwards, we update our data buffer, print the results to the screen, and then directly save the new data to the file. With each round the file will be overwritten to reflect the new history. This keeps a current scoreboard directly after the results are determined.

5. Game Logic

This function is what handles the game logic. Depending on your application, this may be more complex or simple. For rock paper scissors, this function simply compares the choices of the human user and the computer and determines the winner based off the standard rock, paper, scissors rules. Once the results are determined, it returns the results.

6. Creating a client with MQTT

# Create MQTT Client
client = mqtt.Client()

# Assign Callback Functions
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message

# Connect to MQTT Broker
client.connect("mqtt.eclipseprojects.io", 1883, 60)

Create MQTT Client:
- This line creates an instance of the MQTT client. The mqtt.Client() creates a new MQTT client object.
Assign Callback Functions:
- These lines assign callback functions to the client instance. Callback functions are user-defined functions that get executed when certain events occur, like when the client connects to the broker (on_connect), disconnects (on_disconnect), or receives a message (on_message).
Connect to MQTT Broker:
- This line connects the MQTT client to the specified MQTT broker. The parameters are:
  - "mqtt.eclipseprojects.io": The address of the MQTT broker.
  - 1883: The port number on which the MQTT broker is listening.
  - 60: The keep-alive interval in seconds. It defines the maximum time interval between communications with the broker. If no communication happens within this interval, the broker considers the client disconnected.

This code initializes an MQTT client, assigns callback functions to handle connection, disconnection, and message events, and then connects the client to an MQTT broker. Once connected, the client will start listening for incoming messages and trigger the appropriate callback functions when events occur.

7. Starting MQTT client loop

Start MQTT Client Loop:
- client.loop_start(): This line starts a background thread that handles the MQTT client's communication loop. The loop allows the client to efficiently process incoming and outgoing messages without blocking the main thread.
Infinite Loop for Simulating a Game:
- This block contains a while True loop, which runs indefinitely. It simulates the continuous flow of a game. In a real game, this loop would likely wait for user input, make decisions, and then publish those decisions to the MQTT broker.
Publish Player Choices to MQTT Topic:
- Inside the loop, random choices for the player and AI are generated, and a dictionary (player_choices) is created. This dictionary is then converted to JSON format and published to the "game/player_choices" topic on the MQTT broker.
Stop MQTT Client Loop and Disconnect:
- client.loop_stop(): This stops the background thread responsible for the MQTT client's loop.
- client.disconnect(): This disconnects the client from the MQTT broker when the program is interrupted or terminated.

Conclusion

In the world of gaming, where losing progress can be a frustrating setback, the combination of Python, MQTT, and data buffers proves to be a game-changer. This setup ensures that player data is safeguarded, allowing for uninterrupted gaming experiences. MQTT acts as a reliable communication link, facilitating real-time updates and efficient data exchange between players and a central server. The use of data buffers, illustrated through Python dictionaries, becomes essential for temporarily storing player information before it's transmitted or saved persistently. This, coupled with file operations for loading and saving game data, provides a safety net, ensuring that player progress is preserved, even in the face of unexpected disruptions. The incorporation of data buffers not only enhances the reliability of player data but empowers developers to craft games that withstand unforeseen interruptions, offering players a seamless and enjoyable gaming journey. MQTT has many more applications and those details might bog down our use of it here. So, if you are interested in those fine details, please refer to references at the bottom of this wiki page.

Appendix

# 1. Import necessary libraries
import paho.mqtt.client as mqtt
import json
import random
import time

# 2. Define data structure to store game data in memory (buffer)
game_data_buffer = {'human_score': 0, 'AI_score': 0}

# 3. Load existing game data from file
def load_game_data_from_file():
    try:
        with open("game_data.json", "r") as file:
            return json.load(file)
    except FileNotFoundError:
        return {'human_score': 0, 'AI_score': 0}

game_data_buffer = load_game_data_from_file()

# 4. MQTT callbacks
def on_connect(client, userdata, flags, rc):
    print("Connected with result code " + str(rc))
    client.subscribe("game/player_choices")

    # Load existing game data
    load_game_data_from_file()
    
    #display the recovered data
    recover_game_data()


def on_disconnect(client, userdata, rc):
    if rc != 0:
        print('Unexpected Disconnect')
    else:
        print('Expected Disconnect')
        # Save game data to file when the client disconnects
        save_game_data_to_file()

def on_message(client, userdata, msg):
    player_choices = json.loads(msg.payload.decode())
    user_choice = player_choices['user_choice']
    AI_choice = player_choices['AI_choice']

    # Determine the winner and update scores
    user_score, AI_score = determine_winner(user_choice, AI_choice)

    game_data_buffer['human_score'] += user_score
    game_data_buffer['AI_score'] += AI_score

    # Display the tally after each round
    print(f"Human: {user_score} | AI: {AI_score}")
    print("Lifetime Wins Tally:")
    print(f"Human: {game_data_buffer['human_score']} | AI: {game_data_buffer['AI_score']}")

    # Update the persistent game data in the file
    save_game_data_to_file()

#. Define functions to handle file writing and reading 
def save_game_data_to_file():
    # Save game data to a file (game_data.json)
    with open("game_data.json", "w") as file:
        json.dump(game_data_buffer, file)

def recover_game_data():
    # Display the tally at the start of each round
    print("Lifetime Wins Tally:")
    print(f"Human: {game_data_buffer['human_score']} | AI: {game_data_buffer['AI_score']}")

# 5. Function for game logic
def determine_winner(user_choice, AI_choice):
    if user_choice == AI_choice:
        print("It's a tie!")
        return 0, 0
    elif (user_choice == 'R' and AI_choice == 'S') or \
         (user_choice == 'P' and AI_choice == 'R') or \
         (user_choice == 'S' and AI_choice == 'P'):
        print("You win!")
        return 1, 0
    else:
        print("AI wins!")
        return 0, 1

#--------------------------------------------------
# 6. Create client and begin comms with MQTT
# Create an MQTT client
client = mqtt.Client()
client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message

# Connect to the MQTT broker
client.connect("mqtt.eclipseprojects.io", 1883, 60)

# 7. Start the MQTT client loop
client.loop_start()

try:
    while True:
        # In a real game, you would have logic to receive player choices and publish them to "game/player_choices" topic
        # For simplicity, let's assume choices are received from another part of your code
        user_choice = random.choice(['R', 'P', 'S'])
        AI_choice = random.choice(['R', 'P', 'S'])

        # Publish player choices to the MQTT topic
        player_choices = {'user_choice': user_choice, 'AI_choice': AI_choice}
        client.publish("game/player_choices", json.dumps(player_choices))

        time.sleep(5)  # Sleep for 5 seconds between games

except KeyboardInterrupt:
    pass
finally:
    # Disconnect the MQTT client when the program is interrupted
    client.loop_stop()
    client.disconnect()