Robot Framework - amosproj/amos2025ss04-ai-driven-testing GitHub Wiki

Using Robot Framework for AI-Driven Testing

This document explains how to use the open-source Robot Framework in the context of the this project. The goal is to have test cases generated by an AI and automatically executed using Robot Framework.

1. Installation

Install Robot Framework and additional libraries as needed:

pip install robotframework
pip install robotframework-seleniumlibrary    # for UI tests
pip install robotframework-requests           # for API tests

2. Project Structure

Recommended folder structure for your test files:

tests/
  ├─ login_tests.robot
  └─ api_tests.robot
resources/
  ├─ keywords.robot
  └─ variables.robot

3. Example: Writing a Test Case

UI Test (Selenium):

*** Settings ***
Library           SeleniumLibrary

*** Variables ***
${URL}            https://example.com
${BROWSER}        Chrome

*** Test Cases ***
Login With Valid Credentials
    Open Browser    ${URL}    ${BROWSER}
    Input Text      id=username    demo
    Input Text      id=password    demo123
    Click Button    id=login
    Page Should Contain Element    id=logout
    Close Browser

API Test (Requests):

*** Settings ***
Library           RequestsLibrary

*** Test Cases ***
Get User List
    Create Session    myapi    https://jsonplaceholder.typicode.com
    GET Request        myapi    /users
    Status Should Be   200

4. Test Execution

Run tests using the following command:

robot tests/

Generated files:

  • log.html → detailed execution log
  • report.html → summary report
  • output.xml → machine-readable result file

5. AI Integration

The AI (e.g., based on LLMs like Mistral or Ollama) can be configured to automatically create or extend .robot files:

Integration Options:

  • AI directly generates .robot files from source code or specs
  • AI is accessed via API and returns test case code
  • Integration in IDE (e.g., VS Code with Robot plugin)
  • Automated execution via CI/CD or scripts

6. Test Types & Layers

Robot Framework supports various types of tests:

By Test Type:

  • Unit tests
  • Integration tests
  • Acceptance tests (ideal due to readable syntax)

By Software Layer:

  • User interface (Selenium)
  • Business logic (custom keywords)
  • Persistence/database (with DB libraries)

Benefits for Your Project

  • Simple text-based format (ideal for automatic generation)
  • Extensible with libraries
  • Good integration into open-source tools
  • Readable even for non-developers (e.g., product owners)

Next Steps

  • Define which test types the AI should generate
  • Implement an interface that produces .robot files
  • Integrate Robot Framework into your development or CI/CD process

Example Projects and Research on using AI with the Robot Framework

Robotframework AI

RobotFramework-AI is a plugin library that extends the Robot Framework with AI features. It enables:

  • AI-Powered Test Data Generation: Via RealTestDataGenerator, it uses models like OpenAI to create realistic data (e.g., addresses) for tests.

  • Chatbot Integration: Allows dynamic, AI-generated replies within tests (similar to ChatGPT).

  • AI Assistants: Persistent assistants that support context, file input, and tailored instructions.

  • File Handling: Upload files for AI to analyze or reference during test runs.

  • Configurable AI Parameters: Control model, creativity (temperature), penalties, and format.

  • Logging: Built-in logging for test monitoring and debugging.

Supports OpenAI and is extensible for others (e.g., Gemini, Copilot).

AI with Robot Framework

At RoboCon 2024, David Fogl highlighted how Generative AI can assist Robot Framework by automating test data, generating API scenarios, dynamic XPaths, and even predicting failures. Integration is straightforward, but AI lacks full context understanding and needs careful human oversight.

Pros:

  • Fast test data and API scenario generation

  • Helps identify edge cases and test bottlenecks

  • Simplifies XPath and SQL test creation

  • Potential for smarter test strategy and security

Cons:

  • Risk of shallow or incorrect tests

  • Requires training and validation effort

  • Still needs human guidance for quality and accuracy

Other Research

An empirical study on robottest automation framework

Usage of Robot Framework in Automation of Functional Test Regression

MA Thesis on Self Healing Locators in LLMs with Robot Framework