Home - amosproj/amos2025ss04-ai-driven-testing GitHub Wiki

Welcome to the amos2025ss04-ai-driven-testing wiki!

User/Developer Guide

User and Developer Guide

Wiki Pages

Architecture

• Architecture
  Overview of the system architecture and component interactions.

• CLI Pipeline
  Overview over the continous integration pipeline for the project.

• Docker Runner
  A Utility enabling docker containers to run sequentially on python

• Ollama Setup Information
  Possible ways to use Ollama to run LLMs

• Web Interface
  Tutorial on how to start it

• Backend API Design

• Backend Installation and Setup

• Build and Deployment Guide

• CLI Design

• CLI Pipeline

• How to Include a project as context

• Model Configuration Design

• Model System Design

• Iterative Refinement (Multi‐Pass Generation)

Metrics

• Code Complexity
  Description of the code complexity metrics that will be used for evaluation.

• Code Coverage

Language Models Considered

A list of all language models evaluated for use in the project.

• DeepCoder
• DeepSeek-Coder V1
• Google Gemma 3
• Mistral AI
• Phi-4 Mini
• Qwen 2.5 Coder
• StarCoder
• Tinyllama
• Qwen3
• OpenHermes2.5
• Smollm2
• Phi‑4 Reasoning

General LLM Information

• LLMs Incompatible With Our Project
  Overview of models that were evaluated but deemed unsuitable.

• AI-Model Benchmarks
  Standard Benchmarks used to evaluate LLMs

• Docker Performance
  Performance of different docker configurations when running LLMs

• Evaluating LLM Responses to Spelling Errors

• Prompt Design

• Running AI LLM Projects in CI

• Comparison of 1b, 3b, 7b and 14b models

• Benefits of chaining LLMs

• How to finetune a LLM

• How to train a LLM

• How to run on the FAU HPC

• Choosing the Right Dataset for LLM Training on the University HPC

Integration Considerations

• Robot Framework
  Information about potential integration with the Robot Framework.

Project Contribution Infos

• How to be an AMOS Release Manager
  How to do the weekly release

• Contributing

• Prerequisites

The wiki is actively maintained. Additional content may be added as the project evolves.