Iterative Refinement (Multi‐Pass Generation) - amosproj/amos2025ss04-ai-driven-testing GitHub Wiki

This page describes the iterative processing feature, which enables the system to use a Large Language Model's (LLM) output as the input for a subsequent query. This creates a powerful loop for refining results, performing self-correction, or chaining sequential tasks.

1. Overview

The standard workflow of the system is a single pass: a user provides a prompt, and the LLM generates a single response. The iterative refinement feature extends this into a multi-pass workflow.

In this mode, the response from the first pass is automatically used to construct a new prompt, which is then sent back to the same LLM for a second pass. This process can be repeated multiple times. The primary goal is to improve the quality of the final output by allowing the model to review and enhance its own work.

2. How It Works

The iterative process follows this general data flow:

  1. Initial Prompt: The user provides the initial source code and prompt (e.g., "Write unit tests for this code").
  2. Iteration 1: The prompt is sent to the selected LLM, which generates a first-pass response (e.g., the generated test code).
  3. Prepare for Next Iteration: The system extracts the code from the first response. It then constructs a new prompt using this generated code as the new source code, along with a hard-coded "refinement" instruction.
  4. Iteration 2: This new prompt (e.g., "Please review and correct the following code...") is sent to the LLM. The model generates a second, hopefully improved, response.
  5. Final Output: After the specified number of iterations is complete, the final response from the last pass is saved and returned to the user.

3. Usage

This feature is controlled via a command-line flag in the backend.

Command-Line Usage

To enable iterative refinement, use the --iterations flag when running the main script.

python backend/main.py --model <model_index> --prompt_file <your_prompt.txt> --iterations <number_of_passes>
  • --iterations <number_of_passes>: Specifies the total number of passes. For example, --iterations 2 will perform the initial generation and one refinement pass.
  • If the --iterations flag is omitted, it defaults to 1, preserving the standard single-pass behavior.

Example: Two-Pass Refinement

This command will generate a test, then feed that test back to the model for a second pass to correct and improve it.

python backend/main.py --model 0 --iterations 2

4. Use Cases and Examples

Use Case 1: Self-Correction (Current Implementation)

This is the primary use case supported by the current implementation. The second pass explicitly asks the LLM to review and fix the code it just generated.

  • Pass 1 Prompt: "Write unit tests for the following Python function: def add(a, b): return a + b"
  • Pass 1 Response: (LLM generates a test case, which might have a minor bug or lack comments).
  • Pass 2 Prompt: "Please review the following Python code that was generated. If you find any errors, fix them. If there are no errors, improve the code by adding comments, docstrings, or improving clarity. [...code from Pass 1...]"
  • Pass 2 Response: (The LLM returns a corrected or improved version of the test code).

Developer Note: The prompt for the refinement step is currently hard-coded in backend/execution.py. To implement different sequential tasks, this logic would need to be customized.

Use Case 2: Sequential Task Chains (Future Enhancement)

While not fully customizable yet, the iterative framework enables more complex, multi-step tasks. For example:

  1. Pass 1: Generate a Python class based on a description.
  2. Pass 2: Use the generated class from Pass 1 as input, and prompt the LLM to generate unittest tests for it.
  3. Pass 3: Use the code and tests from the previous passes as input, and prompt the LLM to generate Markdown documentation.

This advanced workflow can be achieved in the future by allowing custom prompts for each iteration.

5. Future Enhancements

  • Custom Prompts per Iteration: Allow the user to specify a sequence of different prompts for each pass.
  • Different Models per Iteration: Allow using a different LLM for each pass (e.g., a fast model for initial generation and a more powerful model for refinement).
  • UI Integration: Add a field in the web interface to allow users to specify the number of refinement passes directly from the UI.