Prompting Workflow: Overview - da-moon/llm-playground GitHub Wiki

Table Of Contents

Key Components of the Workflow

1. Brainstorming and Design Phase

  • Purpose: Begin each project by discussing ideas, brainstorming solutions, and iterating on designs. This phase involves back-and-forth discussions to refine the project’s architecture, design patterns, and overall structure.
  • Output: A detailed Summary Document that captures all key decisions, patterns, and contextual information.

2. Summary Document Creation

  • Purpose: After the brainstorming session, create a Summary Document to encapsulate all major decisions, design patterns, and critical details. This document serves as the foundation for future chat threads and helps maintain continuity.
  • Key Content: Project overview, key decisions, design patterns, contextual details, dependencies, and next steps.

3. Task Decomposition

  • Purpose: Break down the project into smaller, self-contained tasks. Each task should be documented in a Task Document that provides detailed instructions, a checklist, and expected outcomes.
  • Key Content: Task overview, detailed instructions, checklist, expected outcomes, and feedback loop.

4. Task Implementation

  • Purpose: Start a new chat thread for each task using the Task Document as the context. This ensures that the task can be completed within a single thread, avoiding context loss and ensuring focus.
  • Output: Completed tasks that align with the overall project design.

5. Iteration and Refinement

  • Purpose: Use the Review and Feedback Document to gather feedback on completed tasks. Iterate on the design and implementation as necessary to refine the project.
  • Key Content: Feedback summary, specific comments, action items, and iterations.

Managing Context Size and Requesting Documents

Objective: To optimize the performance of Large Language Models (LLMs) by managing context size effectively, avoiding the "Lost in the Middle" (LIM) effect, and ensuring that critical details are not buried in large contexts.

Why It Matters: Research indicates that when working with extensive contexts, performance can degrade if relevant details are placed in the middle of the text. This is particularly important when handling large amounts of information, such as a flattened codebase or detailed design discussions, where key information might get lost. Understanding this, it’s crucial to manage how you structure your interactions with LLMs to maintain high efficiency and accuracy. More insights can be found in this blog post on RAG vs. long context.

How to Request Documents and Manage Context Size:

  1. Ask for Summary and Task Documents Regularly:

    • Why: To prevent context overload and ensure that important details are captured without loss, regularly ask the LLM to generate Summary and Task Documents. These documents should encapsulate key decisions, patterns, and the next steps after significant discussions or decisions.
    • How: At the end of each major discussion or design phase, specifically request the creation of a Summary Document to capture the essence of the conversation and a Task Document to outline the next steps.

  2. Use Short, Focused Chat Threads:

    • Why: Shorter, focused chat threads help reduce the risk of important information getting buried in large contexts. This approach keeps the conversation and context concentrated on specific tasks or topics.
    • How: Divide your workflow into smaller, manageable tasks, and start a new chat thread for each distinct task or discussion topic. Reference the relevant documents from previous threads to maintain continuity without overwhelming the context.

  3. Start New Threads for Each Major Phase:

    • Why: Starting new threads for each phase of the project ensures that the context remains relevant and manageable, preventing the LIM effect.
    • How: When transitioning from one phase of the project to another (e.g., from brainstorming to implementation), summarize the current context, request a Summary Document, and then start a new thread to begin the next phase.

  4. Revisit Key Information in New Threads:

    • Why: To ensure that critical information is not lost, begin each new chat thread by revisiting and recapping key points from previous threads.
    • How: Start the new thread by briefly summarizing or asking the LLM to review the relevant Summary Document before continuing with the new discussion or task.

  5. Maintain a Centralized Repository for Documents:

    • Why: A centralized repository ensures that all documents generated during the project are easily accessible and can be referenced across multiple chat threads, preserving continuity.
    • How: Store all generated Summary, Task, and other documents in a central location, such as a cloud-based document management system or a Git repository. Reference these documents when starting new chat threads to maintain a cohesive project narrative.

Example of Workflow Management:

  1. End of Brainstorming Phase:

    • Request a Summary Document summarizing the discussion.
    • Start a new chat thread for task decomposition.

  2. Task Decomposition:

    • Request Task Documents for identified tasks.
    • Start new threads for each task, ensuring context remains focused.

  3. Task Implementation:

    • Begin each implementation thread with a brief recap of the Task Document.

  4. Review and Feedback:

    • At the end of the implementation, request a Review and Feedback Document.
    • Start a new thread for iterations based on feedback.

By following these steps, you can maintain high context relevance, minimize the risk of the LIM effect, and ensure that your interactions with LLMs remain efficient and effective. This approach will help you maximize the utility of LLMs while preserving the integrity and clarity of your project discussions.

Detailed Documentation Structure

1. Summary Document

  • Purpose: Capture all key decisions, patterns, and contextual details from the brainstorming phase.
  • Key Sections:
    • Project Overview
    • Key Decisions
    • Design Patterns and Best Practices
    • Contextual Details
    • Dependencies and Interactions
    • Next Steps

2. Task Document

  • Purpose: Provide detailed instructions and guidelines for completing specific tasks.
  • Key Sections:
    • Task Overview
    • Detailed Instructions
    • Checklist
    • Expected Outcome
    • Feedback Loop

3. Decision Log

  • Purpose: Track important decisions made throughout the project, including the reasoning behind them.
  • Key Sections:
    • Decision Summary
    • Context
    • Alternatives Considered
    • Final Decision

4. Requirements Specification

  • Purpose: Document detailed functional and non-functional requirements for each feature or component.
  • Key Sections:
    • Functional Requirements
    • Non-Functional Requirements
    • Acceptance Criteria

5. Implementation Guide

  • Purpose: Provide detailed instructions on how to implement specific design patterns, frameworks, or technologies.
  • Key Sections:
    • Overview
    • Step-by-Step Instructions
    • Best Practices
    • Examples

6. Review and Feedback Document

  • Purpose: Collect and organize feedback on completed tasks or designs, guiding further iterations.
  • Key Sections:
    • Feedback Summary
    • Specific Comments
    • Action Items
    • Adjustments

7. Test Plan

  • Purpose: Outline the testing strategy, including what needs to be tested, how it will be tested, and the expected outcomes.
  • Key Sections:
    • Test Cases
    • Test Data
    • Expected Results
    • Test Environment

Instructions for Using This Workflow

This section provides a step-by-step guide on how to effectively use this workflow to manage and implement your project. Follow these instructions to ensure that all phases of your project are well-documented, focused, and efficiently executed.

Step 1: Begin with Brainstorming and Design

Objective: Start by discussing and refining ideas. Use this phase to explore different design options, architectures, and patterns.

Example: You are starting a new project that requires a robust API with high security and scalability. Begin by discussing possible architectures, like microservices or monoliths, and explore security patterns such as JWT authentication or OAuth.

Actions:

  1. Create a New Chat Thread: Begin the discussion with the LLM, focusing on exploring different ideas.
  2. Capture Key Points: As decisions are made, note them down. Consider options, pros, cons, and final choices.

Step 2: Create a Summary Document

Objective: Once brainstorming is complete, create a Summary Document to encapsulate all key decisions, patterns, and contextual details.

Example: If the brainstorming led to the decision to use a microservices architecture with JWT authentication, the Summary Document should capture this, explaining why these choices were made and how they will impact the project.

Actions:

  1. Compile Notes: Gather all notes from the brainstorming session.
  2. Document Key Decisions: Use the Summary Document template to detail the project overview, key decisions, design patterns, and next steps.
graph TD;
    A[Brainstorming and Design] --> B[Create Summary Document];
    B --> C[Document Key Decisions];
    C --> D[Detail Design Patterns];
    D --> E[Outline Next Steps];

Step 3: Decompose the Project into Tasks

Objective: Break down the project into manageable, self-contained tasks. Each task should be documented in a Task Document.

Example: For the API project, you might break it down into tasks such as "Set up JWT Authentication," "Create User Registration Endpoint," and "Implement API Rate Limiting."

Actions:

  1. Identify Major Tasks: Based on the Summary Document, identify the primary tasks that need to be completed.
  2. Create Task Documents: For each task, use the Task Document template to provide detailed instructions, expected outcomes, and a checklist.
graph TD;
    A[Create Summary Document] --> B[Identify Major Tasks];
    B --> C[Create Task Documents];
    C --> D[Set Up JWT Authentication];
    C --> E[Create User Registration Endpoint];
    C --> F[Implement API Rate Limiting];

Step 4: Implement Each Task

Objective: Start a new chat thread for each task using the corresponding Task Document. Ensure the task is completed within a single thread to avoid context loss.

Example: When implementing JWT authentication, start a new chat thread, provide the Task Document as context, and work through the implementation.

Actions:

  1. Start a New Thread for Each Task: Use the Task Document as the starting point.
  2. Implement the Task: Follow the detailed instructions in the Task Document.
  3. Complete the Checklist: Ensure all items on the checklist are completed.
graph TD;
    A[Create Task Documents] --> B[Start New Thread for Each Task];
    B --> C[Implement Task];
    C --> D[Complete Checklist];

Step 5: Iterate and Refine

Objective: After implementing a task, use the Review and Feedback Document to gather feedback and iterate on the design and implementation as necessary.

Example: After setting up JWT authentication, review the implementation with feedback from other team members. If necessary, iterate on the implementation to improve security or performance.

Actions:

  1. Gather Feedback: Use the Review and Feedback Document to collect input on the implementation.
  2. Make Adjustments: Based on the feedback, iterate on the design or code.
  3. Document Changes: Update the relevant documents to reflect any changes made.
graph TD;
    A[Implement Task] --> B[Gather Feedback];
    B --> C[Make Adjustments];
    C --> D[Document Changes];

Step 6: Testing and Validation

Objective: Before finalizing the project, ensure that all features and components are thoroughly tested according to the Test Plan.

Example: After completing the API, create test cases to ensure that JWT authentication works correctly, that rate limiting is effective, and that the API meets all functional and non-functional requirements.

Actions:

  1. Create Test Cases: Use the Test Plan template to outline what needs to be tested.
  2. Execute Tests: Run the tests and document the results.
  3. Review Test Results: Validate that the project meets all acceptance criteria.
graph TD;
    A[Implement Task] --> B[Create Test Cases];
    B --> C[Execute Tests];
    C --> D[Review Test Results];

Example Workflow

graph TD;
    A[Brainstorming and Design] --> B[Create Summary Document];
    B --> C[Decompose into Tasks];
    C --> D[Implement Task 1];
    C --> E[Implement Task 2];
    D --> F[Review and Feedback];
    E --> F;
    F --> G[Iterate and Refine];
    G --> H[Testing and Validation];
    H --> I[Final Approval];

6. Example Workflow with Document Requests

This section provides a practical example of how to manage your workflow, request documents, and maintain an organized project structure using LLMs.

graph TD;
    A[Brainstorming and Design] --> B[Create Summary Document];
    B --> C[Decompose into Tasks];
    C --> D[Implement Task 1];
    C --> E[Implement Task 2];
    D --> F[Review and Feedback];
    E --> F;
    F --> G[Iterate and Refine];
    G --> H[Testing and Validation];
    H --> I[Final Approval];

Step-by-Step Example:

  1. [ Optional ] : capture all documents and store them in git. For every iteration, we would create a dedicated directory with all the required files
ITERATION="1"
mkdir -p "prompts/iterations/${ITERATION}"
touch "prompts/iterations/${ITERATION}/00-summary-doc.md"
touch "prompts/iterations/${ITERATION}/01-task-doc.md"
touch "prompts/iterations/${ITERATION}/02-decision-log.md"
touch "prompts/iterations/${ITERATION}/03-requirements-spec.md"
touch "prompts/iterations/${ITERATION}/04-implementation-guide.md"
touch "prompts/iterations/${ITERATION}/05-review-feedback.md"
touch "prompts/iterations/${ITERATION}/06-test-plan.md"

  1. Brainstorming Phase:

    • Action: Start a new chat thread to brainstorm ideas and iterate on design decisions.
    • Document Request: At the end of the brainstorming session, request a Summary Document that captures all the key decisions, design patterns, and contextual information.
    • New Thread: Start a new thread for the next phase—Task Decomposition.

  2. Task Decomposition:

    • Action: In the new thread, use the Summary Document to identify and break down the project into smaller, manageable tasks.
    • Document Request: Request Task Documents for each identified task, detailing the instructions, checklists, and expected outcomes.
    • New Threads: Start separate threads for each task to maintain focus and context relevance.

  3. Task Implementation:

    • Action: Implement each task in its respective thread. Begin each thread by referencing the relevant Task Document.
    • Checklists: Complete the checklist provided in the Task Document to ensure all parts of the task are addressed.

  4. Review and Feedback:

    • Action: After completing each task, initiate a review process by requesting a Review and Feedback Document.
    • Iteration: Based on feedback, make any necessary adjustments and iterate on the implementation.
    • Documentation: Update relevant documents to reflect any changes made during the iteration process.

  5. Testing and Validation:

    • Action: As you approach the final stages, create a new chat thread dedicated to testing and validation.
    • Document Request: Request a Test Plan that outlines the test cases, expected outcomes, and test environment.
    • Execution: Conduct the tests, validate the outcomes, and document the results.
    • Final Approval: Once testing is complete and all criteria are met, move to final approval.

Conclusion

This workflow is designed to optimize the use of Large Language Models (LLMs) in project management and implementation by carefully managing context size, ensuring that key details are captured and easily accessible and avoiding the "Lost in the Middle" (LIM) effect. By regularly requesting Summary and Task Documents, breaking down the project into smaller, manageable tasks, and using focused chat threads for each phase of the project, you can maintain clarity and consistency throughout the project lifecycle.

Additionally, by maintaining a centralized repository for all project documents and being mindful of context size in each chat thread, you can ensure that your interactions with LLMs are as effective and efficient as possible. This approach not only supports the successful completion of the current project but also sets a strong foundation for future projects where the same principles can be applied to achieve high-quality results.

By following this structured workflow, you can leverage the full potential of LLMs, while maintaining rigorous standards for documentation, project management, and implementation.