Structured Output

Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don't need to worry about the model omitting a required key, or hallucinating an invalid enum value.

from litellm import completion
from pydantic import BaseModel


class Step(BaseModel):
    type: str
    path: str
    content: str

class PullRequest(BaseModel):
    steps: list[Step]

completion(model=model, messages=messages, response_format=PullRequest)

This is an example for the CodeMonkey case. Here, we are providing a response_format in the form of the PullRequest class and so the answer will come in that format.

Some benefits of Structed Outputs include:

• Reliable type-safety: No need to validate or retry incorrectly formatted responses;
• Explicit refusals: Safety-based model refusals are now programmatically detectable;
• Simpler prompting: No need for strongly worded prompts to achieve consistent formatting.

However, Structured Outputting is only available in the gpt-4o-mini-2024-07-18 and later and gpt-4o-2024-08-06 and later.

In other models, like gpt-4-turbo and earlier (this includes gpt-4 as well), Structured Outputting is not available and we need to use JSON mode instead.

---

JSON Mode

When JSON mode is turned on, the model's output is ensured to be valid JSON, except for in some edge cases that you should detect and handle appropriately.

To turn on JSON mode with the Chat Completions or Assistants API you can set the response_format to { "type": "json_object" }. If you are using function calling, JSON mode is always turned on.

Important notes:

When using JSON mode, you must always instruct the model to produce JSON via some message in the conversation. If you don't include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit. To help ensure you don't forget, the API will throw an error if the string "JSON" does not appear somewhere in the context.

JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors. You should use Structured Outputs to ensure it matches your schema, or if that is not possible, you should use a validation library and potentially retries to ensure that the output matches your desired schema.