Handy Prompt (hprompt) - atomiechen/HandyLLM GitHub Wiki

Overview

Handy prompt files, or .hprompt files, are self-containing LLM request files in mark-up format. See more examples below.

Minimal example

---
# frontmatter data
model: gpt-3.5-turbo
temperature: 0.5
meta:
  credential_path: .env
  var_map_path: substitute.txt
  output_path: out/%Y-%m-%d/result.%H-%M-%S.hprompt
---

$system$
You are a helpful assistant.

$user$
Your current context: 
%context%

Please follow my instructions:
%instructions%

Key features

  • Frontmatter:
    • Specify request arguments
    • Specify output path and other runtime configurations
  • Body:
    • Construct chat messages with $system$, $user$, $assistant$ and $tool$.

Types of hprompt

.hprompt files are parsed into HandyPrompt objects by hprompt.load_from("xxx.hprompt"), and there are two kinds: ChatPrompt and CompletionsPrompt. The only difference is whether the body part contains message role keys (e.g. $user$).

Play with Hprompt

[!NOTE] For CLI usage, check out CLI Usage.

handyllm.hprompt module contains all you need to play with .hprompt files. Check out test_hprompt.py for details.

Load

from handyllm.hprompt import load_from, load, loads
# load from path
my_prompt = load_from("magic.hprompt")  # ChatPrompt or CompletionsPrompt
# load from file descriptor
with open('magic.hprompt') as fd:
    my_prompt = load(fd)
# load from string
with open('magic.hprompt') as fd:
    text = fd.read()
    my_prompt = loads(text)

You can explicitly specify prompt type:

from handyllm.hprompt import ChatPrompt, CompletionsPrompt
chat_prompt = load_from("chat.hprompt", cls=ChatPrompt) # of ChatPrompt type
completions_prompt = load_from('completions.hprompt', cls=CompletionsPrompt) # of CompletionsPrompt type

In case you want to use the parsed data (messages list or prompt string) to pass on to API calls, use .data:

chat_prompt.data  # this is a list
completions_prompt.data  # this is a str

Run

Both sync and async usages are supported.

Sync version:

result_prompt = my_prompt.run()

Async version:

result_prompt = await my_prompt.arun()

If your .hprompt specifies output paths, data will also be written to those paths.

To replace variables, there is a handy argument var_map which consumes a dict, and you can further use VM to pass in keyword arguments (keywords without %):

from handyllm import VM
result_prompt = my_prompt.run(var_map=VM(
    context='It is raining outside.',
    instructions='Write a poem.'
))

You can specify run_config to override the meta field stored in the hprompt file.

from handyllm.hprompt import RunConfig as RC
...
run_config = RC(
    output_path="new_output.hprompt",
    var_map={
        '%context%': 'It is raining outside.',
        '%instructions%': 'Write a poem.'
    }
)
result_prompt = my_prompt.run(run_config=run_config)

Stream processing

In stream mode, where you have specified stream to true in code or in the frontmatter, you can pass custom on_chunk handler to run_config to process the streamed chunk data (return values are ignored). Note that ChatPrompt and CompletionsPrompt's on_chunk handlers have different signatures.

  • Use on_chunk for ChatPrompt:
def proc_chat_chunk(role: str, text_chunk: Optional[str], new_tool_call: Optional[Dict]):
    print(role, text_chunk, new_tool_call)
    ...

result_prompt: ChatPrompt = chat_prompt.run(run_config=RC(
    on_chunk=proc_chat_chunk,
))
  • Use on_chunk for CompletionsPrompt:
def proc_completions_chunk(text_chunk: str):
    print(text_chunk)
    ...

result_prompt: CompletionsPrompt = completions_prompt.run(run_config=RC(
    on_chunk=proc_completions_chunk,
))

When running in async version (using arun), both sync and async handlers are supported (CompletionsPrompt works the same):

async def async_proc_chat_chunk(role: str, text_chunk: Optional[str], new_tool_call: Optional[Dict]):
    print(role, text_chunk, new_tool_call)
    ...

result_prompt: ChatPrompt = await chat_prompt.arun(run_config=RC(
    # you can use either sync or async handler
    on_chunk=async_proc_chat_chunk,
))

Eval

To get the evaluated hprompt without running it (where variables are substituted, template path strings are evaluated, etc.), use eval():

evaled_prompt = my_prompt.eval()

Pass runtime run_config to override configurations:

from handyllm.hprompt import RunConfig as RC
evaled_prompt = my_prompt.eval(run_config=RC(var_map={
    '%context%': 'It is raining outside.',
    '%instructions%': 'Write a poem.'
}))

Dump

result_prompt.dump_to("result.hprompt")

Chain

# chain result hprompt
prompt += result_prompt
# chain another hprompt
prompt += load_from("another.hprompt")
# chain prompt with manually set role and content
prompt.add_message('assistant', some_str)
prompt.add_message('user', 'continue')

Authoring Hprompt

Frontmatter

Request arguments

The root keywords of the frontmatter are request arguments that will be passed to chat api or completions api. Examples:

[!Tip]

YAML is a superset of JSON, so you definitely can use the JSON format for some fields.

model: gpt-3.5-turbo
temperature: 0.5
response_format: { "type": "json_object" }
stream: true
timeout: 10
tools: [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            },
            "unit": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"]
            }
          },
          "required": ["location"]
        }
      }
    }
  ]

Here you can also specify endpoint information like api_key, organization, api_type, api_base, api_version (used for Azure endpoints). But it is recommended to store these into a separate credential file and point to its path in the meta field (see below).

meta field

meta field of the frontmatter is parsed into a RunConfig dataclass instance.

All options are examplified below:

[!Note]

All relative paths in meta field are resolved relative to the hprompt file directory.

meta:
  # record request arguments in the output file
  # options: blacklist, whitelist, none, all
  # if not specified, use blacklist
  record_request: blacklist
  # if record_request is blacklist, record all request arguments except these
  # if not specified, use DEFAULT_BLACKLIST
  record_blacklist: ["api_key", "organization"]
  # if record_request is whitelist, record only these
  record_whitelist: ["temperature", "model"]

  # variable map
  var_map: 
    # wrap variable name with quotes because it contains special characters
    "%variable1%": data1
    "%variable2%": "data2"
  # alternatively, use variable map file
  var_map_path: var_map.txt

  # output the result to a file; you can use strftime format
  output_path: out/%Y-%m-%d/result.%H-%M-%S.hprompt

  # buffering for opening the output file in stream mode: -1 for system default, 
  # 0 for unbuffered, 1 for line buffered, any other positive value for buffer size
  output_path_buffering: 0

  # output the evaluated input prompt to a file; you can use strftime format
  output_evaled_prompt_path: out/%Y-%m-%d/evaled.%H-%M-%S.hprompt

  # credential file path
  credential_path: credential.env

  # credential type
  # options: env, json, yaml
  # if env, load environment variables from the credential file
  # if json or yaml, load the content of the file as request arguments
  # if not specified, guess from the file extension
  credential_type: env
  
  # verbose output to stderr
  verbose: false

Body

There are two subclasses of HandyPrompt: ChatPrompt and CompletionsPrompt. All of them share the same format of frontmatter.

ChatPrompt

For ChatPrompt, the body part is a markup format of chat messages. Each role key (e.g. $system$ / $user$ / $assistant / $tool$) should be placed in a separate line, and follows the content.

$system$
You are a helpful assistant.

$user$
Please help me merge the following two JSON documents into one.

$assistant$
Sure, please give me the two JSON documents.

$user$
{
    "item1": "It is really a good day."
}
{
    "item2": "Indeed."
}
%output_format%

ChatPrompt extra properties

You can specify extra properties of a message after the role:

$role$ {key1="value1" key2='value2'}

This gives you support for tool calls, image input and extra name fields, etc. See examples below.

CompletionsPrompt

As for CompletionsPrompt, the only difference is that there is no role keys in the body. Variable substitution works the same.

Credentials

You should place credentials in a separate file, and specify its path in meta.credential_path in the frontmatter.

meta.credential_type could be env / json / yaml, you can omit it and leave it inferred from the file extension.

Example .env:

OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxx
OPENAI_ORGANIZATION=org-xxxxxxxxxxxxxxxxxxxxx

Example credential.yaml:

api_key: sk-xxxxxxxxxxxxxxxxxxxxxxxxxxx
organization: org-xxxxxxxxxxxxxxxxxxxxx

Example credential.json:

{
    "api_key": "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "organization": "org-xxxxxxxxxxxxxxxxxxxxx"
}

For Azure API

When using Azure API with hprompt, you can specify api_type="azure" and api_version="2024-02-01" in front matter, RunConfig, or credentials. Meanwhile, provide api_key and api_base in credential file.

For example, originally you would instantiate an Azure client like:

client = AzureOpenAI(
    api_key = "some_azure_key",
    api_version = "2025-01-01",
    azure_endpoint = "https://some_resource.openai.azure.com/"
)

Now for the credential file (yaml for example):

api_key: some_azure_key
api_base: https://some_resource.openai.azure.com/
api_version: 2025-01-01
api_type: azure

Variable Substitution

You can substitute placeholder variables like %output_format%, which can be replaced by the dict var_map specified in the frontmatter meta field. Note that we need to wrap variable names with quotes because it contains special characters.

---
meta:
  var_map:
    '%output_format%': Please output a single YAML object that contains all items from the two input JSON objects.
    '%variable2%': Placeholder text.
    '%variable1%': Placeholder text.
---

You can also store them in text files (specify meta.var_map_path in frontmatter) to make multiple prompts modular. A substitute map substitute.txt looks like this:

%output_format%
Please output a single YAML object that contains all items from the two input JSON objects.

%variable1%
Placeholder text.

%variable2%
Placeholder text.

ChatPrompt Examples

Tool calls

You need to:

  • specify tools in the frontmatter;
  • then the responded type for $assistant$ will be tool_calls: 
    $assistant$ {type="tool_calls"}
  • and the tool calls are in YAML format.
  • Then append $tool$ messages with corresponding tool_call_ids for further chatting.

Full example:

---
model: gpt-4o
tools:
- function:
    description: Get the current weather in a given location
    name: get_current_weather
    parameters:
      properties:
        location:
          description: The city and state, e.g. San Francisco, CA
          type: string
        unit:
          enum:
          - celsius
          - fahrenheit
          type: string
      required:
      - location
      - unit
      type: object
  type: function
---

$user$
Please tell me the weathers in SF and NY.

$assistant$ {type="tool_calls"}
- function:
    arguments: '{"location": "San Francisco, CA", "unit": "celsius"}'
    name: get_current_weather
  id: call_fwXuTQZSjPr966yMrrzymHb3
  index: 0
  type: function
- function:
    arguments: '{"location": "New York, NY", "unit": "fahrenheit"}'
    name: get_current_weather
  id: call_bHFVWmtBk0z8wsEn3k6VHzOo
  index: 1
  type: function


$tool$ {tool_call_id="call_fwXuTQZSjPr966yMrrzymHb3"}
24

$tool$ {tool_call_id="call_bHFVWmtBk0z8wsEn3k6VHzOo"}
75

Image input

You need to specify content_array type for $user$ and then write in YAML array format for the content. Full example:

$user$ {type="content_array"}
- text: What's in this image?
  type: text
- image_url:
    url: https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg
  type: image_url

Local image files are also supported (both absolute and relative):

$user$ {type="content_array"}
- text: What's in this image?
  type: text
- image_url:
    url: file:///Users/username/Documents/my_image.jpg
  type: image_url

$user$ {type="content_array"}
- text: What's in this image?
  type: text
- image_url:
    url: file://my_image.jpg
  type: image_url

To generate such file:// url in code, you can use pathlib.Path: Path("my_file").resolve().as_uri().

For base64 image (replace %base64_image% with actual string during run):

$user$ {type="content_array"}
- text: What's in this image?
  type: text
- image_url:
    url: data:image/jpeg;base64,%base64_image%
  type: image_url

Audio input

Similarly, you can directly specify local audio file path as audio input:

$user$ {type="content_array"}
- type: text
  text: repeat after me
- type: input_audio
  input_audio: 
    data: file://audio.mp3
    format: mp3