Development - atomiechen/HandyLLM GitHub Wiki
Example scripts are placed in examples folder.
Each API method of OpenAIClient returns a Requestor, and you can execute its call() or acall() to get synchronous or asynchronous API calls.
Synchronous API usage:
from handyllm import OpenAIClient
with OpenAIClient(api_key='<your-key>') as client:
response = client.chat(
model="gpt-4-turbo",
messages=[{"role": "user", "content": "please tell me a joke"}]
).call() ## note .call() here
print(response['choices'][0]['message']['content'])Asynchronous API usage:
async with OpenAIClient('async', api_key='<your-key>') as client_async:
response = await client_async.chat(
model="gpt-4-turbo",
messages=[{"role": "user", "content": "please tell me a joke"}]
).acall() ## note .acall() here
print(response['choices'][0]['message']['content'])You can instantiate a client that supports both modes:
client = OpenAIClient('sync') ## only supports sync APIs
client = OpenAIClient('async') ## only supports async APIs
client = OpenAIClient('both') ## supports both versionsImportant
This is not recommended anymore. Use OpenAIClient instead.
Under the hood it connects to a module client and only provides synchronous APIs, without call().
from handyllm import OpenAIAPI
OpenAIAPI.api_key = '<your-key>'
response = OpenAIAPI.chat(
model="gpt-4-turbo",
messages=[{"role": "user", "content": "please tell me a joke"}]
) ## no .call() here
print(response['choices'][0]['message']['content'])Any endpoint that provides APIs compatible with OpenAI API specifications are supported, for example Moonshot AI API and other self-hosted endpoints.
Azure APIs, which unfortunately do not exactly follow OpenAI specifications, are seamlessly supported.
Each API request will connect to an endpoint along with some API configurations, which include:
| Description | Value | |
|---|---|---|
| api_type | API type. Defaults to openai, which means OpenAI API compatible. |
str: openai / azure
|
| api_base | API base url. Defaults to OpenAI base url. | str |
| api_key | API key. | str |
| organization | Organization. | str |
| api_version | API version. Must be provided for Azure end-points. | str |
| model_engine_map | Map model name to engine name. Useful for Azure end-points if you have custom model names. | dict |
An Endpoint object contains these information. An EndpointManager acts like a list and can be used to rotate the next endpoint. See test_endpoint.py.
Methods for configuring endpoint info (values will be inferred in top-town order):
| Configuration method | Description |
|---|---|
| API keyword parameters | e.g.: chat(api_key='xxx', ...)
|
API endpoint keyword parameter |
Providing an Endpoint, e.g.: chat(endpoint=MyEndpoint)
|
API endpoint_manager keyword parameter |
Providing an EndpointManager, e.g.: chat(endpoint_manager=MyEndpointManager)
|
OpenAIClient instance (or OpenAIAPI) variables |
e.g.: client.api_key = 'xxx' / OpenAIAPI.api_key = 'xxx'
|
| Environment variables |
OPENAI_API_KEY, OPENAI_ORGANIZATION/OPENAI_ORG_ID, OPENAI_API_BASE, OPENAI_API_TYPE, OPENAI_API_VERSION, MODEL_ENGINE_MAP. |
Tip
Azure OpenAI APIs are supported: Specify api_type='azure', and set api_base and api_key accordingly. Set model_engine_map if you want to use model parameter instead of engine/deployment_id. See test_azure.py. Please refer to Azure OpenAI Service Documentation for details.
You can pass custom logger and log_marks (a string or a collection of strings) to chat/completions to get input and output logging.
This toolkit supports client-side timeout control:
from handyllm import OpenAIClient
client = OpenAIClient()
prompt = [{
"role": "user",
"content": "please tell me a joke"
}]
response = client.chat(
model="gpt-3.5-turbo",
messages=prompt,
timeout=10
).call()
print(response['choices'][0]['message']['content'])Stream response of chat/completions/finetunes_list_events can be achieved using steam parameter:
from handyllm import OpenAIClient, stream_chat
client = OpenAIClient()
response = client.chat(
model="gpt-3.5-turbo",
messages=prompt,
timeout=10,
stream=True
).call()
# you can use this to stream the response text
for text in stream_chat(response):
print(text, end='')
# or you can use this to get the whole response
# for chunk in response:
# if 'content' in chunk['choices'][0]['delta']:
# print(chunk['choices'][0]['delta']['content'], end='')- chat
- completions
- edits
- embeddings
- models_list
- models_retrieve
- moderations
- images_generations
- images_edits
- images_variations
- audio_speech
- audio_transcriptions
- audtio_translations
- files_list
- files_upload
- files_delete
- files_retrieve
- files_retrieve_content
- finetunes_create
- finetunes_list
- finetunes_retrieve
- finetunes_cancel
- finetunes_list_events
- finetunes_delete_model
Please refer to OpenAI official API reference for details.
The response of non-stream mode is the binary content:
response = client.audio_speech(
model='tts-1',
input="Hello, world! oh yes. This is a test. Sync speech no-stream version.",
voice='alloy',
).call()
with open('output-sync.mp3', 'wb') as f:
f.write(response)Stream mode:
from handyllm import stream_to_file, astream_to_file
response = client.audio_speech(
model='tts-1',
input="Hello, world! oh yes. This is a test. Sync speech stream version.",
voice='alloy',
stream=True,
chunk_size=1024,
).call()
stream_to_file(response, 'output-sync-stream.mp3')
response = await client.audio_speech(
model='tts-1',
input="Hello, world! oh no. This is a test. Async speech stream version.",
voice='alloy',
stream=True,
chunk_size=1024,
).acall()
await astream_to_file(response, 'output-async-stream.mp3')Note: when using Azure, Azure TTS API needs both deployment_id and model arguments (see Azure OpenAI text-to-speech quickstart):
with OpenAIClient(
api_type='azure',
api_base=os.getenv("AZURE_OPENAI_ENDPOINT"),
api_key=os.getenv("AZURE_OPENAI_KEY"),
api_version=os.getenv("AZURE_OPENAI_API_VERSION")
) as client:
response = client.audio_speech(
deployment_id='tts', # replace with your deployment name
model='tts-1', # required: the model you choose
input="Hello, world! oh yes. This is a test. Sync speech no-stream version.",
voice='alloy',
).call()
with open('output-sync-azure.mp3', 'wb') as f:
f.write(response)Important
This is the legacy documentations. Please refer to hprompt files.
PromptConverter can convert this text file prompt.txt into a structured prompt for chat API calls:
$system$
You are a helpful assistant.
$user$
Please help me merge the following two JSON documents into one.
$assistant$
Sure, please give me the two JSON documents.
$user$
{
"item1": "It is really a good day."
}
{
"item2": "Indeed."
}
%output_format%
%misc1%
%misc2%
from handyllm import PromptConverter
converter = PromptConverter()
# `msgs` can be used as the `messages` parameter for OpenAI API
msgs = converter.rawfile2msgs('prompt.txt')
# variables wrapped in %s can be replaced at runtime
new_msgs = converter.msgs_replace_variables(
msgs,
{
r'%misc1%': 'Note1: do not use any bad word.',
r'%misc2%': 'Note2: be optimistic.',
}
)Important
About the prompt format, each role key (e.g. $system$ / $user$ / $assistant) should be placed in a separate line.
PromptConverter can also substitute placeholder variables like %output_format% stored in text files to make multiple prompts modular. A substitute map substitute.txt looks like this:
%output_format%
Please output a SINGLE JSON object that contains all items from the two input JSON objects.
%variable1%
Placeholder text.
%variable2%
Placeholder text.
from handyllm import PromptConverter
converter = PromptConverter()
converter.read_substitute_content('substitute.txt') # read substitute map
msgs = converter.rawfile2msgs('prompt.txt') # variables are substituted already