The AI training software (OneTrainer, kohya scripts, etc.) usually expects captions for each image in a .txt file in the same folder and with the same name as the image.

One of qapyq's key concepts is saving the captions to intermediate .json files. These are structured text files that allow saving multiple captions and other data per image, acting like a small database for each image.

When trying out training with different sets of captions, the contents of the .txt files can quickly be replaced with entries from the .json files. The old captions are not lost, as they remain in the .json files.

This way, captions can also be modular. For example, the originally generated tags can be stored as tags, and a manually refined version as refined. Another set of tags, which are used to guide a VLM, are stored as grounding. Quality descriptions like "blurry" or "noise" can be stored as quality and another entry can be made for different activation tokens. Each entry can be used for filtering and finally, the entries are joined together and exported to the .txt file.

Additionally, different captioning models might recognize different aspects on images. The .json files can store all versions, enabling us to inspect the differences. Afterwards, an LLM can summarize all captions into one, leveraging each model's strengths.

JSON files can be read and edited with any text editor. Just make sure to include all the necessary commas and brackets.

The Caption Window allows manual editing of captions. It works best with lists of comma-separated tags.

The current caption is shown in the text field in the bottom half of the window. Directly above it are the bubbles, which mirror the content from the text field and allow for fast deletion and reordering using the mouse (drag & drop).
Clicking on a bubble will select the respective tag and focus the text field for writing.
Clicking on a bubble while holding CTRL will move that tag after the selected tag (where the text cursor is).

You can move the cursor and navigate between tags inside the caption text by pressing ALT+Arrow keys.
Pressing only the ALT key will select the current tag. ALT+Delete removes the current tag.

The Caption Window can also be used to define rules for the automatic modification of tags.

At the bottom right of the window, select where to load the captions from, and where to save them to. You can select either:

• A .txt file with the same name as the image, in the same folder.
• A named entry in the .json file.

For .json entries you also have to specify a name. The stored caption can later be referenced in templates using this name (e.g. {{tags.name}}).

The save button turns red when a caption is edited, indicating that the displayed text hasn't been saved yet.
Edited captions are kept in memory even when another image is selected. However, when a new file or directory is loaded, changes are lost!

When Ctrl+S is pressed while the Caption Window is focused, or the save button is clicked, the current text is saved to the selected destination.

By default, the save destination is synchronized with the source: Changing the loading source will also change the destination.
Use the toggle button to the left of the save button to change the destination separately.

On the right side there's another toggle button for automatically selecting the next image after saving.

The Gallery Window shows little icons for the caption state:

• A white icon means that a .txt file exists.
• A red icon means that the caption was edited but hasn't been saved yet.
• With a green icon, the caption was edited and saved.

You can enable the CLIP token counter in the burger menu in the bottom left corner of the Caption Window. Whenever a change is made to the text, the counter at the bottom of the window is updated. The tokenizer runs on the CPU and consumes no VRAM.

Additionally, red lines appear inside the text at intervals of 75 tokens - or at the beginning of new lines. They mark the boundaries where the text would be split into separate chunks when used for training with an increased token count of 150, 225, etc.

Other text encoders, like T5, have such a high maximum length that I think it's unnecessary to count the tokens.

When clicking the Apply Rules button at the bottom of the window, it will apply the current rules to the current caption. The groups and conditionals described in the next chapters are also part of the rules.

The rules are applied in this order:

1. Full text search and replace
2. Mutually exclusive tags are removed (as specified in the groups).
3. Banned tags are removed.
4. Conditionals are evaluated and their actions executed from top to bottom.
5. Tags are sorted by groups (if selected).
6. Tags are combined (as specified in the groups).
7. Duplicate tags and subsets are removed (if selected).
8. Prefix is added in the front.
9. Suffix is added at the end.

Enter your search text in the left cell of the table, press Enter, write the replacement text in the right cell, and press Enter again. After adding an entry, a new row will appear.
To delete text from the caption, leave the replacement text empty. You might need to include space characters in the search pattern.

The search patterns support regex and they are matched against the full text of the caption. This is relevant only for advanced regex patterns, as they may match across separators/punctuation. Referencing parts of the pattern in the replacement text is currently not supported.

But note that the . (point) character has a special meaning in regex and will match any single character. If you want to use a . in the search pattern, you have to escape it with a backslash like: \.
For a quick reference and for testing regex patterns see regex101.com.

The order in which the processing happens is defined by the order in which the patterns are added, and currently there's no way to change that order through the GUI (it can be changed in the .json file after saving the rules). As the list is sorted alphabetically, the GUI might display it in the wrong order.

Banned tags are removed from the caption.

To add tags to the banned list, place the text cursor on the tag and click the Ban button.
You can also drag bubbles into the area for the banned tags. This list is sorted alphabetically.

A tag is considered a subset if all its words also appear in another, longer tag.

Example: bamboo, forest, bamboo forest becomes bamboo forest only.
bamboo and forest are removed because they are subsets of bamboo forest.

It also applies to combined tags. For example:
When black pants and denim pants were combined into black denim pants, and another black pants tag is added to the caption, the new tag is considered a subset of black denim pants and is consequently removed.

The order of words doesn't matter. In the example above, denim black would also be removed.

The prefix and suffix are only added to the caption if they don't already exist inside the caption.

If a prefix or suffix already exists, it is sorted to the front or to the back instead (if sorting is enabled). This also applies if the prefix or suffix consists of multiple tags.

With Auto Apply enabled (next to the "Apply Rules" button), the rules are automatically applied when:

• A different image is selected
• Tags are inserted
• Groups are edited
• Caption is generated

Clicking the Reload button will restore the original caption.

Access save and load options via the burger menu in the bottom left corner of the Caption Window.
The current set of rules can also be saved as the defaults. When chosen, these default rules are automatically loaded when qapyq starts or a new tab is opened.

The rules used for new tabs can also be set to From previous Tab. In that case, the default rules are still used when first opening the Caption Window, but new tabs will use the same rules as the previous tab.

Note that the tabs each have their own state. Making changes to the rules in one tab will NOT affect the other tab. Be careful when saving and overwriting rules. I recommend backing up the rules from time to time.

When saving, all the current rules and groups are written into a .json file.

These rule sets can be loaded here in the Caption Window. Or loaded in the Batch Window (Rules tab) and applied to all files.

In the same menu as the save and load options, there is a checkbox called Show refined preview. When enabled, another text field appears above the bubbles. This will show the current caption text with the current rules applied.

Apart for previewing the rules, this is also useful for checking captions without having "Auto Apply" enabled. There's less clutter after the banned tags are removed and similar tags are combined, and with sorting enabled it is much easier to see what is missing and what is wrong.

When you hover the mouse over a tag in the preview, the associated tags are highlighted in the text and bubbles. This makes finding the separate parts of a combined tag easier.

The groups serve multiple purposes:

• When rules are applied, they
  • define the sorting order of tags.
  • define which tags are mutually exclusive.
  • define which tags are combined into one.
• They define the color used for text highlighting.
• Click on a group-tag to quickly toggle tags in the caption.

As a help for building groups, you can auto-generate tags and then use the list of tags in the Stats Window. There, you can right-click on a tag and add it to a group.

Add tags to groups by either:

• Dragging a bubble into a group.
• Or by clicking the Add Tag button after putting the text cursor on the respective tag.

Edit the text of an existing group-tag by right-clicking on it.

To remove a tag from a group, drag it into one of the two trash bins below the groups.
Alternatively, when renaming a tag to an empty name, the tag is also removed from the group.

Reorder tags inside a group or move them between groups by dragging them around.

Add a new group by pressing the Add Group button below the groups.
Each group has a menu on the far right side with options to remove the group, or to create a new group above the current group.

You can reorder the groups themselves by using the drag handle on the far left of each group and dragging it to its new place.

The group's name is only used for display and organizing.
Change the group's color by clicking on the colored rectangle next to the name.

When "Mutually Exclusive" is enabled, only one of the group-tags can be present in the caption. The are 3 modes besides Disabled. This rule is applied before sorting, meaning:

• Keep Last will keep the last tag that occurs in the caption.
  • This will be the tag which was most recently added.
• Keep First will keep the first tag that occurs in the caption.
  • For generated tags, this will be the one with the highest confidence score.
• Priority ignores the tag order in the caption and instead uses the order inside the group to create a priority order.
  • The latest tag of the group will be kept.

When Keep Last is used together with Auto Apply enabled, this effectively becomes a switch where a previous tag is replaced by the newly selected tag. This is useful for correcting colors for example.
Group with tags: blue eyes, green eyes, brown eyes, Mutually Exclusive: Keep Last
When the caption is green eyes and you click on brown eyes, green eyes is removed and it will keep only the last one: brown eyes

Changing the value of "Mutually Exclusive" only works with mouse clicks. The mouse wheel was disabled to prevent accidental changes to the value when scrolling the panel.

When "Combine Tags" is enabled, it will try to merge the tags of the group into one tag. This rule is applied after sorting, so the order inside the group defines the order of words in the resulting tag (when sorting is enabled).

It will only combine tags that have the same last word(s). A preview next to the checkbox shows to which words this applies.

Example:
Group with tags: black pants, denim pants, short hair, blue hair, hair bun, Combine Tags: enabled
Original caption: blue hair, hair bun, denim pants, striped pants, black pants, short hair
Result: black denim pants, short blue hair, hair bun, striped pants

Note how striped pants was not included in the resulting tag because it is not part of the group.
hair bun was not included because the last word is different.

Wildcards are lists of words that can be added to group-tags, acting like a shortcut for adding multiple tags at once. A good example would be color: I have many groups for colored clothing. Adding all the colors to each type of clothing is a hassle and it uses a lot of space.

Instead, a wildcard can be defined:
color: dark, bright, light, yellow, orange, red, purple, pink, blue, aqua, green, silver, gold, beige, brown, grey, black, white, two-tone, multicolored

When used in group-tags like {{color}} shirt, the wildcard variable will expand to all defined values and the group will act as if containing all tags for blue shirt, red shirt, black shirt etc.

Clicking a group-tag with wildcards will open a menu with a list of expanded values. Clicking one of these entries will do the same as clicking any group-tag: Toggle the tag in the caption text. Multiple wildcards can be used in the same group-tag, but beware of combinatorial explosion.

Wildcards are saved as part of the rules preset. To edit the wildcards, press the Wildcards... button below the groups.

When you have lots of groups, you can hide some of them using the filter below the groups. This text field accepts regex and the pattern is matched against the group's name and all group-tags by default. To search for multiple words, enter the text like search1|search2 (no spaces).

When the groups are filtered, the colored text highlighting will be restricted to only the visible groups by default. This helps with finding certain tags inside the text, especially in Multi-Edit mode with many tags.

Click on the menu button to the left to show filter options. You can select to only filter by group names, or to always keep the full highlighting.

This filter also affects the captions in the Gallery. When enabled, filtering the groups in the Caption Window also defines which tags are shown in the Gallery.
See Captions in Gallery.

The conditionals allow to execute actions on the tags if some conditions are met.

They consist of one or multiple conditions on the left, and one or multiple actions on the right. The conditions are evaluated using the defined expression, which defaults to and, meaning all conditions must be met.
When the result of the expression is true, all actions are executed.

The conditions all capture the value they're matching, which then can be referenced in actions (optional). The letter in front of the condition shows the variable name. It is used as {{A}}, {{B}}, etc.

The condition letters and the expression will preview the state for the current caption. Green means it evaluates to true. Red means false.

Conditions and actions that take a list of words as parameters will accept multiple words such as blue sky. They will check if the words are really single words surrounded by whitespace, thus deep blue sky would match, but deepblue sky wouldn't. It's not a substring search.

The text field for the first parameter takes a comma-separated list of tags or words. The separator is always , and independent from the caption separator.

The parameters take a comma-separated list of tags or words. The separator is always , and independent from the caption separator.

The parameters marked with "*" will not strip spaces, so you may want to write it like this: tag1,tag2,tag3 (no spaces)

The default expression is and, which means that all conditions are AND-ed together: They all must be true.
When you set the expression to or, the conditions will be OR-ed together: At least one condition must be true.

The expression can also be written more explicitly, using the conditions' letters. Basic logic operators are supported. Examples:

Conditions that evaluate to false will not capture a value. When such a variable is used in parameters for an action, it will be empty, even when the whole expression evaluates to true.

Captured values from the conditions can be referenced in parameters for the actions using variables.

For example, the following rule will replace boots, black footwear with black boots:
Condition A - Any tags present: shoes, sneakers, high heels, boots, sandals
Condition B - Any words present: footwear
Action - Replace words: footwear → {{A}}

And here, it will combine the two tags sitting, on chair to sitting on chair:
Condition A - Any tags present: on chair, on couch, on floor
Condition B - All tags present: sitting
Action - Replace tags: {{A}} → sitting {{A}}

If multiple values are captured, they can be accessed like:

When multiple images are selected in the Gallery, the Caption Window will automatically switch to Multi-Edit mode.
How to select images in the Gallery: Image Selection

At the bottom of the Caption Window there will be an additional toggle button showing the number of selected files. Click this button to toggle between Multi-Edit and Single-Edit mode without losing the image selection. Right-click that button to clear the selection and return to Single-Edit mode.

In Multi-Edit mode, all captions of the selected images are combined into one text. The bubbles and text highlighting will show bright colors for tags which appear in all files. Tags that are only partially present in a few files are shown in muted colors.

Each tag in the text remembers which files it belongs to.
When you edit a tag, only the files which contain the original tag are changed.

When the text cursor is placed over a tag, or when the mouse hovers a bubble, the images that contain that tag are highlighted in the Gallery with beveled corners and colored background behind the filename.
This highlighting shows which files would be affected if you edit a tag.

Adding a new tag will append it to all selected files.
Removing a tag will remove it from all files that contain it.

Double-clicking a bubble will add the tag to all files where it is missing.
This is also useful for copying tags from one image to another.

Right-clicking a bubble will open a menu for selecting images only with or without the tag.

Changing the order of tags is not 100% accurate, but a tag will be placed into the general region.

Saving the caption using the Save button or by pressing Ctrl+S will save each individual caption to the selected .json key or .txt file.
Pressing the Reload button will reload each individual caption.

Apply Rules will apply the current rules to each individual caption, but the Refined Preview shows the effect of the rules on the combined Multi-Edit text. It is therefore not a preview of individual refined captions.

When the image selection is cleared and Multi-Edit mode exited without saving, the edited captions for each individual image are stored in memory.

Splitting a tag by inserting a comma will add a new tag to all files which contained the original tag.
The original tag is updated to the other half of the split.

The comma has to be inserted in the middle of a tag. If the comma is inserted at the beginning or the end (next to another comma), it will be detected as a completely new tag and appended to all files.

The tags can be edited normally afterwards and only the files which contained the original tag are affected. This way, new tags can be added to only the files that contain a certain other tag.

Example: Adding "long pants" to File 1 and 2 which contain "denim pants".
File 1: denim pants, black pants
File 2: denim pants, black pants, short hair
File 3: black pants, short hair

Multi-Edit text: denim pants, black pants, short hair
Split first tag: denim, pants, black pants, short hair (insert comma)
Further editing: denim pants, long pants, black pants, short hair

Results:
File 1: denim pants, long pants, black pants
File 2: denim pants, long pants, black pants, short hair
File 3: black pants, short hair (unchanged)

Example:
File 1: denim pants, black pants
File 2: denim pants, black pants, short hair
File 3: black pants, short hair

Multi-Edit text: denim pants, black pants, short hair
Edit first tag: long pants denim pants, black pants, short hair
Split first tag: long pants, denim pants, black pants, short hair (insert comma)

Results:
File 1: long pants, denim pants, black pants
File 2: long pants, denim pants, black pants, short hair
File 3: black pants, short hair (unchanged)

You can merge two neighboring tags by removing the comma between them. The resulting tag will appear in the same files as the left AND right tag. It is added to the files of both tags if necessary.

When merging multiple tags for ... pants for example, the resulting tag is present in all captions containing any pants tag. This is a quick method to ensure all pants are tagged the same way.

Before merging multiple tags, you may have to reorder the tags so they are next to each other.
You can click on a bubble while holding CTRL to move it after the currently selected tag (where the text cursor is). CTRL-clicking multiple bubbles will place them next to each other and prepare them for merging.

Example: Change all pants to "long black denim pants".
File 1: denim pants, black pants
File 2: long pants, blue pants
File 3: pants, short hair

Multi-Edit text: denim pants, long pants, pants, black pants, blue pants, short hair
Merge tags: denim pants long pants pants black pants blue pants, short hair (remove commas)
Editing: long black denim pants, short hair

Results:
File 1: long black denim pants
File 2: long black denim pants
File 3: long black denim pants, short hair

Easier: You can also just hold Backspace when the text cursor is inside the last tag to merge the tags after reordering. When characters are deleted, the tags are edited. And when a comma is deleted, two tags are merged.

Whenever a change is made to the caption text in Multi-Edit mode, the difference is calculated and operations are performed to transform the old state into the new one.

When making changes to multiple tags at the same time, by for example pasting text into the middle or overwriting a long text selection, the update can become hard to predict. The state will be consistent, but without knowing the rules in-depth, you might not foresee which files are affected.

Additionally, splits and merges have additional checks and will only be used when the intent is clear (single commas are inserted or removed). Otherwise it will fall back to simply replacing/changing tags.

As general rules:

• Avoid editing text selections that span multiple tags.
• Avoid pasting text containing the separator into a text selection that spans multiple tags.
• Just be considerate whenever you add or remove commas.

When you want to change all files anyway, you can safely paste and edit whatever you want. Some tags might only be partially present after pasting, because they inherited from an existing tag. Double-click on the bubbles to ensure full presence.

When many images with automatically generated tags are selected, the amount of tags can be overwhelming. To get a better overview you can sort the tags:

1. In the Caption Window's burger menu in the bottom left corner, go to the Apply Rules submenu.
2. Unselect all rules but keep Remove banned tags and Sort tags.
3. After selecting images and entering Multi-Edit Mode, click the Apply Rules button.
   • This will apply the selected rules to the individual captions of all selected images.

This presumes you have created groups for sorting.

Focus mode and Multi-Edit mode make little sense together. So when the Focus tab is open, Multi-Edit mode is disabled to prevent adding focus tags to all files by accident.

The Generate tab prefers Single-Edit mode to provide a proper preview of the variables, and will switch modes when opened. It will always load the individual captions when using {{current}} or {{refined}}, and save individual captions, even if Multi-Edit mode was manually enabled and the preview shows the combined text.

Undoing changes with Ctrl+Z is disabled in Multi-Edit mode, as this would also need to restore the tag-file associations. Hooking this into Qt's undo/redo system appears to be a lot of work.
Instead, I recommend saving often and using the reload button to jump back to previous states.

Instead of going image-by-image and tagging them completely, it can be much faster to go vertically, aspect-by-aspect. The Focus tab in the Caption Window provides mouse and keyboard controls for quickly editing tags in this way. As the name suggests, in this mode we're focusing on just a few tags, the keyboard shortcuts are easier to memorize and we can keep our eyes on the image.

The idea is to enter a few tags to focus on. For example, focus on the perspective using the tags straight-on, three quarter view, from side, from behind which are often missed by the tagging models. It will then display bubbles for these tags. When clicking a bubble, the tag will be appended to the current caption if it doesn't already exist.

Additionally, the bubbles show the keyboard shortcut for setting the tag. To enable them, press the Enable Keyboard Shortcuts button below. It will turn green. The keyboard shortcuts are disabled when another GUI element is selected, or when you press ESC.

Up to 9 tags can be controlled using the numbers 1-9 on the keyboard.
Press 0 or click the Unselect All bubble to remove all focus tags from the current caption.

When Mutually Exclusive is enabled, setting a tag will remove all other focus tags from the caption.

Pressing Enter will save the current caption and load the next image.
Use the left and right Arrow keys to navigate between images without saving the caption.
When multiple images are selected, it will only navigate between those images.

While the focus mode is active, text highlighting will use muted colors for all other tags. Switch to another tab in the Caption Window to restore the colors without losing your focus tags.

When Auto Save is enabled, the current caption is saved each time you add or remove a tag through the focus tab.

To automatically load the next (selected) image after saving, you can click and enable the toggle button in the bottom right corner of the Caption Window. This is a fast way for editing lots of captions: Each key press will set a tag, save the caption and skip to the next image.

The title bar of the Main Window shows the current and total image number. Auto-skipping will not loop the images, so when the images stop changing, you know you're done.

AI-assisted batch captioning consists of multiple steps:

1. Generate captions and/or tags.
   • This information is stored in .json files alongside the images, but separate from the final .txt file.
2. (optional) Further transform the entries in the .json file using rules or LLMs.
3. Save the final caption in a .txt file according to a template.

Through prompts you can ask questions and give instructions to the vision models. You can also ask multiple questions that are sent sequentially and guide the AI towards more in-depth answers. Separate these prompts with a line starting with ---:

Describe the image in detail.
---
What can be said about the soil condition and geology. How was this landscape formed?
--- final ---
Condense all this information into a detailed summary using prose without bullet points.

In above example, the name final is given to the last answer. The trailing dashes are optional. During batch processing, the answer is saved under this name and can be referenced later using the {{captions.final}} variable. If no name is specified, the default name (default storage key) is used. When there are duplicate names, an increasing counter is appended.

With correct syntax, the separator line is displayed as bold text.

If the name starts with a ?, the answer is still part of the conversation, but it's excluded from the output:

--- ?
What is the meaning of this image?
--- ? not-saved
What does the subject represent?
---
Summarize your answers.

In this example, only the summary is displayed and saved to the .json file.

When prompts are separated by ---, they will be sent in one conversation and the model will remember previous messages. To start a new conversation, use === instead:

Describe the image in detail.
=== funny
Describe the image using funny language.

As shown in this example, these prompts also can have names.

In the Caption and Batch Window, you can define prompt and save formats using templates. Contents from the .json or .txt file can be referenced with these variables:

Variables are highlighted in different colors, and the preview shows the replacement text in the same color.

You can transform the values of variables by calling functions on them. The functions have a name and they may take arguments. To apply a function, write # after the variable name, followed by the function name. Function arguments are separated by :.

Since # and : have a special meaning, avoid using these characters for separators or for storing captions.

Examples:
{{captions.caption#lower}}
This will convert the caption to lowercase.

{{tags.tags#shufflekeep:3}}
This will randomize the order of tags while keeping the first 3 tags at the beginning. It uses , as the separator by default.

You can chain multiple functions: {{captions.caption#lower#replace:,:#shufflekeep:1:. }}
This will convert the caption to lowercase, remove all commas and shuffle the sentences while keeping the first sentence at the beginning.

{{tags.tags#shuffle#first}}
This will return a random tag.