Plugin Catalog Index Generation - redhat-developer/rhdh-plugin-export-overlays GitHub Wiki
The plugin catalog indexes are the collection of Supported Plugins and curated Optional Extras packages from this repository.
Indexes can be fetched and used from the links at https://github.com/redhat-developer/rhdh-plugin-export-overlays/wiki/Plugin-Catalog-Status-main
They contain all the metadata, OCI image references, and default configuration needed for RHDH to discover and load dynamic plugins. This page explains how the catalog indexes are built, what they contain, and where they are published.
The catalog index generation pipeline reads workspace metadata, queries container registries, and produces a self-contained directory of catalog entities and an index.json file. This directory is then packaged as an OCI image and pushed to a container registry, where RHDH consumes it.
Plugin entities under catalog-entities/extensions/plugins/ can describe plugins that are exported via workspaces/ or plugins whose OCI images are built elsewhere (catalog metadata only). Package bootstrap and registry verification in this pipeline are driven by workspace Package metadata / package lists — catalog-only Plugin YAMLs contribute listing metadata to the index, not an overlay export and not installable Package/OCI resolution. Installable catalog entries still require Model A workspace Package metadata (workspaces/*/metadata/). A future path for catalog-installable external OCI without workspaces/ is out of scope for this document. See 03 - Plugin Owner Responsibilities.
flowchart LR
subgraph Inputs
META["workspaces/{name}/metadata/{plugin}.yaml<br/>(Package entities)"]
CATS["catalog-entities/extensions/<br/>(Plugin entities, Collections)"]
PKGS["Package list files<br/>(default.packages.yaml,<br/>rhdh-supported-packages.txt,<br/>rhdh-community-packages.txt)"]
VERS["versions.json"]
end
subgraph Pipeline
GEN["generate-catalog-index<br/>workflow"]
end
subgraph Outputs
IDX["catalog-index/<br/>(index.json, entities, DPDY)"]
OCI["OCI images<br/>(quay.io, ghcr.io)"]
BRANCH["catalog-index-{branch}<br/>git branch"]
WIKI["Wiki status page"]
end
META --> GEN
CATS --> GEN
PKGS --> GEN
VERS --> GEN
GEN --> IDX
IDX --> OCI
IDX --> BRANCH
GEN --> WIKI
The pipeline runs via the [generate-catalog-index.yaml](../.github/workflows/generate-catalog-index.yaml) workflow, triggered on pushes to main and release-* branches when relevant files change, or manually via workflow_dispatch.
The core orchestrator script is [scripts/update-index.sh](../scripts/update-index.sh), which runs six steps in sequence — four that build the index, and two that check it:
flowchart TB
subgraph "Step 1: Plugin Builds Bootstrap"
S1_IN["workspaces/{name}/metadata/{plugin}.yaml<br/>plugins-list.yaml<br/>versions.json<br/>Package list files"]
S1["bootstrapPluginBuilds.py"]
S1_OUT["plugin_builds/{ws}/{plugin}.json<br/>(workspace path, OCI ref, support level)"]
S1_IN --> S1 --> S1_OUT
end
subgraph "Step 2: Image Metadata Fetch"
S2["generatePluginBuildInfo.py"]
S2_OUT["plugin_builds/{ws}/{plugin}.json<br/>(+ digest, build-date, vcs-ref)"]
S1_OUT --> S2
REG["Container registries<br/>(quay.io, ghcr.io)"] --> S2
S2 --> S2_OUT
end
subgraph "Step 3: DPDY Generation"
S3_IN["default.packages.yaml<br/>metadata appConfigExamples"]
S3["generateDynamicPluginsDefaultYaml.sh<br/>+ injectDpdyTagComments.py"]
S3_OUT["dynamic-plugins.default.yaml"]
S3_IN --> S3
S2_OUT -- "tag & build-date<br/>for comments" --> S3
S3 --> S3_OUT
end
subgraph "Step 4: Catalog Index Generation"
S4["generateCatalogIndex.py"]
S4_OUT["index.json<br/>catalog-entities/<br/>build-report.json"]
S2_OUT --> S4
S3_OUT -- "OCI ref updates &<br/>tag comments" --> S4
CATS2["catalog-entities/extensions/"] --> S4
S4 --> S4_OUT
end
subgraph "Step 5: Validation (no network)"
S5["validateCatalogIndex.py"]
S5_OUT["findings + validate stage<br/>in build-report.json"]
S4_OUT --> S5
S3_OUT --> S5
S2_OUT --> S5
S5 --> S5_OUT
end
subgraph "Step 6: Sanity check (opt-in, --sanity-check)"
S6["smoke-tests-native<br/>yarn smoke --catalog-index"]
S6_OUT["results-catalog-index.json<br/>(install + boot per package)"]
S3_OUT --> S6
REG2["Container registries<br/>(pulls every artifact)"] --> S6
S6 --> S6_OUT
end
Reads each workspaces/*/metadata/*.yaml file and constructs initial plugin_builds/<workspace>/<image-name>.json entries. Each entry includes the workspace path, support level, and a constructed OCI tag reference based on the registry type:
-
ghcr.io:
bs_{backstage_version}__{plugin_version}(e.g.,bs_1.49.4__0.8.2) -
quay.io/rhdh:
{rhdh_version}--{plugin_version}(e.g.,1.10--0.8.2)
Plugins are filtered to only those matching the provided --packages-file list(s).
Orphan cleanup after renames/removals (e.g. lightspeed → intelligent-assistant): JSON files under plugin_builds/ that are no longer produced from metadata are deleted, and matching orphan keys are removed from --report-file (build-report.json) so later steps and renderCatalogStatus.py do not keep advertising obsolete OCI refs.
Queries the container registry for each plugin's OCI image to retrieve:
-
Digest (
sha256:...) for immutable references - Build date and VCS ref from container labels
- Upstream/midstream repo refs from container env vars
Then updates plugin_builds with the relevant metadata.
Images that don't exist in the registry are logged as warnings.
For each plugin, the image metadata fetch follows a three-tier resolution:
-
Exact tag match: The constructed tag (e.g.,
1.10.0--1.5.4) is queried directly. If the image exists, its metadata is used as-is. -
RHDH version alias (quay.io/rhdh only): If the exact tag is not found and the RHDH version prefix has three parts (x.y.z), the patch version is stripped to try the minor-version alias (e.g.,
1.10.2--→1.10--). This is because downstream builds are not repeated for each RHDH patch release if the plugin hasn't changed — a build done during1.10.0produces both1.10.0--1.5.4and1.10--1.5.4tags, and the1.10--alias remains valid for1.10.1,1.10.2, etc. If the exact plugin version is found under the alias, the resolved reference is used without marking it as a fallback. If the alias has tags but not the exact plugin version, the plugin is reported as not found — a new downstream build is needed. -
Fallback to latest version: If the exact plugin version is not found under the original prefix, the latest published plugin version within that prefix is used. This is flagged as a fallback in the output, and the metadata YAML's
version:field is updated to match. Fallback only applies within the original prefix — the alias (minor-version) prefix is only used for exact matches.
Alias resolution does not apply to ghcr.io (community) builds, which use Backstage version prefixes (bs_x.y.z__).
Generates dynamic-plugins.default.yaml — the default plugin configuration shipped with RHDH. This step only runs for the supported tier (requires a YAML-format packages file with enabled/disabled structure).
After generating the DPDY, the script calls injectDpdyTagComments.py to insert # Tag: <tag>, Build date: <date> comments from plugin_builds/*.json (produced by Steps 1-2). Each plugin's registryReference tag and build-date label are extracted from the enriched JSON files and placed as comments above the corresponding - package: lines. This provides traceability from each plugin entry back to the specific OCI image tag and build date, without requiring live registry API calls (which previously caused Quay API timeouts).
Inputs:
-
default.packages.yaml— lists which plugins are enabled vs disabled by default -
workspaces/*/metadata/*.yaml—spec.appConfigExamples[0].contentprovides thepluginConfigfor each plugin -
plugin_builds/*.json— provides tag and build-date metadata for comment injection
Output structure (truncated):
plugins:
# Tag: 1.10--0.8.2, Build date: 2026-05-20T13:45:25Z
- package: oci://quay.io/rhdh/red-hat-developer-hub-backstage-plugin-adoption-insights:1.10--0.8.2
enabled: true
pluginConfig:
dynamicPlugins:
frontend:
red-hat-developer-hub.backstage-plugin-adoption-insights:
# ... frontend wiring config
# Tag: 1.10--1.2.0, Build date: 2026-05-19T09:12:00Z
- package: oci://quay.io/rhdh/backstage-community-plugin-acr:1.10--1.2.0
enabled: falseThe final step that produces the catalog index:
- Copies
catalog-entities/extensions/(Plugin entities, collections) to the output directory - Copies
workspaces/*/metadata/*.yaml(Package entities) to thepackagesdirectory of the catalog index output directory - Scrubs entities to only include packages matching the package list filter
- Verifies each plugin's OCI image exists in the registry
- Generates
index.jsonwith digest-based references - Updates Package entity files and DPDY with OCI references and Tag/Build date comments
- Regenerates
all.yamllocation files
A static check of what the previous steps just produced. No network calls — it reads
dynamic-plugins.default.yaml, index.json and plugin_builds/ and reports where they
disagree, so it is cheap enough to run on every generation, upstream and in the midstream
Konflux pipeline alike.
It exists because the generator is deliberately forgiving: when a plugin's image is not
found in the registry it logs a warning and carries on, so the index can go out declaring
an oci:// ref that was never confirmed to exist. Nothing downstream re-checked that.
Run python3 scripts/validateCatalogIndex.py --list-rules for the full list. The ones
that matter most:
| Rule | Severity | What it means |
|---|---|---|
unresolved-image |
error | The index ships a package whose image was never resolved. Enabling it fails at pull time. |
unknown-image |
error | An oci:// ref names an image with no plugin_builds/ entry — the index and build metadata disagree. |
digest-mismatch |
error | A digest-pinned ref does not match the digest plugin_builds/ recorded. |
registry-not-allowed |
error | A ref points at a registry this index is not built against (the ghcr.io-into-quay.io/rhdh leak). |
duplicate-ref |
error | The same ref appears twice; the later entry silently shadows the earlier one's pluginConfig. |
fallback-tag |
warning | The requested build was missing and an older tag was substituted — the index ships a stale build. |
not-digest-pinned |
warning | A ref carries a tag rather than a digest, so what it resolves to can change under the index. |
index-missing-entry |
warning | A resolved package is in the DPDY but absent from index.json, so the Extensions UI will not list it. |
Modes. --validate-mode controls what a finding does to the build:
-
report(default) — always runs, prints the findings, never fails the build. -
gate— fails the build on any error. -
off— skips the step.
The default is report on purpose. The check is new, and the indexes it runs against
today already carry findings nobody has triaged; landing it as a hard gate would turn
those into a red build for whoever happens to push next. Flip it to gate once the
standing findings are fixed or allowlisted — that is the point of shipping it.
Allowlist. Known and accepted findings go in
scripts/catalog-index-validation-allowlist.txt,
using the same TODO(TICKET) discipline as the smoke harness's exclusion files: a
pattern with no ticket is a parse error, so exceptions get removed rather than
accumulated. Patterns are matched against the OCI image name
(backstage-community-plugin-quay).
Findings are also recorded per plugin in build-report.json as a validate stage, so
they reach the generated status page. Only errors set that stage to fail — a stale tag
should not turn a plugin red and drown the ones that really are broken.
It only checks a tier that has a DPDY. Every rule is driven by
dynamic-plugins.default.yaml, and Step 3 generates that file only when a
default.packages.yaml is among the --packages-file arguments. The community tier is
generated with --packages-file rhdh-community-packages.txt alone, so for that tier
Step 5 logs nothing to validate and exits 0 — including under --validate-mode gate.
It says so in the log rather than passing silently, but do not read a green community
run as a validated one.
Installs and boots every package the generated index declares, using the Docker-free
smoke-tests-native harness — the install CLI plus
startTestBackend, no container and no cluster. This is the upstream half of RHDH's
cluster-free plugin sanity check (RHIDP-13508), running against the index as it is about
to be published rather than against an index image that already exists.
It is off by default (--sanity-check opts in) because it pulls every artifact the
index declares and needs Node 24 and Yarn 4, which the midstream update-index GitLab
job has neither the budget (1 CPU / 2Gi, 30-minute script timeout, today doing metadata
lookups only) nor the toolchain for.
Two things about how it reads the index are worth knowing:
-
The index's
enabled:flags are ignored. Most packages shipenabled: falsebecause that is RHDH's out-of-the-box default, which says nothing about whether the artifact works. Honouring them would validate almost nothing, so the check generates an enable-everything config — the same reasoning behind RHDH'spopulate-catalog-index.sh. -
pluginConfigblocks are dropped. They reference${ENV_VAR}placeholders that exist in a deployed RHDH and nowhere here. The harness supplies its own dummy root config instead, with--app-configlayered on top.
Packages that genuinely cannot be validated go in
smoke-tests-native/catalog-index-sanity-excludes.txt,
again with a tracking ticket per entry.
# Against the index this repo is about to generate
scripts/update-index.sh --registry quay.io/rhdh --validate-mode gate --sanity-check
# Against an index that is already published
scripts/extractCatalogIndex.sh quay.io/rhdh/plugin-catalog-index:next /tmp/dpdy.yaml
cd smoke-tests-native && yarn smoke --catalog-index /tmp/dpdy.yamlOn a schedule, this runs as the
Catalog Index Sanity workflow
(04:00 UTC daily, plus workflow_dispatch with a catalog_index_image input for RC
verification). It is not a PR check, for the same reason RHDH's is not: the index changes
on its own, so a PR here cannot change what the job validates, and running it per-PR would
fail unrelated work whenever the index drifts.
When validating a published index there is no plugin_builds/ alongside it, so pass
--no-build-metadata to validateCatalogIndex.py. It skips the rules that need the build
metadata — each one declares that in its own row of the rule table, so the list cannot
drift — and names them in its output, so a pass cannot be mistaken for a full check.
The generated catalog-index/<tier>/ directory contains:
| File | Purpose |
|---|---|
index.json |
Main index mapping plugin image names to OCI refs with digests |
catalog-entities/extensions/packages/*.yaml |
Package entity definitions with resolved OCI references |
catalog-entities/extensions/plugins/*.yaml |
Plugin entity definitions (descriptions, icons, categories) |
catalog-entities/extensions/collections/*.yaml |
Collection groupings (featured, recommended, etc.) |
dynamic-plugins.default.yaml |
Default plugin configuration (supported tier only) |
build-info.json |
Build metadata (date, versions, source commit) |
build-report.json |
Detailed per-plugin build status with stage tracking (not included in final OCI artifact) |
The pipeline runs twice per build — once for each tier.
Inclusion in either catalog is optional for plugin owners and requires RHDH PM approval. For the owner-facing files and checklist (Plugin entity YAML, collections if applicable, all.yaml, package list files, and GA-only default.packages.yaml with enabled: / disabled:), see 03 - Plugin Owner Responsibilities.
Plugins with Red Hat support (GA or Tech Preview heading to GA).
-
Filter: Union of
default.packages.yaml(GA only) +rhdh-supported-packages.txt(GA + TP) -
Registry:
quay.io/rhdh(plugins registry) /quay.io/rhdh-community(catalog index image registry) -
Published to:
quay.io/rhdh-community/plugin-catalog-index -
Tags:
bs_{backstage_version}-{short_sha},bs_{backstage_version},latest - Includes DPDY: Yes
Curated Optional Extras (community, developer preview, and non-core) plugins. The package-list file selects this catalog tier; support level remains in Plugin/Package metadata YAML.
-
Filter:
rhdh-community-packages.txt -
Registry:
ghcr.io/redhat-developer/rhdh-plugin-export-overlays -
Published to:
ghcr.io/redhat-developer/rhdh-plugin-export-overlays/plugin-catalog-index -
Tags:
bs_{backstage_version}-{short_sha},bs_{backstage_version},latest - Includes DPDY: No
After each build, a status page is automatically pushed to the GitHub Wiki:
- Index page: Plugin Catalog Index Status — links to all branch status pages
-
Per-branch pages:
Plugin-Catalog-Status-{branch}— shows which plugins passed/failed and why
The status page includes:
- Build metadata (date, commit, versions)
- Summary counts per tier (total, passed, failed)
- Per-plugin tables with links to metadata files, OCI references, and failure reasons
- Links to the workflow run for debugging failures
The raw build-report.json files are also available on the catalog-index-{branch} git branch in catalog-index/supported/build-report.json and catalog-index/community/build-report.json.
Pushes to main or release-* branches that modify any of these paths trigger the workflow:
workspaces/*/metadata/*.yamlcatalog-entities/extensions/**-
default.packages.yaml,rhdh-supported-packages.txt,rhdh-community-packages.txt scripts/**versions.json
# Build from main, push to catalog-index-main
gh workflow run generate-catalog-index.yaml
# Build from a specific branch
gh workflow run generate-catalog-index.yaml -f source-branch=release-1.9
# Build from one branch, push catalog to a custom target branch
gh workflow run generate-catalog-index.yaml \
-f source-branch=main \
-f target-branch=catalog-index-customTo pull just dynamic-plugins.default.yaml out of a published index — which is what the
sanity check needs, and what RHDH's own check reimplements — use
scripts/extractCatalogIndex.sh:
scripts/extractCatalogIndex.sh quay.io/rhdh/plugin-catalog-index:next /tmp/dpdy.yamlThe index image is FROM scratch, so there is nothing in it to exec and the file has to
come out of a layer. The script walks the manifest top-down and takes the first layer
carrying the file: layers are listed base-first, and an index rebuilt as an overlay keeps
a stale copy in a lower layer — reading that one would validate the previous index while
reporting on the current one.
To browse the whole tree instead, run this script:
unpack () {
if [[ ! $1 ]]; then
echo "Usage: unpack reg/org/container:tagorsha"
else
local IMAGE="$1"
DIR="${IMAGE//:/_}"
DIR="/tmp/${DIR//\//-}"
rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}")
podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar
echo "Unpacked $IMAGE into $DIR"
cd $DIR; tree -d -L 3 -I "usr|root|buildinfo"
fi
}
unpack ghcr.io/redhat-developer/rhdh-plugin-export-overlays/plugin-catalog-index:1.11-bs_1.49.4
unpack quay.io/rhdh-community/plugin-catalog-index:<some tag>
Once unpacked, you should see a tree of metadata files to browse, as well as index.json and build-info.json.
.
└── catalog-entities
└── extensions
├── collections
├── packages
└── plugins