Demo Data Developer Guide - edmozley/freeitsm GitHub Wiki

Demo Data โ€” Developer Guide

How the demo-data importer works, and how to add demo data for a new module. The user-facing page is Demo Data.

The design goal: a module's sample data is a single JSON file of plain rows, and one generic importer turns it into a correct, cross-referenced, repeatable database seed. Adding demo data should mean writing JSON โ€” not code.


1. ๐Ÿ“ The files involved

Colour key: ๐Ÿ“‹ data ยท โš™๏ธ importer ยท ๐Ÿ–ฅ๏ธ UI ยท ๐Ÿ”Œ diagnostic ยท ๐Ÿ“„ docs

File What it does
๐Ÿ“‹ database/demo-data/<module>.json the sample rows for one module (e.g. core.json, tickets.json)
โš™๏ธ api/system/import_demo_data.php the engine. Loads the JSON, resolves references + tokens, inserts in a transaction
๐Ÿ–ฅ๏ธ system/demo-data/index.php the System โ†’ Demo Data page โ€” one card/button per module
๐Ÿ”Œ api/system/debug-tools/D001_demo_core_import.php the Demo Core Data Import diagnostic (traces a failing core import)
๐Ÿ“„ this wiki you are here

There is no per-module import code โ€” the importer is generic. Everything specific to a module lives in its JSON.


2. The shape of a demo file

A demo file is an object of tiers, each tier an object of tables, each table a list of rows:

{
  "tier1": {
    "departments": [
      { "_ref": "it", "name": "IT Support", "is_active": 1, "display_order": 1 }
    ]
  },
  "tier2": {
    "teams": [
      { "_ref": "servicedesk", "name": "Service Desk", "is_active": 1 }
    ]
  },
  "tier3": {
    "analyst_teams": [
      { "analyst_id": "@analysts.jsmith", "team_id": "@teams.servicedesk" }
    ]
  }
}
  • The table key is the real database table name. Row fields are its columns. The importer builds the INSERT from whatever keys are present โ€” so a row is just "the columns I want to set".
  • Tiers are ordering buckets, processed tier1 โ†’ tier5. Their only job is dependency order: a row that references another must live in a later tier (or a later position in the same tier) than the row it points at. Name them for meaning if you like, but the importer only cares about order.

3. Meta fields (underscore-prefixed)

These are stripped before insert โ€” they control the importer, they are not columns.

Field Purpose
_ref Name this row so others can reference it. Becomes @<table>.<ref> (see ยง4).
_skip_insert + _match_by + _match_value Don't insert โ€” match an existing row and map its id. Used for the admin account: { "_ref": "admin", "_skip_insert": true, "_match_by": "username", "_match_value": "admin" }. The row is looked up, its id recorded under @<table>.<ref>, and it is preserved when the table is cleared.

4. Cross-references โ€” @table.ref

Any string value of the form @<table>.<ref> is replaced with the real primary-key id of the row that declared that _ref, at insert time:

{ "analyst_id": "@analysts.jsmith", "team_id": "@teams.servicedesk" }

Because ids are assigned as rows insert, the referenced row must already have been inserted โ€” hence tiers. An unresolved reference throws and rolls back the whole import, so a typo fails loudly rather than writing a bad row.

๐Ÿ”‘ Referencing rows in the same table you're currently inserting works too, as long as the target appears earlier in the list.


5. Tokens โ€” computed values

Any string value matching one of these is expanded per row:

Token Expands to
__BCRYPT:secret__ a bcrypt hash of secret (demo passwords โ€” the analysts use __BCRYPT:demo1234__)
__RELATIVE_DATE:-60d__ a UTC datetime N days from now; optional hours: __RELATIVE_DATE:-60d+3h__
__RELATIVE_DATEONLY:-7d__ a UTC date (no time) N days from now
__NOW__ the current UTC datetime
__GENERATE__ a fresh demo ticket number (tickets only)
__UNIQUE__ replaced with a uniqid() (embed it in a larger string)

Relative dates are why a freshly imported ticket looks recent: stamp demo dates relatively, never with a literal date.


6. String โ†’ id translation for normalised tables

Some tables store a status/priority/etc. as an id into a lookup table, but the demo JSON is written with the human-readable name for legibility. The importer swaps them using a small table in import_demo_data.php:

$lookupTranslations = [
    'tickets' => [
        ['status',   'status_id',   'ticket_statuses'],
        ['priority', 'priority_id', 'ticket_priorities'],
    ],
    // changes, tasks, status_incidents, ...
];

So "status": "Open" in tickets.json becomes status_id = <id of 'Open'>. An unknown name throws. If your new module normalises a field into a lookup table, add a row here โ€” otherwise write the real *_id (or an @ref) in the JSON.

A couple of bespoke cases live near it: morningChecks_Results resolves Status via Label (not name), and tickets drops legacy requester_email / requester_name. Follow that pattern only if a table genuinely needs it.


7. What the importer guarantees

  • One transaction. Any error โ†’ full rollback, nothing changes. The response is { success, imported: {table: count}, total }.
  • Repeatable. Before inserting, every table that has insertable rows is cleared (DELETE, FK checks off, reverse tier order). _skip_insert rows are preserved via a WHERE NOT (...) guard โ€” that's how the admin account survives.
  • Primary keys. Ids are mapped via getPrimaryKeyColumn(), which defaults to id. Override it there for a table whose PK isn't id (e.g. morningChecks_Checks โ†’ CheckID), or map to null for a pure join table with no id worth referencing.

8. โœ… Adding demo data for a new module

  1. Write database/demo-data/<module>.json โ€” tiers, tables, rows, using _ref / @refs / tokens above. Order tables so referenced rows insert first.
  2. Allowlist it โ€” add <module> to $allowedModules in api/system/import_demo_data.php (the import refuses any module not on this list).
  3. Add a card for it in system/demo-data/index.php so it appears on the Demo Data page, with a one-line description of what it seeds.
  4. Lookup translations (only if needed) โ€” if the module stores a name-as-id, add its columns to $lookupTranslations (ยง6).
  5. Primary key (only if needed) โ€” if a table's PK isn't id, add it to getPrimaryKeyColumn() (ยง7).
  6. Test on a fresh install (a throwaway Docker instance is ideal): import Core Data first, then your module. Confirm the counts, and that re-importing gives a clean copy, not duplicates.

๐Ÿ”‘ Core Data first, always. Nearly every module references analysts, teams, departments or users by @ref, and those refs are only in scope once core.json has been imported in the same database. Document any cross-module dependency in the module's Demo Data card.

If a core import misbehaves, the D001 โ€” Demo Core Data Import debug tool (System โ†’ Debug tools) runs the whole thing in-process and reports table-by-table what happened, including a write-probe inside a rolled-back transaction.


9. Worked example โ€” the RBAC seed (#48)

core.json seeds a realistic role-based-access example purely as data โ€” no importer changes were needed, which is the whole point of the generic design. It:

  • sets demo analysts non-admin with can_access_all_modules = 0;
  • gives the Escalation Team can_access_all_modules = 1 and an Escalation Team rbac_role holding every capability (rbac_role_capabilities, one row per key from capAll());
  • scopes the Service Desk team to a 12-module subset via team_modules, with a blank Service Desk role;
  • links roles to teams via rbac_team_roles.

All of it is @ref wiring across rbac_roles, rbac_role_capabilities, rbac_team_roles, team_modules, teams and analysts โ€” a good template for any permissions-related demo. See the Demo Data user page for what it looks like in the UI.

โš ๏ธ **GitHub.com Fallback** โš ๏ธ