JENNER BVT Testing Notes - laser-base/laser-core GitHub Wiki

Problem Statement

We love our LASER AI, JENNER, but as software, we need a way of testing it. And that means not manual testing but automated. In short, assuming we can define a set of prompts that we expect to "work", we want to have a way of automatically sending those prompts to the AI, getting code back, and making sure that code "is good".

Problem Statement Unpacked

For our set of test prompts, we want to:

  • Send them to an AI webservice (enterprise hosted MCP server), and get python code back.
  • Run the code in an environment with laser(*) fully installed (docker image).
  • Capture the output from running the code.
  • Compare the output of the code to an expected result.
  • Report pass/fail.
  • Note that we do not here propose testing the code by inspection, just by running and inspecting the output.

We want these prompts/tests to go from something very simple to become progressively more demanding.

Prompts

Here’s a curated set of 15 progressively complex prompt ideas you could use to test and reuse laser-core features. Each prompt is designed to return runnable Python code that imports and invokes at least one function or class from laser-core. This suite gradually introduces core concepts, starting with minimal LaserFrame usage and building toward demographic modeling, components, squashing, and migration.


🧪 LASER Test Prompt Suite (1–15)


Level 1: Basics of LaserFrame

  1. Prompt 1: "Create a minimal example that initializes a LaserFrame with a capacity of 10 and adds a scalar property named age, then populates it with 5 agents and prints the age array."

Sample Result 1

python3 jenner_bvt1.py
Age array:
[25 30 18 45 60]
  1. Prompt 2: "Add a vector property called position with length 2 to a LaserFrame of capacity 20. Populate 3 agents and set each agent's position manually, then print the property."

Sample Result 2

python3 jenner_bvt2.py
Position array (2D):
[[1. 3. 5.]
 [2. 4. 6.]]

Position array (agent-wise rows):
[[1. 2.]
 [3. 4.]
 [5. 6.]]
  1. Prompt 3: "Write a function that adds 100 agents to a LaserFrame with a scalar state property, sets the state of all agents to 1, and verifies that all entries are equal to 1."

Sample Result 3

python3 jenner_bvt3.py
After step 1: age = [1 1 1 1 1]
After step 2: age = [2 2 2 2 2]
After step 3: age = [3 3 3 3 3]
After step 4: age = [4 4 4 4 4]
After step 5: age = [5 5 5 5 5]

Level 2: Demographic Utilities

  1. Prompt 4: Sample Initial Ages from Population Pyramid

Use AliasedDistribution and load_pyramid_csv() to assign realistic ages (in days) to a new population of 1,000 agents using LASER’s bundled US 2023 pyramid.

Key Features:

  • Tests load_pyramid_csv()
  • Exercises AliasedDistribution.sample()
  • Converts age bins to age-in-days
  • Stores in scalar age property

Sample Result 4

python3 jenner_bvt4_realisticages.py
First 10 sampled agent ages (in days):
[ 3457  4743  1204 18044 27217   889 26394 26076  3436  1024]

Converted to years:
[ 9 12  3 49 74  2 72 71  9  2]
  1. Prompt 5: Set date_of_birth Based on Sampled Age

Convert sampled agent ages into date_of_birth (i.e. -age_in_days) and add date_of_birth as a scalar property in a LaserFrame. Show the first 10 values.

Key Features:

  • Reinforces LASER's design: “born in the past”
  • Introduces date_of_birth as a required field for demographic realism
  • Ensures compatibility with later use in mortality models
  • Keeps age dynamic (instead of static scalar)

Sample Result 5

python3 jenner_bvt5_dob.py
First 10 agents:
  Agent 0: age=1436 days, date_of_birth=-1436
  Agent 1: age=25241 days, date_of_birth=-25241
  Agent 2: age=5825 days, date_of_birth=-5825
  Agent 3: age=9828 days, date_of_birth=-9828
  Agent 4: age=12434 days, date_of_birth=-12434
  Agent 5: age=21007 days, date_of_birth=-21007
  Agent 6: age=15258 days, date_of_birth=-15258
  Agent 7: age=12134 days, date_of_birth=-12134
  Agent 8: age=24851 days, date_of_birth=-24851
  Agent 9: age=25432 days, date_of_birth=-25432
  1. Prompt 6: Predict date_of_death with KaplanMeierEstimator

Using KaplanMeierEstimator, predict date_of_death for all agents based on their age in days, and store it as a property in the LaserFrame.

Key Features:

  • Uses create_cumulative_deaths() helper or synthetic data
  • Introduces date_of_death
  • Sets up agents for mortality-aware step functions in Level 3

Sample Result 6

python3 jenner_bvt5_dob.py
First 10 agents:
  Agent 0: age=1436 days, date_of_birth=-1436
  Agent 1: age=25241 days, date_of_birth=-25241
  Agent 2: age=5825 days, date_of_birth=-5825
  Agent 3: age=9828 days, date_of_birth=-9828
  Agent 4: age=12434 days, date_of_birth=-12434
  Agent 5: age=21007 days, date_of_birth=-21007
  Agent 6: age=15258 days, date_of_birth=-15258
  Agent 7: age=12134 days, date_of_birth=-12134
  Agent 8: age=24851 days, date_of_birth=-24851
  Agent 9: age=25432 days, date_of_birth=-25432

Level 3: Components + Model Class Integration

  1. Prompt 7: "Define a Model class with a LaserFrame, add infection_state and a component that randomly infects 1% of agents at each step. Run 10 steps and count infections per step."

Sample Result 7

python3 jenner_bvt7.py
Step 1: infected = 10
Step 2: infected = 20
Step 3: infected = 30
Step 4: infected = 40
Step 5: infected = 50
Step 6: infected = 60
Step 7: infected = 70
Step 8: infected = 80
Step 9: infected = 90
Step 10: infected = 100
  1. Prompt 8: "Create a LASER model that includes a component for aging (increments age each day), and a component that kills agents when their age exceeds a lifespan threshold."

Sample Result 8

Step 1: alive = 100
Step 2: alive = 100
<snip!>
Step 16: alive = 100
Step 17: alive = 100
Step 18: alive = 100
Step 19: alive = 100
Step 20: alive = 100
Step 21: alive = 99
Step 22: alive = 98
Step 23: alive = 96
Step 24: alive = 95
<snip!>
Step 74: alive = 16
Step 75: alive = 15
Step 76: alive = 10
Step 77: alive = 8
Step 78: alive = 4
Step 79: alive = 2
Step 80: alive = 0

  1. Prompt 9: "Create a LASER model that which has 2 components that interact with each other. Maybe a timer which counts down and changes state when the counter reaches 0."

Sample Result 9:

Step 1: infected = 20, recovered = 0
Step 2: infected = 20, recovered = 0
Step 3: infected = 20, recovered = 0
Step 4: infected = 16, recovered = 4
Step 5: infected = 14, recovered = 6
Step 6: infected = 9, recovered = 11
Step 7: infected = 6, recovered = 14
Step 8: infected = 0, recovered = 20

  1. Prompt 10:

"Create a LASER model that includes a LaserFrame for storing time-series results (per timestep). Add a reporting component that counts the number of agents in each infection state — Susceptible (0), Infected (1), and Recovered (2) — and records their fractions in a results frame. Run the model for 20 timesteps and print the values stored in the result frame at the end."

Sample Result 10:

python3 jenner_bvt10.py
Timestep |   S     I     R
       1 | 0.990 0.010 0.000
       2 | 0.980 0.020 0.000
       3 | 0.970 0.030 0.000
       4 | 0.960 0.040 0.000
       5 | 0.950 0.050 0.000
       6 | 0.940 0.060 0.000
       7 | 0.930 0.070 0.000
       8 | 0.920 0.080 0.000
       9 | 0.910 0.090 0.000
      10 | 0.900 0.100 0.000
      11 | 0.890 0.110 0.000
      12 | 0.880 0.120 0.000
      13 | 0.870 0.130 0.000
      14 | 0.860 0.140 0.000
      15 | 0.850 0.150 0.000
      16 | 0.840 0.160 0.000
      17 | 0.830 0.170 0.000
      18 | 0.820 0.180 0.000
      19 | 0.810 0.190 0.000
      20 | 0.800 0.200 0.000

Level 4: Squashing, Snapshots, and Migration

  1. Prompt 11: "Write a component that sets agents with status = 2 as recovereds and squashes them from the LaserFrame. Print the reduced agent count."

Sample Results 11:

python3 jenner_bvt_squash.py
Before squashing: count = 100
After squashing: count = 70
  1. Prompt 12: "Demonstrate saving a population snapshot to HDF5 using save_snapshot() and reloading it with load_snapshot(). Show that data matches before and after."

TBD

  1. Prompt 13: "Create a migration matrix using laser_core.migration.gravity() with 3 nodes and use it to simulate movement of agents between nodes. Print before/after node distributions."

Sample Result 13: (Might not be correct)

python3 jenner_bvt_migration.py
Before migration:
  Node 0: 100 agents
  Node 1: 100 agents
  Node 2: 100 agents

After migration:
  Node 0: 153 agents
  Node 1: 147 agents
  Node 2: 0 agents

Level 5: Full Micro-Model

  1. Prompt 14: Spatial

    "Build a spatial SIR disease model on a 1-D ring of patches, e.g., 10 patches where the ends are connected (patch 0 neighbors patch 1 and patch 9). Population: Distribute populations heterogeneously across patches (e.g., patches 0–2 and 7–9 are high-density, others are sparse). Seeding: Infect a small fraction of agents in one high-density patch (e.g., patch 0). Migration: Implement agent migration between adjacent patches, with ring wraparound. Use a fixed per-tick migration probability. Transmission: Model SIR infection dynamics within each patch. Transmission depends on local susceptible and infectious fractions. Objective: Show that the infection spreads to distant patches via agent migration, not external seeding. The contagion should eventually appear in patches not directly connected to the seed."

Sample Result 14:

Step 1
  Patch 0: S=1226 I=11 R=0
  Patch 1: S=1248 I=2 R=0
  Patch 2: S=1243 I=0 R=0
  Patch 3: S=644 I=0 R=0
  Patch 4: S=625 I=0 R=0
  Patch 5: S=632 I=0 R=0
  Patch 6: S=624 I=0 R=0
  Patch 7: S=1220 I=0 R=0
  Patch 8: S=1261 I=0 R=0
  Patch 9: S=1263 I=1 R=0
----------------------------------------
<snip!>
Step 50
  Patch 0: S=508 I=213 R=486
  Patch 1: S=507 I=194 R=389
  Patch 2: S=534 I=243 R=252
  Patch 3: S=681 I=136 R=113
  Patch 4: S=649 I=65 R=27
  Patch 5: S=715 I=11 R=17
  Patch 6: S=786 I=59 R=36
  Patch 7: S=713 I=173 R=125
  Patch 8: S=630 I=242 R=329
  Patch 9: S=528 I=198 R=441

(This prompt produced 150 lines of perfectly working LASER spatial SIR code, with above output.)

  1. Prompt 15: All The Things

    "Build a minimal SIR model using LaserFrame, PropertySet, and 3 components: SeedInfection, Transmission, and Recovery. Run it over 30 timesteps and print S, I, R counts per day."

Sample Result 15:

TBD

Conclusion and Way Ahead

We see above that we can enumerate prompts which return code which can be run and produces outputs which can be measured against a reference output. The next step is to host the JENNER-MCP service at some known endpoint and run these tests automatically. It's possible that some outputs may need to be compared to the reference within some statistical bound.

⚠️ **GitHub.com Fallback** ⚠️