Plans and policy - FullmetalBober/Indexterity GitHub Wiki

Plans and policy

What a cluster owner can configure, what each plan entitles them to, and why the defaults are where they are. The engine that acts on all of it is described in Architecture; this page is the surface a customer touches.

Policy knobs (per cluster)

knob effect default
readOnly compute everything, never write on
workloadAnalysis the create/merge/update engine; turn it OFF to silence the create side on
instantCreate build critical missing indexes without approval off
observeWindowDays baseline bake time for a hidden index; scaled per index (below) 30
maxCollectionSizeBytes size ceiling for building new indexes โ€”
autoApplyScore the one auto-approval control: empty = you approve everything, 0 = everything auto-approves, 1โ€“100 = a confidence floor. 70 is the suggested setting. Advisories never auto-approve. The settings page says how many of your current findings a given number would approve โ€” read that rather than guessing, because the scores are not evenly spread and two adjacent numbers can differ by a lot empty
changeWindowStartHour / EndHour elective changes run only in this UTC window; safety rollbacks never wait engine-chosen

Policy knobs (per organization)

One so far, and it lives apart from the table above for a reason worth stating: a per-cluster policy row is created after its cluster exists, so it cannot decide anything about connecting one.

knob effect default
requireLeastPrivilege refuse to store credentials that can create users or roles, at connect and at rotate off

Settings โ†’ Organization โ†’ Credential policy, owner-only to change and visible to every member โ€” somebody refused a connect has to be able to read the rule that refused them.

What it refuses, and what it does not

Two shapes, and the second is not a narrower version of the first.

  • The credentials can create users or roles. Refused with a 422 whose message points at the provisioning path, which is one button along on the same screen: the admin string is used once to create a scoped user and is never stored. The connect form withdraws Use these credentials as-is entirely rather than greying it out, because a disabled control is still an offer and sends the reader hunting for what unlocks it instead of pressing the button beside it that works.
  • The deployment has authentication disabled. Also refused, and provisioning is not offered โ€” a server that asks for no credentials grants every privilege to anyone who can reach it, no grant narrows that, and there would be nothing to authenticate a scoped user as. The remedy is on the server.

A rotation whose diagnosis could not be taken is not refused. The string pinged, so the credentials work; what failed is our ability to describe them, and refusing on that would mean a cluster with a flaky diagnosis cannot have its password changed โ€” the one operation an incident needs most. The posture is recorded as unknown instead, and the connection card says so.

It never reaches backwards

Switching it on cannot un-store an admin string that is already sealed, and it deliberately does not try. Clusters connected under the old rule keep collecting โ€” an organization that had to choose between its policy and its analysis would never switch the policy on โ€” and each one's connection card marks it out of policy with a rotation as the fix. A cluster whose posture was never recorded gets its own sentence rather than the violation's, because rotating records the posture and may well record that it was fine all along.

Turning it on or off writes an ORG_POLICY_CHANGED row to the security trail with both sides of the change, and only when the value actually moved: a form saved twice is one decision.

The observe window scales to the index

observeWindowDays is the baseline; each pending drop gets its own, decided once at hide time and recorded with its reason โ€” which the dashboard shows on the drop date, so a 7-day window beside a 60-day one reads as measured rather than as arbitrary. The window is set by whichever question is still open, and the two run on different clocks โ€” will anything want this again at the cadence of the workload, did hiding it hurt at the rate the index is queried:

the index window
periodic usage (monthly report, weekly batch) 2ร— the largest activity gap, โ‰ค 90d might be wanted next cycle
queried about daily, and still when hidden 7d a regression arrives in days, not weeks
in place โ‰ฅ 2ร— the policy and used in that time 1.5ร— the policy, โ‰ค 90d may have a cadence longer than we have watched
zero usage across โ‰ฅ 2ร— the policy half the policy, โ‰ฅ 7d the history already was the observation
appeared on our watch, never used since about as long as it has existed, โ‰ฅ 7d its whole life is on record
anything else the policy

An owner can end an observation early โ€” "Drop sooner" on a hidden row โ€” which shortens the window to the time already served. It never lengthens one, and it never skips a gate: the drop still waits for the change window and still passes the regression check, so what is skipped is the waiting and nothing else.

Every row above turns on activity, and activity is a difference. MongoDB's $indexStats counter is cumulative โ€” it climbs while the index is queried and holds perfectly still while it is not โ€” so one reading only says the index was used at some point, never when. Indexterity reads usage as the change between two consecutive readings, which is why an index whose counter has been frozen for a month counts as a month of silence rather than as a month of use. A counter that restarts under us โ€” a node restart, a stepdown โ€” is counted in full rather than as a difference, so a restarted member never reads as a burst of activity or as negative usage.

Order matters in the first two rows. A quarterly job that runs densely for a week is periodic, not busy, so the cadence rule is checked first. And "still" is narrow: an index that was busy and went quiet a week ago gets no fast verdict from being hidden, because nothing is querying it to notice โ€” that one is back to a cadence question, and waiting is the only answer.

The last two handle a hand-made ad-hoc index: created, used once, forgotten. Its whole life is on record, so it leaves in about a week instead of a month. Age only counts when the index appeared after we started watching โ€” snapshots begin at onboarding, so an index in the first one may be five years old.

Decided in D17.

The score

Every recommendation carries 0โ€“100, and the scale is calibrated so 100 is reachable and means "as sure as this engine gets".

  • Drops โ€” the argument is worth 55 (redundant), 50 (never used) or 35 (was periodic, went quiet); a month of unbroken history adds up to 25; reclaimable space up to 20.
  • Creates โ€” 40 for a live collection scan, up to 35 for frequency, up to 25 for collection size.
  • Each past regression on the same index subtracts 40, so one is nearly disqualifying and two are โ€” while it still counts. A regression carries full weight for as long as the cooldown it caused, then fades to nothing over the same span again, so an index is not disqualified for the life of the cluster by one failed experiment against a workload that has since been rewritten. The count itself is kept either way, and the parked panel still shows it.

The score gates entry only. Every safety stage runs regardless of it.

Every five minutes, a health probe

Every five minutes is the probe's schedule, whichever clock drives it. FAST_PASS_INTERVAL_MINUTES can stretch it, and on a free serverless database fifteen is the number, for the reason in Running Indexterity โ†’ Configuration, under Choosing the interval on free serverless tiers.

Two things, in order.

First the server as a whole. CPU is not available โ€” mongod reports none outside FTDC โ€” but serverStatus reports what the query engine is doing: collection scans, documents and index keys walked, sorts run with no index to order by, and operations queued behind the global lock. The signal is documents walked per index key, which is volume-independent, so it reads the same on a small cluster and a large one. A loaded CPU could be a backup or a noisy neighbour; thousands of documents per key is a missing index and nothing else. This catches a scan storm spread thinly across many collections that no single collection's latency would show.

Then per collection: a missing index shows up as average read latency climbing while the collection keeps serving traffic. When reads get sharply slower than their own baseline, the workload pass runs immediately instead of waiting for the hourly one. Only the busiest 20 collections are probed, and the readings are never written to latency_samples โ€” that table's hourly cadence is what the activity gate and the change-window inference reason over.

serverStatus is the one privilege that reads beyond index metadata, so it is optional: a cluster without it onboards clean and simply loses the first half of the probe. Connecting a cluster says exactly what it exposes.

The change window picks itself

Left unset, the engine buckets the cluster's own traffic into the four 6h slots of the UTC day and takes the quietest, re-deriving after every collect. It declines to guess on a flat day or thin history, and an explicit setting always wins.

Plans

clusters seats index suggestions unattended changes history
FREE 1 3 yes โ€” 90 days
PRO 5 15 yes yes 183 days
SCALE unlimited unlimited yes yes 365 days
SELF_HOSTED 1 unlimited yes yes 365 days

A plan is per organization. Every number above is that org's, so a customer with a free side project and a paid production team holds two orgs on two plans and is billed for one of them. How many organizations you make is therefore not metered โ€” capping it would cap how much you can buy โ€” and the free tier is held by the cluster limit, applied inside each org one at a time. What has been spent of that limit is drawn beside the connect form, before anything is typed into it: a quota nobody sees until it refuses them is a 402 in the middle of someone's work.

Free gives away the analysis and sells the automation. Every plan sees every recommendation, with the reasoning, and can approve any of them by hand. What a paid plan adds is not having to: autoApplyScore (approve by score) and instantCreate (build a critical missing index immediately). The safety pipeline โ€” hide, observe, regression-gate, roll back โ€” is what makes unattended changes safe to run, and it is the part that took the work.

The rules live in one table in apps/api/src/billing/plans.ts; nothing else decides them. Limits are enforced by the api, not drawn in the dashboard, and a refusal comes back as 402 rather than 403 โ€” the caller is an owner, so "forbidden" would send them looking for a permissions problem they do not have. Seats count members plus outstanding invites, so an org cannot invite past its plan and leave the refusal for whoever clicks the link; better-auth's own membershipLimit, which counts members only, is deliberately left unset so that there is one limit with one name. A downgrade never deletes anything: an org over its new limit keeps what it has and simply cannot add more, and an auto-approve score saved on a paid plan stops being obeyed without being erased โ€” it comes back on upgrading.

History is kept for the plan's window, and no longer

Each org's history is kept for its own plan's window, and the daily sweep at 03:00 UTC deletes whatever is older (#549). The same window is applied at every read of the time-series tables: the engine's reads as much as the dashboard's, since a longer series is precisely what lets the engine call an index unused at all. That read filter also covers the day a row can sit past its window between two sweeps.

It used to be split. Deletion ran one cutoff for the whole deployment (the longest window any plan could claim), and each plan's window was applied only on the way out. That kept an upgrade's history ready: a free org moving to a paid plan saw its extra months at once. The price was storing history no reader was entitled to. On a deployment of FREE and PRO orgs that meant everything past 183 days of a 365-day sweep, and on a small database that is what reaches the storage limit first.

So, stated plainly:

  • An upgrade starts from what the old plan kept. A free org moving to PRO has its 90 days, and the rest of the new window fills as time passes.
  • A downgrade deletes history at the next sweep, and moving back up does not return it. set-plan.js prints both windows when it moves an org down.

Settled recommendations age out on the same clock as the history they were made from.

The mechanism is Architecture ยง11.3.

SELF_HOSTED is not a tier anyone buys

It is the BUSL Additional Use Grant expressed as entitlements โ€” one production cluster, everything else on โ€” and it is what the chart ships. The licence caps production clusters and says nothing about features, seats or history, so neither does this. Shipping self-hosters the hosted free tier would restrict them further than the licence they are complying with, on hardware they pay for themselves.

Which plan a new organization lands on

DEFAULT_ORG_PLAN, validated against the plan list at boot, stamped onto the row by the organization-create hook. The chart sets it to SELF_HOSTED; a deployment that says nothing gets FREE, because a process that has not been told where it runs should offer the least.

organizations.plan deliberately has no database default. It had one โ€” FREE โ€” and for a release that default was the decision rather than a fallback: the create hook wrote no plan, so every self-hosted install ran on the hosted free tier (three seats, no unattended apply, ninety days) while the chart asked for SELF_HOSTED and the variable reached the container and stopped (#132). A column that cannot fall back is what makes a path that forgets fail at the insert instead of silently choosing the most restrictive plan.

Upgrading an install that has been running a while? Its organizations predate that fix and are sitting on FREE. List them:

node apps/api/dist/set-plan.js --backfill SELF_HOSTED

Nothing changes until you add --apply. It moves only organizations that are on FREE โ€” the plan the database default used to impose โ€” and whose plan was never chosen deliberately: no set-plan.js run, no webhook, no billing subscription attached. So it can raise a plan and never lower one, and the same command is safe to hand a hosted operator: a paying customer's org is skipped by construction.

The FREE guard is there because "nobody ever chose this" is not quite what a null timestamp means. Anything that writes the column without also writing plan_updated_at โ€” an integration test, a hand-run UPDATE โ€” looks identical to an untouched organization, and without that guard a SCALE org would have been silently downgraded by a command called backfill.

No payment provider is wired, on purpose

Plans are set with node apps/api/dist/set-plan.js <org> <PLAN> [note] โ€” enough to charge by invoice today, from anywhere, with no processor account. Whoever eventually takes the money only decides which plan an org is on; a webhook would write the same column that CLI does, and the entitlements above would not change.

โš ๏ธ **GitHub.com Fallback** โš ๏ธ