KB Agent improvements - SS67/project-docs GitHub Wiki

Short version: Phase 1 is right in principle and wrong in format. Phase 2 is the highest-value work and mostly a data-architecture problem, not an agent problem. Phase 3 is viable but only if you stop thinking of it as "the agent triggers Ansible." Built the obvious way, Phase 3 will eventually cause an outage and get the whole thing switched off.

Detail below.

Phase 1 — SharePoint + verified PDFs

Keeping SharePoint as the system of record is a good call. You get versioning, Purview/DLP, permission trimming, and an approval surface your compliance people already accept. Don't move off it.

The problem is that SharePoint is also your retrieval store, and PDF is your retrieval format. Copilot Studio's SharePoint knowledge goes through Graph search. You don't control chunk size, overlap, embeddings, or reranking, and the PDF text layer flattens tables, multi-column layouts, and diagrams into noise. For infra runbooks, which are mostly tables and step lists, this hurts more than for prose.

Two fixes, in order of effort:

Split the corpus. The PDF stays as the human-facing signed artifact. A derived, retrieval-optimised version (Markdown or DOCX, one heading per answerable question, tables restated as sentences, front-matter metadata) is what the agent indexes. Same source, two renders. This is exactly what your Phase 2 pipeline should produce.

If quality still isn't there, move the knowledge source to Azure AI Search. You get chunking control, hybrid search, semantic ranker, and metadata filters. The last one matters most: you'll want to filter by OS, environment, and applicability rather than hoping the model infers it.

Two things I'd add regardless:

Staleness is your biggest liability. A KB agent confidently citing a runbook last verified 18 months ago is worse than no agent. Put last_verified and review_by in metadata, have the agent state it in every answer, and have the pipeline flag anything past review date.

Check your authentication mode. If the agent authenticates as the user, SharePoint permission trimming applies. If it uses maker or service credentials, every user sees every document. For infra runbooks with hostnames, service accounts, and escalation paths in them, that difference is the whole security model.

Phase 2 — CI/CD and live MCP data

Docs-as-code is straightforwardly correct. Git, PR review by the SME who owns the domain, pipeline renders and publishes. Your "verified" gate becomes a merge approval, which is auditable in a way a SharePoint approval flow isn't.

The MCP work is where this stops being a KB bot and becomes an operations assistant. Three things will decide whether it works.

Don't query vCenter live. vSphere inventory queries are slow and the payloads are large. Copilot Studio has per-call and per-turn timeouts you will hit. Run a scheduled collector that writes patch state, VM tags, and custom attributes into a compliance store (Azure SQL, Data Explorer, or Log Analytics), and point MCP at that. It's faster, cacheable, historically queryable, doesn't hammer vCenter, and gives you trend data for free. Accept the freshness lag and display it: "as of 04:00 today."

Authorization belongs in the tool, not the model. You wrote "VMs user own." That filter must be applied server-side using the caller's propagated identity. If the MCP server connects with a service account and you rely on instructions to make the agent only show the user's VMs, you have built an inventory disclosure endpoint with a polite front end. Verify how identity flows into your MCP connection before you build anything on top of it, and treat "the model was told not to" as not a control.

Design narrow tools. get_patch_status(hostname) and get_vm_compliance(vm_name) beat a generic run_query. Overlapping or vague tools are the main cause of wrong-tool selection, and wrong-tool selection is what your users will experience as "the agent is unreliable."

You'll also need explicit precedence rules in the instructions once the agent has both KB and live state. It now knows how things should be and how they are, and it must never blend the two into one unattributed answer.

One caveat: MCP support in Copilot Studio has moved fast and my picture of the current limits may be stale. Confirm the current timeout, payload, and auth behaviour against Microsoft's docs before you design around them.

Phase 3 — Remediation

Technically viable. The architecture you described is not.

The failure mode is putting the LLM in the authorization path: model diagnoses, model picks playbook, playbook runs with privileged credentials. Two problems with that.

First, prompt injection is a live threat here, not a theoretical one. Your agent reads KB content and tool output. vSphere custom attributes and annotations are user-editable free text. So are hostnames, ticket descriptions, and tag values. Any of those can carry instructions into a context window that has remediation tools attached. You are one crafted VM annotation away from an unintended playbook run.

Second, the model's diagnosis will sometimes be wrong, and "domain join is broken, so rejoin the domain" is a textbook way to make an outage worse. Most domain auth failures aren't a broken join. They're clock skew past the Kerberos tolerance, DNS SRV resolution, an expired machine account password, or a stale keytab. Blind rejoin on a healthy-but-misbehaving host can drop every user off it.

The shape that works: the agent proposes, a control plane decides.

The agent never calls Ansible. It calls an intent API: request_remediation(type, target, reason). That API is your policy enforcement point. It validates the target is in the caller's scope, checks change freeze and maintenance window, caps blast radius (one host, never a pattern or wildcard from the model), creates a change record, and routes to approval or auto-approves by risk class. Execution goes through AAP job templates with surveys and RBAC, using AAP's own credential store, so Copilot never touches credentials.

Sequence your three use cases by risk, not by ease:

• Defender onboarding is additive and reversible. Start here.
• Sudoers via AD group is medium. Drop-in files in /etc/sudoers.d only, visudo -c validation before activation, never edit /etc/sudoers. A malformed sudoers file locks out every admin on the box.
• Domain rejoin is the riskiest and should be last. Make the playbook diagnose first (time sync, DNS, machine account age, keytab) and rejoin only as a final branch.

Every playbook: idempotent, check-mode dry run first, pre-flight validation, post-change verification, automatic rollback on verification failure.

And run it in shadow mode for a couple of months before it executes anything. Log "would have recommended X for Y" against real tickets and measure how often it was right. That number tells you whether Phase 3 is safe, and it's the number your change board will ask for.

Honest overall read

The concept is sound and the phasing is sensible. Phase 2 alone, done well, is probably worth more than Phases 1 and 3 combined, because "is my VM compliant and why" is a question people ask constantly and currently answer by opening four consoles.

The two things most likely to kill it aren't technical. One is KB freshness with no named owner. The other is the on-call story for Phase 3: when a playbook fires at 2am and something breaks, whose pager goes off, and can they reconstruct what the agent did and why? Build the audit trail (who asked, what was retrieved, what the model concluded, what tool ran, who approved, job ID, outcome) before you build the remediation, not after.

Also worth naming: Copilot Studio is a good chat surface with excellent M365 identity and Teams distribution. It is a weak orchestration engine for multi-step operational workflows. Resist putting your control logic in agent instructions. Keep the brain in a service you own and let Copilot Studio be the face.