Product Pilot
A Product Knowledge Agent Built on Microsoft Copilot Studio
Pilot · In active beta
The Problem
In a managed service provider with a broad and growing product portfolio, knowledge lives in documents. Service descriptions, competitive positioning guides, objection handling frameworks, onboarding steps, SLA definitions — all of it carefully written, properly maintained, and almost entirely invisible to the people who need it most.
The sales team. The delivery consultants. The people on calls.
Not because the content doesn't exist — it does, and it's good. The problem is discoverability. A salesperson with five minutes before a customer call doesn't have time to navigate a SharePoint library, open three documents, and extract the specific answer they need. They either wing it, ask a colleague, or go into the call underprepared.
The knowledge gap isn't a content problem. It's an access problem.
The Idea
An AI agent that knows the product library — and can answer questions about it in plain English, instantly, from wherever the team is working.
Ask it for a pre-call briefing on a specific service. Ask it how to handle a common objection. Ask it to quiz you on a product before an important meeting. Ask it what's on the roadmap. Get a grounded, accurate answer in seconds, with a reference to the source document so you can verify if it matters.
The agent wouldn't browse the web or make things up. It would be grounded exclusively on content the product team had published — no hallucination risk from external sources, no answers that contradict the official position. What the product team publishes is what the agent knows.
The platform choice was straightforward given the organisation's existing stack: Microsoft Copilot Studio, grounded on SharePoint document libraries, deployed via SharePoint and Microsoft Teams.
What Was Built
Product Pilot — a Microsoft Copilot Studio agent grounded on the organisation's product and security services document library. Six primary use cases were designed and tested:
Pre-call preparation
A one-question briefing on any product: what it does, who it's for, what problem it solves. Designed for the five minutes before a call, not for deep research — a quick orientation before picking up the phone.
Objection handling
Surfacing documented responses to common customer objections for a specific service. The agent draws on whatever the product team has published on this; the more complete the content, the better the answer.
Competitive positioning
How a specific service compares to what competitors offer, based on published positioning material. Consistent, authoritative answers that reflect the official position rather than individual interpretation.
Roadmap awareness
What's coming for a specific service, based on published roadmap content. Keeps the team current without requiring them to track every product update announcement manually.
Knowledge testing
Quiz mode. Ask the agent to test your knowledge on a product and it generates questions from the product library. The user responds with text answers — building genuine depth rather than surface familiarity.
Rapid briefing
A 60-second summary of any product, designed to be read immediately before jumping on a call. The short-form answer to "just tell me what I need to know right now."
Delivery use cases were also built out: onboarding steps, service scope queries, documentation finder, and SLA lookups — giving the delivery and implementation teams their own entry points alongside the sales-focused use cases.
The agent was deployed in two places: embedded directly in the UK Product Hub SharePoint site for occasional one-off lookups, and installable in Microsoft Teams for teams who wanted it always one click away.
The Build Experience
Building in Copilot Studio works. The output is a functional, deployed agent that sits inside the Microsoft ecosystem your users are already in — Teams, SharePoint, existing permissions. That's a genuine advantage. There's no new login, no separate tool to adopt, no change management problem. The agent uses the user's existing SharePoint read permissions and appears in their Teams chat list like any other contact.
But the build experience itself is clunky compared to other AI tooling.
The iteration speed that you get from tools like Claude — where you can describe what you want an agent to say, get a response formatted exactly as needed, paste it in, and refine from there — doesn't translate directly to Copilot Studio. The platform has its own structure: topics, trigger phrases, responses, adaptive cards. Getting that structure right takes more manual work than it should, and the AI assistance within the studio doesn't yet match the speed and flexibility of external tools.
The quiz use case illustrated this clearly. The ideal experience for a knowledge quiz is a card with clickable answer buttons — the user selects A, B, or C rather than typing. In most modern agent frameworks, generating that card format is straightforward: describe the interaction, let the AI design the card structure, adjust and ship. In Copilot Studio, building adaptive cards with button inputs involves enough steps and configuration that it becomes a meaningful time investment for what should be a simple UI pattern. After working through the options, the decision was made to accept text-based answers instead. It works — users type A, B, or C — but it's a less polished experience than was originally intended.
The lesson here is one that applies across the Microsoft ecosystem generally: the platform is powerful and deeply integrated, but it rewards patience and familiarity with its own conventions. Rapid prototyping and iteration is harder than on purpose-built AI tools.
What the Demo Revealed
The live demonstration covered five questions: a pre-call prep query, an objection handling query, a product quiz, a roadmap question, and a deliberate limitation — a vendor query that the agent correctly couldn't answer and redirected to SharePoint.
The first four worked well. The agent surfaced accurate, grounded answers quickly. The quiz functioned as designed. The redirection on the vendor query was handled cleanly.
Two friction points showed up that were worth noting honestly.
Authentication prompts. On first use, the agent shows two to three authentication confirmation dialogs — the Microsoft ecosystem confirming the user's SharePoint read permissions before the agent can access the document library. These are expected, and the permission model is actually a strength: the agent inherits existing access controls rather than requiring new ones to be set up. But for a user who doesn't know to expect them, the prompts can feel like something going wrong. Managing that expectation upfront — as the slides and briefing did — is important.
Token limits. During the live demo, the conversation hit a token limit — the point at which the accumulated context of the session becomes too large for the model to handle cleanly. The workaround is a /clear command to start a fresh conversation. This works, but it's not a natural concept for most business users. "Clear your context window" is not a phrase that means anything to a salesperson. The immediate fix was documenting the /clear command clearly in the getting started guidance. The longer-term fix — whether through session management improvements in Copilot Studio or model updates that handle longer contexts — is something to monitor and investigate.
Results
The presentation to the UK sales team landed well. The use cases resonated — particularly pre-call preparation and objection handling, which address pain points the team recognises immediately. The feedback channel (thumbs up/down on every response) was set up from day one, with the explicit message that there is no fixed feature list — the roadmap is shaped entirely by usage and feedback.
That approach to the pilot is deliberate. Rather than building out every conceivable feature before launch and shipping something complex, the beta release focused on core use cases that demonstrably work, with honest communication about known limitations and a clear path for users to influence what gets built next.
15–20 minutes per query reduced to approximately 30 seconds
Estimated time saving measured against the previous manual document search workflow. Analytics collection is active from day one to build a reflective data set for future development priorities.
Key Takeaways
Grounding matters more than capability
An agent is only as good as the content it's grounded on. The quality of the product library — how complete it is, how current it is, how well structured the documents are — directly determines the quality of the agent's answers. The technology is the easy part; the content is the work.
Deploy where your users already are
The decision to build in Copilot Studio and deploy via Teams wasn't about the platform being the best tool for AI development — it was about meeting users where they work. Zero friction to access, existing credentials, familiar interface. Adoption is much harder when it requires a behaviour change.
Expect platform friction and plan for it
Copilot Studio produces a real, enterprise-grade agent. But the build experience is slower and less intuitive than purpose-built AI tools. Budget for that. If you're used to the iteration speed of tools like Claude, Copilot Studio will feel slow. That's not a reason not to use it — the integration advantages are real — but it changes how you plan the build.
Surface limitations honestly
The token limit, the auth prompts, the vendor query gap — all of these were communicated openly in the launch presentation rather than hidden or glossed over. Users who understand the tool's limitations use it better and give more useful feedback when things don't work as expected.
Start small, iterate fast
A focused beta with a clear feedback loop beats a comprehensive launch with no mechanism to improve. The agent that ships is the agent that gets better. The roadmap is shaped entirely by usage and feedback — not by a fixed feature list decided before launch.
What Comes Next
Product Pilot is the first agent. It won't be the last.
The broader opportunity — using AI agents to surface information and automate repeatable tasks across a large organisation — is significant. The product knowledge use case is a relatively contained starting point: well-defined content source, clear user need, measurable outcome. It's a good foundation to build on.
For future agents, one approach worth exploring is to prototype and iterate rapidly in purpose-built AI tools first — where the feedback loop is faster and the iteration cost is lower — before working out how to implement the final version in the Microsoft ecosystem, or how to bridge between the two using Power Automate or Power BI where the business process requires it.
The principle is the same one that applies to any AI development: use the right tool for each job. Copilot Studio's strength is deep integration with the Microsoft stack and enterprise-grade access controls. Other tools' strength is speed of iteration and flexibility of design. Understanding which matters most at which stage of a build is the thing that makes both more effective.
Curious about Copilot?
Microsoft Copilot is the platform behind Product Pilot. If your organisation is exploring how AI agents can surface knowledge that's buried in documents, Microsoft's own adoption resources are the best place to start.
Explore Microsoft Copilot resources