Solution Atlas
SpecialisedUser storyConsultative playbook

We need an AI that answers from our product docs, not the open internet

A software company's support team wants an AI assistant that answers from product documentation, internal playbooks, and ticket history — and only from those. The CTO insists on the guardrails — content filtering, an evaluation process, and a clear record of what fed each answer — before anything goes live to customers.

Trigger
Support costs rising; the product team has a clear use case.
Good outcome
Customers ask a question and get an accurate answer from the product's own documentation. Support agents handle a smaller, harder set of tickets, and the company can show how every answer was produced.
Discovery — signals and questions

Signals validating this story

  • ·Customer-support volume rising; product complexity growing
  • ·Product team has clear documentation surface (Confluence, SharePoint, internal wiki)
  • ·CISO insists on responsible-AI guardrails before any customer-facing launch
  • ·Existing Azure footprint and Azure OpenAI access already in place
  • ·Internal LLM experiments running without lineage or governance

Discovery questions

  1. 1.What are your support team's top question categories, and which are documented?

    WhyDocumented topics are the RAG sweet-spot. Undocumented ones are different (training data) projects.

  2. 2.Where do product docs live — SharePoint, Confluence, Notion, custom?

    WhyDetermines source-content indexing strategy and AI Search configuration.

  3. 3.What's your RAG pipeline today, if any — notebook prototypes, ad-hoc tooling, nothing?

    WhySurfaces maturity. Most customers are at "notebook prototype" stage.

  4. 4.How do you evaluate model output quality?

    WhyWithout evaluation, scale is impossible. Surfaces the responsible-AI gap.

  5. 5.What content filtering is configured on Azure OpenAI today?

    WhyDefault filtering is rarely enough for customer-facing scenarios. Tests CISO readiness.

  6. 6.Has Legal reviewed the proposed launch surface (internal beta vs customer-facing)?

    WhyCustomer-facing AI carries materially different obligations.

Baseline architecture → Target architecture
Baseline architecture

Support team handling rising volume manually. Product documentation fragmented. Ad-hoc Azure OpenAI experiments running on default settings. No evaluation harness. No content filtering beyond defaults. No lineage of what content the model sees.

Typical concerns

  • ·Support cost rising faster than headcount budget
  • ·Documentation surface fragmented
  • ·AI experiments without responsible-AI guardrails
  • ·No evaluation framework for output quality
  • ·Customer-facing risk if launched without governance

Capability gaps

  • ·RAG pipeline with grounded retrieval
  • ·Evaluation harness with quality gates
  • ·Content filtering tuned to scenario
  • ·Identity-bound endpoint access
  • ·Lineage of grounding content
Target architecture

Azure AI Foundry hub with AI Search indexing product documentation. RAG pipeline grounded on indexed content. Evaluation harness with quality gates (relevance, faithfulness, hallucination rate). Content filtering tuned for customer-facing tier. Entra ID-bound access for internal users and (later) customer-facing endpoints. Optional Databricks for data engineering on ticket history.

Key capabilities

  • AI Search-backed RAG pipeline
  • Evaluation harness and quality gates
  • Content filtering tuned to scenario
  • Identity-bound endpoint access
  • Responsible AI build-time discipline
Architecture decisions

  1. 1.Retrieval — Azure AI Search vs custom vector store

    Azure AI Search

    Fits whenAzure-native estate; native Foundry integration; mixed dense + keyword retrieval needed.

    Trade-offsAI Search consumption cost at scale.

    Custom vector store (e.g. pgvector, Pinecone)

    Fits whenExisting vector-store investment; very specific retrieval patterns.

    Trade-offsMore integration work; less native Foundry tooling.

    Default recommendationAzure AI Search for the first RAG workload; revisit only if scale demands it.

  2. 2.Model substrate — Foundry-hosted vs Mosaic AI on Databricks

    Foundry-hosted

    Fits whenAzure-native; need broad model catalogue; pro-code AI engineering.

    Trade-offsTwo governance planes if Databricks also in use.

    Mosaic AI on Databricks

    Fits whenData and ML workloads already on Databricks; lakehouse-native lineage needed.

    Trade-offsSmaller GenAI ecosystem than Foundry.

    Default recommendationFoundry where the data lives in Azure; Mosaic AI where the data is already in Databricks.

  3. 3.Content filtering strictness — default vs custom

    Default

    Fits whenInternal-only beta; risk surface low.

    Trade-offsGeneric filtering may block legitimate domain queries.

    Custom (scenario-tuned)

    Fits whenCustomer-facing launch; specific compliance requirements.

    Trade-offsMore tuning effort; needs iterative refinement.

    Default recommendationDefault for the internal beta; transition to custom before customer-facing launch.

Low-risk trial — proof of value

45-day grounded-LLM prototype for support agents

~6 weeks

Foundry hub provisioned. Product documentation indexed via AI Search. RAG endpoint live with content filtering. Evaluation harness covering relevance, faithfulness, hallucination rate. Internal beta to 20 support agents. Telemetry captured for grounding hit-rate, response time, escalation rate.

Success criteria

  • Response relevance score above 80% on evaluation harness
  • Hallucination rate below 5% on evaluation harness
  • Support-agent NPS on the tool above baseline
  • Zero content-filter false positives causing escalation gaps

InvestmentFoundry token consumption + AI Search capacity. Estimated ~€2–4k/month for the trial scope. No customer-facing launch decisions made during trial.

Proof metrics

  • ·Relevance score, faithfulness, hallucination rate tracked daily
  • ·Internal-agent NPS on the tool above baseline
  • ·Reduction in time-to-answer for indexed topics
  • ·Evaluation harness covering the top ten question categories

Recommended cards

The SKUs and capabilities most likely to be part of the solution, with the editorial rationale for each in the context of this story. Add the ones that fit your situation.

Why for this story

The pro-code AI development surface. AI Search for RAG, agent service for stateful agents, evaluation harness for the quality gates the CISO insists on.

Why for this story

Identity scope for both internal users and (when launched) customer-facing endpoints. Conditional Access enforces who can call what.

Why for this story

For data engineering on ticket history, support session logs, and any structured data feeding RAG. Optional unless data preparation is heavy.

Back to Custom generative AI with Azure AI Foundry