Quality engineering,
with AI built in.

Problem. An ITSM virtual agent (Microsoft Teams + ServiceNow) answered employee requests using live enterprise data — and sometimes answered wrong. No playbook existed for testing it.

Approach. Built the AI/LLM testing strategy from scratch: 100+ structured prompt variations, hallucination detection, response-accuracy scoring, data-retrieval validation against ServiceNow records, and regression-safe prompt refinement.

Result. A measurably more accurate and safer agent in production, plus a reusable evaluation playbook the team still runs.

Problem. Test design was the bottleneck: manual case creation ate hours every sprint, and ambiguous requirements kept surfacing late as rework.

Approach. Engineered internal Copilot-based agents for automated test-case generation and early requirement validation — flagging ambiguity before development, not after.

Result. Test design stopped being the bottleneck; requirement gaps now surface in refinement instead of QA.

Problem. Should we build an automation framework (Playwright MCP + GitHub Copilot) or buy an enterprise platform (Katalon / AccelQ)? Opinions were cheap; the decision wasn't.

Approach. Built a six-sheet cost-benefit workbook: 3-year horizon, vendor evaluation (security certs, Entra auth, CRM integration, AI capabilities), sensitivity analysis, and a risk register — executive-ready.

Result. Clear, defensible numbers: Buy returned ~373% ROI with year-one payback against ~17% for Build. The decision made itself.

Problem. Which of two anonymous model responses is actually better? Rank them — accurately, consistently, at speed, in a 24-hour evaluation sprint.

Approach. RLHF-style pairwise preference ranking with rubric-driven scoring: accuracy, completeness, structure, instruction-following, hallucination risk — every judgment backed by a written rationale.

Result. Hands-on experience with the exact evaluation methodology behind modern model training, from the human side of the loop.

Problem. A year of knowledge scattered across nine AI accounts. How do you build one queryable brain out of it — without leaking secrets into embeddings?

Approach. Full architecture: Supabase schema (messages + chunks), 1536-dim embeddings with an HNSW cosine index, a redaction module scrubbing keys/tokens/PII before persistence, and CI/CD config.

Result. An honest lesson in product thinking: a structured markdown system solved 80% of the problem with none of the code — so that's what shipped. Full design available on request.