Dev Tools
Collect and compare LLM responses
In this example, we built an app that uses Nimble's Extract Templates to search the web for ChatGPT, Perplexity, and Gemini responses to the same question, then has Claude judge whether the models agree.
Quick Start
Inputs
- Question Any question to send to all three AI platforms simultaneously — typed into the Live tab in the dashboard.
Outputs
- Pre-analyzed question set with per-model verdicts and consensus labels
- Consensus breakdown: Strong / Moderate / Split
- Category breakdown by topic domain
- Live tab for real-time queries
- Per-question detail: raw responses and verdict per model
How it works
A 5-phase pipeline. Read the blog here for a deeper explanation.
- Send The same question is sent to ChatGPT, Perplexity, and Gemini simultaneously via three parallel Nimble Extract Template calls — all three fire at once via ThreadPoolExecutor, so round-trip time is bounded by the slowest platform.
- Collect All three raw responses are stored as-is with no preprocessing — Claude Haiku receives the raw output exactly as each platform produced it, including Gemini's freeform markdown.
- Judge Claude Haiku reads all three responses in a single prompt and extracts each model's core position regardless of format, tone, or verbosity.
- Label Consensus is labeled as Strong (all three agree), Moderate (two agree, one differs), or Split (clear disagreement) — assigned semantically by Claude, not by keyword matching.
- Render Results are displayed with per-model verdict, reason, and consensus label — browse the 100 pre-loaded questions or enter any question in the Live tab (~65 seconds end-to-end).
Stack
Nimble primitives plus the full runtime stack.
Nimble APIs
What it does
- chatgpt Extract Template Fetches a live response from ChatGPT’s web interface for any prompt. Returns structured text.
- perplexity Extract Template Fetches a live response from Perplexity’s search interface for any prompt. Returns structured text.
- gemini Extract Template Fetches a live response from Gemini’s interface for any prompt. Returns freeform markdown — no structured format enforced.
3rd Party Tools
Role
- claude-haiku-4-5 Anthropic Claude API — reads all three raw responses per question in a single pass and judges consensus semantically.
- streamlit Dashboard — Browse tab for the pre-loaded dataset and Live tab for real-time queries.
- python 3.9+ Parallel fetch with ThreadPoolExecutor — 300 total Extract Template calls across 100 questions.
Reach out if you have any questions.
Talk to an Expert