Dev Tools
Collect and compare LLM responses
A pipeline that sends the same question to three major AI platforms simultaneously — ChatGPT, Perplexity, and Gemini — via Nimble's AI agents, then uses Claude Haiku to read all three raw responses and judge whether they agree. Ships with 100 pre-loaded questions and a live tab for real-time queries.
Quick Start
Inputs
- Question Any question to send to all three AI platforms simultaneously — typed into the Live tab in the dashboard.
Outputs
- Pre-analyzed question set with per-model verdicts and consensus labels
- Consensus breakdown: Strong / Moderate / Split
- Category breakdown by topic domain
- Live tab for real-time queries
- Per-question detail: raw responses and verdict per model
How it works
A 5-phase pipeline. Read the blog here for a deeper explanation.
- Send The same question is sent to ChatGPT, Perplexity, and Gemini simultaneously via three parallel Nimble agent calls — all three fire at once via ThreadPoolExecutor, so round-trip time is bounded by the slowest platform.
- Collect All three raw responses are stored as-is with no preprocessing — Claude Haiku receives the raw output exactly as each platform produced it, including Gemini's freeform markdown.
- Judge Claude Haiku reads all three responses in a single prompt and extracts each model's core position regardless of format, tone, or verbosity.
- Label Consensus is labeled as Strong (all three agree), Moderate (two agree, one differs), or Split (clear disagreement) — assigned semantically by Claude, not by keyword matching.
- Render Results are displayed with per-model verdict, reason, and consensus label — browse the 100 pre-loaded questions or enter any question in the Live tab (~65 seconds end-to-end).
Stack
Nimble primitives plus the full runtime stack.
Nimble APIs
What it does
- chatgpt Fetches a live response from ChatGPT’s web interface for any prompt. Returns structured text.
- perplexity Fetches a live response from Perplexity’s search interface for any prompt. Returns structured text.
- gemini Fetches a live response from Gemini’s interface for any prompt. Returns freeform markdown — no structured format enforced.
3rd Party Tools
Role
- claude-haiku-4-5 Anthropic Claude API — reads all three raw responses per question in a single pass and judges consensus semantically.
- streamlit Dashboard — Browse tab for the pre-loaded dataset and Live tab for real-time queries.
- python 3.9+ Parallel fetch with ThreadPoolExecutor — 300 total agent calls across 100 questions.
Reach out if you have any questions.
Talk to an Expert