In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans.&nbsp;This system uses your smartest AI model to evaluate the performance of other AIs&mdash;by scoring them across a range of tasks and comparing them to human intelligence levels.
I talk about:
1. Using One AI to Evaluate Another The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.
2. A Human-Centric Grading System Models are scored on a human scale&mdash;from &ldquo;uneducated&rdquo; and &ldquo;high school&rdquo; up to &ldquo;PhD&rdquo; and &ldquo;world-class human.&rdquo; Stronger models consistently rate higher, while weaker ones rank lower&mdash;just as expected.
3. Custom Prompts That Push for Deeper Evaluation The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.
Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest&mdash;but the framework and methodology still work perfectly with current models.
Subscribe to the newsletter at: <a href="https://danielmiessler.com/subscribe">https://danielmiessler.com/subscribe</a>
Join the UL community at: <a href="https://danielmiessler.com/upgrade">https://danielmiessler.com/upgrade</a>
Follow on X: <a href="https://x.com/danielmiessler">https://x.com/danielmiessler</a>
Follow on LinkedIn: <a href="https://www.linkedin.com/in/danielmiessler">https://www.linkedin.com/in/danielmiessler</a>
See you in the next one!

Unsupervised Learning is about ideas and trends in Cybersecurity, National Security, AI, Technology, and Culture—and how best to upgrade ourselves to be ready for what's coming.

Unsupervised Learning is about ideas and trends in Cybersecurity, National Security, AI, Technology, and Culture&mdash;and how best to upgrade ourselves to be ready for what's coming.

Exploring the fascinating intersection of security, technology, and humans.

Unsupervised Learning

Using the Smartest AI to Rate Other AI

https://api.omny.fm/orgs/070af456-729b-4a0f-9c09-a6c100397b59/clips/8a5bf7e7-84d2-4b9b-9080-b2c1013e0fd5/transcript

Advertisement