2 posts tagged with "evaluation"

GPT-6 Astra does not prove that OpenAI has reached AGI. What it does prove is that the harness can no longer be treated as plumbing.

Astra scored 54.8% on ARC-AGI-3 at high reasoning in the standard harness. The same model scored 99.9% when the harness preserved its reasoning state and compacted long conversations. That is a 45.1-point difference without changing the model.

My takeaway is simple: if you evaluate only the model name, you are evaluating the wrong product. For long-running agent work, the product is the model, memory, tools, context management, and control loop together.

That is why Astra matters even if you are not interested in arguing about AGI.

Our Chapter 8: Evaluating What You Built lab uses a second LLM call as a judge: ask the judge model to compare a generated answer against a reference answer and reply PASS or FAIL. It works, but it's one technique among several, and the lab's own real output caught the judge writing PASS on the first line while its own one-sentence explanation said the answer "fails to provide accurate information about its location." A team of researchers built a toolkit, SIRIN (arXiv:2608.00033), around exactly that gap: instead of picking one hallucination-detection method and living with its blind spots, run three unrelated methods on the same answer and see where they agree.