Skip to main content

Evaluate AI systems and choose your strategy

Is this AI system suitable for my task, and what work should remain mine?

Read its intended use and limits, inspect how its claims were tested, and try representative tasks with clear success and failure criteria. Use that evidence to set a division of work you can explain.

Scope
System evaluation, product claims, human expertise and personal strategy
Reviewed
5 October 2026

In brief

  • A benchmark score answers a question about a stated test, not every future use.
  • The model and the product around it can have different failure paths.
  • Choose what you delegate, how you verify it, and when a person needs to decide.

Evaluate the task you have​

You want help locating a definition in a course collection. A system that finds the right passage can be useful even if it cannot solve a difficult programming task. For an agent that edits your project, retrieval accuracy alone would leave most of the task untested.

Write a testable purpose first: "Find the course definition of a binary search tree and show the passage that supports it." Then name a failure: returning a plausible definition from elsewhere while claiming it came from the course. This gives you something concrete to check.

Keep a record of the model, application version where available, date, context, tools and permissions. Changes to any of these can change behavior. A small test gives evidence for the tested task and conditions; it does not produce a universal reliability rate.

Read both model and system cards​

The model-card framework proposes reporting intended use, evaluation conditions, relevant groups, limitations and ethical considerations for a trained model. It encourages readers to examine performance across conditions instead of accepting one overall number. Mitchell et al., 2019

Scroll horizontally to view all columns.

Read the document for its actual scope
DocumentQuestions it can help answerQuestions that may need more evidence
Model cardWhich trained model is described? What uses, evaluations and limitations are reported?How does a particular application select context, call tools, store data and handle errors?
System card or system documentationWhich models and product features are combined? What safeguards, evaluations and failure paths are described?Does the published test cover my users, task, deployment and current version?
Evaluation reportWhat was tested, against what criterion, under which conditions?What was not tested, and how do failures affect the intended use?

These names are conventions, not guarantees of complete or consistent reporting. Open the document and inspect what it actually covers. An application can add retrieval, instructions, tools, permissions and interfaces around a model; those choices can change what the user can do and what can go wrong.

For a hypothetical campus tutor, ask whether a retrieved passage belongs to the right course and whether the answer extends beyond it. For a coding agent, also ask what files and commands it can change, whether consequential actions need confirmation, and how you inspect the result.

Work through a real limitation​

OpenAI's GPT-4o System Card, published on 8 August 2024, includes clinical-knowledge evaluations. Its limitations section says those tests do not measure utility in real-world workflows. Read the card and its limitation

Suppose someone proposes: "Those scores show it is suitable as a health adviser for students." The stated evidence does not establish that use. A further evaluation would need to examine the actual tasks, users, oversight, failure handling and escalation route. This is a reading exercise, not advice to use a chatbot for health decisions.

Try the same reading method on a card relevant to your task:

  1. Find the intended use and exact system version.
  2. Select one reported test and identify its inputs and success criterion.
  3. Find a limitation that affects your proposed use.
  4. Propose a test or design change that addresses that limitation.
  5. Name a question the card still leaves unanswered.

The publication is vendor-authored and dated. A public card is useful evidence to inspect, rather than certification that every later version or application is suitable.

Review Iris using public documentation​

This is a review worksheet based on public documentation, not an official system card or a measured evaluation. Record the deployed version and course configuration before making a decision.

Review questionStarting point and unresolved check
Intended useIris provides learning support within Artemis. Ask the course team which exercises and material are available to the deployed tutor.
Context and boundariesInspect the material retrieved for a response. Check whether an answer preserves a missing passage or a source's stated limit.
Data accessRead the deployment's data information. Confirm which learner and exercise context is processed, where it is processed, and who can inspect the interaction.
Evidence of learningA fluent hint is not proof of improved learning. Compare an unaided explanation or new exercise with the stated learning goal.
Failure and escalationKeep an example of an unsupported answer and identify the teaching assistant or course contact who can resolve it.

Use the public Iris documentation for students and Iris project information as starting points. These pages were checked on 6 October 2026. This worksheet reports no test results.

Check a benchmark before comparing scores​

The HELM research framework evaluates several properties, including accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. It makes the coverage and missing areas visible and releases prompts and completions. This supports reading an evaluation as a collection of tests and trade-offs. Its historical rankings do not identify the best current study tool. Liang et al., HELM

Scroll horizontally to view all columns.

What to inspect behind a headline score
QuestionWhy it changes the interpretation
Which task and benchmark version?A changed dataset, grading rule or environment can change what the score measures.
Which model and surrounding system?Tools, retrieved context, prompts and agent setup are part of the tested conditions.
How many attempts and what compute or tool budget?A best result after several attempts differs from a single-attempt result.
What counts as success?Passing a narrow check may leave other requirements unexamined.
Which failures and groups were examined?An average can hide a failure that matters for a particular user or task.
Who ran and selected the evaluation?A provider report, independent study and customer testimonial offer different evidence.

Prefer a report that lets you inspect conditions and failed examples. A launch demo can show that a system performed a task in that demonstration. To judge suitability for your work, test the task and failure conditions you expect to encounter.

A dated launch-page example​

Anthropic's Claude Sonnet 4.6 launch page, dated 17 February 2026, includes a comparison on OSWorld. Its footnote states that scores before Sonnet 4.5 used the original benchmark, while later scores used OSWorld-Verified, released in July 2025 with changes to task quality, grading and infrastructure. Read the launch-page footnote

The footnote matters when interpreting a progress chart. It identifies a change in the test, so the series is not simply repeated measurement under one unchanged procedure. It does not establish that no improvement occurred. Ask what comparison remains supported and what additional evaluation would make the trend easier to interpret.

This is a dated example of reading a provider claim, not a recommendation to buy that product or a claim that its benchmark performance predicts your coursework outcomes. Customer quotations and provider preference studies should retain those labels.

Separate a token price from your cost​

The same release listed starting API rates of $3 per million input tokens and $15 per million output tokens. Anthropic's pricing documentation still listed those base rates for Sonnet 4.6 when checked on 5 October 2026. Release announcement, pricing documentation

For a hypothetical API workload of 100,000 uncached input tokens and 20,000 output tokens:

Input:   100,000 / 1,000,000 × $3  = $0.30
Output:   20,000 / 1,000,000 × $15 = $0.30
Base token cost:                      $0.60

This is arithmetic using listed rates, not an observed student bill. It excludes tools, caching, repeated attempts, taxes and other features. A subscription is a different pricing arrangement. An agent may perform several calls for one visible task, so a token rate alone does not tell you the cost of completing it.

For a comparison, include the whole useful task: the service cost, the time you spend specifying and reviewing it, and whether the result meets the criterion. The resource-cost guide explains why environmental estimates also need a workload and boundary.

Build a small evaluation you can explain​

Choose examples you are allowed to use and whose answers you can check. Include a normal task, a missing-information case and an edge case. Avoid uploading private course or personal data to an unsuitable service.

For the definition-finding task, an illustrative evaluation set could contain:

  • A definition that is explicitly present in the supplied course material.
  • A related concept that the collection does not cover, where the system should state the limit.
  • Two passages with different scopes, where the answer should preserve the distinction.

Write the expected behavior before running the examples. Check the original passages rather than scoring the answer's style. Keep failed runs and note whether repeated runs differ. No results are presented here; these are proposed cases, not measured performance.

For an agent, inspect the changes and exercise the behavior. A passing generated test does not prove the agent chose the right requirement or tested the relevant failure. The verification guide gives checks for code, calculations, sources and policy claims.

If you plan to build or deploy a system, legal responsibilities depend on the actor and use. Read the roles and scope and current dates before applying a general statement about "AI rules" to a student project or deployed service.

Practice the human work you need​

The learning workflows provide worked examples for practising recall, checking code and defending a proof yourself.

You need domain knowledge to notice a missing assumption, distinguish a sensible test from an irrelevant one, and explain why a result is acceptable. A tool can suggest each of those things, but someone still needs to decide whether the suggestion holds for the task.

For a data-analysis project, practice defining the population, checking the denominator and explaining an exclusion. For code, practice tracing a failure and choosing an edge case. For a proof, practice stating the assumptions and defending each step. Those are concrete learning goals you can act on now.

For career planning, compare actual role requirements, speak with people working in the field and use TUM's Career Service. This guide's learning goals do not predict which jobs will disappear or promise job security. Keep your plans open to new evidence while retaining responsibility for work you submit.

Write your personal strategy​

The course ends with a one-page strategy: what you delegate, what you do yourself, how you verify outputs, and how you declare material assistance. It is private, not submitted and not marked. It can change when your course rules, tools or needs change.

Use the worksheet for a task you actually expect to do. Include a point at which you stop asking the system and return to the work, plus the person you will consult when necessary.

Write a plan for one task. These notes stay in this page’s memory. They are not submitted or saved automatically. Copy or download them before leaving.

Source notes and dates​

Sources checked on 5 October 2026. Examples retain their publication dates. No product was tested and no benchmark result was reproduced for this page.