Build an AI review skill and test its missed issues and false positives
The verification lesson repeated a useful set of questions: who owns the pending state, can a second call start, what happens after rejection, and which checks actually ran? Another save flow can benefit from the same review.
A skill records that procedure so an assistant can follow it again. Its value depends on the specificity of the review and the evidence in its output.
Give the skill one bounded job
Our review-save-flow procedure is report-only. It reviews the supplied save/submission flow for concurrency and retry problems. It does not edit code, certify the whole app or claim that native checks were run.
The Agent Skills specification defines the skill directory and SKILL.md metadata structure. Our sample names the procedure review-save-flow and describes save concurrency and retry as its trigger. Supporting references belong beside the procedure when they are useful.
The lesson explicitly asks the assistant to read the file. Automatic discovery and installation behavior depend on the chosen tool and are not measured by that explicit request.
Write a procedure that follows behavior
The reviewer should:
- Identify the owner and lifetime of the in-flight state, including the call site.
- Follow two calls before the first settles and determine whether a second callback starts.
- Trace success, rejection and a synchronous throw.
- Inspect caller-facing errors and visible feedback separately from admission.
- Run the supplied checks when execution is available and report the actual command.
“Review carefully” does not tell the assistant how to inspect any of these paths. The procedure should expose reasoning another developer can follow.
Require findings with a triggering sequence
A useful Case B finding identifies the reset after await, explains that rejection skips it, and states the effect: later attempts through this owner stay blocked. Its correction is to release the retained flag in finally.
The finding format asks for a file and line, triggering sequence, expected behavior, observed code path, likely effect and smallest corrective suggestion. Separate supported findings from questions needing runtime evidence. No arbitrary finding count is required.
Include valid code in the evaluation
The fresh cases are prepared teaching inputs. Their expected outcomes differ:
| Case | Expected review |
|---|---|
| A: gate created inside the press handler | Identify independent state on every press |
| B: reset only after successful await | Identify blocked retry after rejection |
| C: retained tested gate | Do not invent a concurrency/retry defect within the exercise assumptions |
| D: enabled button plus retained handler guard | Separate UI feedback concerns from duplicate-task claims |
A disabled={false} prop alone does not prove that another task can start. The retained handler guard may still reject that call. If the requirements include busy feedback, report that concern specifically.
The exercise assumes caller error handling outside the displayed snippets for C and D. Do not invent missing host code to manufacture a finding.
Keep expected answers away from the reviewer
For an actual blind run, use blind-evaluation.zip in a clean workspace. It contains the procedure and inputs, without the answer key or this lesson. Ask for a report using the evaluation prompt from the prompt pack.
Save the assistant's actual report before comparing it with the answer key. The table above is an expected result, not evidence of a real model run. Avoid supplying the lesson article itself to the blind reviewer, since it reveals the answers.
Count missed defects and false positives separately. A skill that flags every snippet may find A and B while generating unnecessary work on C. If it misses the lifetime problem, improve that check and evaluate a different lifetime example rather than repeating only the same case.
Four cases can expose weaknesses. They do not establish universal accuracy or test automatic invocation.
Apply the workflow to a different action
Try retrying a failed refresh. Define behavior while pending, recovery after failure, ownership and preservation of already visible data. Then decide which save-review questions transfer and which requirements change.
The complete storyboard contains the spoken lesson. English narration uses an AI clone of Ibrahim Fathi's voice, with English and Arabic subtitle tracks. No actual agent evaluation score or native execution result is claimed.
Comments