Amazon AI Token Usage Metrics: Build a Delivery Scorecard

The memo still considers adoption and input metrics useful when interpreted carefully. It recommends examining both low-adoption teams and heavy users, asking whether delivery improves. It also recognizes that a documented task where current AI tooling does not help can be a useful finding. That is a more nuanced position than either ranking employees by tokens or discarding usage telemetry.

• Amazon: engineering leadership memo

Start with a concrete task in your own application: produce a valid shot-list document from a product brief. Define acceptance before selecting a model. Each item must name the required product feature, fit the requested duration and refer only to supplied assets. A syntactically valid response can still fail if it invents a logo or omits the final shot.

Keep the task identifier stable across retries. One brief that takes three attempts is one task with three attempts, not three delivered briefs. Save a separate attempt identifier for debugging and billing reconciliation. If a human finishes a rejected draft, record that repair work and retain the original rejection reason.

Avoid a single department-wide denominator when tasks differ substantially. A one-line caption and a multi-scene production brief are different workloads. Group tasks by a small number of meaningful classes, keep their acceptance rules stable, and compare the same classes over time. Otherwise a change in task mix can look like a productivity improvement.

Imagine two equal batches of eight briefs. Approach A consumes 24,000 reported tokens, incurs $0.48 of model charges, takes twelve active review minutes and delivers six accepted briefs. Approach B consumes 40,000 tokens, incurs $0.80, takes six review minutes and delivers eight. These are invented observations for explaining the scorecard, not prices or measured results.

At an illustrative review rate of $0.50 per minute, A costs ($0.48 + $6.00) / 6 = $1.08 per accepted brief. B costs ($0.80 + $3.00) / 8 = $0.475, or about $0.48. A has two unfinished briefs, so its final batch-delivery cost is still unresolved. Neither raw token count nor model charges alone tells the whole story.

accepted_rate = accepted_tasks / attempted_tasks
accepted_unit_cost = (charged_model_cost + active_review_cost) / accepted_tasks
when accepted_tasks == 0: report no accepted delivery; do not divide by zero
keep remaining_tasks and elapsed_deadline_time beside the averages

Do not reverse the mistake and optimize only for fewer tokens. Shortening the supplied brief can lower consumption while removing essential constraints. Investigate a change in usage with the corresponding artifacts: what information was supplied, what failed, what the reviewer repaired, and whether the final customer requirement was met.

For a documented language-model route, SJolt exposes Gemini 4 Argon through Chat Completions and the native Gemini API. The catalog documents usage in the Chat response and usageMetadata in the Gemini response. Preserve the native response fields with the model and protocol so your measurement retains its meaning.

• Gemini 4 Argon model and API documentation
• SJolt LLM API guide
• Current SJolt prices

Normalize usage only after checking the selected protocol’s definitions. A total and its component counts are not independent quantities to add together. Cached tokens, reasoning and different billing categories can make a single displayed count a poor estimate of money spent. Reconcile charged cost separately rather than declaring that twice the tokens always means twice the bill.

If the accepted shot list later drives video generation, give that stage a separate task record. A language response is not a video asset, and media generation can have a different pricing unit. Link the brief revision to the generation task and downloaded result. Report accepted briefs and accepted clips separately so a successful planning stage cannot conceal failed production.

• Asynchronous media task documentation

Create six local fixtures: a valid brief, a missing product requirement, an invented asset, a repeated attempt, a late result and a failed request. Have a reviewer label them with the written rubric. Verify that a duplicate attempt does not create a second accepted task, that rejected work remains in cost totals, and that an unfinished batch remains visibly unfinished.

For a later model comparison, use the same frozen briefs, a declared retry allowance and equivalent review conditions. Rotate evaluation order if reviewers might learn from earlier outputs. Keep the raw examples available behind aggregate numbers. A small pilot can guide your application choice, but it does not establish a company-wide effect or an employee ranking.

End the review with one actionable decision: keep the current prompt, repair a recurring missing requirement, or evaluate another documented route. Assign an owner and a date for the next check. The scorecard is useful when it explains a delivery problem and changes what the team does; a rising activity chart alone does neither.