Skip to main content

Model Intake and Benchmarking

Agent Harness can discover local models, check whether they fit the current machine, run approved benchmark work, and save results for later comparison. This guide covers the public workflow only. Keep private task definitions, prompts, evaluation matrices, pass thresholds, routing weights, and internal evidence outside public documentation and repositories.

1. Inspect available benchmark suites

From the Agent Harness source checkout, list the suites registered in the current config:

npm run harness -- benchmark-suites

Use the installed command output as the authority for suite names and versions. Do not guess a suite identifier or copy a private suite into public material.

2. Preview model intake

Intake is a dry run by default. It enumerates models from a local provider and reports which models fit the detected hardware tier:

npm run harness -- intake --provider ollama

For LM Studio, select the provider and endpoint explicitly when needed:

npm run harness -- intake \
  --provider lm-studio \
  --endpoint http://localhost:1234/v1

Use --tier <band> only when the detected hardware tier needs an explicit override. The default benchmark database is .harness/db/benchmark.db; use --benchmark-db <path> when the deployment requires another location.

3. Run intake benchmarks

After reviewing the dry-run candidates, add --run to benchmark fitting local models and record their results in the capability map:

npm run harness -- intake --provider ollama --run

Use --all with --run only when you intentionally want to remeasure models that already have records. Intake can also run on a schedule with --watch; the interval comes from the configured intake schedule and defaults to 60 minutes.

Signed submission records are an explicit opt-in. Use --attest only when the result is intended for the project’s signed evidence workflow:

npm run harness -- intake --provider ollama --run --attest

Do not publish the underlying private task set or evaluation design with an attestation.

4. Run a known benchmark directly

To benchmark a task path or a suite registered in the current config:

npm run harness -- benchmark <task-path-or-suite-id>

Useful options include:

npm run harness -- benchmark <task-path-or-suite-id> \
  --config harness.config.json \
  --suite <public-suite-name> \
  --benchmark-dir ./benchmarks

Replace placeholders with values that exist in the checkout. Keep private benchmark paths and suite names private.

5. Review saved results

List models with saved benchmark records:

npm run harness -- benchmark-list

For model comparison and routing decisions, use the operator UI or the benchmark and capability-map commands available in the installed CLI. Treat pass rates, timing, model rankings, and capability coverage as evidence for the current machine and task set—not as universal model claims.

Safe operating rules

  • Start with intake dry-run mode before spending model time.
  • Keep benchmark work bounded and reproducible.
  • Inspect verification evidence, not only a model’s reported result.
  • Keep credentials in the configured credential store or environment variables.
  • Store benchmark databases and private task definitions outside public docs.
  • Re-run measurements after changing models, providers, hardware, or verifier behavior.
  • Publish only the methodology and results that are intentionally approved for public release.