Canonset: Export formats, raw JSONL, preference pairs and Harbor tasks

Approved answers export in three formats: raw JSONL for every task type, preference pairs (dpo) for preference projects, and a Harbor package (harbor) for coding-task projects. Known-answer tasks, rejected answers and answers still in review are never included; the export buttons and the API give the same files.

Raw JSONL (format=raw)

One line per approved answer, in task order:

• task_id, task_type, input: the task as you uploaded it.
• answer: the expert's answer, in the task type's fields.
• annotator: the expert's per-project pseudonym, never a name.
• review_score: the reviewer's score, 1 to 5.
• time_spent_sec: the expert's time, measured on our server.
• billed_minutes, on hourly projects: the measured time, rounded up to the minute and capped.
• submitted_at, approved_at: ISO 8601 times.

Coding tasks add checks, {status, language, runner, fail_to_pass, pass_to_pass, checked_at}: the sandbox's verdict, with the tests that fail on the starting code and pass on the reference solution, and the tests that pass on both. They also add pass_at_k, {model, effort, setting, attempts, solved, pass_rate, measured_at}: how often a named model solved the task from the statement and starting code alone; null until measured, only where you allow that provider's models to read the project's answers.

Environment tasks add environment_task, the task in the engine's own format, and checks with the reference solution's score and each reward hack's.

Preference pairs (format=dpo)

One line per task with a clear majority: {task_id, prompt, chosen, rejected, votes, agreement}. chosen is the response the experts preferred, votes how many experts answered, and agreement the winning share. The majority rule: one side must win more than half of all votes. Ties, and tasks where neither side has more than half, are left out.

Harbor tasks (format=harbor)

For coding-task projects only (other task types get a 400): a zip of Harbor task directories (schema 1.4), one per approved answer, with a README. Each directory:

• instruction.md: the title and problem statement the agent is told, and where the code is (/app).
• task.toml: the task's name, the time limits, and metadata: the project's codename, our task and answer IDs, the expert's pseudonym and difficulty rating, the review score, the approval and export dates, the sandbox's verdict and pass_at_k where measured.
• environment/: a Dockerfile and, in app/, the starting code. The runtime is the one our sandbox verified the task in: python:3.12-slim with pytest, or node:24-slim.
• tests/: test.sh lays the hidden tests in files/ over /app, runs them as the sandbox did, and writes 1 or 0 to /logs/verifier/reward.txt.
• solution/: solve.sh lays the reference solution in files/ over /app; notes.md, where there is one, is the expert's note on it.

With Harbor, harbor run -p <task directory> -a oracle checks the reference solution; name an agent instead of oracle to run it. Without Harbor: build environment/, start a container from it with tests/ at /tests and solution/ at /solution, run bash /solution/solve.sh and then bash /tests/test.sh, and read /logs/verifier/reward.txt.

Nothing in the package names your company, your project's own name or the expert. The public coding samples ship as the same package at https://canonset.com/samples/coding.

Pulling only new answers

Add since=<ISO 8601 time> to the export endpoint to get only answers approved after that time; keep the largest approved_at you have seen and pass it next time. For preference pairs, since returns the pairs of tasks with a newer approval, built over all of that task's votes.

Every export response carries a Link header, rel="describedby", to the datasheet; see The datasheet and The Canonset API.

Questions: support@canonset.com.