Run

How to run DrugTargetBench v1.0

01 Install

DrugTargetBench runs in Harbor.

Shell

uv tool install harbor

02 Run the benchmark

Harbor pulls the task and the image, downloads and checksum-verifies the world data, starts the experiment service, runs your agent, then scores it in a separate verifier container.

Shell

harbor run \
  -d drugtargetbench/drugtargetbench@v1.0 \
  -a claude-code \
  -m claude-opus-5

03 Options

Appended to harbor run.

Shell

--limit 1  # one task instead of all 60
-n 2       # concurrency, default 4; each trial needs ~17 GB of disk
-k 3       # 3 attempts per task

# a single specific task
harbor run -p drugtargetbench/hard-02-full-program -a claude-code -m <model>

harbor agent list shows the available agents: claude-code, codex, aider, swe-agent, terminus, oracle and others.

04 What to expect

Tasks: 60 — 20 worlds x 3 budget regimes
First run per world: Downloads ~17 GB
Full sweep: ~345 GB unique, each world once rather than once per task
Disk needed: 345 GB plus Docker overhead
Scoring: CardioSeek rubric v0.9, 0-100, normalised to 0-1 for Harbor

Results land in jobs/. Each trial writes reward.txt and score.json with the full component breakdown: target identification, causal confidence, discrimination, direction of effect, phenotype construction, safety penalty.

05 The task

The agent gets a biobank of 54,000 synthetic participants in /app/data: proteomics, transcriptomics, metabolomics, genotypes, EHR, ECG, coronary CT, survival and raw cine-MRI. No phenotype column.

It must derive a cardiac phenotype from the imaging, identify which proteins causally drive disease against ten planted trap mechanisms, and say which direction a drug should move each. It can buy experiments within its budget.

Episode

request-experiment --kind knockdown --protein PROT_0123          # $400k
request-experiment --kind cell_perturbation --protein PROT_0123  # $150k
request-experiment --balance

Output goes to /app/results/submission.json and /app/results/phenotype.csv.

06 Links

Harbor dataset: hub.harborframework.com/datasets/drugtargetbench/drugtargetbench
Hugging Face data: huggingface.co/datasets/sammargolis/drugtargetbench-assets
Container image: github.com/users/sammargolis/packages/container/package/drugtargetbench-base
Source: github.com/sammargolis/DrugTargetBench/
Example task: hub.harborframework.com/tasks/drugtargetbench/hard-02-full-program

07 Citation

Cite DrugTargetBench, and ACDC if you use the imaging.

O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, et al. “Deep Learning Techniques for Automatic MRI Cardiac Multi-structures Segmentation and Diagnosis: Is the Problem Solved?” IEEE TMI 37(11):2514–2525, 2018.

Status
No full 17 GB trial has run end to end. Every component is verified: the pinned revision resolves, checksums match, the experiment service loads, the verifier scores 43.33 to reward 0.4333, and all 60 tasks download clean from the registry. Bulk materialisation inside a Harbor-managed build has not been exercised. Run one complete task before pointing anyone else at it.