(4/n) Package Arena-Hard Task Variants

[kargibora]

Summary

This PR migrates Arena-Hard v0.1 and v2.0 to the declarative task registry introduced in the previous PR.

Arena-Hard configuration was previously distributed across dataset utilities, revision maps, baseline constants, prompt
handling, and pairwise runner conditions. The two versions also share most of their configuration but require different
dataset variants and baseline policies.

The task definitions now provide a single source of truth for both versions.

Task structure

The definitions are organized as:

judgearena/tasks/definitions/arena_hard/
├── _base.yaml
├── arena-hard-v0.1.yaml
└── arena-hard-v2.0.yaml

_base.yaml contains the configuration shared by both versions:

• Dataset repository and pinned revision
• Arena-Hard dataset adapter
• Canonical instruction fields
• Pairwise runner
• Generation mode
• Judge prompt and parser
• Swap policy
• Scoring configuration
• Reference implementation

Each public task YAML extends the base and defines only its version-specific differences.

Version-specific behavior

Arena-Hard v0.1 declares:

• The arena-hard-v0.1 dataset variant
• The official gpt-4-0314 default baseline

Arena-Hard v2.0 declares:

• The arena-hard-v2.0 dataset variant
• The category field used for baseline routing
• Category-specific official baselines for hard prompts, coding, mathematics, and creative writing

Both versions allow users to provide a runtime baseline override when they want to compare against another model.

Dataset adapter

This PR adds an arena_hard dataset adapter responsible for:

Task specification
-> download pinned Arena-Hard JSONL files
-> normalize questions into instruction rows
-> normalize official answers into model-output rows
-> return canonical data to the pairwise runner

The adapter handles the different answer formats found in the upstream files while keeping those details out of the
generic pairwise runner.

A dataset-adapter registry now connects the adapter ID declared in YAML to its Python implementation.

Removed duplication

Arena-Hard-specific information is removed from legacy baseline, revision, prompt, and dataset dispatch mappings. The
task definition now owns this information, while the Python adapter only owns loading and normalization behavior.

This demonstrates how related benchmark versions can share a base definition while keeping their differences explicit
and easy to review.

Notes

• As can be seen from this PR, now wiring datasets are much easier.
• All default values/revisions/definitions of a task stays on a single file. The file spec can always be changed without affecting others, and the value can always propogate in the pipeline without any difficulty
• This also unifies the logic: pairwise runner is used for both dataset, but they can be parsed differently. If required new runner is defined. This keeps all the logic inherent and easy to understand, and all customazibility to not affect unrelated files. (any change in arena-hard should not affect functions in evaluations.py, instead, it should affect arena-hard specific functions only which should be resolved before reaching evaluations.py)

[kargibora]

Note: we have this because we are adding tasks one-by-one and don't want to break the old pipeline.

[kargibora]

adapter handles how to load the dataset now. So we dont need a specific functions as before, different way of loading should be only implemented to adapter classes so executing the pipeline flow is seperated from how to load it in a graceful way.

[kargibora]

As we want to add new tasks, if it requires a special download function or load function, we will add it here. This will normalize all the instances in the pipeline.

[kargibora]

Similarly we can write this as a dict object. However writing like this also shows what we expect as data type for its keys. Compared to previous TaskRegistry, I think we should let it be like this.