MAGAR-Bench opens 1,000 3D assembly challenges for multimodal agents
The new public export includes prompts and image references for assembly decisions, GUI actions and minimal rework. Reference answers and the scorer remain private.
MAGAR-Bench has posted 1,000 test inputs aimed at a demanding agent workflow: understanding a mechanical assembly, choosing the next GUI action and repairing it while moving as few parts as possible. The November 5 public export on Hugging Face includes a JSONL table and an archive described as containing 1,140 PNG files across six assembly families.
The English release documentation divides the inputs into 238 Assembly Decision tasks, 662 Atomic Action tasks and 100 Minimal Rework tasks. AITrending’s inspection of the downloaded table matched those row counts and found 1,140 distinct referenced image paths. That check confirms the table structure, rather than the quality of a model’s answers or the contents and rights of every archived image.
Three different demands on an agent
Assembly Decision requires a structured choice of operation, involved parts and next subgoal. Atomic Action asks for one complete native GUI primitive, such as a click or drag, rather than a future action sequence. Minimal Rework asks an agent to choose a recovery route and identify the distinct parts it would move.
The last task has a deliberately bounded definition: the reference minimizes moved parts within a validated candidate recovery set. It does not claim a globally shortest solution across every possible GUI action sequence. Matching route identifiers also does not execute the recovery.
Public inputs, private scoring
The package includes neither reference answers nor training records. Its integrity checker validates the release; it is not a benchmark scorer. The card says answer-based scores require separately maintained private references and an evaluator, so this download alone cannot produce a comparable scored leaderboard result.
The documented loader requires Python 3.11 or newer and no third-party packages. Loading the JSONL with Hugging Face Datasets reads the table only; the image archive must be extracted separately. The public export preserves the frozen test prompts, row order and image bytes while reducing metadata. It is not a newly constructed test split.
The release also has an unresolved reuse boundary. Its English asset document says image ownership and third-party redistribution clearance still await maintainer attestation. CC BY-NC 4.0 applies only where the maintainers own or can redistribute the material; helper scripts use Apache-2.0. A verified author list and public identifier for the associated manuscript are not supplied. For researchers, the immediate value is inspectable task inputs and explicit contracts, with scoring access and image-rights clearance still to resolve.
