A team can produce a wrong answer even when one member did useful work. A search subagent might find the missing evidence, only for the coordinator to discard its report or the answerer to misread it. Giving the entire trajectory the same failure signal leaves that searcher with little guidance about what to change.
MultiAgent-OPD gives coordination and answering outcome rewards, while a teacher scores search tokens on the student’s own trajectories. Training first produces a search teacher, then jointly updates the hierarchy with these role-specific signals.
Feedback for search subagents
We study a Leader–Searcher–Answerer team. The Leader decomposes a question and delegates subtasks. Searchers retrieve passages and write reports. The Leader can request another wave of searches before selecting reports for the Answerer. Compared with a fixed search-then-answer workflow, this lets the system revise its division of work as evidence arrives.
The flexibility also increases the distance between a search decision and the final answer. A report’s value depends on later selection and reasoning. Different Searchers in one trajectory can receive the same outcome-based advantage even if one finds decisive evidence and another repeats an unhelpful query.
When all trajectories for a question succeed or all fail, their outcome rewards have no group-relative variation. Search queries and reports may still differ, so search needs a signal that can distinguish these processes even when final correctness cannot.
A report-level LLM judge provides another local signal, but its reliability depends on its own assessment. One report score also covers a sequence of earlier decisions. Our alternative is to first train a search-and-summary teacher, then use its probabilities to guide the tokens generated by the current student.
Teacher training
Stage I uses a Searcher–Answerer cascade. The Searcher receives the original question, queries a fixed retriever, and writes a report that goes directly to the Answerer. The connection between retrieved evidence and the answer is shorter than in the full hierarchy, making outcome-based learning a useful starting point.
Both roles share parameters. During training, each answering request produces five candidates. Each candidate receives a normalized exact-match score, and the workflow outcome is one if any candidate is correct. The Searcher receives mostly outcome reward, with a small behavior bonus.
The behavior bonus is 0.5 for producing a summary after one search, one after two or more searches, and zero otherwise. Invalid action formats receive a total reward of −1. Correctness remains the main signal. The behavior term encourages searching and reporting, but does not directly establish whether the retrieved evidence is relevant.
The resulting policy has two uses. Evaluated in its original cascade, it is the C-MAS + RL baseline. Frozen, it becomes the Stage-II teacher. We therefore obtain guidance from a policy trained through the same search interface, rather than ask a separate general-purpose model to grade reports.
On-policy student distillation
The student starts from the base instruction model rather than the teacher’s weights. Its Leader changes task decomposition during training, exposing Searchers to new subtasks and evidence. Feedback must follow these states instead of remaining attached to fixed teacher demonstrations.
The student generates the team trajectory first. The frozen teacher reads the same Searcher prompt, subtask, passages, and generated prefix, then scores the sampled tokens. It neither substitutes its own trajectory nor generates a reference report to copy.
In this on-policy distillation, the current student supplies histories and the teacher supplies preferences at those histories. Changes in delegation produce new states to score. Both stages give Searchers the same question-to-report interface, allowing the cascade-trained teacher to guide the hierarchy.
Token-level advantages
The distillation objective uses reverse KL from the student to the teacher. Its mask retains only Searcher-generated tokens. Passages supplied by the retriever and padding are excluded.
For the PPO update, the sampling policy is π_old. A Searcher token receives the difference between its teacher and sampling-policy log probabilities, multiplied by β. The reported setting uses β = 0.01.
When the teacher gives a sampled token higher probability than the sampling policy does, its advantage is positive and the update encourages it. The signal covers generated search text and summaries rather than assigning one score to the complete report. It can remain nonzero on failed trajectories and on questions whose sampled outcomes are all identical.
The signal expresses the teacher’s local preference, with the teacher’s own limitations. It is not a causal estimate of a token’s contribution to the final answer; OPD supplies finer process supervision without solving that full attribution problem.
Leader and Answerer rewards
The Leader still learns mainly from team correctness, with a bounded bonus for search activity. The activity count measures environment steps containing at least one Searcher search action. It is capped at six; it is not simply the number of subagents launched.
The Answerer learns from candidate correctness. Leader rewards are normalized across team trajectories for the same question. Answerer candidate rewards are normalized within their parent workflow. These are different comparison groups, even though both use group-relative advantages.
We assign each generated token its role’s advantage and update the shared student with the PPO clipped objective and an entropy bonus. Leader and Answerer rewards also include a KL penalty toward the teacher with coefficient 10⁻³. Searcher advantages already come from OPD, so their policy loss receives no additional KL penalty.
Only the student is needed at inference. Different role prompts select its three roles; the teacher is absent. Sharing parameters also means that an update from one role changes the model used by the others. Role-specific losses do not create independently isolated models.
Evaluation setup
All systems use Qwen3-4B-Instruct-2507 and the same E5 retriever, returning three passages per query. Training uses 79,168 NQ and 90,447 HotpotQA examples, with veRL and SGLang, batch size 32, five trajectories per question, learning rate 10⁻⁶, and five PPO minibatch updates.
The cascade permits five environment steps; the hierarchy permits twelve and up to five Leader delegation rounds. Comparisons between those architectures include different execution budgets. Prompted Critic and MultiAgent-OPD instead share the hierarchical setup and Leader objective, with the Searcher learning signal as their main difference.
The fixed test set contains 5,000 questions across NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, and Bamboogle. Each question gets four independent workflows. At evaluation, an answering request generates one candidate, not the five used during training.
- avg@4 averages correctness over the four runs, estimating single-attempt performance.
- pass@4 counts questions answered correctly at least once. It is a coverage upper bound without a procedure for selecting the correct answer.
- Macro gives each of the seven datasets equal weight rather than pooling every question into one average.
Scoring uses normalized exact match. Missing or malformed answers count as wrong; unsuccessful workflows are retained. Normalized question-overlap checks separate training, development, and test questions, but do not establish the absence of paraphrases or pretraining contamination.
Main results
| System | Macro avg@4 | Macro pass@4 |
|---|---|---|
| C-MAS base | 25.09% | 34.02% |
| C-MAS + RL | 36.07% | 45.97% |
| H-MAS base | 28.60% | 39.32% |
| Prompted Critic | 34.62% | 45.96% |
| MultiAgent-OPD | 38.68% | 48.73% |
Before task training, the hierarchy outperforms the cascade. After training, C-MAS + RL scores above Prompted Critic. The structural advantage therefore does not automatically become a training gain; Searcher supervision affects the result.
MultiAgent-OPD improves macro avg@4 by 4.06 percentage points over Prompted Critic and 2.61 over C-MAS + RL. The paired 95% intervals are [2.98, 5.12] and [1.49, 3.73]. These intervals resample test questions for fixed retained policies. They do not measure variability across independent training runs.
Its avg@4 exceeds Prompted Critic on all seven datasets, but does not exceed the cascade on every dataset. NQ is 42.3% versus 43.7% for C-MAS + RL, while Bamboogle rises from 39.4% to 46.8%. The aggregate gain should not become a claim that more complicated teams help every kind of question.
The paper also compares five RL-based search systems under the same backbone and protocol. The closest, Tree-GRPO, reaches 37.37% macro avg@4, a gap of 1.31 percentage points. Those external comparisons have aggregate scores rather than the question-level paired intervals available for the main systems, so they remain descriptive comparisons under this protocol.
Consistency improves too. The fraction answered correctly in all four runs is 29.27% for MultiAgent-OPD, 27.05% for C-MAS + RL, and 23.24% for Prompted Critic. Both coverage and repeated success matter, although neither supplies an answer-selection method.
Ablation: search collapse
| Searcher training configuration | Macro avg@4 | Macro pass@4 |
|---|---|---|
| Full method: task-trained teacher OPD | 38.68% | 48.73% |
| Outcome-based Searcher RL | 17.78% | 22.79% |
| Base-model teacher | 29.50% | 40.86% |
| No direct Searcher supervision | 29.82% | 39.86% |
The outcome-reward variant records no searches in 20,000 test workflows, although 99.51% contain subtasks and reports and 99.03% produce a final answer. Completion remains high while retrieval disappears.
Between development steps 300 and 400, workflows without a search rise from 0.5% to 100%, while EM falls from 26.0% to 21.8%. Completion rate misses the retrieval failure, which becomes visible only by checking intermediate behavior.
Outcome rewards did not sustain retrieval in this configuration. The variants retain the main settings but use different training batches, and the comparison does not isolate every training choice responsible for the collapse.
Replacing the task-trained teacher while retaining OPD lowers performance, as does removing direct Searcher losses. The teacher and the direct supervision both matter in these runs. Removing those losses does not freeze Searchers, since the other roles still update their shared parameters.
Retrieval cost
| System | Search actions per workflow | Recorded input tokens |
|---|---|---|
| C-MAS + RL | 2.86 | 3,886 |
| Prompted Critic | 4.45 | 6,786 |
| MultiAgent-OPD | 5.39 | 9,172 |
Relative to Prompted Critic, search actions increase 21.4% and input tokens increase 35.2%, while output tokens fall from 885 to 697. Removing the teacher at inference does not make the improvement free: the learned student changes how much work the team performs.
Recorded tokens count histories repeatedly across calls. They do not measure end-to-end latency, billing, or training cost, and the architectures have different step limits. The figure compares correctness with accompanying workload rather than efficiency at equal compute.
The paper also bounds the effect of format failures. If every Prompted Critic format failure became correct, the remaining gap would be 0.33 percentage points, with an interval crossing zero. This assumed repair bound is not a measured gain; it shows why the entire score difference cannot be attributed to evidence reasoning.
A two-hop QA example
As an interface example, take “Where was the director of a particular film born?” The Leader can first request the director, then assign a birthplace search using that report. With some facts already known, it can delegate a narrower lookup. This illustrates how earlier evidence shapes the next task; it is not another experiment.
The Searcher and teacher see the same subtask and local history, without hindsight access to the correct answer. If the team identifies the wrong person upstream, local feedback may not repair that delegation. Outcome rewards still train coordination and evidence use.
If a Searcher finds the right director but the Leader omits its report, the team can still fail. Local teacher feedback can retain a useful distinction in that failed trajectory, while final correctness remains the team objective.
Groups within a training batch
Training samples five team trajectories per question and five candidates per answering request. These are different comparisons. The first exposes alternative collaboration paths for one question; the second compares answers given a particular workflow’s reports. Mixing all answer candidates across teams for normalization would change the Answerer objective defined here.
Searcher feedback uses teacher and sampling-policy probabilities at each local history. A single trajectory score copied across search tokens would lose that distinction. Retaining role, workflow, candidate, and call ownership lets the trainer assign each advantage to the right tokens.
The frozen teacher supplies preferences, the old student supplies samples, and the updating student supplies the PPO probability ratio. Student states change even while the teacher stays fixed. Scoring those new states is part of on-policy distillation; a static cache of teacher demonstrations implements a different procedure.
Completion and evidence use
The main scorer extracts the answer tag from the last Answerer output. Without a valid answer, the workflow scores zero. All failures stay in the denominator, while formatting becomes part of the task score. A separate check uses preselected flexible extraction rules on saved outputs to see whether formatting concealed correct content. It does not use reference answers to select a candidate.
Flexible extraction re-reads saved content; the perfect-repair bound assumes every failure becomes correct. Neither changes the primary score. The measured extraction result and assumed upper bound need separate reporting.
Search counts also need evidence checks. Recording calls, passages, reports, and answers can expose completion without retrieval. Diagnosis then needs passage relevance, facts retained in reports, and Leader report selection. Workload alone does not measure evidence quality.
These checks help locate regressions at an interface. A flat average score can reflect different changes in format, query generation, or evidence selection. Identify the changed behavior before adjusting its reward.
Aligning trajectories and feedback
A reproduction should inspect roles, local histories, generated tokens, and masks along complete trajectories. A Searcher-written report receives supervision; the same report becomes an observation when received by the Leader. Teacher scoring must use the evidence and prefix available to the student at that decision.
Next I would verify normalization groups. Leaders compare teams, Answerers compare candidates within a workflow, and Searchers receive local teacher guidance. The sampling student, the current update policy, and the frozen teacher also have different jobs. Treating all their probabilities as one “current model probability” obscures the actual update.
A task-trained teacher supplies finer feedback for repeated delegation and search than a final score alone. Teacher preferences remain, and a larger hierarchy may not justify its cost. Adoption therefore depends on correctness, sustained retrieval, completion, and the workload of the trained policy.