We moved business rules, tools, and prompts into a Harness so that every operational change would not require retraining. Then we met a second problem: the compact model did not always follow the updated runtime.
Large models could often interpret new configurations, but were too slow. Small models were fast, yet fixed-configuration fine-tuning encouraged them to memorize tool names, Skill identifiers, and prompt wording.
Harness-Aware Training (HAT) addresses that mismatch: the environment keeps changing after deployment, while training usually exposes the model to one version.
Constraints of livestreaming
A digital-avatar host answers product questions, handles casual conversation, and invokes tools during a live stream. Unlike a private chat assistant, it may speak to the entire room. A wrong price, unsupported product claim, or invented promise can therefore affect many viewers at once. Each request must combine the viewer message with product information, live-room state, and dialogue history, then produce a useful reply within a tight latency budget.
The runtime has four editable parts. Skills contain strategies, tool guidance, and reply rules. The System Prompt Pipeline combines them with product specifications, retrieved FAQs, and room state. Hooks check inputs, arguments, and output formats, requesting correction or retry when needed. The tool registry connects product, inventory, and commerce services. Each has version history separate from the weights, allowing a rule or interface update to be evaluated in replay before deciding whether another training run is needed.
In single-comment mode, the runtime handles one viewer message independently. In multi-comments mode, it aggregates related messages before responding. An available internal traffic summary groups the main viewer intents as shown in Table 1.
The Harness exposes built-in interfaces for retrieval, commerce operations, and flow control. Additional marketing interfaces are discovered dynamically from an external service through the Model Context Protocol (MCP). Table 11 groups the release inventory by responsibility.
Each Skill is a versioned Markdown module with YAML metadata and behavioral directives. The runtime requires at least one Reply Skill before a substantive final response. Strategy Skills are optional and can be composed when an interaction requires a specialized tool chain. Table 12 records the release inventory. Module names and versions are reconciled against the same manifest before promotion.
A request is normalized into product, FAQ, room, and dialogue context. The prompt pipeline assembles the active instructions; a bounded agent loop loads Skills and invokes tools; Hooks check parameters, required Skills, format, and factual risks. The runtime then strips internal reasoning from the spoken reply while retaining the structured trajectory for inspection.
Harness design and iteration
Harness Evolution edits the runtime with model weights fixed. The model groups failures, proposes causes, and drafts Skill, prompt, Hook, or tool changes. A developer reviews the proposal and evaluation, then decides whether to publish, revise, roll back, or stop. Parts of diagnosis can be automated; these decisions remain under human review.
Evolution 1 raises Accuracy from 82.40 to 92.13 while Effectiveness falls to 84.16. Evolution 2 reaches 92.55 and 92.75. Long-tail fixes in Evolutions 3–4 regress one or both metrics, so we retain Evolution 2. Edits need comparison across the full development set because a fix can hurt another category. The selected Harness revision is kept with its model checkpoint, while the judging agent’s own editable Harness is calibrated separately.
Harness Evolution cycles through diagnosis, human review, editing, evaluation, and regression checks. Models cluster failures and propose changes; developers decide whether to publish, revise, roll back, or stop. When remaining failures are isolated long-tail cases, another rule may cause more regressions than it fixes.
Harness-state augmentation
In production,
Our training objective is twofold: (1) improve policy effectiveness when deployment draws a new harness
A model trained on one Harness can memorize Skill names or schema strings. HSA varies these cues while preserving the task: Skills are renamed, descriptions rewritten, and plausible distractors added. Routing then requires more attention to the current functional description. The goal is to preserve behavior when production edits change how a rule is expressed.
We perturb the harness along five dimensions, each targeting a distinct aspect of surface-form dependence:
Skill Identifier: We expand the skill set with synthetic plausible skills and noise skills, randomly mask subsets of existing skills, rename skills, and rewrite skill descriptions by a strong language model. These changes reduce the usefulness of a fixed skill identifier.
Skill Content: Within each skill, we paraphrase individual rules, randomly mask a subset, and reorder them. This variation is intended to reduce reliance on rule position and exact wording.
Tool Definition: We rename tools, rewrite descriptions. This transformation makes name matching less reliable while preserving the tool’s function.
System Prompt: We reorder top-level instruction blocks and items within blocks, and perturb numeric constraints (e.g., response length limits, maximum interaction rounds) within operationally valid ranges.
Hook: We design controlled variants of retry behavior, message structure, and hook-triggered modifications to simulate the changes that the model may encounter after hooks are updated in production.
Three-stage training
HSA-SFT: task training
Start from real live-room requests containing viewer messages, room state, and product context. HSA creates additional task-preserving Harness variants. A strong teacher generates candidate trajectories under both the original and augmented states, so supervision spans several ways of expressing the same operational rules.
Quality filtering checks Accuracy and Effectiveness before a trajectory reaches the student. The supervised corpus therefore teaches both how to act in the task and how to interpret the current configuration, rather than associating one tool name with one fixed behavior.
Hook retries need special treatment. Most demonstrations remove intermediate failures and retain the final valid behavior, avoiding a policy that routinely makes an avoidable error before fixing it. A smaller set keeps complete retry histories so the student can still understand a Hook message and recover when something does go wrong.
General OPD: recovering general capabilities
General OPD follows HSA-SFT to recover general instruction following. Its teacher is the pre-SFT base model, and inputs come from Tulu3. The student generates replies and minimizes KL divergence toward that base distribution, recovering capabilities that domain training may have weakened.
HSA-RL: interaction training
Starting from HSA-SFT plus General OPD, the policy rolls out complete interactions in the simulator. Each rollout can involve Skill selection, several tool calls, result interpretation, a retry, and the final response. This stage optimizes the student’s own behavior under changed Harness states rather than asking it to imitate another fixed collection of teacher traces.
Within the simulated environment, we optimize the policy based on Group Relative Policy Optimization (GRPO) . For each prompt,
Accuracy: whether the final response contains factually correct information and follows the instructions defined in prompts or skills. Applied on reply state;
Effectiveness: whether the response adequately addresses the user’s intent with appropriate detail and actionability. Applied on reply state;
Tool Rationality: whether the tool invocation chain is logically sound and efficiently structured (no redundant calls, correct parameter usage), this reward can prevent model from overfitting to specific tools. Applied on tool state;
Skill Selection: whether the model correctly identifies the user’s intent and the current context to load the appropriate Skills, this reward can prevent model from overfitting to specific skills. Applied on tool state.
Tool and reply segments form separate groups for advantage computation. Tool rationality and Skill selection assess the interaction path; Accuracy and Effectiveness assess the final reply. The reported implementation combines GRPO with reward-dimension normalization and sequence-level importance sampling. An auxiliary length term discourages unnecessary reasoning without replacing the task rewards.
Simulation and failure recovery
Demonstrations cover the teacher’s successful paths, but a deployed student also encounters its own mistakes: a wrong Skill, malformed arguments, a tool timeout, or a Hook requesting revision. HSA-RL lets the student experience these consequences and practice recovery instead of learning only the final successful trajectory.
The simulator implements the production control flow rather than returning a single static answer. Its four parts are input sampling, Harness scheduling, sandboxed tool execution, and configuration augmentation.
Inputs come from processed live-stream scenarios, including product inquiries, casual chat, purchase intent, after-sales questions, and messages containing several intents. Each is paired with product, dialogue, and live-room context.
Scheduling handles Skill routing, parameter and response checks, required Skill loading, retries, and the bounded multi-turn loop. The sequence of state transitions is retained for training and inspection.
Sandboxed tool executors replay product details, prices, and inventory from controlled data. Fault injection adds timeouts, malformed results, or rejected operations so recovery is represented in training.
The same augmented Harness state governs both the policy and its environment. Renaming an interface without updating the executor would create a broken task, not a useful robustness example.
Evaluation design
Each evaluation set tests a different part of the system. Live-Stream QA checks reply quality on real interactions; Harness-Variant QA changes their runtime context. A separate real-traffic cohort calibrates the Judge against human labels. Synthetic Live-Stream QA expands tool and prompt variation, while IFEval checks general instruction following. Sources and purposes remain separate when comparing checkpoints, so a business gain cannot hide lost instruction following or difficulty with an updated configuration.
The two real-traffic QA sets score Accuracy and Effectiveness separately. Binary Accuracy covers factual errors, unsupported product claims, false promises, and capability overreach. Effectiveness uses 0, 0.5, or 1 for relevance and usefulness. AVG is the per-example arithmetic mean before rounding. Both components are retained to distinguish avoiding errors from helping the viewer.
For IFEval, we use its official evaluator and report prompt-level and instruction-level accuracy (abbreviated as IFE-P and IFE-I hereafter). IFE-P measures whether the model satisfies the full prompt-level instruction set, while IFE-I measures whether individual instructions are followed.
The data-generation teacher, RL reward judge, final evaluation judge, and human calibration cohort have distinct roles. Their models, prompts, evidence access, and data overlap need to be recorded. Shared model families or rubric conventions can still create correlated errors, so automatic scores are complemented by human checks rather than treated as an independent ground truth.
| Measure | Base | HAT |
|---|---|---|
| Live-Stream QA | 80.3 | 94.8 |
| Harness-Variant QA | 75.4 | 94.6 |
HAT retains 83.5 on IFEval Prompt, while Fixed-Harness SFT loses 7.7 points from the base. These comparisons apply to the reported evaluation sets, rather than every agent task.
Training-stage ablations
Fixed-Harness SFT improves business performance but damages general instruction following. In the reported pathway, IFEval Prompt drops by 7.7 points and synthetic prompt robustness also declines. The task can therefore look better on its original configuration while the model becomes less capable of obeying the next configuration.
HSA-SFT and General OPD address different parts of this problem. Variation during SFT prevents much of the surface-form dependence; OPD recovers capability that was still lost. General OPD restores 8.5 IFEval points after fixed-Harness SFT but only 0.2 after HSA-SFT, which had already preserved more of the base model’s general ability.
RL raises the business metrics in both branches, but a fixed training environment can again sacrifice generalization. The stronger result comes from applying HSA during RL as well as SFT. This is a stage-wise ablation, not evidence that adding any augmentation to any task will necessarily help.
Testing Harness changes
The Harness-edit test uses 1,500 new production cases: 500 development and 1,000 held out. Each retains comments, history, room context, product evidence, tool traces, and replies. An automatic check identifies wording unsuitable for a conversational live host, providing cases for diagnosis and editing.
Only the 500 development cases guide diagnosis. We revise the system prompt and Skills with weights fixed, freeze the edits, then apply the same changes to Fixed-Harness SFT and HAT. The 1,000 cases remain unseen until evaluation, avoiding edits fitted to the test examples.
Held-out errors fall by 18.1% for Fixed-Harness SFT and 51.7% for HAT, each against its own pre-edit count. Both benefit, with a larger reduction for HAT. The report’s examples illustrate whether a model follows the new wording or continues exposing internal uncertainty; the aggregate test supplies the measured comparison, limited to this edit protocol.
The 500 development cases guide edits; the 1,000 held-out cases evaluate them. Error reductions of 51.7% and 18.1% support adaptation within this protocol. New tools or business domains still require separate evaluation.
Reasoning compression and MTP
Figure 6 traces the auxiliary CoT-length term in the selected HSA-RL run. From the first to the last 100-step window, mean extracted length decreases from 290.6 to 171.2 tokens for tool-call CoT and from 93.6 to 37.0 for final-reply CoT. The mean logged penalty score
Table 5 reports deployment inference performance in the application’s low-concurrency operating range. Every row uses the same fixed 100-case sample and complete Agent workflow; the API rows traverse vendor-managed serving routes, while all H20 rows use one local GPU.
At concurrency one and two, the selected MTP configuration reaches P95 8.114 and 9.047 seconds respectively, with every measured request below 15 seconds in the fixed replay set. The comparison uses complete agent executions, so tool calls, retries, and generation are included rather than timing one isolated decode.
The task policy has moved away from its base-model distribution. We initialize a grafted MTP head and adapt it using sequences produced by HSA-RL rollouts. The standalone head update uses multi-token prediction supervision from those sequences; it is not a second RL objective on the draft head.
The initially higher MTP loss moves toward the native-head training range as adaptation proceeds. That is an optimization diagnostic, not sufficient evidence of serving speed. We also measure draft acceptance, accepted length, response quality, and complete-request latency before selecting a deployment configuration.
In the C=1–2 operating range, the selected checkpoint’s decode throughput improves by 1.69× and 1.52× with MTP. The gains fall to 1.27× and 1.19× under C=4–8 stress. Quality checks use the same checkpoint with MTP switched off and on, separating the decoder change from model training gains.
The selected single-H20, MTP-enabled deployment reaches P50 3.407 seconds and P95 8.114 seconds at concurrency one. Low-concurrency replay is the relevant operating regime in this report. The acceleration declines under the higher-concurrency stress tests; it should not be extrapolated to every workload or serving engine.
Production results and limitations
Of 100 blind-test pairs, 64 are ties, 35 favor Harness, and one favors ReAct. The 97.2% win rate uses the 36 non-ties as its denominator. Annotators inspect full traces alongside replies to assess input understanding, evidence handling, and scenario-appropriate behavior.
The production A/B reports approximately 0.91% higher UV-normalized item-page views than ReAct. This engagement metric does not measure purchase conversion. Offline evaluation, the blind test, and the online experiment use different baselines and answer different questions.
Each result needs its model checkpoint, Harness revision, tool registry, and serving configuration. HAT trains the model on runtime changes from this business. Held-out evaluation and deployment replay then test whether expressing those changes in the Harness and exposing the model to them improves the system.