LiveThinking: efficient reasoning for real-time QA

A large reasoning model can answer product questions well—and finish after the conversation has moved on.

LiveThinking addresses this in two stages. Rejection Sampling Fine-Tuning (RFT) transfers capability to a compact model. GRPO then makes its reasoning shorter without sacrificing correctness or helpfulness.

A short comment may depend on the current product, an earlier turn, and retrieved documents. Reasoning helps combine that evidence, but its duration must fit a live conversation. The internal trace also needs to stay separate from the short response spoken by the avatar.

LiveThinking first transfers product-answering behavior to a compact student, then uses RL to reduce unnecessary reasoning while retaining response quality. The budget is chosen for real-time commerce; other tasks may need different reasoning lengths.

Correctness and helpfulness

Responses are evaluated along two dimensions using LLM-as-a-Judge:

Correctness (C)

Measures factual alignment with product knowledge:

C(D_{t},Q_{t},H_{t},R_{t})=\begin{cases}1&\text{if }R_{t}\text{ accurately reflects }D_{t},\\ 0&\text{otherwise.}\end{cases}

A correct reply must have every claim supported by documents $D_{t}$ , remain consistent with history $H_{t}$ , and directly answer question $Q_{t}$ . This excludes both factual errors and off-topic replies.

Helpfulness (H)

Optimizing correctness alone can encourage minimal, cautious replies that avoid errors without giving the viewer enough help. We therefore score helpfulness separately.

To counteract this tendency, we introduce Helpfulness as a complementary metric. It incentivizes the model to provide richer, more valuable information, striking a balance between accuracy and utility:

H(Q_{t},H_{t},R_{t})=\begin{cases}1&\text{if }R_{t}\text{ proactively assists the user }Q_{t},\\ 0&\text{otherwise.}\end{cases}

A response is considered helpful if, in addition to correctly answering the query, it proactively offers supplementary, relevant information from the documents that anticipates and addresses the user’s implicit needs within the conversational context ( $H_{t}$ ).

LLM Judge validation

We calibrated the Judge against several hundred human-labeled replies before using it in training or data production. Two annotators scored each reply for correctness and helpfulness; disagreements were excluded, and each task’s positive and negative labels were balanced to 1:1. Refining the judging prompt brought agreement to 83% for correctness and 84% for helpfulness. Later automatic scores remain subject to this evaluator’s reliability, including on ambiguous cases.

Two-stage training: RFT distillation followed by GRPO for efficient reasoning.

Trajectory distillation and RL

Our two-stage methodology first distills e-commerce reasoning ability into a lightweight model via RFT. Subsequently, we employ GRPO to compress the model’s reasoning path, optimizing for efficiency while maintaining response quality.

Rejection sampling fine-tuning

To transfer capabilities from the 670B reasoning MoE teacher model (37B activated parameters) to the lightweight student model, we employ a three-stage distillation pipeline formalized as:

Teacher-generated data

We start with the unlabeled commerce corpus $\mathcal{U}=\{(D^{(i)}_{t},Q^{(i)}_{t},H^{(i)}_{t})\}_{i=1}^{N}$ and sample from the teacher $P_{\theta_{\text{teacher}}}$ with temperature. For each input we collect $k$ candidate trajectories, $\{(T^{(i,j)}_{t},R^{(i,j)}_{t})\}_{j=1}^{k}$ , indexed by $i$ . Every record retains its question, evidence, and generated output. That lets us inspect whether a plausible-looking reasoning trace actually ends in a grounded and useful reply before offering it to the student as a demonstration.

LLM Judge filtering

DeepSeek-R1 filters teacher trajectories using the correctness and helpfulness criteria above. Only examples passing both checks enter student training.

For input $i$ , we inspect $k$ candidates. If trajectory $(T_{t}^{(i,j)},R_{t}^{(i,j)})$ passes both checks, one valid example enters $\mathcal{D}_{\text{distill}}$ . If all $k$ fail, the input is excluded from this training round. The resulting set is:

\mathcal{D}_{\text{distill}}\subseteq\bigcup_{i=1}^{N}\bigcup_{j=1}^{k}\left\{\tau^{(i,j)}\mid C(\tau^{(i,j)})=1\land H(\tau^{(i,j)})=1\right\},

where $\tau^{(i,j)}=(D_{t}^{(i)},Q_{t}^{(i)},H_{t}^{(i)},T_{t}^{(i,j)},R_{t}^{(i,j)})$ , and the judgment functions for correctness $C(\cdot)$ and helpfulness $H(\cdot)$ are applied accordingly. The subset notation ( $\subseteq$ ) reflects our strategy of selecting at most one trajectory per original input instance.

MoE student fine-tuning

Since our student model $\theta^{\text{student}}$ is a Mixture of Experts (MoE) model, we train it by minimizing a composite loss function. This objective includes the primary sequence-to-sequence task loss and an auxiliary loss for load balancing. The total optimization objective is:

\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{SFT}}+\lambda_{\text{aux}}\cdot\mathcal{L}_{\text{aux}.}

The main task loss $\mathcal{L}_{\text{SFT}}$ trains the model to reconstruct teacher outputs:

\mathcal{L}_{\text{SFT}}=-\mathbb{E}_{(D,Q,H,T,R)\sim\mathcal{D}_{\text{distill}}}\left[\sum_{i=1}^{L_{y}}\log P_{\theta^{\text{student}}}\left(y_{i}\mid\mathbf{x},y_{<i}\right)\right],

where the input $\mathbf{x}=[D;Q;H]$ is the concatenated context, and the target sequence $y=[T;R]$ encodes the thought $T$ and response $R$ . The term $\mathcal{L}_{\text{aux}}$ represents the MoE auxiliary loss, which promotes balanced routing of tokens across experts. Its contribution is scaled by the hyperparameter $\lambda_{\text{aux}}$ , ensures stable training and effective expert specialization.

Reasoning quality and length

While RFT (as detailed in §4.1) transfers knowledge, it also inherits the teacher’s verbose reasoning. We address this using RL to optimize for reasoning efficiency, correctness, and helpfulness.

RL formulation

We frame response generation as a Markov Decision Process (MDP). The state $s_{t}$ is the concatenated input context $[D_{t};Q_{t};H_{t}]$ , and the action $a_{t}$ is a token sequence $[T_{t};R_{t}]$ generated autoregressively by the policy $\pi_{\theta}$ , a student model with parameters $\theta$ . An episode consists of the complete generation of an action $a_{t}$ and terminates upon producing an end-of-sequence (EOS) token.

Reward design

Now we need to shorten generation without losing the behavior we just taught. We use three rewards together: length tracks generation cost, correctness checks product facts, and helpfulness checks whether the reply meets the viewer’s need. Each addresses a different shortcut. A policy that says little can avoid factual errors; a policy that answers quickly can also stop before checking the evidence. Keeping all three signals in the loop makes those trade-offs visible during training rather than only after deployment.

Length Reward ( $r_{\mathrm{length}}$ ): Optimizes reasoning efficiency relative to reference model outputs:

\begin{aligned} \displaystyle L_{\mathrm{upper}}&\displaystyle=\lambda_{\mathrm{upper}}L_{\mathrm{ref}},\quad L_{\mathrm{lower}}=\lambda_{\mathrm{lower}}L_{\mathrm{ref},}\\[6pt] \displaystyle d&\displaystyle=\begin{cases}\max(L_{\mathrm{lower}}-L_{\mathrm{policy}},0)&L_{\mathrm{policy}}<L_{\mathrm{lower},}\\ \max(L_{\mathrm{policy}}-L_{\mathrm{upper}},0)&L_{\mathrm{policy}}>L_{\mathrm{upper},}\\ 0&\text{otherwise,}\end{cases}\\[6pt] \displaystyle r_{\mathrm{length}}&\displaystyle=\max\left(0,1-\frac{d}{\epsilon}\right). \end{aligned}

The length reward $r_{\mathrm{length}}$ compares the current output $L_{\mathrm{policy}}$ with an acceptable interval. Its reference length $L_{\mathrm{ref}})$ comes from the student after RFT. We set the bounds to $L_{\mathrm{upper}}=\lambda_{\mathrm{upper}}L_{\mathrm{ref}}$ and $L_{\mathrm{lower}}=\lambda_{\mathrm{lower}}L_{\mathrm{ref}}$ : $\lambda_{\mathrm{upper}}$ adjusts the compression ratio, while $\lambda_{\mathrm{lower}}$ guards against over-compression. The deviation $d$ records how far $L_{\mathrm{policy}}$ falls outside $[L_{\mathrm{lower}},L_{\mathrm{upper}}]$ , and $r_{\mathrm{length}}$ applies the corresponding penalty. The tolerance $\epsilon$ controls how quickly that reward falls after the output crosses a bound. This gives us a budget to tune rather than asking every question to use the fewest possible tokens. A short price lookup and a comparison involving several product constraints need not spend the same amount of reasoning.

The length reward allows an acceptable range instead of demanding the shortest possible output.

Correctness Reward ( $r_{\mathrm{correct}}$ ): Reduces hallucinations by verifying response fidelity to product documents:

r_{\mathrm{correct}}=C(D_{t},Q_{t},H_{t},R_{t})\in\{0,1\},

where 1 indicates $R_{t}$ is fully consistent with $D_{t}$ and answers $Q_{t}$ without fabrication.

Helpfulness Reward ( $r_{\mathrm{helpful}}$ ): Prevents over-conservatism by assessing practical utility:

r_{\mathrm{helpful}}=H(Q_{t},H_{t},R_{t})\in\{0,1\},

where 1 denotes $R_{t}$ provides actionable shopping guidance addressing implicit needs in $H_{t}$ .

Group-relative advantages

We combine the three rewards into $r_{j}$ and normalize the $K$ replies within each group. Advantage $A_{j}$ uses $r_{j}$ , group mean $\mu$ , and standard deviation $\sigma$ , comparing candidates for the same input rather than relying only on absolute reward. The denominator includes $\epsilon_{\mathrm{std}}$ , such as $10^{-8}$ , for stability when scores are close.

\begin{aligned} \displaystyle r_{j}&\displaystyle=w_{c}r_{j}^{\mathrm{correct}}+w_{h}r_{j}^{\mathrm{helpful}}+w_{l}r_{j}^{\mathrm{length}}, \end{aligned}

(1)

\begin{aligned} \displaystyle A_{j}&\displaystyle=\frac{r_{j}-\mu}{\sigma+\epsilon_{\mathrm{std}}},\quad\text{where }\mu,\sigma=\text{mean, stddev}\left(\{r_{i}\}_{i=1}^{K}\right). \end{aligned}

(2)

Intuitively, this mechanism rewards responses that perform better than average within a mini-batch, making training more stable and less sensitive to absolute reward scales.

Policy optimization

The GRPO objective maximizes advantage-regularized returns:

\begin{aligned} \mathcal{L}^{\mathrm{GRPO}}(\theta)&=\frac{1}{K}\sum_{j=1}^{K}\Bigg[\min\Bigg(\frac{\pi_\theta(a_j|s_j)}{\pi_{\theta_{\mathrm{old}}}(a_j|s_j)}A_j,\\[5pt] &\quad\mathrm{clip}\!\left(\frac{\pi_\theta(a_j|s_j)}{\pi_{\theta_{\mathrm{old}}}(a_j|s_j)},1-\eta,1+\eta\right)A_j\Bigg)\\[5pt] &\quad-\beta D_{\text{KL}}\!\left(\pi_{\theta_{\mathrm{old}}}\|\pi_\theta\right)\Bigg]. \end{aligned}

where $\eta=0.2$ is the clip range, $\beta=0.01$ controls KL penalty, and $s_{j}=(D_{t},Q_{t},H_{t})$ , $a_{j}=(T_{t},R_{t})$ . We choose GRPO over standard PPO due to its improved training efficiency and stability, particularly in scenarios with complex, multi-objective rewards.

Data and training setup

Datasets

We evaluate our approach on two distinct datasets, each containing 8,000 training and 400 test samples.

Tblive-E-Commerce QA

This dataset comes from real shopping streams. Each example keeps the viewer question, dialogue history, and structured product information retrieved for its intent. We use it because it preserves two problems we actually have to solve: comments arrive with omissions and conversational noise, while the reply still needs to stay inside the product evidence. Clean, self-contained questions would hide much of that difficulty.

MuSiQue

We also wanted a view beyond commerce, so we added the public multi-hop benchmark MuSiQue. One paragraph is deliberately insufficient to answer its questions; the model has to connect facts across material. We use the distractor setting, with roughly 10 documents per question and mostly irrelevant information. It tests evidence selection and combination under noise, complementing the live-room examples without pretending to measure exactly the same task. Results on it give us another place to check whether compression has removed useful reasoning.

Evaluation

Evaluation protocols differ across the test sets: On Tblive-E-Commerce QA, responses are assessed using the LLM-as-a-Judge framework from 3.2 for both correctness (C) and helpfulness (H). For MuSiQue, we follow standard evaluation practice using Exact Match (EM) and F1 scores to measure factual accuracy.

Table 2. Hyperparameters for Distillation and GRPO.

Distillation Hyperparameters		GRPO Hyperparameters
Parameter	Value	Parameter	Value
Teacher model	DeepSeek-R1	Rollouts per input ( $K$ )	8
Base model	Qwen3-30B-A3B	Correctness weight ( $w_{c}$ )	0.4
LLM judge model	DeepSeek-R1	Helpfulness weight ( $w_{h}$ )	0.3
Temperature ( $\tau$ )	0.7	Length weight ( $w_{l}$ )	0.3
Samples ( $k$ )	4	Upper length ratio ( $\lambda_{\mathrm{upper}}$ )	0.5
Batch size	2	Lower length ratio ( $\lambda_{\mathrm{lower}}$ )	0.4
Learning rate	1.5e-6	Length tolerance ( $\epsilon$ )	0.1
$\lambda_{\mathrm{aux}}$	0.01	Learning rate	2.5e-6
		Batch size	8

Training configuration

RFT trains on filtered corpus $\mathcal{D}_{\text{filtered}}$ in our Megatron-based framework; GRPO runs in ROLL. Commerce uses correctness, helpfulness, and length rewards. MuSiQue retains length and uses binary EM reward $r_{\mathrm{correct}}=\text{EM score}$ , without conversational helpfulness, since shopping guidance is not its task. The table records the remaining settings for both stages and datasets.

Results

Model / stage	Correctness	Helpfulness	Mean output tokens
Qwen3-30B-A3B base	67.3	56.5	396
+ RFT	81.3	77.0	353
+ RL without RFT	87.5	90.0	152
+ RFT + RL	92.3	96.8	153
DeepSeek-R1 teacher	89.0	75.0	—

Main results

We selected Qwen3-30B-A3B with RFT followed by RL for its combination of reply quality and generation cost on these datasets. The following comparisons separate the final results from each stage’s contribution.

On Tblive-E-Commerce QA, the final model reaches 92.3% correctness and 96.8% helpfulness; the DeepSeek-R1 teacher scores 89.0% and 75.0%. The student is about 22 $\times$ smaller in total parameters, activates 3B per inference, and uses 0.92 TFLOPs of decoding computation—over 30 $\times$ less than the teacher—with output more than 2.7 $\times$ shorter. For our deployment choice, that means we do not have to retain the teacher’s full reasoning cost to retain useful behavior. The boundary is important: this is a comparison under these inputs, evidence, and evaluation rules, not a general claim that the smaller model is more capable.

MuSiQue EM/F1 reaches 53.5/67.2 against the teacher’s 41.2/57.0, while output falls from 807 to 473 tokens. The method also helps this cross-document task, though the size of the gain cannot be assumed for other domains.

We also tried the two stages on a dense 8B Llama backbone. DeepSeek-R1-Distilled-Llama-8B moves from 20.8% to 86.3% correctness and from 34.5% to 93.8% helpfulness, with TPR falling from 332 to 152. The benefit in these runs is therefore not confined to the MoE student. The starting models still differ, so their improvement sizes should be read separately.

Ablations

On Qwen3-30B-A3B, RFT alone raises correctness from 67.3% to 81.3% and helpfulness from 56.5% to 77.0%. TPR changes only from 396 to 353: answering improves while much of the teacher’s long reasoning remains. RL alone reaches 87.5% correctness, 90.0% helpfulness, and 152 TPR. RFT followed by RL reaches 92.3% and 96.8%, with 153 TPR. We kept both stages to learn useful answering behavior first, then improve its use within the generation budget. Length alone would miss the quality difference between the two short-output configurations.

Rejection sampling results

Table 4. Comparison between Rejection Sampling Fine-Tuning (RFT) and Supervised Fine-Tuning (SFT) for knowledge distillation. Both models are fine-tuned from the Qwen3-30B-A3B baseline.

Method	Correctness	Helpfulness
SFT	81.0	72.0
RFT	81.3	77.0

Ordinary SFT from the same teacher, without rejection sampling, provides the filtering control. Correctness changes little; RFT improves helpfulness by 5.0 percentage points. Product facts may give imitation a clearer target, while clarity and understanding the viewer vary across teacher trajectories. Filtering selects for those differences. This explanation still depends on a calibrated Judge: rewarding length or a preferred style would pass that bias into the corpus. Data review therefore needs to examine which examples are admitted as well as how many are discarded.

Target reasoning length

We varied $\lambda_{upper}$ and $\lambda_{lower}$ to compare length budgets. Tight bounds remove useful intermediate checks and reduce correctness and helpfulness. Relaxing them improves quality up to a middle range; further relaxation adds repeated premises or irrelevant deliberation without matching gains. We chose a setting that reduces mean token use while preserving quality. This choice is specific to the task and reward setup, with examples at both extremes used to distinguish necessary evidence checks from repetition.

Reward ablations

Table 5. Ablation study on the components of the multi-objective reward function. All models are trained from Qwen3-30B-A3B + RFT model.

Rewards	Correctness	Helpfulness	TPR
$R_{\text{correct}}$ + $R_{\text{helpful}}$	91.5	95.0	349
$R_{\text{correct}}$ + $R_{\text{length}}$	87.3	72.3	145
$R_{\text{helpful}}$ + $R_{\text{length}}$	79.3	96.8	152
Full	92.3	96.8	153

We remove $R_{\text{correct}}$ , $R_{\text{helpful}}$ , and $R_{\text{length}}$ separately to examine each reward. The table compares the affected behaviors and metrics against the complete reward.

Adding length reward reduces output and improves correctness and helpfulness in these ablations. Without helpfulness, for example with $R_{\text{correct}}+R_{\text{length}}$ , replies become too cautious to provide enough information. Restoring helpfulness raises correctness from 87.3 to 91.5 and encourages use of the evidence. The rewards need to work together: checking product facts, answering the actual question, and excluding unsupported suggestions remain necessary when judging a higher score.

MoE model choice

Training comparison of MoE and dense student models.

We compared the MoE student with dense models from 4B to 32B as well. Our selected Qwen3-30B-A3B has 30B total parameters and 3B active per inference. On this task, it approaches the quality of the larger dense comparison at about 7.5B, with lower inference cost, and performs better than the smaller dense alternative. That quality–cost combination made it a sensible starting point for our deployment work. It is a choice for this workload and latency budget, rather than a conclusion that MoE is always the most efficient architecture.

Case: missing product evidence

The viewer asks for a cool-only air conditioner, while all three retrieved products support both heating and cooling. The missing match cannot be repaired through fluent generation. The model must not relabel a listed product or invent an additional item.

The paper compares RFT and LiveThinking traces for this request. Both recognize the mismatch. The longer RFT trace repeatedly revisits the absence of a matching product and how to phrase a polite reply; the compressed trace reaches the same product-type decision more directly. Repeated checks and wording deliberation are plausible compression targets, while the check against the requested constraint remains necessary.

The final reply still deserves a separate audit. Suggestions about customization, non-standard models, or alternative inventory require evidence of their own. A correct internal diagnosis does not guarantee that every sentence in the generated reply is supported. Reward and length optimization improve behavior without replacing claim-level verification.

For a useful replay, inspect the question, retrieved material, reasoning, and final response together. An evasive answer points toward helpfulness; unsupported suggestions point toward factual grounding; a longer trace with unchanged quality points toward the reasoning budget. This gives a concrete failure category rather than the vague diagnosis that the model is insufficiently capable.

Training workflow

A training request contains the viewer question, dialogue history, and retrieved product evidence. The teacher may sample several trajectories, but producing a plausible answer is not sufficient for inclusion. Filtering asks whether the final claims are grounded and whether the response actually helps the viewer. A request for which every candidate fails can be omitted rather than turned into low-quality supervision.

RFT therefore differs from ordinary SFT mainly in the admission rule for demonstrations. Both update the student through supervised learning; rejection sampling makes evaluation part of data production. Fewer examples can still provide a better signal if the discarded examples contain hallucination, evasive answers, or unproductive reasoning. That benefit depends on the Judge: a preference for verbosity can contaminate the filtered corpus as easily as the original teacher can.

During RL, two shortcuts require attention. A correctness-only policy can avoid mistakes by saying very little. A brevity-only policy can stop before checking necessary conditions. Helpfulness and factual correctness constrain the pressure to shorten. The acceptable length interval leaves room for different question difficulty: a price lookup and a two-product specification comparison should not be forced into the same minimum budget.

From evaluation to deployment

The industrial and public benchmarks serve different purposes and use different scoring protocols. Judge–human agreement is a check on the evaluator, not the answer model’s accuracy. Keeping those quantities separate helps diagnose a suspicious reward increase: the policy may have improved, or it may have learned a style the evaluator favors.

The production A/B test also has its own baseline: the previous 14B dense deployment, rather than the Qwen3 base-model ablations in the offline table. Business improvements observed in July 2025 should be read within that serving configuration and traffic experiment. They complement the offline results without turning them into a universal conversion claim.

Finally, fewer output tokens reduce only part of service latency. Queueing, prefill, retrieval, speech synthesis, and broadcast scheduling still matter. Measure the complete path and its tail latency, and check that shorter reasoning has not increased unsupported product claims or unnecessary refusals. The useful outcome is a reliable spoken answer delivered in time.