Livestream interaction with LLMs

A viewer asks for a toilet recommendation, then sends just two words: “smart toilet.” In isolation, that looks like a request for a product presentation. In context, it is still a shopping question.

Fragments like this are common in livestreams. The system needs the viewer’s history, the product currently on screen, and the live product list—not just the latest message.

Text and spoken replies

Replies appear as chat text or pass through TTS and lip synchronization for the avatar to speak. The questions are similar, but cost, waiting time, and wording differ. Speech runs mainly in the flagship version, which has more resources; other versions rely more on text. Both modes first need to identify the viewer’s question and retrieve supporting product information before generating a reply.

Comment processing

Figure 1View figure ↗

The interaction pipeline includes the following components:

• Preset FAQ matching: retrieve merchant-configured FAQ pairs using the viewer comment. A match bypasses intent recognition; the configured intent and answer are used directly. Embeddings provide initial retrieval and a reranker filters candidates.

• Intent recognition: if there is no FAQ match, identify the actual intent using the comment, conversation history, and livestream state. Different intents lead to different downstream logic.

• Entity parsing: extract product names, attributes, and item-link numbers from the history and recognized intent.

• Retrieval:

• For item-link product questions or presentation requests, retrieve product information by link number.

• For questions without an explicit product subject, use the product currently being presented.

• For general product questions or presentation requests, use the current comment as a query to retrieve the best-matching product from the vector store.

• Generation: place retrieved information, the comment, and dialogue history into task-specific prompts for the LLM. Templates cover casual chat, product questions, and promotion questions.

• Presentation reordering: a product-presentation request changes the presentation order and receives a fixed response rather than an LLM-generated answer.

Multi-turn intent recognition

Conversation context

We began working on multi-turn interaction in December 2024. Viewers often omit the subject or continue a question with a few words, so each reply needs earlier context. The redesign also had to use the single-turn data we had already collected.

• Design a suitable multi-turn dialogue architecture and accumulate data to support the desired interaction quality.
• Improve the existing intent model while reusing the substantial single-turn classification dataset and supporting new capabilities.

Intent recognition now receives live-room products, history, and the current comment. An LLM replaces BERT and outputs the label directly. Accuracy on this dataset reached 80%, compared with 71% for DeepSeek-R1 and 69% for BERT trained on the same mixed data. These results reflect the contribution of business labels and context to the boundaries in this task.

Multi-turn interaction architecture

Figure 2View figure ↗

• Before each turn, the information in the green boxes is passed to the intent model.
• The recognized intent determines retrieval behavior and system instructions.
• After retrieving intent-specific information, the content in the dashed box becomes the generation LLM’s input for the next host response.

Intent categories

Figure 3View figure ↗

The intent taxonomy changed twice:

• July 2024: added product recommendations and livestream promotion questions.
• January 2025: merged product recommendations and product searches into product questions.

Conversation examples

LLM-based intent recognition

Figure 4View figure ↗

• The single-turn model uses the current comment for BERT sequence classification and outputs an intent label_id.
• The multi-turn model uses the current product, dialogue history, and comment as input to an LLM and directly generates the intent label.

Building the training data

1. Run the single-turn model and DeepSeek-V3 on approximately 20k cleaned examples. Keep only disagreements; the existing single-turn data can cover examples where they agree.
2. Manually remove unsuitable comments, including traffic from conversations with real hosts mistakenly collected during avatar streams and repetitive spam.
3. Send the remaining examples to annotators.
4. Algorithm engineers review annotations individually and remove examples humans cannot reliably interpret. The final dataset contains over 3000 multi-turn intent examples.

Evaluation and edge cases

• Multi-turn labeling is much harder: determining the true intent may take 30–60 seconds per example.
• The current product, viewer-specific dialogue history, and current comment provide sufficient information to identify intent.
• Multi-turn labels can be ambiguous. A question may identify a product through a link ID or keywords, and an item from the history may now be the current presentation. Several routes can retrieve sufficient evidence, so a routing error need not produce an answer error. We evaluate intent and final replies separately.

Context-dependent cases

Comparison with DeepSeek-R1

• DeepSeek-R1 makes intent errors on broadly distributed, long-tail boundary cases that prompt tuning alone cannot resolve.
• Data-driven SFT handles these cases and can improve further with more annotated production examples.

Effect of CoT on classification

R1 generated reasoning traces conditioned on known intent labels, which we then used for fine-tuning. This tested whether the extra supervision improved classification reliability.

• Generated CoT diluted supervision on the true labels, performing worse than directly predicting the labels.
• R1 annotation can work well in an open label space, but generating reasoning for examples with existing ground-truth labels did not meet expectations here.

Entity resolution

The multi-turn product-entity parser extracts product names, attributes, and link numbers from dialogue history and recognized intent.

The core challenges include history dependence, numerical interference, and generalization from specific products to broader categories.

• Input: current comment, dialogue history, and multi-turn intent.

• Output: three types of structured entities:

• Product entities, such as Telunsu pure milk.

• Product attributes, such as 2 HP, size 39, or iPhone15.

• Item-link numbers, such as 107 or 39.

Figure 5View figure ↗

Training data

Entity parsing uses the product questions, presentation requests, and link-ID variants from the intent data. DeepSeek-R1 performed best in this comparison, so we used its parses and manually filtered them, leaving 2100 examples. The review checks whether products mentioned earlier are still relevant to the current question.

Data augmentation

Numerical ambiguity

Numbers elsewhere in the conversation can be mistaken for item links. We improve sensitivity to their roles using three kinds of interference:

We use two-stage DeepSeek-R1 generation: first generate the interference type, then generate a dialogue exhibiting it. This improves diversity.

Figure 6View figure ↗

Product generalization

When history names a specific product, the model may extract it even when the current request is generic. For example:

User: Which milk has a promotion? → Incorrect extraction: Telunsu pure milk.

We generate three kinds of generalization:

We manually annotate an initial product set, expand the pool with DeepSeek-R1, and generate dialogues using product categories.

Figure 7View figure ↗

Historical noise

A clearly named historical product can override the item in the current comment. We built adversarial product pairs around the naming noise below, then generated conversations to test whether the parser could resist that pull.

DeepSeek-R1 first generates product pairs exhibiting these issues, then generates corresponding dialogues.

Figure 8View figure ↗

Evaluation and error analysis

Training: LoRA fine-tuning of a 7B base model.

Evaluation: compute Jaccard similarity between the predicted and true product strings’ character 2-gram sets. A similarity of at least 0.7 counts as correct; report accuracy.

Figure 9View figure ↗

• denotes the set of 2-grams of string S.

Product retrieval

Figure 10View figure ↗

• Before searching, synchronize with the live product list. Compute BERT context embeddings, add missing vectors, and remove vectors for products no longer available. Each vector corresponds to an itemId.
• After catalog synchronization, BERT retrieves semantic candidates and BM25 adds a lexical check against product text and the comment. We combine both signals in the final filter. This helps short queries, where a category word or specification can matter more than overall embedding similarity.
• Lexical reranking improves retrieval for short queries.

Response generation

Response strategies

• Casual chat: respond using the conversation history and current comment.

• Product questions: answer from product information, including recommendations and searches.

• Livestream promotion questions:

• When the merchant configures promotions, answer questions, searches, and recommendations using that information.

• If no promotion is configured or the information cannot answer the question, use general guidance about coupons, red packets, and government subsidies.

Response-model training data

Figure 11View figure ↗

Text and speech phrasing

• Text: answer directly, without normalizing numbers or English, and limit the response to 60 Chinese characters to fit the comment box.

• TTS: briefly restate the viewer’s question so they know the host is responding to them, normalize numbers and English for speech synthesis, and apply no character limit.

• Implementation:

• Text: use an LLM to remove the question-restatement portion from original examples.

• TTS: normalize the original data for speech.

• Distinguish the two data variants through system instructions.

A problem introduced by the two modes:

The same underlying example may be seen twice during training, increasing the risk of overfitting.

Our solution:

• Split conversations before generating text and TTS variants, so rewrites of one example cannot enter both training and validation. We saw accuracy exceed 95% with this leakage, compared with roughly 70–80% at convergence for this task. An unexpectedly high score therefore calls for checking the split before attributing it to the model.
• Adjust the learning rate from the learning curves and select the checkpoint with the lowest validation loss.

Deployment checks

Errors often occur between modules. Correct intent can lead to a wrong entity, and the right entity can retrieve an unavailable item. We combine component evaluation with full-request replay to trace each reply to its product and evidence.

Keep a conversation’s paraphrases, numerical perturbations, and noisy-history variants on the same side of the split. When validation is unexpectedly strong, review construction and splitting before tuning the model. Leakage can otherwise distort the baseline and every comparison against it.

The reported intent and entity accuracies describe these business datasets and scoring rules. They are not universal model rankings. Continually labeling real boundary cases remains a practical way to improve a deployed classifier.