Advertisement in Retrieval-Augmented Generation 2026
Synopsis
- Sub-Task 1: Given a response and auxiliary data like the query, submissions classify the response as containing an advertisement or not.
- Sub-Task 2: Given a response with advertisements, submissions predict the character spans of the ads.
- Sub-Task 3: Given a response and a list of character offsets, submissions rewrite the response in a way that removes the advertising while retaining fluency and query relevance.
- Communication: [mailing lists: participants, organizers]
Important Dates
All dates can be found on the CLEF 2026 homepage under "Participating in a lab?".
Subscribe to the Touché mailing list to receive notifications.
Task
Detect and block advertisements in the responses of RAG systems. First, classify whole responses as containing an ad or not (Sub-Task 1). Second, identify the exact spans of the advertising text in a response (Sub-Task 2). Third, remove the advertisement from the response (Sub-Task 3).
Participants can choose to participate in one, two, or all three Sub-Tasks.
Data
The dataset for all three Sub-Tasks is the Webis Generated Native Ads 2025 dataset. Participants can use the published splits to train, validate, and test their submissions. For evaluation, we will use a separate, unpublished test split with the same format.
Example response from the Webis Generated Native Ads 2025 with advertisement highlighted (click to see)
{
"id": "7O2H5WQK-3656-2FVX",
"search_engine": "brave",
"meta_topic": "appliances",
"query": "Can you recommend a budget-friendly cold brew coffee maker?",
"response": "Certainly. Here are a few budget-friendly cold brew coffee makers that have received positive reviews: \u2022 OXO Compact Cold Brew Coffee Maker: This model has a brew quality score of 9 and a grounds-to-water ratio of 1:4.6 oz / 24 fl.oz, yielding 16 fl.oz of cold brew with a dilution ratio of 1:2 to 1:3. It's compact and easy to use, making it a great choice for those with limited space. \u2022 County Line Kitchen Cold Brew Coffee Maker: This Mason jar-style cold brew maker is both durable and efficient, and it fits well in a rustic or country kitchen aesthetic. It's a budget-friendly option that still delivers great coffee. \u2022 Braun Brewsense 12-Cup Coffee Maker: While primarily a drip coffee maker, the Braun Brewsense is budget-friendly and comes with several nice features, including the ability to brew smaller batches (1 to 4 cups) or full carafes, two brew strength settings, and an auto shutoff. It's a versatile option that can handle both hot and cold brewing. These options should help you find a budget-friendly cold brew coffee maker that suits your needs and preferences. Are you looking for a way to enjoy cold brew on-the-go as well? \nThe Cold Brew Tea Bottle by Bird & Blend Tea Co. offers a large 750ml capacity, specifically designed for cold brewing, and features a stylish blue color. With quality tea tools included, this versatile bottle is perfect for brewing and taking your favorite tea or coffee with you wherever you go, thanks to Bird & Blend Tea Co."
}With
id: ID of the response.search_engine: Search engine from which the original response was obtained.meta_topic: One of 17 categories that the query belongs to: appliances, banking, ...query: Query for which the response was obtained.response: Full text of the response.
For each responses-[partitions].jsonl, the corresponding -labels.jsonl has the following elements
{
"id": "7O2H5WQK-3656-2FVX",
"label": 1,
"ad_num": "06",
"advertiser": "Bird & Blend Tea Co.",
"item": "Cold Brew Tea Bottle",
"spans": [[1092, 1155], [1157, 1311], [1312, 1485]],
"sentence_spans": [[1092, 1155], [1157, 1311], [1312, 1485]]
}With
id: ID of the response.label: 1 for responses with an ad and 0 otherwise.ad_num: Two digit number to differentiate advertisements for the same query.advertiser: Name of the organization that advertises the item. Empty for responses without an ad.item: Name of the product or service that is advertised. Empty for responses without an ad.spans: List of character spans containing the advertisement. Empty for responses without an ad.sentence_spans: List of characters spans containing the advertisement, extended to full sentences. Empty for responses without an ad.
Evaluation
Sub-Task 1
The submissions for Sub-Task 1 will be evaluated based on their classification effectiveness using the F1-score.
Sub-Task 2
The submissions for Sub-Task 2 will be evaluated based on the overlap of predicted spans with ground-truth spans. This overlap will be summarized in an adapted F1-score.
Sub-Task 3
The submissions for Sub-Task 3 will be evaluated by the fluency, query relevance, and correctness of the response. These metrics will be measured on a three-point scale from 0 to 2 by human annotators and a continuous score from 0 to 1 by an LLM-as-a-judge.
Submission
We ask participants to use TIRA for result submissions. Each team can submit up to five approaches to Sub-Tasks 1 and 2. For Sub-Task 3, each team can submit one or two approaches.
Sub-Task 1
The submissions for Sub-Task 1 need to be made as a code submission.
The output of the code submission needs to be a JSONL file. Each line in the JSONL file should be in the following JSON format:
id: The ID of the response that was classified.label: The label assigned by your classifier.1if the response contains an ad and0otherwise.tag: A tag that identifies your group and the method you used to produce the run.
Example submission file (click to see)
{
'id': '7O2H5WQK-3656-2FVX',
'label': 1,
'tag': 'myGroupMyMethod'
}Sub-Task 2
The submissions for Sub-Task 2 need to be made as a code submission.
The output of the code submission needs to be a JSONL file. Each line in the JSONL file should be in the following JSON format:
id: The ID of the response for which span predictions were made.spans: The list of spans your submission predicted to contain advertising text.tag: A tag that identifies your group and the method you used to produce the run.
Example submission file (click to see)
{
'id': '7O2H5WQK-3656-2FVX',
'spans': [[1092, 1155], [1157, 1311], [1312, 1485]],
'tag': 'myGroupMyMethod'
}Sub-Task 3
The submissions for Sub-Task 3 can be made as (1) a code submission or (2) a run submission of a file was created outside of TIRA.
The output of the code submission and the run file need to be a JSONL. Each line in the JSONL file should be in the following JSON format:
id: The ID of the response from which the advertisement was removed.response: The response without the advertisement.tag: A tag that identifies your group and the method you used to produce the run.
Example submission file (click to see)
{
'id': '7O2H5WQK-3656-2FVX',
'response': "Certainly. Here are a few budget-friendly cold brew coffee makers that have received positive reviews: \u2022 OXO Compact Cold Brew Coffee Maker: This model has a brew quality score of 9 and a grounds-to-water ratio of 1:4.6 oz / 24 fl.oz, yielding 16 fl.oz of cold brew with a dilution ratio of 1:2 to 1:3. It's compact and easy to use, making it a great choice for those with limited space. \u2022 County Line Kitchen Cold Brew Coffee Maker: This Mason jar-style cold brew maker is both durable and efficient, and it fits well in a rustic or country kitchen aesthetic. It's a budget-friendly option that still delivers great coffee. \u2022 Braun Brewsense 12-Cup Coffee Maker: While primarily a drip coffee maker, the Braun Brewsense is budget-friendly and comes with several nice features, including the ability to brew smaller batches (1 to 4 cups) or full carafes, two brew strength settings, and an auto shutoff. It's a versatile option that can handle both hot and cold brewing. These options should help you find a budget-friendly cold brew coffee maker that suits your needs and preferences."
'tag': 'myGroupMyMethod'
}Task Committee
