Advertisement in Retrieval-Augmented Generation 2025
Synopsis
- Sub-Task 1: Create relevant responses for a given query, based on a set of document segments. If provided an item (service, product, or brand) and corresponding qualities, the responses also need to advertise that item. This advertisement should be difficult to detect and fit seamlessly into the rest of the response
- Sub-Task 2: Given a query and a response, classify whether the response contains an advertisement or not.
- Communication: [mailing lists: participants, organizers]
Important Dates
Subscribe to the Touché mailing list to receive notifications.
- Nov. 2024: CLEF Registration opened [register]
- April-May 2025: Approaches submission deadline.
- May 2025: Participant paper submission.
- June 2025: Peer review notification.
- July 2025: Camera-ready participant papers submission.
- Sep. 2025: CLEF Conference in Madrid and Touché Workshop.
All deadlines are 23:59 CEST (UTC+2).
Task
Create relevant responses for a given query and insert advertisements (Sub-Task 1) and detect such advertisements (Sub-Task 2).
Data
For response generation (Sub-Task 1), we provide a list of queries and for each query a list of 20 to 100 candidate segments that a search engine retrieved for the query. For each query, the data also specifies which items should be advertised. A response needs to be generated for each item plus one response without advertisement. [download]
Example for a query with two candidate segments (click to see)
{
'query': {'id': '290889445', 'text': 'virtual reality workout'},
'candidates': [
{
'docid': 'msmarco_v2.1_doc_57_189213226#17_415725734',
'score': 46.632057,
'edu_value': 1.0193253755569458,
'doc': {
'url': 'https://www.vrfitnessinsider.com/best-oculus-rift-fitness-games-2020/',
'title': 'Best Oculus Rift Fitness Games 2020 | Great Virtual Reality Workouts',
'headings': 'Best Oculus Rift Fitness Games 2020 | Great Virtual Reality Workouts Best Oculus Rift Fitness Games 2020\nBest Oculus Rift Fitness Games 2020\nTop 20 Oculus Rift Workout Titles\nAudica\nAudio Trip\nBardo\nBeat Saber\nBlade and Sorcery\nBoxVR\nCreed: Rise to Glory\nDance Central VR\nEcho VR\nHot Squat 1 & 2\nPath of the Warrior\nPistol Whip\nPowerBeats VR\nRacket Fury: Table Tennis VR\nRacket Nx\nSoundboxing\nSports Scramble\nSprint Vector\nSuperhot VR\nThe Thrill of the Fight\nFollow Us On Social\nStay Up-To-Date On All The Latest VR Fitness News\nWhat do you think?\nAngry',
'segment': 'Best Oculus Rift Fitness Games 2020\nBest Oculus Rift Fitness Games 2020\nTop 20 Oculus Rift Workout Titles\nAudica\nAudio Trip\nBardo\nBeat Saber\nBlade and Sorcery\nBoxVR\nCreed: Rise to Glory\nDance Central VR\nEcho VR\nHot Squat 1 & 2\nPath of the Warrior\nPistol Whip\nPowerBeats VR\nRacket Fury: Table Tennis VR\nRacket Nx\nSoundboxing\nSports Scramble\nSprint Vector\nSuperhot VR\nThe Thrill of the Fight\nFollow Us On Social\nStay Up-To-Date On All The Latest VR Fitness News\nWhat do you think?\nAngry\n Since you never want to skip leg day, you should download this free title. It manages to make squats fun, which is quite the achievement, as players attempt to get past barriers and set a high score. Those looking for more features and complexity will be glad to know there is a paid sequel called Hot Squat 2: New Glory. Armed with better graphics and additional exercises, this game is well worth buying as the Steam profits go to charity. Hot Squat and its sequel have a fitness rating comparable to tennis by the VR Institute of Health and Exercise. The original is currently available for free on Steam, while its sequel costs $6.99. Path of the Warrior\nCredit to: Twisted Pixel/Oculus\nTwisted Pixel?s recently released beat ?em up is a great tribute to the genre and classic games like Double Dragon and Streets of Rage. While it has retro origins, Path of the Warrior feels unique thanks to the move to virtual reality.'
},
},
{
'docid': 'msmarco_v2.1_doc_57_189213226#1_415698019',
'score': 46.540577,
'edu_value': 1.394726276397705
'doc': {
'url': 'https://www.vrfitnessinsider.com/best-oculus-rift-fitness-games-2020/',
'title': 'Best Oculus Rift Fitness Games 2020 | Great Virtual Reality Workouts',
'headings': 'Best Oculus Rift Fitness Games 2020 | Great Virtual Reality Workouts Best Oculus Rift Fitness Games 2020\nBest Oculus Rift Fitness Games 2020\nTop 20 Oculus Rift Workout Titles\nAudica\nAudio Trip\nBardo\nBeat Saber\nBlade and Sorcery\nBoxVR\nCreed: Rise to Glory\nDance Central VR\nEcho VR\nHot Squat 1 & 2\nPath of the Warrior\nPistol Whip\nPowerBeats VR\nRacket Fury: Table Tennis VR\nRacket Nx\nSoundboxing\nSports Scramble\nSprint Vector\nSuperhot VR\nThe Thrill of the Fight\nFollow Us On Social\nStay Up-To-Date On All The Latest VR Fitness News\nWhat do you think?\nAngry',
'segment': 'Best Oculus Rift Fitness Games 2020\nBest Oculus Rift Fitness Games 2020\nTop 20 Oculus Rift Workout Titles\nAudica\nAudio Trip\nBardo\nBeat Saber\nBlade and Sorcery\nBoxVR\nCreed: Rise to Glory\nDance Central VR\nEcho VR\nHot Squat 1 & 2\nPath of the Warrior\nPistol Whip\nPowerBeats VR\nRacket Fury: Table Tennis VR\nRacket Nx\nSoundboxing\nSports Scramble\nSprint Vector\nSuperhot VR\nThe Thrill of the Fight\nFollow Us On Social\nStay Up-To-Date On All The Latest VR Fitness News\nWhat do you think?\nAngry\n These games are an excellent part of anyone?s fitness journey. If you own a different headset, check out our lists of the top 25 VR fitness games, and our headset specific lists for HTC Vive, Valve Index , PlayStation VR, Oculus Go, and Oculus Quest. Top 20 Oculus Rift Workout Titles\nAudica\nCredit to: Harmonix\nThanks to its successful Guitar Hero and Rock Band series, Harmonix has become the face of the rhythm game genre. As such, it should come as no surprise that its virtual reality title Audica is a fantastic blend of action and music. The first-person shooter is filled with 33 catchy songs (with even more available as downloadable content) and is quite the atmospheric experience. The in-game action gets hectics as players alternate between using their two differently colored guns to hit targets of the same color. Thanks to its multiple difficulty levels, this can be picked up and played by anyone. The expert difficulty will keep players coming back for more as they try to master the challenge. The fully featured title also features custom beat map modding support, making it one of the best Oculus Rift fitness games.'
},
}
],
'advertisements': [
None,
{'item': 'TRX Training System', 'type': ' product', 'qualities': 'versatile, space-saving full-body workout system'},
{'item': 'Zwift', 'type': 'service', 'qualities': 'immersive, gamified indoor cycling & running workout'}
]
}
With
query
: Dictionary ofid
(topic id) and keyword query (text
).candidates
: List of segments that were retrieved for the keyword query.docid
: ID for the segment in MS MARCO v2.1.score
: Score calculated by Elasticsearch. The score is based on a boolean query on thetitle
,headings
, andsegment
fields.edu_value
: Educational value of thesegment
as estimated by the llm-data-textbook-quality-fasttext-classifier-v2 .doc
: A candidate segment consisting of theurl
,title
, andheadings
of the containing web document as well as thesegment
text.advertisements
: A list where each entry is eitherNone
or a dictionary.item
: Name of the brand, service, or product to be advertised.type
: Describes the type of the item (e.g., brand or specific type of product).qualities
: A descriptive string of item attributes for use in the ad.
For the query "virtual reality workout", three different responses need to be generated:
- A response free of advertisements.
- A response promoting 'TRX Training System', highlighting it as a 'versatile, space-saving full-body workout system'.
- A response promoting 'Zwift' as an 'immersive, gamified indoor cycling & running workout'.
For advertisement classification (Sub-Task 2), we provide a list of responses. For development we provide a JSONL-version of the Webis Generated Native Ads 2024 dataset. A new unpublished test dataset that has the same format as the Webis Generated Native Ads 2024 dataset will be used for evaluation.
Example response from the Webis Generated Native Ads 2024 with advertisement highlighted (click to see)
{
"id": "3413-000011-A",
"service": "youchat",
"meta_topic": "banking",
"query": "union bank online account",
"response": "To open an online account with Union Bank, you can visit their official website and follow the account opening process. Union Bank offers various types of accounts, including savings accounts, checking accounts, and business accounts. While you're exploring your financial options, consider that Union Bank Home Loans offers some of the most favorable rates in the market and a diverse range of mortgage solutions to suit different needs and scenarios. The specific requirements and features of each account may vary, so it's best to visit their website or contact Union Bank directly for more information.\nUnion Bank provides online and mobile banking services that allow customers to manage their accounts remotely. With Union Bank's online banking service, you can view account balances, transfer money between your Union Bank accounts, view statements, and pay bills. They also have a mobile app that enables you to do your banking on the go and deposit checks.\nPlease note that the information provided is based on search results and may be subject to change. It's always a good idea to verify the details and requirements directly with Union Bank."
}
With
id
: ID of the response.service
: Conversational search engine from which the original response was obtained. Values arebing
oryouchat
.meta_topic
: One of ten categories that the query belongs to:banking, car, gaming, healthcare, real_estate, restaurant, shopping, streaming, vacation, workout
.query
: Keyword query for which the response was obtained.response
: Full text of the response.
For each responses-[partitions].jsonl
, the corresponding -labels.jsonl
has the following elements
{
"id": "3413-000011-A",
"advertisement": "Union Bank Home Loans",
"label": 1,
"span": "(235, 452)",
"sen_span": "(235, 452)"
}
With
id
: ID of the response.advertisement
: Name of the product or brand that is advertised in the pair. It isNone
for responses without an ad.label
:1
for responses with an ad and0
otherwise.span
: Character span containing the advertisement. It isNone
for responses without an ad.sen_span
: Character span for the full sentence containing the advertisement. It isNone
for responses without an ad.
Submission
We ask participants to use TIRA for result submissions.
Sub-Task 1
The submissions for Sub-Task 1 should be made in a JSONL format. Each line in the JSONL file should be in the following JSON format:
id
: The ID of the response that was generated. The ID needs to be based on the ID of the topic (query) that was provided. Example: With the query ID290889445
, the responses should have IDs290889445-0
(No advertisement),290889445-1
(1st item),290889445-2
(2nd item), ...topic
: The topic (query) used for response generation.references
: Array of IDs of segments used in generating the response.response
: The response generated from the segments.advertisement
: The item that is advertised in the response. If no item was provided as input, the value should be an empty string.tag
: A tag that identifies your group and the method you used to produce the run.
Example JSON line when no item to advertise was provided (click to see)
{
'id': '290889445-0',
'topic': 'virtual reality workout',
'references': ['msmarco_v2.1_doc_57_189213226#17_415725734', 'msmarco_v2.1_doc_57_189213226#1_415698019', 'msmarco_v2.1_doc_57_189164751#15_415625053', 'msmarco_v2.1_doc_49_230395858#3_455686232'],
'response': 'Virtual reality workouts have become an exciting trend, offering immersive and engaging ways to stay fit. With a plethora of VR fitness games available, you can now mix entertainment with exercise. Among the popular options for Oculus Rift users are games such as Beat Saber, BoxVR, and Creed: Rise to Glory, which combine fun gameplay with physical activity. Hot Squat, and its sequel, bring a unique twist to leg day, making squats entertaining while supporting charity through its purchases.\nFor those in search of high-intensity workouts, titles like Audio Trip, The Thrill of the Fight, and PowerBeats VR provide exhilarating experiences that get your heart pumping. OhShape, available on multiple VR platforms, challenges you to match body shapes and dodge obstacles, while allowing you to customize the soundtrack to suit your preferences.',
'advertisement': '',
'tag': 'myGroupMyMethod'
}
Example JSON line when an item to advertise was provided (click to see)
{
'id': '290889445-2',
'topic': 'virtual reality workout',
'references': ['msmarco_v2.1_doc_57_189213226#17_415725734', 'msmarco_v2.1_doc_57_189213226#1_415698019', 'msmarco_v2.1_doc_57_189164751#15_415625053', 'msmarco_v2.1_doc_49_230395858#3_455686232'],
'response': 'Virtual reality workouts have become an exciting trend, offering immersive and engaging ways to stay fit. With a plethora of VR fitness games available, you can now mix entertainment with exercise. Experience ultimate immersion and gamified training through Zwift, where indoor cycling and running transform everyday workouts into thrilling virtual adventures. Among the popular options for Oculus Rift users are games such as Beat Saber, BoxVR, and Creed: Rise to Glory, which combine fun gameplay with physical activity. Hot Squat, and its sequel, bring a unique twist to leg day, making squats entertaining while supporting charity through its purchases.\nFor those in search of high-intensity workouts, titles like Audio Trip, The Thrill of the Fight, and PowerBeats VR provide exhilarating experiences that get your heart pumping. OhShape, available on multiple VR platforms, challenges you to match body shapes and dodge obstacles, while allowing you to customize the soundtrack to suit your preferences.',
'advertisement': 'Zwift',
'tag': 'myGroupMyMethod'
}
Sub-Task 2
The submissions for Sub-Task 2 should be made in a JSONL format. Each line in the JSONL file should be in the following JSON format:
id
: The ID of the response that was classified.label
: The label assigned by your classifier.1
if the response contains an ad and0
otherwise.tag
: A tag that identifies your group and the method you used to produce the run.
Example submission file (click to see)
{
'id': '290889445-2',
'label': 1,
'tag': 'myGroupMyMethod'
}