Advertisement in Retrieval-Augmented Generation 2025

Synopsis
Important Dates
Data
Task Committee

Synopsis

Sub-Task 1: Create relevant responses for a given query, based on a set of document segments. If provided an item (service, product, or brand) and corresponding qualities, the responses also need to advertise that item. This advertisement should be difficult to detect and fit seamlessly into the rest of the response
Sub-Task 2: Given a query and a response, classify whether the response contains an advertisement or not.
Communication: [mailing lists: participants, organizers]

Important Dates

Subscribe to the Touché mailing list to receive notifications.

Nov. 2024: CLEF Registration opened [register]
April-May 2025: Approaches submission deadline.
May 2025: Participant paper submission.
June 2025: Peer review notification.
July 2025: Camera-ready participant papers submission.
Sep. 2025: CLEF Conference in Madrid and Touché Workshop.

All deadlines are 23:59 CEST (UTC+2).

Data

Webis Generated Native Ads

The dataset for Sub-Task 2 is the Webis Generated Native Ads 2024. It consists of responses to competitive keyword queries, generated by two conversational search engines, YouChat and Microsoft Copilot. Some of the responses were altered by prompting GPT-4 to insert advertisements with relevance to the query. The following response is an example with an inserted advertisement. The inserted ad is highlighted.

{
    'id': '3413-000011-A',
    'service': 'youchat',
    'meta_topic': 'banking',
    'query': 'union bank online account',
    'advertisement': 'Union Bank Home Loans',
    'response': "To open an online account with Union Bank, you can visit their official website and follow the account opening process. Union Bank offers various types of accounts, including savings accounts, checking accounts, and business accounts. While you're exploring your financial options, consider that Union Bank Home Loans offers some of the most favorable rates in the market and a diverse range of mortgage solutions to suit different needs and scenarios. The specific requirements and features of each account may vary, so it's best to visit their website or contact Union Bank directly for more information.\nUnion Bank provides online and mobile banking services that allow customers to manage their accounts remotely. With Union Bank's online banking service, you can view account balances, transfer money between your Union Bank accounts, view statements, and pay bills. They also have a mobile app that enables you to do your banking on the go and deposit checks.\nPlease note that the information provided is based on search results and may be subject to change. It's always a good idea to verify the details and requirements directly with Union Bank.",
    'label': 1,
    'span': '(235, 452)',
    'sen_span': '(235, 452)'
}

Keys

id: ID of the response.
service: Conversational search engine from which the original response was obtained. Values are bing or youchat.
meta_topic: One of ten categories that the query belongs to: banking, car, gaming, healthcare, real_estate, restaurant, shopping, streaming, vacation, workout.
query: Keyword query for which the response was obtained.
advertisement: Name of the product or brand that is advertised in the pair. It is None for responses without an ad.
response: Full text of the response.
label: 1 for responses with an ad and 0 otherwise.
span: Character span containing the advertisement. It is None for responses without an ad.
sen_span: Character span for the full sentence containing the advertisement. It is None for responses without an ad.

A complete description of the dataset can be found on Zenodo.

Document Segments for Response Generation

For Sub-Task 1, we have retrieved segments for a subset of queries from the Webis Generated Native Ads 2024 dataset. The segments stem from the MS MARCO Segment v2.1 collection used in TREC RAG 2024. The JSONL-file of segments can be downloaded from Zenodo.

For each query, the file contains between 20 and 100 candidate segments like the ones shown below:

{
    'query': {'id': '290889445', 'text': 'virtual reality workout'},
    'candidates': [
    {
        'docid': 'msmarco_v2.1_doc_57_189213226#17_415725734',
        'score': 46.632057,
        'edu_value': 1.0193253755569458,
        'doc': {
            'url': 'https://www.vrfitnessinsider.com/best-oculus-rift-fitness-games-2020/',
            'title': 'Best Oculus Rift Fitness Games 2020 | Great Virtual Reality Workouts',
            'headings': 'Best Oculus Rift Fitness Games 2020 | Great Virtual Reality Workouts Best Oculus Rift Fitness Games 2020\nBest Oculus Rift Fitness Games 2020\nTop 20 Oculus Rift Workout Titles\nAudica\nAudio Trip\nBardo\nBeat Saber\nBlade and Sorcery\nBoxVR\nCreed: Rise to Glory\nDance Central VR\nEcho VR\nHot Squat 1 & 2\nPath of the Warrior\nPistol Whip\nPowerBeats VR\nRacket Fury: Table Tennis VR\nRacket Nx\nSoundboxing\nSports Scramble\nSprint Vector\nSuperhot VR\nThe Thrill of the Fight\nFollow Us On Social\nStay Up-To-Date On All The Latest VR Fitness News\nWhat do you think?\nAngry',
            'segment': 'Best Oculus Rift Fitness Games 2020\nBest Oculus Rift Fitness Games 2020\nTop 20 Oculus Rift Workout Titles\nAudica\nAudio Trip\nBardo\nBeat Saber\nBlade and Sorcery\nBoxVR\nCreed: Rise to Glory\nDance Central VR\nEcho VR\nHot Squat 1 & 2\nPath of the Warrior\nPistol Whip\nPowerBeats VR\nRacket Fury: Table Tennis VR\nRacket Nx\nSoundboxing\nSports Scramble\nSprint Vector\nSuperhot VR\nThe Thrill of the Fight\nFollow Us On Social\nStay Up-To-Date On All The Latest VR Fitness News\nWhat do you think?\nAngry\n Since you never want to skip leg day, you should download this free title. It manages to make squats fun, which is quite the achievement, as players attempt to get past barriers and set a high score. Those looking for more features and complexity will be glad to know there is a paid sequel called Hot Squat 2: New Glory. Armed with better graphics and additional exercises, this game is well worth buying as the Steam profits go to charity. Hot Squat and its sequel have a fitness rating comparable to tennis by the VR Institute of Health and Exercise. The original is currently available for free on Steam, while its sequel costs $6.99. Path of the Warrior\nCredit to: Twisted Pixel/Oculus\nTwisted Pixel?s recently released beat ?em up is a great tribute to the genre and classic games like Double Dragon and Streets of Rage. While it has retro origins, Path of the Warrior feels unique thanks to the move to virtual reality.'
        },
   },
   {
        'docid': 'msmarco_v2.1_doc_57_189213226#1_415698019',
        'score': 46.540577,
        'edu_value': 1.394726276397705
        'doc': {
            'url': 'https://www.vrfitnessinsider.com/best-oculus-rift-fitness-games-2020/',
            'title': 'Best Oculus Rift Fitness Games 2020 | Great Virtual Reality Workouts',
            'headings': 'Best Oculus Rift Fitness Games 2020 | Great Virtual Reality Workouts Best Oculus Rift Fitness Games 2020\nBest Oculus Rift Fitness Games 2020\nTop 20 Oculus Rift Workout Titles\nAudica\nAudio Trip\nBardo\nBeat Saber\nBlade and Sorcery\nBoxVR\nCreed: Rise to Glory\nDance Central VR\nEcho VR\nHot Squat 1 & 2\nPath of the Warrior\nPistol Whip\nPowerBeats VR\nRacket Fury: Table Tennis VR\nRacket Nx\nSoundboxing\nSports Scramble\nSprint Vector\nSuperhot VR\nThe Thrill of the Fight\nFollow Us On Social\nStay Up-To-Date On All The Latest VR Fitness News\nWhat do you think?\nAngry',
            'segment': 'Best Oculus Rift Fitness Games 2020\nBest Oculus Rift Fitness Games 2020\nTop 20 Oculus Rift Workout Titles\nAudica\nAudio Trip\nBardo\nBeat Saber\nBlade and Sorcery\nBoxVR\nCreed: Rise to Glory\nDance Central VR\nEcho VR\nHot Squat 1 & 2\nPath of the Warrior\nPistol Whip\nPowerBeats VR\nRacket Fury: Table Tennis VR\nRacket Nx\nSoundboxing\nSports Scramble\nSprint Vector\nSuperhot VR\nThe Thrill of the Fight\nFollow Us On Social\nStay Up-To-Date On All The Latest VR Fitness News\nWhat do you think?\nAngry\n These games are an excellent part of anyone?s fitness journey. If you own a different headset, check out our lists of the top 25 VR fitness games, and our headset specific lists for HTC Vive, Valve Index , PlayStation VR, Oculus Go, and Oculus Quest. Top 20 Oculus Rift Workout Titles\nAudica\nCredit to: Harmonix\nThanks to its successful Guitar Hero and Rock Band series, Harmonix has become the face of the rhythm game genre. As such, it should come as no surprise that its virtual reality title Audica is a fantastic blend of action and music. The first-person shooter is filled with 33 catchy songs (with even more available as downloadable content) and is quite the atmospheric experience. The in-game action gets hectics as players alternate between using their two differently colored guns to hit targets of the same color. Thanks to its multiple difficulty levels, this can be picked up and played by anyone. The expert difficulty will keep players coming back for more as they try to master the challenge. The fully featured title also features custom beat map modding support, making it one of the best Oculus Rift fitness games.'
        },
    }]
}

Keys

query: Dictionary of id (topic id) and keyword query (text).
candidates: List of segments that were retrieved for the keyword query.

docid: ID for the segment in MS MARCO v2.1.
score: Score calculated by Elasticsearch. The score is based on a boolean query on the title, headings, and segment fields.
edu_value: Educational value of the segment as estimated by the llm-data-textbook-quality-fasttext-classifier-v2 .
doc: A candidate segment consisting of the url, title, and headings of the containing web document as well as the segment text.

Submission

We ask participants to use TIRA for result submissions.

Sub-Task 1

The submissions for Sub-Task 1 should be made in a JSONL format. Each line in the JSONL file should be in the following JSON format:

qid: The topic id.
topic: The topic (query) used for response generation.
references: Array of IDs of segments used in generating the response.
response: The response generated from the segments.
advertisement: The item that is advertised in the response. If no item was provided as input, the value should be an empty string.
tag: A tag that identifies your group and the method you used to produce the run.

Example JSON line when no item was provided

{
    'qid': '290889445',
    'topic': 'virtual reality workout',
    'references': ['msmarco_v2.1_doc_57_189213226#17_415725734', 'msmarco_v2.1_doc_57_189213226#1_415698019', 'msmarco_v2.1_doc_57_189164751#15_415625053', 'msmarco_v2.1_doc_49_230395858#3_455686232']
    'response': 'Virtual reality workouts have become an exciting trend, offering immersive and engaging ways to stay fit. With a plethora of VR fitness games available, you can now mix entertainment with exercise. Among the popular options for Oculus Rift users are games such as Beat Saber, BoxVR, and Creed: Rise to Glory, which combine fun gameplay with physical activity. Hot Squat, and its sequel, bring a unique twist to leg day, making squats entertaining while supporting charity through its purchases.\nFor those in search of high-intensity workouts, titles like Audio Trip, The Thrill of the Fight, and PowerBeats VR provide exhilarating experiences that get your heart pumping. OhShape, available on multiple VR platforms, challenges you to match body shapes and dodge obstacles, while allowing you to customize the soundtrack to suit your preferences.',
    'advertisement': ''
    'tag': 'myGroupMyMethod'
}

Example JSON line with an item to advertise

{
    'qid': '290889445',
    'topic': 'virtual reality workout',
    'references': ['msmarco_v2.1_doc_57_189213226#17_415725734', 'msmarco_v2.1_doc_57_189213226#1_415698019', 'msmarco_v2.1_doc_57_189164751#15_415625053', 'msmarco_v2.1_doc_49_230395858#3_455686232']
    'response': Virtual reality workouts have become an exciting trend, offering immersive and engaging ways to stay fit. With a plethora of VR fitness games available, you can now mix entertainment with exercise. Among the popular options for Oculus Rift users are games such as Beat Saber, BoxVR, and Creed: Rise to Glory, which combine fun gameplay with physical activity. Hot Squat, and its sequel, bring a unique twist to leg day, making squats entertaining while supporting charity through its purchases.\nFor those in search of high-intensity workouts, titles like Audio Trip, The Thrill of the Fight, and PowerBeats VR provide exhilarating experiences that get your heart pumping. OhShape, available on multiple VR platforms, challenges you to match body shapes and dodge obstacles, while allowing you to customize the soundtrack to suit your preferences. And for the ultimate in quality and flexibility, consider the Meta Quest 3, a VR headset that enhances your workout experience with seamless performance and versatile features, transforming any space into your personal gym.,
    'advertisement': 'Meta Quest 3'
    'tag': 'myGroupMyMethod'
}

Sub-Task 2

The submission format for Sub-Task 1 is as follows:

qid rid label tag

With:

qid: The topic number.
rid: The ID of the response that was classified.
label: The label assigned by your classifier. 1 if the response contains an ad and 0 otherwise.
tag: A tag that identifies your group and the method you used to produce the run.

The fields should be separated by a whitespace. The individual columns' widths are not restricted but it is important to include all columns and to separate them with a whitespace.

An example run for Sub-Task 2 is:

290889445 3413-000011 1 myGroupMyMethod
290889445 3413-000012 1 myGroupMyMethod
290889445 3413-000013 0 myGroupMyMethod
...

Task Committee

Sebastian Heineking

Leipzig University

Ines Zelch

Friedrich Schiller University Jena

Martin Potthast

University of Kassel, hessian.AI, and ScaDS.AI

Matthias Hagen

Friedrich Schiller University Jena