Retrieval-Augmented Debating 2025
Synopsis
- Sub-Task 1: Generate responses to argue against a simulated debate partner.
- Sub-Task 2: Evaluate systems of sub-task 1.
- Communication: [mailing lists: participants, organizers]
- Data: [arguments: api, index, image:
registry.webis.de/code-lib/public-images/touche25-rad-claimrev-index:latest, example queries] [training claims]
Important Dates
Subscribe to the Touché mailing list to receive notifications.
- 2024-11: CLEF Registration opened [register]
- 2025-05-10: Approaches submission deadline
- 2025-05-30: Participant paper submission
- 2025-06-10: Peer review notification
- 2025-07-07: Camera-ready participant papers submission
- 2025-09: CLEF Conference in Madrid and Touché Workshop
All deadlines are 23:59 CEST (UTC+2).
Task
This task serves to develop generative retrieval systems that argue against their users to support users in forming or confirming opinions or to train their debating skills. Participating systems are debated by simulated users in multiple turns (following the procedure shown below) and evaluated based on their responses.
Sub-Task 1: Participants submit systems that respond with an utterance Si, to a (simulated) user's utterance Ui by (1) retrieving either counterarguments (to Ui) or supporting evidence (for the attacked S) from a provided argument collection and (2) generating a response (of at most 60 words) from the retrieved data.
Sub-Task 2: Participants submit systems that assess the participating systems of sub-task 1 in terms of one or more of these criteria:
- Quantity: be informative. Does the response contain at least one (attack or defense) argument, and at most one of each type of defense and attack?
- Quality: be truthful. Can the response be deduced from the retrieved arguments?
- Relation: be relevant. Is the response coherent with the conversation and does it express a contrary stance to the user?
- Manner: be clear. Is the response clear and precise?
Join the Touché mailing list to stay up-to-date.
Data
Debate systems (Sub-Task 1) have to use our argument index to retrieve arguments for the simulated debates. We further provide a set of 100 curated claims sourced from the ChangeMyView subreddit and simulated debates (judged by us) for training debate systems (Sub-Task 1) or evaluation systems (Sub-Task 2).
Argument Index
We provide a collection of about 300 000 arguments from the ClaimRev dataset. Arguments can be retrieved either by making requests to our publicly available Elasticsearch API at https://touche25-rad.webis.de/arguments/ (example queries) or by deploying a local instance of Elasticsearch either using our Docker image with the index included (registry.webis.de/code-lib/public-images/touche25-rad-claimrev-index:latest) or our index snapshot.
An argument from the index consists of the following fields:
id: Id of the argument from the Claimrev dataset.topic: A general topic the arguments belongs to (e.g., Does pineapple belong on pizza?).tags: A list of categories associated with the topic (e.g., [Food, Pizza]).attacks: An argument that is attacked by this argument.attacks_embedding_stella: A 1024-dimensional dense embedding vector of theattacksfield.supports: An argument that is supported by this argument.supports_embedding_stella: A 1024-dimensional dense embedding vector of thesupportsfield.text: The actual text of the argument.text_embedding_stella: A 1024-dimensional dense embedding vector of thetextfield.references: A list of references to back up the argument.original: This field indicates if eitherattacksorsupportswas part of the ClaimRev dataset or was automatically generated by us.
stella_en_400M_v5 model which is available on Huggingface. These embeddings enable dense retrieval through kNN search with Elasticsearch (example query).
Example query to find arguments that support pineapple belongs on pizza using BM25.
Request
curl -X POST "https://touche25-rad.webis.de/arguments/claimrev/_search?pretty&size=10" -H "Content-Type: application/json" -d '
{
"query": {
"match": {
"supports": {
"query": "pineapple on pizza",
"operator": "or"
}
}
},
"_source": {
"excludes": ["text_embedding_stella", "supports_embedding_stella", "attacks_embedding_stella"]
}
}'
Response
{
"took": 35,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score": 35.958282,
"hits" : [
{
"_index" : "claimrev",
"_id" : "10104.4",
"_score": 35.958282,
"_source" : {
"id": "10104.4",
"topic": "Does pineapple belong on pizza?",
"tags": [
"Food",
"Pizza",
"Hawaiian",
"Pineapple"
],
"attacks": "It's not always a good thing to have another recipe to make.",
"supports": "Pineapple belongs on pizza.",
"text": "From a culinary point of view, it's always a good thing to have another recipe to make.",
"references": [],
"original": "supports"
}
},
{
"_index" : "claimrev",
"_id" : "10104.349",
"_score": 35.958282,
"_source" : {
"id": "10104.349",
"topic": "Does pineapple belong on pizza?",
"tags": [
"Food",
"Pizza",
"Hawaiian",
"Pineapple"
],
"attacks": "Pineapples do not make pizza aesthetically pleasing.",
"supports": "Pineapple belongs on pizza.",
"text": "Pineapples make pizza aesthetically pleasing.",
"references": [],
"original": "supports"
}
},
{
"_index" : "claimrev",
"_id" : "10104.425",
"_score": 35.958282,
"_source" : {
"id": "10104.425",
"topic": "Does pineapple belong on pizza?",
"tags": [
"Food",
"Pizza",
"Hawaiian",
"Pineapple"
],
"attacks": "It is a coincidence that pineapple and pizza both start with pi.",
"supports": "Pineapple belongs on pizza.",
"text": "It's no coincidence that pineapple and pizza both start with pi.",
"references": [],
"original": "supports"
}
}
]
}
}
Claims and Debates
We provide 100 claims from ChangeMyView, debates simulated based on these, and human judgment of the system responses on Zenodo (see there for more information).
Example claim about school uniforms.
{
"id": "touche25-rad-cmv-13fhkeg",
"claim": "School uniforms are a good thing.",
"description": "Firstly, school uniforms help to promote a sense of unity and belonging among students. When everyone is dressed in the same attire, it eliminates any competition or judgement based on clothing choices. This can lead to a more inclusive and positive school environment.\nAdditionally, school uniforms can help to reduce bullying and discrimination. When everyone is wearing the same clothes, there is less opportunity for students to make fun of others who may not have the latest fashion trends or expensive clothing brands.\nFurthermore, school uniforms can help to increase academic performance. Without the pressure of trying to keep up with fashion trends, students can focus more on their studies and less on their appearance. This can lead to better grades and overall success in school.\n\nOverall, while some may argue that school uniforms limit individual expression, I believe that the benefits they provide in terms of promoting unity, reducing bullying, and improving academic performance make them a worthwhile investment. It's school, not a fashion show."
}
Task Committee
