Image Retrieval for Arguments 2023
Synopsis
- Task: Given a controversial topic, the task is to retrieve images (from web pages) for each stance (pro/con) that show support for that stance.
- Input: [topics] [data]
- Submission: [submit]
Data
This task uses a focused crawl of about 56K images (and associated web pages) as document collection. The crawl is created from several image search queries for each of the 50 topics. The relevance judgments of last year are available for training supervised approaches. The output of last year's approaches can be inspected on images.args.me. Registered teams were invited in 2022 to submit search queries that were then used to extend the crawl. [data]
Evaluation
Like last year, systems are evaluated on 50 topics by the ratio of images among the 20 retrieved images for each topic (10 images for each stance) that are relevant for the respective topic and stance. [topics]
Submission
This task uses TIRA for submissions, which allows for both run file upload and Docker image submission. For each topic and stance, include 10 retrieved images. Each team can submit up to 5 different runs.
The submission format adapts the standard TREC format. Each line corresponds to an image retrieved for some topic and stance at a certain rank, making a run file 1000 lines long (50 topics, 2 stances, 10 ranks). Each line contains the following fields, separated by single whitespaces:
- The topic number (51 to 100).
- The stance ("PRO" or "CON").
- The image's ID (corresponds to the name of the image's directory in the collection; always 17 characters long and starts with "I").
- The rank (1 to 10 in increasing order per topic and stance). Not used in this year's evaluation.
- A score (integer or floating point; non-increasing per topic and stance). Not used in this year's evaluation.
- A tag that identifies your group and the method you used to produce the run.
1 PRO I000330ba4ea0ad13 1 17.89 myGroupMyMethod 1 PRO I0005e6fe00ea17fd 2 16.43 myGroupMyMethod ... 1 CON I0009d5f038fe6f2e 1 15.89 myGroupMyMethod 1 CON I000f34bd3f8cb030 2 14.43 myGroupMyMethod ...
TIRA Tutorial and TIRA Baselines
We provide a TIRA tutorial that provides baselines that can be executed in TIRA at https://github.com/touche-webis-de/touche-code/tree/main/clef23.
In case of problems or questions concerning TIRA, please use the TIRA forum.
Results
| Team | Tag | Precision@10 | ||
|---|---|---|---|---|
| On topic | Argumentative | On stance | ||
| Neville Longbottom | clip_chatgpt_args.raw | 0.785 | 0.338 | 0.222 |
| Neville Longbottom | clip_chatgpt_args.debater | 0.684 | 0.341 | 0.216 |
| Hikaru Sulu | Keywords | 0.664 | 0.350 | 0.185 |
| Hikaru Sulu | Topic-title | 0.770 | 0.335 | 0.179 |
| Neville Longbottom | bm25_chatgpt_args.raw | 0.572 | 0.274 | 0.166 |
| Jean-Luc Picard | No stance detection (0) | 0.523 | 0.292 | 0.162 |
| Neville Longbottom | bm25_chatgpt_args.diff | 0.442 | 0.240 | 0.150 |
| Jean-Luc Picard | Text and image text stance detection (3) | 0.502 | 0.272 | 0.144 |
| Jean-Luc Picard | BM25 Baseline (-1) | 0.536 | 0.268 | 0.141 |
| Jean-Luc Picard | Text stance detection (1) | 0.498 | 0.262 | 0.136 |
| Neville Longbottom | bm25_chatgpt_args.debater | 0.416 | 0.201 | 0.128 |
| Minsc | Aramis | 0.376 | 0.194 | 0.102 |
| Jean-Luc Picard | Image text stance detection (2) | 0.369 | 0.196 | 0.098 |
Related Work
- Johannes Kiesel, Nico Reichenbach, Benno Stein, and Martin Potthast. Image Retrieval for Arguments Using Stance-Aware Query Expansion. 8th Workshop on Argument Mining (ArgMining 2021) at EMNLP, November 2021.
- Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images. 15th International Workshop on Semantic Evaluation (SemEval 2021), August 2021.
- Keiji Yanai. Image collector III: a web image-gathering system with bag-of-keypoints. 16th International Conference on World Wide Web (WWW 2007), May 2007.
Task Committee
