Image Retrieval for Arguments 2023

Synopsis

  • Task: Given a controversial topic, the task is to retrieve images (from web pages) for each stance (pro/con) that show support for that stance.
  • Input: [topics] [data]
  • Submission: [queries (until Dec. 31): info, baselines code, validator] [submit]
Register now

Data

This task uses a focused crawl of about 30,000 images (and associated web pages) as document collection. The crawl is created from several image search queries for each of the 50 topics. The relevance judgments of last year are available for training supervised approaches. The output of last year's approaches can be inspected on images.args.me. There is already a version of the crawl available, but it will be constantly improved until about mid January (feedback welcome!). Registered teams are invited to submit additional search queries to extend the crawl (see below). [data]

Query Submission

Registered teams can submit queries on each of the 50 topics until December 31, 2022. Both automatically and manually created queries are welcome. We will integrate the results of submitted queries into the dataset in early 2023. submitted queries will be evaluated using precision@k, similar to but separate from the main task, and the results will be published for each topic. a valid submission may include queries on fewer than all 50 topics.

Submission uses TIRA, specifically the query-submission-2023 dataset. Either substitute "PRO" and "CON" in the query submission form and use "upload submission" in TIRA to submit it, or provide a generic query expansion method as Docker container and use TIRA's "Docker submission". See the baselines code to get you started with Docker submissions. Up to two valid submissions are allowed per team (please write us a mail if you did more than two submissions). [topics] [form] [submit]

In case of problems or questions concerning TIRA, please use the TIRA forum. Note: At the moment, the TIRA web interface sometimes fails to provide feedback on actions. Reload the page in such cases.

Evaluation

Like last year, systems are evaluated on 50 topics by the ratio of images among the 20 retrieved images for each topic (10 images for each stance) that are relevant for the respective topic and stance. [topics]

Submission

This task uses TIRA for submissions, which allows for both run file upload and Docker image submission. For each topic and stance, include 10 retrieved images. Each team can submit up to 5 different runs.

The submission format adapts the standard TREC format. Each line corresponds to an image retrieved for some topic and stance at a certain rank, making a run file 1000 lines long (50 topics, 2 stances, 10 ranks). Each line contains the following fields, separated by single whitespaces:

  • The topic number (51 to 100).
  • The stance ("PRO" or "CON").
  • The image's ID (corresponds to the name of the image's directory in the collection; always 17 characters long and starts with "I").
  • The rank (1 to 10 in increasing order per topic and stance). Not used in this year's evaluation.
  • A score (integer or floating point; non-increasing per topic and stance). Not used in this year's evaluation.
  • A tag that identifies your group and the method you used to produce the run.
For example:
1 PRO I000330ba4ea0ad13 1 17.89 myGroupMyMethod
1 PRO I0005e6fe00ea17fd 2 16.43 myGroupMyMethod
...
1 CON I0009d5f038fe6f2e 1 15.89 myGroupMyMethod
1 CON I000f34bd3f8cb030 2 14.43 myGroupMyMethod
...

In case of problems or questions concerning TIRA, please use the TIRA forum.

Related Work

Task Committee