Intra-Multilingual Multi-Target Stance Classification 2023

Synopsis

  • Task: Given a proposal on a socially important issue, the task is to classify whether a comment is in favor, against, or neutral towards the proposal.
  • Input: data
  • Submission: [submit].

Task

The goal of Task 4 is to support to support opinion formation on socially important topics. Given a proposal on a socially important issue, its title and topic, the task is to classify whether a comment is in favor, against, or neutral towards the proposal. The proposals and titles can be written in any of the 24 EU languages (plus Catalan and Esperanto) and will come with their automatic English translation.

Subtask 1: Cross-Debate Classification. In the first subtask, the participants cannot use in their training set the examples from debates that are in the test set.

Subtask 2: All-data-available. In the second subtask, the participants can use all the available data, labeled or unlabeled.

Register now

Data

The data used in this Shared Task comes from the CoFE dataset, presented at AACL-IJCNLP 2022 [oral presentation] [slides].

CF_S dataset is composed of 7K comments from CoFE that are annotated with stance by the writer. The annotation is binary: in favor or against. It can be found in the column "alignment" .

CF_U dataset is composed of 12K unlabeled comments from CoFE. All the "alignment" are "None".

CF_E-Dev dataset is composed of multiligual comments that have been annotated by external coders (not the writer of the comment) in a 3-class fashion. It is consider as silver-standard because of a low inter-annotator agreement. The labels are in the columns "label". Note that all the "alignment" of those comments are "None".

CF_E-Test dataset is composed of 1.2K comments in English, French, German, Italian, Greek, and Hungarian that have been annotated by external coders (not the writer of the comment) in a 3-class fashion. It is considered as gold-standard because of a higher inter-annotator agreement. This dataset will used for the test phase. The test set is now available [here].

The datasets to use for the Subtask 1: Cross-Debate Classification:

  • CF_U without the comments from the debates of the test set CF_E-Test is available [here].
  • CF_S without the comments from the debates of test set CF_E-Test is available [here].
  • CF_E-Dev without the comments from debates of the test set CF_E-Test is available [here].

The datasets to use for the Subtask 2: All-data-available:

  • CF_U is available [here].
  • CF_S is available [here].
  • CF_E-Dev is available [here].

The teams are free to use every other datasets they want to.

All the proposals are in a separate file containing all the 4.2K proposals. The column "id" of the proposal file corresponds to the column "id_prop" of the comment files. [download].

Additional metadata:

  • For the proposals: the title of the proposal, the topic, the ID of the writter, the language of the text, and the number of endorsments.
  • For the comments: the ID of the writter, the language of the text and the number of up/downvote.

Example proposal instance for Task 4:

id title proposal proposal_en title_en topic lan endors.
43992 Massentierhalter EU-Agrarsubventionen Die EU hat "schöne" Ideen und arbeitet an Gesetzte für den Tierschutz und die Umwelt. Doch solange immer noch Massentierbetriebe von der EU durch Subventionen unterstützt werden, ist dies alles nicht glaubwürdig! The EU has "beautiful" ideas and is working on legislation for animal welfare and the environment. But as long as EU subsidies still support mass animal farms, all of this is not credible! Mass livestock farmers EU agricultural subsidies GreenDeal de 1
11138 Författningsdomstol För att säkra skyddet av konstitutionen i varje EU-land bör krav ställas av EU att en oberoende konstitutionsdomstol finns i varje land. Exempelvis saknas en sådan i Sverige vilket leder till låg kvalité på stiftade lagar samt lagar som inte är anpassade till grundlagen. In order to ensure the protection of the Constitution in each EU country, the EU should be required to have an independent constitutional court in each country. For example, there is no one in Sweden, which results in a low quality of enacted laws and laws that are not in line with the Constitution. Constitutional court ValuesRights sv 0

Example comment instance for Task 4:

id id_prop alignment comment depth thread_id last_ comment_ in_thread upvote downvote Topic lan time
comment_ 90817 89992 Against Europäische Waffenexporte leisten einen großen Beitrag zum zum Elend in der Welt [...] 0 comment_ 90817 True 0 0 EUInTheWorld de 2021-09-14 T10:28:46+02:00
comment_ 1330 417 Against Je ne comprend pas très bien ce que l'on reproche a l'UE (vous devriez pouvoir m'éclairer) [...] 0 comment_ 1330 True 0 ValuesRights fr 2021-04-22 T18:55:25+02:00

Evaluation

Results will be evaluated using a macro-averaged F1 over the 3 stance classes.

Submission

We ask participants to use TIRA for result submissions.

TIRA Tutorial and TIRA Baselines

We provide a TIRA tutorial that provides baselines that can be executed in TIRA at https://github.com/touche-webis-de/touche-code/tree/main/clef23.

Task Committee