Intra-Multilingual Multi-Target Stance Classification 2023

Synopsis

  • Task: Given a proposal on a socially important issue, the task is to classify whether a comment is in favor, against, or neutral towards the proposal.
  • Input: data
  • Submission: TBA
  • Evaluation: TBA

Task

The goal of Task 4 is to support to support opinion formation on socially important topics. Given a proposal on a socially important issue, its title and topic, the task is to classify whether a comment is in favor, against, or neutral towards the proposal.

Subtask 1: Cross-Debate Classification. In the first subtask, the participants cannot use in their training set examples from debates that are in the test set.

Subtask 2: Only-labeled-data Training. In the second subtask, the participants can use all the available data, labeled or unlabeled.

Register now

Data

The data used in this Shared Task comes from the CoFE dataset, presented at AACL-IJCNLP 2022 [oral presentation] [slides].

CFS dataset is composed of 7K comments from CoFE that are annotated with stance by the writer. The annotation is binary: in favor or against [download].

CFU dataset is composed of 12K unlabeled comments from CoFE [download].

CFE-Dev dataset is composed of multiligual comments that have been annotated by external coders (not the writer of the comment) with 3 stance labels. This dataset will be available in December.

CFE-Test dataset is composed of 1.2K comments in English, French, German, Italian, Greek, and Hungarian that have been annotated by external coders (not the writer of the comment) with 3 stance labels. This dataset will be used for testing.

All the proposals are in a separate file containing all the 4.2K proposals [download].

Additional metadata:
  • For the proposals: the title of the proposal, the topic, the ID of the writter, the language of the text and the number of endorsments.
  • For the comments: the ID of the writter, the language of the text and the number of up/downvote.

Example proposal instance for Task 4:

id user_name title proposal proposal_en title_en topic lan endors.
43992 user_4897 Massentierhalter EU-Agrarsubventionen Die EU hat "schöne" Ideen und arbeitet an Gesetzte für den Tierschutz und die Umwelt. Doch solange immer noch Massentierbetriebe von der EU durch Subventionen unterstützt werden, ist dies alles nicht glaubwürdig! The EU has "beautiful" ideas and is working on legislation for animal welfare and the environment. But as long as EU subsidies still support mass animal farms, all of this is not credible! Mass livestock farmers EU agricultural subsidies GreenDeal de 1
11138 user_7109 Författningsdomstol För att säkra skyddet av konstitutionen i varje EU-land bör krav ställas av EU att en oberoende konstitutionsdomstol finns i varje land. Exempelvis saknas en sådan i Sverige vilket leder till låg kvalité på stiftade lagar samt lagar som inte är anpassade till grundlagen. In order to ensure the protection of the Constitution in each EU country, the EU should be required to have an independent constitutional court in each country. For example, there is no one in Sweden, which results in a low quality of enacted laws and laws that are not in line with the Constitution. Constitutional court ValuesRights sv 0

Example comment instance for Task 4:

id user_name id_prop alignment comment depth thread_id last_ comment_ in_thread upvote downvote Topic lan time
comment_ 90817 user_5 89992 Against Europäische Waffenexporte leisten einen großen Beitrag zum zum Elend in der Welt [...] 0 comment_ 90817 True 0 0 EUInTheWorld de 2021-09-14 T10:28:46+02:00
comment_ 1330 user_60 417 Against Je ne comprend pas très bien ce que l'on reproche a l'UE (vous devriez pouvoir m'éclairer) [...] 0 comment_ 1330 True 0 ValuesRights fr 2021-04-22 T18:55:25+02:00

Evaluation

Results will be evaluated using a micro-averaged F1 over the 3 stance classes.

Submission

We ask participants to use TIRA for result submissions.

Task Committee