Intra-Multilingual Multi-Target Stance Classification 2023
- Task: Given a proposal on a socially important issue, the task is to classify whether a comment is in favor, against, or neutral towards the proposal.
- Input: data
- Submission: [submit].
The goal of Task 4 is to support to support opinion formation on socially important topics. Given a proposal on a socially important issue, its title and topic, the task is to classify whether a comment is in favor, against, or neutral towards the proposal. The proposals and titles can be written in any of the 24 EU languages (plus Catalan and Esperanto) and will come with their automatic English translation.
Subtask 1: Cross-Debate Classification. In the first subtask, the participants cannot use in their training set the examples from debates that are in the test set.
Subtask 2: All-data-available. In the second subtask, the participants can use all the available data, labeled or unlabeled.Register now
The data used in this Shared Task comes from the CoFE dataset, presented at AACL-IJCNLP 2022 [oral presentation] [slides].
CF_S dataset is composed of 7K comments from CoFE that are annotated with stance by the writer. The annotation is binary: in favor or against. It can be found in the column "alignment" .
CF_U dataset is composed of 12K unlabeled comments from CoFE. All the "alignment" are "None".
CF_E-Dev dataset is composed of multiligual comments that have been annotated by external coders (not the writer of the comment) in a 3-class fashion. It is consider as silver-standard because of a low inter-annotator agreement. The labels are in the columns "label". Note that all the "alignment" of those comments are "None".
CF_E-Test dataset is composed of 1.2K comments in English, French, German, Italian, Greek, and Hungarian that have been annotated by external coders (not the writer of the comment) in a 3-class fashion. It is considered as gold-standard because of a higher inter-annotator agreement. This dataset will used for the test phase. This test set will be available a few days before the test phase.
The datasets to use for the Subtask 1: Cross-Debate Classification:
- CF_U without the comments from the debates of the test set CF_E-Test is available [here].
- CF_S without the comments from the debates of test set CF_E-Test is available [here].
- CF_E-Dev without the comments from debates of the test set CF_E-Test is available [here].
The datasets to use for the Subtask 2: All-data-available:
The teams are free to use every other datasets they want to.
All the proposals are in a separate file containing all the 4.2K proposals. The column "id" of the proposal file corresponds to the column "id_prop" of the comment files. [download].
- For the proposals: the title of the proposal, the topic, the ID of the writter, the language of the text, and the number of endorsments.
- For the comments: the ID of the writter, the language of the text and the number of up/downvote.
Example proposal instance for Task 4:
|43992||Massentierhalter EU-Agrarsubventionen||Die EU hat "schöne" Ideen und arbeitet an Gesetzte für den Tierschutz und die Umwelt. Doch solange immer noch Massentierbetriebe von der EU durch Subventionen unterstützt werden, ist dies alles nicht glaubwürdig!||The EU has "beautiful" ideas and is working on legislation for animal welfare and the environment. But as long as EU subsidies still support mass animal farms, all of this is not credible!||Mass livestock farmers EU agricultural subsidies||GreenDeal||de||1|
|11138||Författningsdomstol||För att säkra skyddet av konstitutionen i varje EU-land bör krav ställas av EU att en oberoende konstitutionsdomstol finns i varje land. Exempelvis saknas en sådan i Sverige vilket leder till låg kvalité på stiftade lagar samt lagar som inte är anpassade till grundlagen.||In order to ensure the protection of the Constitution in each EU country, the EU should be required to have an independent constitutional court in each country. For example, there is no one in Sweden, which results in a low quality of enacted laws and laws that are not in line with the Constitution.||Constitutional court||ValuesRights||sv||0|
Example comment instance for Task 4:
|id||id_prop||alignment||comment||depth||thread_id||last_ comment_ in_thread||upvote||downvote||Topic||lan||time|
|comment_ 90817||89992||Against||Europäische Waffenexporte leisten einen großen Beitrag zum zum Elend in der Welt [...]||0||comment_ 90817||True||0||0||EUInTheWorld||de||2021-09-14 T10:28:46+02:00|
|comment_ 1330||417||Against||Je ne comprend pas très bien ce que l'on reproche a l'UE (vous devriez pouvoir m'éclairer) [...]||0||comment_ 1330||True||0||ValuesRights||fr||2021-04-22 T18:55:25+02:00|
Results will be evaluated using a micro-averaged F1 over the 3 stance classes.
We ask participants to use TIRA for result submissions.
TIRA Tutorial and TIRA Baselines
We provide a TIRA tutorial that provides baselines that can be executed in TIRA at https://github.com/touche-webis-de/touche-code/tree/main/clef23.