Intra-Multilingual Multi-Target Stance Classification 2023

Synopsis
Task
Data
Evaluation
Submission
Task Committee

Synopsis

Task: Given a proposal on a socially important issue, the task is to classify whether a comment is in favor, against, or neutral towards the proposal.
Input: data
Submission: [submit].

Task

The goal of Task 4 is to support to support opinion formation on socially important topics. Given a proposal on a socially important issue, its title and topic, the task is to classify whether a comment is in favor, against, or neutral towards the proposal. The proposals and titles can be written in any of the 24 EU languages (plus Catalan and Esperanto) and will come with their automatic English translation.

Subtask 1: Cross-Debate Classification. In the first subtask, the participants cannot use in their training set the examples from debates that are in the test set.

Subtask 2: All-data-available. In the second subtask, the participants can use all the available data, labeled or unlabeled.

Data

The data used in this Shared Task comes from the CoFE dataset, presented at AACL-IJCNLP 2022 [oral presentation] [slides].

CF_S dataset is composed of 7K comments from CoFE that are annotated with stance by the writer. The annotation is binary: in favor or against. It can be found in the column "alignment" .

CF_U dataset is composed of 12K unlabeled comments from CoFE. All the "alignment" are "None".

CF_E-Dev dataset is composed of multiligual comments that have been annotated by external coders (not the writer of the comment) in a 3-class fashion. It is consider as silver-standard because of a low inter-annotator agreement. The labels are in the columns "label". Note that all the "alignment" of those comments are "None".

CF_E-Test dataset is composed of 1.2K comments in English, French, German, Italian, Greek, and Hungarian that have been annotated by external coders (not the writer of the comment) in a 3-class fashion. It is considered as gold-standard because of a higher inter-annotator agreement. This dataset will used for the test phase. The test set is now available [here].

The datasets to use for the Subtask 1: Cross-Debate Classification:

CF_U without the comments from the debates of the test set CF_E-Test is available [here].
CF_S without the comments from the debates of test set CF_E-Test is available [here].
CF_E-Dev without the comments from debates of the test set CF_E-Test is available [here].

The datasets to use for the Subtask 2: All-data-available:

CF_U is available [here].
CF_S is available [here].
CF_E-Dev is available [here].

The teams are free to use every other datasets they want to.

All the proposals are in a separate file containing all the 4.2K proposals. The column "id" of the proposal file corresponds to the column "id_prop" of the comment files. [download].

Additional metadata:

For the proposals: the title of the proposal, the topic, the ID of the writter, the language of the text, and the number of endorsments.
For the comments: the ID of the writter, the language of the text and the number of up/downvote.

Example proposal instance for Task 4:

id	title	proposal	proposal_en	title_en	topic	lan	endors.
43992	Massentierhalter EU-Agrarsubventionen	Die EU hat "schöne" Ideen und arbeitet an Gesetzte für den Tierschutz und die Umwelt. Doch solange immer noch Massentierbetriebe von der EU durch Subventionen unterstützt werden, ist dies alles nicht glaubwürdig!	The EU has "beautiful" ideas and is working on legislation for animal welfare and the environment. But as long as EU subsidies still support mass animal farms, all of this is not credible!	Mass livestock farmers EU agricultural subsidies	GreenDeal	de	1
11138	Författningsdomstol	För att säkra skyddet av konstitutionen i varje EU-land bör krav ställas av EU att en oberoende konstitutionsdomstol finns i varje land. Exempelvis saknas en sådan i Sverige vilket leder till låg kvalité på stiftade lagar samt lagar som inte är anpassade till grundlagen.	In order to ensure the protection of the Constitution in each EU country, the EU should be required to have an independent constitutional court in each country. For example, there is no one in Sweden, which results in a low quality of enacted laws and laws that are not in line with the Constitution.	Constitutional court	ValuesRights	sv	0

Example comment instance for Task 4:

id	id_prop	alignment	comment	depth	thread_id	last_ comment_ in_thread	upvote	downvote	Topic	lan	time
comment_ 90817	89992	Against	Europäische Waffenexporte leisten einen großen Beitrag zum zum Elend in der Welt [...]	0	comment_ 90817	True	0	0	EUInTheWorld	de	2021-09-14 T10:28:46+02:00
comment_ 1330	417	Against	Je ne comprend pas très bien ce que l'on reproche a l'UE (vous devriez pouvoir m'éclairer) [...]	0	comment_ 1330	True		0	ValuesRights	fr	2021-04-22 T18:55:25+02:00