Human Value Detection 2023

Synopsis

• Task: Given a textual argument and a human value category, classify whether or not the argument draws on that category. [paper]
• Communication: [mailing lists: task, organizers] [twitter]
• Input: [data download, paper]
• Submission: [validator] [evaluator] [baselines] [paper template: overleaf, download, tables, radar plots] [tira]
• ValueEval'24: [website]

Video

Important Dates

• In November, 2022: Run submission opens.
• December 16, 2022: Early-bird run submission closes.
• January 25, 2023: Registration closes.
• January 27, 2023: Run submission closes (extended).
• January 31, 2023: Run evaluation results out. [tira]
• February 28, 2023: Paper submission deadline. [template: overleaf, download, tables, radar plots] [submission: requirements, site]
• March 31, 2023: Paper notification.
• April 24, 2023: Camera-ready paper submission deadline.
• July 13-14, 2023: SemEval Workshop (co-located with ACL'23 in Toronto, Canada).

All deadlines are 23:59 anywhere on earth (UTC-12).

Task

Given a textual argument and a human value category, classify whether or not the argument draws on that category. This task uses a set of 20 value categories compiled from the social science literature and described in our ACL paper. Participants can submit runs (also known as approaches or systems) that detect one, a subset, or all of these values in arguments. Arguments are given as premise text, conclusion text, and binary stance of the premise to the conclusion ("in favor of" or "against").

The 20 value categories are shown here on Schwartz' value continuum (click to see description):

If you want to focus on a few categories but are unsure which to pick, we suggest to take (from) the most frequent ones in our initial dataset: Self-direction: action, Achievement, Security: personal, Security: societal, Benevolence: caring, Universalism: concern.

Data

Data is provided as tab-separated values files with one header line. Stay up-to-date and report problems on the task mailing list. [download] [description paper]

The arguments-training/validation/test.tsv files contain one argument per line: its unique argument ID, the conclusion, the premise's stance towards the conclusion, and the premise itself. Example with tab-separated columns highlighted:

Argument ID	Conclusion	Stance	Premise
A01010	We should prohibit school prayer	against	it should be allowed if the student wants to pray as long as it is not interfering with his classes
A01011	We should abolish the three-strikes laws	in favor of	three strike laws can cause young people to be put away for life without a chance to straight out their life
A01012	The use of public defenders should be mandatory	in favor of	the use of public defenders should be mandatory because some people don't have money for a lawyer and this would help those that don't

The labels-training/validation/test.tsv files also contain one argument per line: its unique argument ID and one column for each of the 20 value categories with a 1 meaning that the argument resorts to the value category and a 0 that not. Example with tab-separated columns highlighted:

Argument ID	Self-direction: thought	Self-direction: action	Stimulation	Hedonism	Achievement	Power: dominance	Power: resources	Face	Security: personal	Security: societal	Tradition	Conformity: rules	Conformity: interpersonal	Humility	Benevolence: caring	Benevolence: dependability	Universalism: concern	Universalism: nature	Universalism: tolerance	Universalism: objectivity
A01010	1	1	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	0	0	0
A01011	0	0	0	0	1	0	0	1	0	0	0	0	0	0	1	0	0	0	1	1
A01012	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0

In addition, we provide three datasets for evaluating the robustness of your approaches: validation-zhihu from the recommendation and hotlist section of the Chinese question-answering website Zhihu (labels available), test-nahjalbalagha from and based on the Nahj al-Balagha (labels were kept secret during competition), and test-nyt from New York Times articles related to the Coronavirus (labels were kept secret during competition; articles have to be downloaded unless doing a Docker submission).

Evaluation

Runs are evaluated on the basis of F1-score, Precision, and Recall in TIRA: averaged over all value categories and for each category individually. For the shared task leaderboard, the runs are ranked according to the averaged F1-score on the "arguments-test" dataset, corresponding to the arguments-test.tsv of the dataset.

Results

Download the results for all runs submitted for the deadline as [tab-separated values file].

Main

Nahj al-Balagha

New York Times

Task Committee