Human Value Detection 2024

Synopsis

• Sub-Task 1: Given a text, for each sentence, detect which human values the sentence refers to.
• Sub-Task 2: Given a text, for each sentence and human value this sentence refers to, detect whether this reference (partially) attains or (partially) constrains the value.
• Communication: [mailing lists: task, organizers] [twitter/x]
• Data: [download] [project] [annotation guidelines]
• Submission: [example approaches] [web ui] [evaluator] [forum] [submit]
• ValueEval'23: [website] [demo]

Important Dates

Subscribe to the mailing list to receive notifications.

• May 20, 2024: EXTENDED approaches submission deadline. [register via submission system (SUBMIT)]
• May 31, 2024: Participant paper submission [paper template + submission instructions].
• June 24, 2024: Peer review notification.
• July 8, 2024: Camera-ready participant papers submission.
• Sep. 9-12, 2024: CLEF Conference in Grenoble and Touché Workshop.

All deadlines are 23:59 CEST (UTC+2).

Task

Identify human values (sub-task 1) and their attainment (sub-task 2) in long texts in eight languages.

The task employs a collection of roughly 3000 human-annotated texts between 400 and 800 words that is created as part of ValuesML. The annotated values are those of the Schwartz' value continuum (see below and click to see the value's description):

Each referred value (detected in sub-task 1) can either be mentioned as something that is or should be attained or something that is not attained or constrained (detected in sub-task 2). Attainment would mean that whatever is described in the sentence will help lead to fulfilling the value. For Security (personal or societal), attainment would mean that something is made safer or healthier. In contrast, an event can be stated in a way that thwarts/constrains safety or health.

Data

Data is provided as tab-separated values files with one header line. The data stems from the ValuesML project. In addition to the original files in nine languages we provide a machine-translated version in English. Stay up-to-date and report problems on the task mailing list.

The sentences.tsv file contains one sentence per line: the Text-ID identifies the text that contains the sentence, the Sentence-ID gives the index of the sentence in the text, and Text is the sentence text itself. Toy example with tab-separated columns highlighted: [toy example]

Text-ID	Sentence-ID	Text
A	1	First text first sentence.
A	2	First text second sentence.
A	3	First text third sentence.
B	1	Second text first sentence.
B	2	Second text second sentence.

The labels.tsv file contains one sentence per line: the Text-ID and Sentence-ID as above, and for each of the 19 values two columns—one column value attained with a 1 meaning that the sentences refers to this value and (partially) attains it; and one column value constrained with a 1 meaning that the sentences refers to this value and (partially) constrains it. If both are 0 the sentence does not refer to that value at all. If both are 0.5 the sentence refers to the value but it is unclear whether it (even partially) attains or constrains it. Toy example with tab-separated columns highlighted: [toy example]

Text-ID	Sentence-ID	Self-direction: thought attained	Self-direction: thought constrained	Self-direction: action attained	Self-direction: action constrained	Stimulation attained	Stimulation constrained	Hedonism attained	Hedonism constrained	Achievement attained	Achievement constrained	Power: dominance attained	Power: dominance constrained	Power: resources attained	Power: resources constrained	Face attained	Face constrained	Security: personal attained	Security: personal constrained	Security: societal attained	Security: societal constrained	Tradition attained	Tradition constrained	Conformity: rules attained	Conformity: rules constrained	Conformity: interpersonal attained	Conformity: interpersonal constrained	Humility attained	Humility constrained	Benevolence: caring attained	Benevolence: caring constrained	Benevolence: dependability attained	Benevolence: dependability constrained	Universalism: concern attained	Universalism: concern constrained	Universalism: nature attained	Universalism: nature constrained	Universalism: tolerance attained	Universalism: tolerance constrained
A	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0
A	2	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0
A	3	0.5	0.5	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1
B	1	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0
B	2	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1

For sub-task 1 an approach thus has to identify the values for each sentence where at least one is not 0. For sub-task 2 an approach has to identify for those values for a sentence where at least one is 1, whether the 1 is in the attained or constrained column.

Submission

Submit your approach via TIRA. Ask in the Forum if you need help. You need to register your team (in addition to a registration at CLEF) and pick an alias for your team name (submission is anonymous; you can reveal you true team name after final paper acceptance). You submit a Docker image or submit from your Github repository (via automated Docker building). In case of trouble, you can also submit via run file upload (not recommended due to poor reproducibility; rather contact us if you need help with Dockerization). You can submit on the validation dataset to check how submission works or the test dataset. You will not be able to see your results on the test dataset until after the deadline. Datasets are provided multilingual or in machine-translated English (see Data). [forum] [submit]

We recommend to start your approach from one of our example approaches (in Python), which include the code for reading and writing the files and make it easy to later deploy your approach as server or submit and distribute it as Docker image. If you run them as a local server, you can use our web ui to use them interactively. [random baseline: script, notebook] [bert baseline] [ollama baseline]

Approaches need to produce run files that have the same format as the labels.tsv, but the numbers can be between 0 and 1 and are interpreted as the confidence of the approach (employed for evaluation via ROC-curves): [toy example]

• For sub-task 1: For each sentence and value, the sum of the numbers in the attained and constrained columns should be the confidence of your approach in that the sentence references the value. A sum ≥ 0.5 is treated as a positive prediction for purposes of evaluation with precision, recall, and F1-score.
  Example: An approach predicts for some sentence, "Face attained" = 0.1 and "Face constrained" = 0.2 ⇒ approach's confidence (sentence resorts to "Face") = 0.1 + 0.2 = 0.3 ⇒ approach's confidence < 0.5 ⇒ the approach predicted the sentence does not refer to "Face".

• For sub-task 2: For each sentence and value, the number for attained normalized (i.e., divided) by the sum should be the confidence of your approach in that the reference (partially) attains the value. A normalized number ≥ 0.5 is treated as prediction for attained (and otherwise constrained) for purposes of evaluation with precision, recall, and F1-score. Thus the higher number is treated as the prediction.
  Example: An approach predicts for some sentence, "Face attained" = 0.1 and "Face constrained" = 0.2 ⇒ approach's confidence (sentence rather attains than constrains "Face") = 0.1 / (0.1 + 0.2) = 0.33 ⇒ approach's confidence < 0.5 ⇒ the approach predicted the sentence rather constrains than attains "Face".

Note that you submit for both sub-tasks with the same file. If you want to participate only in sub-task 1, always set the number for constrained to 0. If you want to participate only in sub-task 2, the sum for attained and constrained for a value does not matter, only their ratio. If you want to participate in both sub-tasks, avoid submitting the same number for attained or constrained for a value even if your approach is certain that the value is not referenced: if the approach is wrong and the value is actually referenced, it still matters for sub-task 2 which number is the larger one.

Evaluation

For both sub-tasks, the submission system will evaluate runs automatically using F1-score, Precision, Recall, and ROC-curves (for each value and averaged). Runs on the sub-task leaderboards are ranked according to averaged F1-score. Use our evaluator to produce a detailed report (as HTML) for your runs, including ROC curves and tables for identifying the sentences your approach got the most wrong. [evaluator]

Results

Related Work

• Johannes Kiesel, Milad Alshomary, Nailia Mirzakhmedova, Maximilian Heinrich, Nicolas Handke, Henning Wachsmuth, and Benno Stein. SemEval-2023 Task 4: ValueEval: Identification of Human Values behind Arguments. In Ritesh Kumar et al., editors, 17th International Workshop on Semantic Evaluation (SemEval 2023), pages 2287-2303, July 2023. Association for Computational Linguistics.

• Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein. Identifying the Human Values behind Arguments. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022), pages 4459-4471, May 2022. Association for Computational Linguistics.

• Mario Scharfbillig, Laura Smillie, David Mair, Marta Sienkiewicz, Julian Keimer, Raquel Pinho Dos Santos, Hélder Vinagreiro Alves, Elise Vecchione, and Laurenz Scheunemann. Values and Identities - a Policymaker’s Guide. Technical Report KJ-NA-30800-EN-N, 2021. European Commission’s Joint Research Centre.

• Shalom H. Schwartz, Jan Cieciuch, Michele Vecchione, Eldad Davidov, Ronald Fischer, Constanze Beierlein, Alice Ramos, Markku Verkasalo, Jan-Erik Lönnqvist, Kursad Demirutku, Ozlem Dirilen-gumus, and Mark Konty. Refining the Theory of Basic Individual Values. Journal of Personality and Social Psychology, 103(4), pages 663-688, 2012.

Task Committee

Johannes Kiesel

GESIS - Leibniz Institute for the Social Sciences

Milad Alshomary

Leibniz University Hannover

Nailia Mirzakhmedova

Bauhaus-Universität Weimar

Nicolas Handke

Leipzig University

Bertrand De Longueville

European Commission Joint Research Centre

Theresa Reitis-Münstermann

European Commission Joint Research Centre

Mario Scharfbillig

European Commission Joint Research Centre

Nicolas Stefanovitch

European Commission Joint Research Centre

Henning Wachsmuth

Leibniz University Hannover

Benno Stein

Bauhaus-Universität Weimar