Human Value Detection 2023

SemEval 2023 Task 4. ValueEval: Identification of Human Values behind Arguments

Synopsis

Register now

Video

Important Dates

  • In November, 2022: Run submission opens.
  • December 16, 2022: Early-bird run submission closes.
  • January 24, 2023: Run submission closes.
  • January 31, 2023: Run evaluation results out.
  • In February 2023: Paper submission deadline.
  • In March 2023: Paper notification.
  • In April 2023: Camera-ready paper submission deadline.
  • In Summer 2023: SemEval Workshop (co-located with a major NLP conference).

All deadlines are 23:59 anywhere on earth (UTC-12).

Task

Given a textual argument and a human value category, classify whether or not the argument draws on that category. This task uses a set of 20 value categories compiled from the social science literature and described in our ACL paper. Participants can submit runs (also known as approaches or systems) that detect one, a subset, or all of these values in arguments. Arguments are given as premise text, conclusion text, and binary stance of the premise to the conclusion ("in favor of" or "against").

The 20 value categories are shown here on Schwartz' value continuum (click to see description):

Self-direction: Power: Security: Conformity: Benevolence: Universalism: thought action Stimulation Hedonism Achievement dominance resources Face personal societal Tradition rules interpersonal Humility caring dependability concern nature tolerance objectivity

Self-direction: thought

It is good to have own ideas and interests.

Contained values and associated arguments (examples):

  • Be creative: arguments towards more creativity or imagination
  • Be curious: arguments towards more curiosity, discoveries, or general interestingness
  • Have freedom of thought: arguments toward people figuring things out on their own, towards less censorship, or towards less influence on thoughts

Self-direction: action

It is good to determine one's own actions.

Contained values and associated arguments (examples):

  • Be choosing own goals: arguments towards allowing people to choose what is best for them, to decide on their life, and to follow their dreams
  • Be independent: arguments towards allowing people to plan on their own and not ask for consent
  • Have freedom of action: arguments towards allowing people to be self-determined and able to do what they want
  • Have privacy: arguments towards allowing for private spaces, time alone, and less surveillance, or towards more control on what to disclose and to whom

Stimulation

It is good to experience excitement, novelty, and change.

Contained values and associated arguments (examples):

  • Have an exciting life: arguments towards allowing people to experience foreign places and special activities or having perspective-changing experiences
  • Have a varied life: arguments towards allowing people to engage in many activities and change parts of their life or towards promoting local clubs (sports, ...)
  • Be daring: arguments towards more risk-taking

Hedonism

It is good to experience pleasure and sensual gratification.

Contained values and associated arguments (examples):

  • Have pleasure: arguments towards making life enjoyable or providing leisure, opportunities to have fun, and sensual gratification

Achievement

It is good to be successful in accordance with social norms.

Contained values and associated arguments (examples):

  • Be ambitious: arguments towards allowing for ambitions and climbing up the social ladder
  • Have success: arguments towards allowing for success and recognizing achievements
  • Be capable: arguments towards acquiring competence in certain tasks, being more effective, and showing competence in solving tasks
  • Be intellectual: arguments towards acquiring high cognitive skills, being more reflective, and showing intelligence
  • Be courageous: arguments towards being more courageous and having people stand up for their beliefs

Power: dominance

It is good to be in positions of control over others.

Contained values and associated arguments (examples):

  • Have influence: arguments towards having more people to ask for a favor, more influence, and more ways to control events
  • Have the right to command: arguments towards allowing the right people to take command, putting experts in charge, and clearer hierarchies of command, or towards fostering leadership

Power: resources

It is good to have material possessions and social resources.

Contained values and associated arguments (examples):

  • Have wealth: arguments towards allowing people to gain wealth and material possession, show their wealth, and exercise control through wealth, or towards financial prosperity

Face

It is good to maintain one's public image.

Contained values and associated arguments (examples):

  • Have social recognition: arguments towards allowing people to gain respect and social recognition or avoid humiliation
  • Have a good reputation: arguments towards allowing people to build up their reputation, protect their public image, and spread reputation

Security: personal

It is good to have a secure immediate environment.

Contained values and associated arguments (examples):

  • Have a sense of belonging: arguments towards allowing people to establish, join, and stay in groups, show their group membership, and show that they care for each other, or towards fostering a sense of belonging
  • Have good health: arguments towards avoiding diseases, preserving health, or having physiological and mental well-being
  • Have no debts: arguments towards avoiding indebtedness and having people return favors
  • Be neat and tidy: arguments towards being more clean, neat, or orderly
  • Have a comfortable life: arguments towards providing subsistence income, having no financial worries, and having a prosperous life, or towards resulting in a higher general happiness

Security: societal

It is good to have a secure and stable wider society.

Contained values and associated arguments (examples):

  • Have a safe country: arguments towards a state that can better act on crimes, and defend or care for its citizens, or towards a stronger state in general
  • Have a stable society: arguments towards accepting or maintaining the existing social structure or towards preventing chaos and disorder at a societal level

Tradition

It is good to maintain cultural, family, or religious traditions.

Contained values and associated arguments (examples):

  • Be respecting traditions: arguments towards allowing to follow one's family's customs, honoring traditional practices, maintaining traditional values and ways of thinking, or promoting the preservation of customs
  • Be holding religious faith: arguments towards allowing the customs of a religion and to devote one's life to their faith, or towards promoting piety and the spreading of one's religion

Conformity: rules

It is good to comply with rules, laws, and formal obligations.

Contained values and associated arguments (examples):

  • Be compliant: arguments towards abiding to laws or rules and promoting to meet one's obligations or recognizing people who do
  • Be self-disciplined: arguments towards fostering to exercise restraint, follow rules even when no-one is watching, and to set rules for oneself
  • Be behaving properly: arguments towards avoiding to violate informal rules or social conventions or towards fostering good manners

Conformity: interpersonal

It is good to avoid upsetting or harming others.

Contained values and associated arguments (examples):

  • Be polite: arguments towards avoiding to upset other people, taking others into account, and being less annoying for others
  • Be honoring elders: arguments towards following one's parents or showing faith and respect towards one's elders

Humility

It is good to recognize one's own insignificance in the larger scheme of things.

Contained values and associated arguments (examples):

  • Be humble: arguments towards demoting arrogance, bragging, and thinking one deserves more than other people, or towards emphasizing the successful group over single persons and giving back to society
  • Have life accepted as is: arguments towards accepting one's fate, submitting to life's circumstances, and being satisfied with what one has

Benevolence: caring

It is good to work for the welfare of one's group's members.

Contained values and associated arguments (examples):

  • Be helpful: arguments towards helping the people in one's group and promoting to work for the welfare of others in one group
  • Be honest: arguments towards being more honest and recognizing people for their honesty
  • Be forgiving: arguments towards allowing people to forgive each other, giving people a second chance, and being merciful, or towards providing paths to redemption
  • Have the own family secured: arguments towards allowing people to have, protect, and care for their family
  • Be loving: arguments towards fostering close relationships and placing the well-being of others above the own, or towards allowing to show affection, compassion, and sympathy

Benevolence: dependability

It is good to be a reliable and trustworthy member of one's group.

Contained values and associated arguments (examples):

  • Be responsible: arguments towards clear responsibilities, fostering confidence, and promoting reliability
  • Have loyalty towards friends: arguments towards being a dependable, trustworthy, and loyal friend, or towards allowing to give friends a full backing

Universalism: concern

It is good to strive for equality, justice, and protection for all people.

Contained values and associated arguments (examples):

  • Have equality: arguments towards fostering people of a lower social status, helping poorer regions of the world, providing all people with equal opportunities in life, and resulting in a world were success is less determined by birth
  • Be just: arguments towards allowing justice to be 'blind' to irrelevant aspects of a case, promoting fairness in competitions, protecting the weak and vulnerable in society, and resulting a world were people are less discriminated based on race, gender, and so on, or towards fostering a general sense for justice
  • Have a world at peace: arguments towards nations ceasing fire, avoiding conflicts, and ending wars, or promoting to see peace as fragile and precious or to care for all of humanity

Universalism: nature

It is good to preserve the natural environment.

Contained values and associated arguments (examples):

  • Be protecting the environment: arguments towards avoiding pollution, fostering to care for nature, or promoting programs to restore nature
  • Have harmony with nature: arguments towards avoiding chemicals and genetically modified organisms (especially in nutrition), or towards treating animals and plants like having souls, promoting a life in harmony with nature, and resulting in more people reflecting the consequences of their actions towards the environment
  • Have a world of beauty: arguments towards allowing people to experience art and stand in awe of nature, or towards promoting the beauty of nature and the fine arts

Universalism: tolerance

It is good to accept and try to understand those who are different from oneself.

Contained values and associated arguments (examples):

  • Be broadminded: arguments towards allowing for discussion between groups, clearing up with prejudices, listening to people who are different from oneself, and promoting to life within a different group for some time, or towards promoting tolerance between all kinds of people and groups in general
  • Have the wisdom to accept others: arguments towards allowing people to accept disagreements and people even when one disagrees with them, to promote a mature understanding of different opinions, or to decrease partisanship or fanaticism

Universalism: objectivity

It is good to search for the truth and think in a rational and unbiased way.

Contained values and associated arguments (examples):

  • Be logical: arguments towards going for the numbers instead of gut feeling, towards a rational, focused, and consistent way of thinking, towards a rational analysis of circumstances, or towards promoting the scientific method
  • Have an objective view: arguments towards fostering to seek the truth, to take on a neutral perspective, to form an unbiased opinion, and to weigh all pros and cons, or towards providing people with the means to make informed decisions

If you want to focus on a few categories but are unsure which to pick, we suggest to take (from) the most frequent ones in our initial dataset: Self-direction: action, Achievement, Security: personal, Security: societal, Benevolence: caring, Universalism: concern.

Data

Data is provided as tab-separated values files with one header line. We will provide an extended training set, a validation set, and the test set (the latter without labels) end of November 2022. Stay up-to-date and report problems on the task mailing list. [download]

The arguments-training/validation/testing.tsv files contain one argument per line: its unique argument ID, the conclusion, the premise's stance towards the conclusion, and the premise itself. Example with tab-separated columns highlighted:

Argument ID	Conclusion	Stance	Premise
A01010	We should prohibit school prayer	against	it should be allowed if the student wants to pray as long as it is not interfering with his classes
A01011	We should abolish the three-strikes laws	in favor of	three strike laws can cause young people to be put away for life without a chance to straight out their life
A01012	The use of public defenders should be mandatory	in favor of	the use of public defenders should be mandatory because some people don't have money for a lawyer and this would help those that don't

The labels-training/validation/testing.tsv files also contain one argument per line: its unique argument ID and one column for each of the 20 value categories with a 1 meaning that the argument resorts to the value category and a 0 that not. Example with tab-separated columns highlighted:

Argument ID	Self-direction: thought	Self-direction: action	Stimulation	Hedonism	Achievement	Power: dominance	Power: resources	Face	Security: personal	Security: societal	Tradition	Conformity: rules	Conformity: interpersonal	Humility	Benevolence: caring	Benevolence: dependability	Universalism: concern	Universalism: nature	Universalism: tolerance	Universalism: objectivity
A01010	1	1	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	0	0	0
A01011	0	0	0	0	1	0	0	1	0	0	0	0	0	0	1	0	0	0	1	1
A01012	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0

Evaluation

Runs are evaluated on the basis of F1-score, Precision, and Recall: averaged over all value categories and for each category individually. The runs are ranked according to the averaged F1-score.

Submission

This task uses TIRA for submissions, which allows for both run file upload and Docker image submission. The latter allows for excellent reproducibility of your run. Registered teams will receive links to register with TIRA up to one work day after registration.

For each topic and stance, include 10 retrieved images. Each registered team can submit up to 4 different runs. Teams are allowed to submit one additional "early bird" run on the final data until December 16. The submission format is the same as the labels-training/validation/testing.tsv with value category columns being optional (skip categories for which the run contains no prediction).

To submit a run file, go to this task's TIRA page. Log in. Click "SUBMIT". Make sure "UPLOAD SUBMISSION" is selected. Specify your file and the dataset and "UPLOAD".

To submit a Docker image, go to this task's TIRA page. Log in. Click "SUBMIT". Make sure "DOCKER SUBMISSION" is selected. Click on "UPLOAD IMAGES" and follow the instructions. Once uploaded, click on "ADD CONTAINER". Specify the command and the Docker image and "ADD CONTAINER". A randomly named tab is created. Click on it, specify the dataset and "RUN CONTAINER". You can copy the random baseline to get started with Docker submissions.

In case of problems or questions concerning TIRA, please use the TIRA forum. Note: At the moment, the TIRA web interface sometimes fails to provide feedback on actions. Reload the page in such cases.

Task Committee