[CPM-SPIRE-L] Call for participation: WASSA-2017 Shared Task on Emotion Intensity

[Felipe Bravo]

Part of the 8th Workshop on Computational Approaches to Subjectivity,
Sentiment and Social Media Analysis (WASSA-2017), which is to be held in
conjunction with EMNLP-2017.

Task webpage: http://saifmohammad.com/WebPages/EmotionIntensity-SharedTask
.html

Task mailing group: EmotionIntensity at googlegroups.com
<https://groups.google.com/forum/#%21forum/emotionintensity>

Background and Significance:

Existing emotion datasets are mainly annotated categorically without an
indication of degree of emotion. Further, the tasks are almost always
framed as classification tasks (identify 1 among n emotions for this
sentence). In contrast, it is often useful for applications to know the
degree to which an emotion is expressed in text. In this task, systems have
to automatically determine the intensity of emotions in tweets.

Task:

Given a tweet and an emotion X, determine the intensity or degree of
emotion X felt by the speaker -- a real-valued score between 0 and 1. The
maximum possible score 1 stands for feeling the maximum amount of emotion X
(or having a mental state maximally inclined towards feeling emotion X).
The minimum possible score 0 stands for feeling the least amount of emotion
X (or having a mental state maximally away from feeling emotion X). The
tweet along with the emotion X will be referred to as an instance. Note
that the absolute scores have no inherent meaning -- they are used only as
a means to convey that the instances with higher scores correspond to a
greater degree of emotion X than instances with lower scores.

Data:

Training, development, and test datasets are provided for four emotions:
joy, sadness, fear, and anger. For example, the anger training dataset has
tweets along with a real-valued score between 0 and 1 indicating the degree
of anger felt by the speaker. The test data includes only the tweet text.
Gold emotion intensity scores will be released after the evaluation period.

Evaluation:

For each emotion, systems are evaluated by calculating the Pearson
Correlation Coefficient with Gold ratings. The correlation scores across
all four emotions will be averaged to determine the bottom-line competition
metric by which the submissions will be ranked.

The official evaluation script (which also acts as a format checker) is
available for download. You may want to run it on the training set to
determine your progress, and eventually on the test set to check the format
of your submission.

Web Hosting of the Competition:

The entire competition will be hosted on CodaLab Competitions (
https://competitions.codalab.org/). A direct link to the Emotion Intensity
CodaLab competition is here: https://competitions.codalab.o
rg/competitions/16380

(CodaLab has been used in many research evaluation competitions in the past
such as Microsoft COCO Image Captioning Challenge and SemEval-2017.)

Paper:

Participants will be given the opportunity to write a system-description
paper that describes their system, resources used, results, and analysis.
This paper will be part of the official WASSA-2017 proceedings. The paper
is to be four pages long plus two pages at most for references. The papers
are to follow the format and style files provided by EMNLP-2017.

Schedule:

Training data ready: Data for anger, fear, and joy are already available;
data for sadness will be made available in the second half of February 2017

Evaluation period starts: May 02, 2017

Evaluation period ends: May 14, 2017

Results posted: May 21, 2017

Workshop paper submission deadline: June 10, 2017

Author notifications : July 9, 2017

Camera ready submissions due: July 23, 2017

Baseline Weka System for Determining Emotion Intensity:

You are free to build a system from scratch using any available software
packages and resources, as long as they are not against the spirit of fair
competition. In order to assist testing of ideas, we also provide a
baseline emotion intensity system that you can build on. The use of this
system is completely optional. The system is available here:
https://github.com/felipebravom/AffectiveTweets


Organizers of the shared task:

Saif M. Mohammad

saif.mohammad at nrc-cnrc.gc.ca

National Research Council Canada

Felipe Bravo-Marquez

fjb11 at students.waikato.ac.nz

The University of Waikato

Alexandra Balahur

alexandra.balahur at jrc.ec.europa.eu

European Commission, Brussels

Contact:

Saif M. Mohammad, saif.mohammad at nrc-cnrc.gc.ca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://fenris.cs.ucr.edu/pipermail/cpm-spire-l/attachments/20170316/14268352/attachment.html