| Category: ||E-Announce |
| Subject: ||Answer Validation Exercise 2008: Call for Participation |
| From: || |
| Email: ||alvarory_(on)_lsi.uned.es |
| Date received: ||05 May 2008 |
| Deadline: ||01 Jun 2008 |
| Start date: ||17 Sep 2008 |
Question Answering at Cross-Language Evaluation Forum
ANSWER VALIDATION EXERCISE (AVE) 2008
Call for Participation
Answer Validation (AV), Question Answering (QA)
The objective of the AVE is to promote the development and evaluation of
subsystems aimed at validating the correctness of the answers given by
QA systems [1, 2, 3]. This automatic Answer Validation would be useful
for improving QA systems performance, helping humans in the assessment
of QA systems output, improving QA systems self-score, developing better
criteria for collaborative QA systems, etc. Systems must emulate human
assessment of QA responses and decide whether an answer is correct or
not according to a given snippet.
Participant systems will receive a set of triplets (Question, Answer,
Supporting Text) built from QA_(at)_CLEF 2008 responses and they must return
a validation value (SELECTED, ACCEPTED or REJECTED) for each triplet.
Results will be evaluated against the QA human assessments. Please visit
the "Exercise Description" tab at http://nlp.uned.es/clef-qa/ave/ for
There will be a subtask for each language involved in QA_(at)_CLEF 2008:
Basque, Bulgarian, German, English, Spanish, French, Italian, Dutch,
Portuguese, Romanian and Greek. The number of triplets in each subtask
depends on the number of participants in the corresponding Question
Answering subtrack. If there are no participants in a language at the
QA_(at)_CLEF, there will not be a subtask for this language at AVE.
Evaluation measures will permit to compare AV systems performance and
the potential gain with respect to the QA systems.
Registration at CLEF http://www.clef-campaign.org (Registration Page)
Registration for the at AVE sub-exercises (languages):
Before June 1, 2008 at http://nlp.uned.es/clef-qa/ave/
Test set release
June 23, 2008
Submission of runs
June 27, 2008
Release of individual results
July 15, 2008
Submission of Paper for Working Notes
August 15, 2008
CLEF - ECDL 2008 Workshop
September 17-19, 2008
The development collections for Answer Validation Exercise (AVE) 2008
are now available at the AVE site (http://nlp.uned.es/QA/ave/) under the
"2008" tab of your menu, after login as a registered participant.
 Anselmo Peñas; Alvaro Rodrigo; Valentin Sama; Felisa Verdejo.
Testing the Reasoning for Question Answering Validation. Journal of
Logic and Computation 2007
 Anselmo Peñas, Álvaro Rodrigo, Valentín Sama, Felisa Verdejo.
Overview of the Answer Validation Exercise 2006. CLEF 2006, Lecture
Notes in Computer Science LNCS 4730. Springer, Berlin.
 Anselmo Peñas, Álvaro Rodrigo, Felisa Verdejo. Overview of the
Answer Validation Exercise 2007. CLEF 2007, Lecture Notes in Computer
Science. Springer, Berlin. To appear.
Thank you for your interest.
For further information please visit the website or contact:
Elsnet-list mailing list