ELSNET-list archive

Category:   E-Announce
Subject:   Answer Validation Exercise 2008: Call for Participation
Email:   alvarory_(on)_lsi.uned.es
Date received:   05 May 2008
Deadline:   01 Jun 2008
Start date:   17 Sep 2008

Question Answering at Cross-Language Evaluation Forum ________________________________________________________________________ ANSWER VALIDATION EXERCISE (AVE) 2008 Call for Participation ________________________________________________________________________ http://nlp.uned.es/clef-qa/ave/ KEYWORDS Answer Validation (AV), Question Answering (QA) INTRODUCTION The objective of the AVE is to promote the development and evaluation of subsystems aimed at validating the correctness of the answers given by QA systems [1, 2, 3]. This automatic Answer Validation would be useful for improving QA systems performance, helping humans in the assessment of QA systems output, improving QA systems self-score, developing better criteria for collaborative QA systems, etc. Systems must emulate human assessment of QA responses and decide whether an answer is correct or not according to a given snippet. EXERCISE Participant systems will receive a set of triplets (Question, Answer, Supporting Text) built from QA_(at)_CLEF 2008 responses and they must return a validation value (SELECTED, ACCEPTED or REJECTED) for each triplet. Results will be evaluated against the QA human assessments. Please visit the "Exercise Description" tab at http://nlp.uned.es/clef-qa/ave/ for more details. There will be a subtask for each language involved in QA_(at)_CLEF 2008: Basque, Bulgarian, German, English, Spanish, French, Italian, Dutch, Portuguese, Romanian and Greek. The number of triplets in each subtask depends on the number of participants in the corresponding Question Answering subtrack. If there are no participants in a language at the QA_(at)_CLEF, there will not be a subtask for this language at AVE. Evaluation measures will permit to compare AV systems performance and the potential gain with respect to the QA systems. IMPORTANT DATES Registration at CLEF http://www.clef-campaign.org (Registration Page) Already open Registration for the at AVE sub-exercises (languages): Before June 1, 2008 at http://nlp.uned.es/clef-qa/ave/ Test set release June 23, 2008 Submission of runs June 27, 2008 Release of individual results July 15, 2008 Submission of Paper for Working Notes August 15, 2008 CLEF - ECDL 2008 Workshop September 17-19, 2008 TRAINING DATA The development collections for Answer Validation Exercise (AVE) 2008 are now available at the AVE site (http://nlp.uned.es/QA/ave/) under the "2008" tab of your menu, after login as a registered participant. REFERENCES [1] Anselmo Peñas; Alvaro Rodrigo; Valentin Sama; Felisa Verdejo. Testing the Reasoning for Question Answering Validation. Journal of Logic and Computation 2007 doi: 10.1093/logcom/exm072 http://logcom.oxfordjournals.org/cgi/reprint/exm072 [2] Anselmo Peñas, Álvaro Rodrigo, Valentín Sama, Felisa Verdejo. Overview of the Answer Validation Exercise 2006. CLEF 2006, Lecture Notes in Computer Science LNCS 4730. Springer, Berlin. doi: 10.1007/978-3-540-74999-8_32 http://www.springerlink.com/content/a1x43676577483u4/ [3] Anselmo Peñas, Álvaro Rodrigo, Felisa Verdejo. Overview of the Answer Validation Exercise 2007. CLEF 2007, Lecture Notes in Computer Science. Springer, Berlin. To appear. http://www.clef-campaign.org/2007/working_notes/penasCLEF2007.pdf Thank you for your interest. For further information please visit the website or contact: Anselmo Peñas anselmo_(at)_lsi.uned.es Álvaro Rodrigo alvarory_(at)_lsi.uned.es _______________________________________________ Elsnet-list mailing list Elsnet-list_(at)_elsnet.org http://mailman.elsnet.org/mailman/listinfo/elsnet-list

[print/pda] [no frame] [navigation table] [navigation frame]     Page generated 08-05-2008 by Steven Krauwer Disclaimer / Contact ELSNET