ELSNET-List Message

ELSNET-List Message

Subject: [ E-Conf ] Living Labs for Information Retrieval Evaluation Workshop
From: <lkelly_(on)_computing.dcu.ie>
Date received: 23 Jul 2013
Deadline: -
Start date: 01 Nov 2013

Living Labs for Information Retrieval Evaluation Workshop


At CIKM 2013, November 1 2013, San Francisco, USA

Call for 2 & 4 page paper and demo submissions

Extended Deadline - July 27

750? award - for best demo submission (see ll2013.dcu.ie for full

Call for Papers

In the past few years, a new evaluation methodology known as
living labs has been proposed as a way for researchers to be able
to perform in-situ evaluation which involve and integrate users
within the research process. The basic idea of living labs for
information retrieval (IR) is that rather than individual
research groups independently developing experimental search
infrastructures and gathering their own groups of test searchers
for IR evaluations, a central and shared experimental environment
is developed to facilitate the sharing of resources in a live
setting (most important of all: users). (see ll2013.dcu.ie for
further details)

Living labs would offer huge benefits to the community, such as:
availability of, potentially larger, cohorts of real users and
their behaviours, e.g. querying behaviours, for experiment
purposes; cross-comparability across research centres; and
greater knowledge transfer between industry and academia, when
industry partners are involved. The need for this methodology is
further amplified by the increased reliance of IR approaches on
proprietary data; living labs are a way to bridge the data divide
between academia and industry. Progress towards realising actual
living labs has nevertheless been limited. There are many
challenges to be overcome before the benefits associated with
living labs for IR can be realised, including challenges
associated with living labs architecture and design, hosting,
maintenance, security, privacy, participant recruiting, and
scenarios and tasks for use development.

This workshop aims to bring together for the first time people
interested in progressing the living labs for IR evaluation
methodology. An interactive forum for researchers to share ideas
and initiate collaborations will be provided, with the explicit
goal of determining means for progressing towards living labs for
IR and formulating practical next steps for progression.

Possible use cases in this space include, but are not limited to:

· The e-commerce domain (i.e., product search and recommendation)

· The personal search space (search of personal computer files,
emails, web pages looked at, etc)

· Medical information retrieval (e.g., patients searching for
medical information on the Internet)

· Searching Wikipedia (an open-source collection with fewer
privacy concerns than, say, personal search)

Workshop topics include, but are not limited to:

· Technical challenges:

o Designing and implementing living labs architecture

o Cost of implementation

o Maintenance and adoption

o Managing living labs infrastructure

· Evaluation challenges:

o Evaluation framework and methodology

o Evaluation measures

· Privacy and security:

o Hosting data on secure server

o Gaining subjects trust

o Coping with individuals need for privacy

o Alternates when individuals will not share their data

· Legal and ethical issues:

o User consent

o Ethics approval

o Legalities regarding release of data

o Trust between parties

o Copyright issues

o Commercial sensitivity of interaction data

· Practical challenges:

o Forming living labs for IR partners within the research

o Obtaining commercial partners

o Alternates when commercial partners cannot be obtained

o Sharing of the physical resources

o Defining tasks and scenarios for evaluation purposes

Paper & Demo Submissions

Short papers (4 pages), position papers (2 pages), and posters (2
pages) describing approaches or ideas / challenges on the topics
of the workshop are invited.

Demo papers (max. 4 pages) based on an implemented system that
pursues one or more aspects relevant to the interest areas of the
workshop are also invited.

Accepted submissions will be included in the CIKM 2013 Living
Labs for Information Retrieval Evaluation Workshop proceedings,
which will be indexed in the ACM Digital Library.

Further submission details on the workshop website:

Important Dates

July 27, 2013: Deadline for paper submissions (midnight Pacific
Daylight Time)

July 27, 2013: Deadline for demo submissions (midnight Pacific
Daylight Time)

August 6, 2013: Paper and demo notification to authors

August 11, 2013: Camera-ready papers due

November 1, 2013: Workshop

Further Information

Further information is available on the workshop website at
http://LL2013.dcu.ie/ or by emailing the workshop organisers.

Workshop Organisers

Krisztian Balog - University of Stavanger, Norway
(krisztian.balog (at) uis.no)

David Elsweiler - University of Regensburg, Germany (david (at)

Evangelos Kanoulas - Google, Switzerland (ekanoulas (at)

Liadh Kelly - Dublin City University, Ireland (liadh.kelly (at)

Mark Smucker - University of Waterloo, Canada (msmucker (at)

Liadh Kelly, PhD Centre for Next Generation Localisation Dublin
City University, Ireland email: liadh.kelly_(at)_computing.dcu.ie web:

Email Disclaimer: This email and any files transmitted with it
are confidential and are intended solely for use by the
addressee. Any unauthorised dissemination, distribution or
copying of this message and any attachments is strictly
prohibited. If you have received this email in error please
notify the sender and delete the message. Any views or opinions
presented in this email may solely be the views of the author and
cannot be relied upon as being those of Dublin City University.
E-mail communications such as this cannot be guaranteed to be
virus free, timely, secure or error free and Dublin City
University do not accept liability for any such matters or their
consequences. Please consider the environment before printing
this Email.

Séanadh Ríomhphoist: Tá an ríomhphost seo agus aon chomhad a
sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus sin
amháin é. Tá cosc iomlán ar scaipeadh, dháileadh nó chóipeáil
neamhúdaraithe ar an teachtaireacht seo agus ar aon cheangaltán
atá ag dul leis. Má tá an ríomhphost seo faighte agat trí
dhearmad cuir sin in iúl le do thoil don seoltóir agus scrios an
teachtaireacht. D'fhéadfadh sé gurb iad tuairimí an údair agus
sin amháin atá in aon tuairimí no dearcthaí atá curtha i láthair
sa ríomhphost seo agus níor chóir glacadh leo mar thuairimí nó
dhearcthaí Ollscoil Chathair Bhaile Atha Cliath. Ní ghlactar leis
go bhfuil cumarsáid ríomhphoist den sórt seo saor ó víreas, in
am, slán, nó saor ó earráid agus ní ghlacann Ollscoil Chathair
Bhaile Atha Cliath le dliteanas in aon chás den sórt sin ná as
aon iarmhairt a d'eascródh astu. Cuimhnigh ar an timpeallacht le
do thoil sula gcuireann tú an ríomhphost seo i gcló.

- ELSNET mailing list Elsnet-list_(at)_elsnet.org
- To manage your subscription go to:


[ Search | Events calendar | Deadline calendar ]


Page generated 23-04-2014 by Steven Krauwer Disclaimer / Contact ELSNET