ELSNET-List Message

ELSNET-List Message

Subject: [ E-Conf ] CIKM Workshop on Living Labs for IR Evaluation
From: <lkelly_(on)_computing.dcu.ie>
Date received: 20 Jun 2013
Deadline: -
Start date: 01 Nov 2013

Living Labs for Information Retrieval Evaluation Workshop


At CIKM 2013, November 1 2013, San Francisco, USA

Paper and Demo Submission deadlines - Extended to July 22

Call for Papers

In the past few years, a new evaluation methodology known as
living labs has been proposed as a way for researchers to be able
to perform in-situ evaluation which involve and integrate users
within the research process. The basic idea of living labs for
information retrieval (IR) is that rather than individual
research groups independently developing experimental search
infrastructures and gathering their own groups of test searchers
for IR evaluations, a central and shared experimental environment
is developed to facilitate the sharing of resources in a live
setting (most important of all: users). (see ll2013.dcu.ie for
further details)

Living labs would offer huge benefits to the community, such as:
availability of, potentially larger, cohorts of real users and
their behaviours, e.g. querying behaviours, for experiment
purposes; cross-comparability across research centres; and
greater knowledge transfer between industry and academia, when
industry partners are involved. The need for this methodology is
further amplified by the increased reliance of IR approaches on
proprietary data; living labs are a way to bridge the data divide
between academia and industry. Progress towards realising actual
living labs has nevertheless been limited. There are many
challenges to be overcome before the benefits associated with
living labs for IR can be realised, including challenges
associated with living labs architecture and design, hosting,
maintenance, security, privacy, participant recruiting, and
scenarios and tasks for use development.

This workshop aims to bring together for the first time people
interested in progressing the living labs for IR evaluation
methodology. An interactive forum for researchers to share ideas
and initiate collaborations will be provided, with the explicit
goal of determining means for progressing towards living labs for
IR and formulating practical next steps for progression.

Possible use cases in this space include, but are not limited to:

· The e-commerce domain (i.e., product search and recommendation)

· The personal search space (search of personal computer files,
emails, web pages looked at, etc)

· Medical information retrieval (e.g., patients searching for
medical information on the Internet)

· Searching Wikipedia (an open-source collection with fewer
privacy concerns than, say, personal search)

Workshop topics include, but are not limited to:

· Technical challenges:

o Designing and implementing living labs architecture

o Cost of implementation

o Maintenance and adoption

o Managing living labs infrastructure

· Evaluation challenges:

o Evaluation framework and methodology

o Evaluation measures

· Privacy and security:

o Hosting data on secure server

o Gaining subjects trust

o Coping with individuals need for privacy

o Alternates when individuals will not share their data

· Legal and ethical issues:

o User consent

o Ethics approval

o Legalities regarding release of data

o Trust between parties

o Copyright issues

o Commercial sensitivity of interaction data

· Practical challenges:

o Forming living labs for IR partners within the research

o Obtaining commercial partners

o Alternates when commercial partners cannot be obtained

o Sharing of the physical resources

o Defining tasks and scenarios for evaluation purposes

Paper & Demo Submissions

Short papers (4 pages), position papers (2 pages), and posters (2
pages) describing approaches or ideas / challenges on the topics
of the workshop are invited.

Demo papers (max. 4 pages) based on an implemented system that
pursues one or more aspects relevant to the interest areas of the
workshop are also invited.

Accepted submissions will be included in the CIKM 2013 Living
Labs for Information Retrieval Evaluation Workshop proceedings,
which will be indexed in the ACM Digital Library.

Further submission details on the workshop website:

Important Dates

July 22, 2013: Deadline for paper submissions (midnight Pacific
Daylight Time)

July 22, 2013: Deadline for demo submissions (midnight Pacific
Daylight Time)

August 4, 2013: Paper and demo notification to authors

August 11, 2013: Camera-ready papers due

November 1, 2013: Workshop

Further Information

Further information is available on the workshop website at
http://LL2013.dcu.ie/ or by emailing the workshop organisers.

Workshop Organisers

Krisztian Balog - University of Stavanger, Norway
(krisztian.balog (at) uis.no)

David Elsweiler - University of Regensburg, Germany (david (at)

Evangelos Kanoulas - Google, Switzerland (ekanoulas (at)

Liadh Kelly - Dublin City University, Ireland (liadh.kelly (at)

Mark Smucker - University of Waterloo, Canada (msmucker (at)

- ELSNET mailing list Elsnet-list_(at)_elsnet.org
- To manage your subscription go to:


[ Search | Events calendar | Deadline calendar ]


Page generated 21-01-2018 by Steven Krauwer Disclaimer / Contact ELSNET