ELSNET-List Message

ELSNET-List Message

Subject: [ E-CFP ] CfP: CIKM Workshop on Living Labs for Information Retrieval Evaluation
From: <lkelly_(on)_computing.dcu.ie>
Date received: 03 May 2013
Deadline: 21 Jun 2013
Start date: 01 Nov 2013

Living Labs for Information Retrieval Evaluation Workshop


At CIKM 2013, November 1 2013, San Francisco, USA

Poll: should we have a living labs IR challenge using a
medium-sized online retailer?



Call for Papers

Submission deadline: June 21

In the past few years, a new evaluation methodology known as
living labs has been proposed as a way for researchers to be able
to perform in-situ evaluation. The basic idea of living labs for
information retrieval (IR) is that rather than individual
research groups independently developing experimental search
infrastructures and gathering their own groups of test searchers
for IR evaluations, a central and shared experimental environment
is developed to facilitate the sharing of resources.

Living labs would offer huge benefits to the community, such as:
availability of, potentially larger, cohorts of real users and
their behaviours, e.g. querying behaviours, for experiment
purposes; cross-comparability across research centres; and
greater knowledge transfer between industry and academia, when
industry partners are involved. The need for this methodology is
further amplified by the increased reliance of IR approaches on
proprietary data; living labs are a way to bridge the data divide
between academia and industry. Progress towards realising actual
living labs has nevertheless been limited. There are many
challenges to be overcome before the benefits associated with
living labs for IR can be realised, including challenges
associated with living labs architecture and design, hosting,
maintenance, security, privacy, participant recruiting, and
scenarios and tasks for use development.

This workshop aims to bring together for the first time people
interested in progressing the living labs for IR evaluation
methodology. An interactive forum for researchers to share ideas
and initiate collaborations will be provided, with the explicit
goal of determining means for progressing towards living labs for
IR and formulating practical next steps for progression.

Possible use cases in this space include, but are not limited to:

· The e-commerce domain (i.e., product search and recommendation)

· The personal search space (search of personal computer files,
emails, web pages looked at, etc)

· Medical information retrieval (e.g., patients searching for
medical information on the Internet)

· Searching Wikipedia (an open-source collection with fewer
privacy concerns than, say, personal search)

Workshop topics include, but are not limited to:

· Privacy and security:

o Hosting data on secure server

o Gaining subjects trust

o Coping with individuals need for privacy

o Alternates when individuals will not share their data

· Legal and ethical issues:

o User consent

o Ethics approval

o Legalities regarding release of data

o Trust between parties

o Copyright issues

o Commercial sensitivity of interaction data

· Technical challenges:

o Designing and implementing living labs architecture

o Cost of implementation

o Maintenance and adoption

o Managing living labs infrastructure

· Practical challenges:

o Forming living labs for IR partners within the research

o Obtaining commercial partners

o Alternates when commercial partners cannot be obtained

o Sharing of the physical resources

o Defining tasks and scenarios for evaluation purposes Poll -
Should we have a living labs IR challenge using a medium-sized
online retailer?

As part of the Living Labs for Information Retrieval Evaluation
workshop, we are considering organising a challenge in the
e-commerce domain with the involvement of a medium-sized online
retailer. The goal of this challenge would be to (i) allow
academics to work with real users and data (esp. those who
otherwise would have no access to such data) and (ii) to provide
a starting point for the discussions at the workshop.

We will set up and run this challenge if there is sufficient
interest in the community. So please let us know what you think
on the poll at http://bit.ly/LL13poll.

Paper Submissions

The workshop is now accepting paper submissions. Short papers (4
pages), position papers (2 pages), and posters (2 pages)
describing approaches or ideas / challenges on the topics of the
workshop are invited. Submissions should be in ACM SIGS format.
LaTeX and Word templates are available at
http://www.acm.org/sigs/publications/proceedings-templates (for
LaTeX, use the "Option 2" style).

Papers should be anonymised for double blind review and submitted
in pdf format through the EasyChair system
https://www.easychair.org/conferences/?conf=ll2013 no later than
midnight Pacific Daylight Time on June 21, 2013. Submissions will
be reviewed by members of the workshop program committee.
Accepted papers will be included in the CIKM 2013 Living Labs for
Information Retrieval Evaluation Workshop proceedings.

Important Dates

June 21, 2013: Deadline for paper submission (midnight Pacific
Daylight Time)

July 22, 2013: Notification to authors

August 11, 2013: Camera-ready paper due

November 1, 2013: Workshop

Further Information

Further information is available on the workshop website at
http://LL2013.dcu.ie/ or by emailing the workshop organisers.

Workshop Organisers

Krisztian Balog - University of Stavanger, Norway
(krisztian.balog (at) uis.no)

David Elsweiler - University of Regensburg, Germany (david (at)

Evangelos Kanoulas - Google, Switzerland (ekanoulas (at)

Liadh Kelly - Dublin City University, Ireland (liadh.kelly (at)

Mark Smucker - University of Waterloo, Canada (msmucker (at)

- ELSNET mailing list Elsnet-list_(at)_elsnet.org
- To manage your subscription go to:


[ Search | Events calendar | Deadline calendar ]


Page generated 21-06-2018 by Steven Krauwer Disclaimer / Contact ELSNET