NTCIR-17 FairWeb-1 (グループフェアネスを考慮したウェブ検索タスク)
の参加案内を共有させていただきます。
タスク参加登録締切は4/15です。ご検討いただければ幸いです。
SECOND CALL FOR TASK PARTICIPATION: NTCIR-17 FairWeb-1 Task (Registration due April 15, 2023 AoE)
http://sakailab.com/fairweb1/
# Overview
Intro slide deck: https://waseda.box.com/fairweb1intro2023feb
The FairWeb-1 task is a new English web search task that considers both relevance and group fairness.
Each of our search topic seeks information about specific entity types: R (researher), M (movies), and Y (YouTube contents); accordingly, we have four topic types:
R-topics (e.g., information retrieval researchers)
M-topics (e.g., Daniel Craig 007 movies)
Y-topics (e.g., Coldplay covers on YouTube)
For evaluating group fairness, we consider the following attribute sets containing either ordinal or nominal groups:
R-topics: h-index (ordinal) AND gender (nominal)
M-topics: #ratings on IMDb (ordinal) AND geographic region (nominal)
Y-topics: #subscribers of the YouTube account (ordinal)
For R- and M-topics, we will evaluate intersectional groupfairness.
For example, for each R-topic, we want the search engine result page (SERP)
to contain information about researchers with varying levels of h-index (not just high h-index people), AND with different genders (not just "men").
# Task input/ouput
INPUT:
- a search topic (R, M, or Y)
- attribute set(s) and target distribution(s)
OUTPUT:
- a TREC-style run (a SERP for each topic)
# Evaluation method
We will use the Group Fairness and Relevance (GFR) framework: please see the slide deck for details.
# Timeline
February 15, 2023 Pilot relevance assessments for the sample topics and a few pilot runs released
February 1-March 10, 2023 Topic development
March 15, 2023 Topics released
April 15, 2023 Task registrations due
May 16, 2023 Run submissions due
May 17-July 31, 2023 Entity annotations; runs evaluated
August 1, 2023 Evaluation results and draft overview released
September 1, 2023 Draft participant papers due
November 1, 2023 Camera ready papers due
December 2023 NTCIR-17@NII, Tokyo, Japan
# Inquiries:
fairweb1org@list.waseda.jp
# Organisers:
Sijie Tao, Nuo Chen, Tetsuya Sakai (Waseda University, Japan)
Zhumin Chu (Tsinghua University, P.R.C.)
Nicola Ferro (University of Padua, Italy)
Maria Maistro (University of Copenhagen, Denmark)
Ian Soboroff (NIST, USA)
Hiromi Arai (RIKEN AIP, Japan)