日本データベース学会の皆様
兵庫県立大学の山本岳洋です．
今年から2023年12月にかけて，NTCIR-17が行われます．
NTCIR-17では，公平性や密検索に関する情報検索タスクや，
医療，金融，政治情報を対象とした自然言語処理タスクなどの
6つのタスクが採択されました．
この度，これらのタスクに加えて，追加タスクを募集することに
いたしましたので，下記の通りご案内いたします．
締切は10月31日となっております．
ぜひご提案をご検討いただければ幸いです．
*********************************************
CALL FOR NTCIR-17 ADDITIONAL TASK PROPOSALS
Task Proposals Due: October 31, 2022 (AoE)
Submission Link:
https://easychair.org/my/conference?conf=ntcir17taskproposal
*********************************************
NTCIR (NII Testbeds and Community for Information access Research) is a
series of
evaluation conferences that mainly focus on information access with East
Asian languages
and English. The first NTCIR conference (NTCIR-1) took place in
August/September 1999,
and the latest NTCIR-16 conference took place in June 2022. Research
teams from all
over the world participate in one or more NTCIR tasks to advance the
state of
the art and to learn from one another's experiences.
At the NTCIR-17, 6 tasks have already been accepted, and the kickoff
meeting has been
held on September 28, 2022. Among the 6 tasks, three are core tasks
(FinArg-1,
MedNLP-SC, and QA Lab-PoliInfo-4), and 3 are pilot tasks (FairWeb-1,
Transfer,
and UFO). Since there are still many diverse problems emerging in
information access
research, we would like to call for additional task proposals.
Additional task proposals will be reviewed by the NTCIR-17 Program
Committee,
following the schedule below:
* IMPORTANT DATES for Additional Task Proposals:
October 31 (AoE), 2022 Task Proposal Submission Due
December 5, 2022 Acceptance Notification of Additional Task Proposals
* NTCIR-17 TENTATIVE SCHEDULE:
January 2023: Dataset Release*
January-June 2023: Dry Run*
March 2023-July 2023: Formal Run*
August 1, 2023: Evaluation Result Release
August 1, 2023: Draft Task Overview Paper Release
September 1, 2023: Draft Participant Paper Submission Due
November 1, 2023: All Camera-ready Paper Submission Due
December 2023: NTCIR-17 Conference in NII, Tokyo, Japan
(* indicates that the schedule can be different for different tasks)
Please submit your task proposal to
https://easychair.org/my/conference?conf=ntcir17taskproposal
by October 31, 2022 (AoE).
If you have any questions regarding proposing additional tasks to NTCIR-17,
please feel free to contact Program Co-Chairs (ntc-17(a)nii.ac.jp).
* WHO SHOULD SUBMIT NTCIR-17 TASK PROPOSALS?
To organize an evaluation task is to identify important research
problems, tackle
them strategically by collaborating with other researchers (both
co-organizers and
participants), build the necessary evaluation framework to advance the
state of the
art, and make an impact on the research community and to the future.
We encourage applicants to emphasize real-world applications of the
proposed task
by using real-world data, real tasks, and addressing real-world
problems, and address
a challenge in information access technology evaluation such as a large
number of
assessments required for evaluation, privacy-preserving use of
proprietary data,
and live test with real users.
* PROPOSAL TYPES:
We will accept two types of task proposals:
- Proposal of a Core task:
This is for fostering research on a particular information access
problem by
providing researchers with a common ground for evaluation. New test
collections and
evaluation methods may be developed through the collaboration between
task organizers(proposers)
and task participants. At NTCIR-16, the core tasks are Data Search 2,
DialEval-2, FinNum-3, QA Lab-PoliInfo-3, WWW-4, and Lifelog-4 (Details
can be found at
http://research.nii.ac.jp/ntcir/ntcir-16/tasks.html).
- Proposal of a Pilot task:
This is recommended for organizers who proposed to focus on a novel
information
access problem and there are uncertainties either in task designing or
organization.
It may focus on a sub-problem of an information access problem and may
attract a
smaller group of participating teams than core tasks. However, it may
grow into
a core challenging task in the next round of NTCIR. At NTCIR-16, the
pilot tasks
are RCIR, Real-MedNLP, SS, and ULTRE (Details can be found at
http://research.nii.ac.jp/ntcir/ntcir-16/tasks.html).
Organizers are expected to run their tasks mainly with their own funding
and to make
the task as self-sustaining as possible. A part of the fund can be
supported by NTCIR,
which is called "seed funding". It is usually used for some limited
purposes such as
hiring relevance assessors. The amount of seed funding allocated to each
task varies
depending on requirements and the total number of accepted tasks.
Typical cases would
be around 1M JPY for a core task and around 0.5M JPY for a pilot task
(note that the
amount is subject to change).
* TASK PROPOSAL FORMAT:
The proposal should not exceed six pages in A4 single-column format. The
first five
pages should contain the main part and appendix, and the last page
should contain only
a description of the data to be used in the task. Please describe the
data in as much
detail as possible so that we can help your data release process after
the proposal is
accepted. In the past NTCIRs, it took much time to create memorandums
for data release,
which sometimes slowed down the task organization.
Main part
- Task name and short name
- Task type (core or pilot)
- Abstract
- Motivation
- Methodology
- Expected results
Appendix
- Names and contact information of the organizers
- Prospective participants
- Data to be used and/or constructed
- Budget planning
- Schedule
- Other notes
Data (to be used in your task)
- Details
(Please describe the details of the data, which should include the
source of the data,
methods to collect the data, range of the data, etc.)
- License
(Please make sure that you have a license to distribute the data, and
details of the
license should be provided. If you do not have permission to release the
data yet,
please describe your plan to get the permission.)
- Distribution
(Please describe how you plan to distribute the data to participants.
There are mainly
three choices: distributed by the data provider, distributed by
organizers, and
distributed by NII.)
- Legal / Ethical issues
(If the data can cause legal or ethical problems, please describe how
you propose to
address them. e.g. some medical data may need approval from an ethical
committee. e.g.
some Web data may need filtering for excluding discriminative messages.)
* REVIEW CRITERIA:
- Importance of the task to the information access community and to the
society
- Timeliness of the task
- Organizers’ commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)
- Soundness of the evaluation methodology
- Detailed description about the data to be used
- Language scope
* NTCIR-17 PC Members:
Chung-Chi Chen, National Institute of Advanced Industrial Science and
Technology
Hsin-Hsi Chen, National Taiwan University
Gareth Jones, Dublin City University
Noriko Kando, National Institute of Informatics
Makoto P. Kato University of Tsukuba
Yiqun Liu, Tsinghua University
Alistair Moffat, The University of Melbourne
Jian-Yun Nie, University de Montreal
Douglas Oard, University of Maryland
Tetsuya Sakai, Waseda University
Mark Sanderson, RMIT University
Ian Soboroff, NIST
* NTCIR-17 PROGRAM Co-Chairs
Takehiro Yamamoto (Co-chair, University of Hyogo, Japan)
Zhicheng Dou (Co-chair, Renmin University of China)
* NTCIR-17 GENERAL Chairs:
Charles Clarke (University of Waterloo, Canada)
Noriko Kando (National Institute of Informatics, Japan)
Makoto P. Kato (University of Tsukuba, Japan)
Yiqun Liu (Tsinghua University, China)
--
Takehiro Yamamoto, Associate Professor
School of Social Information Science, University of Hyogo
Email: t.yamamoto(a)sis.u-hyogo.ac.jp
Tel: 078-794-5212
https://rerank-lab.org/
