6 Mar
2025
6 Mar
'25
4:58 p.m.
日本データベース学会の皆様
＃重複してお受け取りの場合はご容赦ください
奈良先端大の若宮と申します．
今年9月から2026年12月にかけて開催されますNTCIR-19のタスクを
募集しておりますので，下記の通りご案内いたします．
NTCIR-19では，大規模言語モデル (LLM) を用いた情報アクセスや
評価などのタスク提案も積極的に募集しています．
締切は3月31日ですので，ぜひご提案をご検討いただければ幸いです．
よろしくお願いいたします．
若宮
*********************************************
CALL FOR NTCIR-19 TASK PROPOSALS
Task Proposals Due: March 31, 2025 (AoE)
Submission Link:
https://easychair.org/conferences/?conf=ntcir19proposal
*********************************************
We are delighted to announce the call for task proposals for NTCIR-19.
NTCIR (NII Testbeds and Community for Information Access Research) is a
series of evaluation conferences that mainly focus on information access
with East Asian languages and English. The first NTCIR conference (NTCIR-1)
took place in August/September 1999, and the latest NTCIR-18 conference
will be held on June 10-13, 2025. Research teams from all over the world
participate in one or more NTCIR tasks to advance the state of the art and
to learn from one another's experiences.
It is time to call for task proposals for the next NTCIR (NTCIR-19), which
will start in September 2025 and conclude in December 2026. Task proposals
will be reviewed by the NTCIR Program Committee, and organizers of accepted
tasks will have a chance to present their proposed tasks at the NTCIR-18
Conference held in NII, Tokyo, Japan, from June 10-13, 2025.
* IMPORTANT DATES:
*March 31, 2025: Task Proposal Submission Due (Anywhere on Earth) *May 15,
2025: Acceptance Notification of Task Proposals
June 10-13, 2025: NTCIR-18 Conference (Organizers of accepted tasks have a
chance to present their proposed tasks)
* SUBMISSION LINK:
*https://easychair.org/conferences/?conf=ntcir19proposal
<https://easychair.org/conferences/?conf=ntcir19proposal>*
* NTCIR-19 TENTATIVE SCHEDULE:
January 2026: Dataset release*
January-June 2026: Dry run*
March-July 2026: Formal run*
August 1, 2026: Evaluation results return
August 1, 2026: Task overview release (draft)
September 1, 2026: Submission due of participant papers (draft)
November 1, 2026: Camera-ready participant paper due
December 2026: NTCIR-19 Conference at NII, Tokyo, Japan
(* indicates that the schedule can be different for different tasks)
* WHO SHOULD SUBMIT NTCIR-19 TASK PROPOSALS?
We invite new task proposals within the expansive field of information
access. Organizing an evaluation task entails pinpointing significant
research challenges, strategically addressing them through collaboration
with fellow researchers (including co-organizers and participants),
developing the requisite evaluation framework to propel advancements in the
state of the art, and generating a meaningful impact on both the research
community and future developments.
Prospective applicants are urged to underscore the real-world applicability
of their proposed tasks by utilizing authentic data, focusing on practical
tasks, and solving tangible problems. Additionally, they should confront
challenges in evaluating information access technology, such as the
extensive number of assessments needed for evaluation, ensuring privacy
while using proprietary data, and conducting live tests with actual users.
In the era of large language models (LLMs), these models are anticipated to
significantly influence daily human activities. Nonetheless, the content
produced by LLMs often exhibits issues, such as hallucinations. NTCIR-19
encourages tasks that focus on the evaluation of the quality of content
generated by LLMs continued from NTCIR-18 as well as information access
exploiting LLMs, including generative information retrieval (IR), IR using
generative queries, conversational search using generated utterances,
evaluation using LLM (relevance judgements or language annotation using
LLM), and RAG.
* PROPOSAL TYPES:
We will accept two types of task proposals:
- Proposal of a Core task:
This is for fostering research on a particular information access problem
by providing researchers with a common ground for evaluation. New test
collections and evaluation methods may be developed through the
collaboration between task organizers (proposers) and task participants. At
NTCIR-18, the core tasks are AEOLLM, FairWeb-2, FinArg-2, Lifelog-6,
MedNLP-CHAT, RadNLP, and Transfer-2. Details can be found at
http://research.nii.ac.jp/ntcir/NTCIR-18/tasks.html.
- Proposal of a Pilot task:
This is recommended for organizers who propose to focus on a novel
information access problem, and there are uncertainties either in task
design or organization. It may focus on a sub-problem of an information
access problem and attract a smaller group of participating teams than core
tasks. However, it may grow into a core challenging task in the next round
of NTCIR. At NTCIR-18, the pilot tasks are HIDDEN-RAD, SUSHI, and U4.
Details can be found at http://research.nii.ac.jp/ntcir/NTCIR-18/tasks.html.
Organizers are expected to run their tasks mainly with their own funding
and to make the task as self-sustaining as possible. A part of the fund can
be supported by NTCIR, which is called "seed funding." It is usually used
for some limited purposes such as hiring relevance assessors. The seed
funding allocated to each task varies depending on requirements and the
number of accepted tasks. Typical cases would be around 1M JPY for a core
task and around 0.5M JPY for a pilot task (note that the amount is subject
to change).
Please submit your task proposal as a PDF file via EasyChair by March 31,
2025 (Anywhere on Earth).
https://easychair.org/conferences/?conf=ntcir19proposal
* TASK PROPOSAL FORMAT:
The proposal should not exceed four pages in A4 single-column format. The
first three pages should contain the main part and appendix, and the last
page should contain only a description of the data to be used in the task.
Please describe the data in as much detail as possible so that we can help
your data release process after the proposal is accepted. In the past
NTCIRs, it took much time to create memorandums for data release, which
sometimes slowed down the task organization.
Main part
- Task name and short name
- Task type (core or pilot) - Abstract
- Motivation
- Methodology
- Expected results
Appendix
- Names and contact information of the organizers - Prospective participants
- Data to be used and/or constructed
- Budget planning
- Schedule
- Other notes
Data (to be used in your task) - Details
(Please describe the details of the data, which should include the source
of the data, methods to collect the data, range of the data, etc.)
- License
(Please make sure that you have a license to distribute the data, and
details of the license should be provided. If you do not have permission to
release the data yet, please describe your plan to get the permission.)
- Distribution
(Please describe how you plan to distribute the data to participants. There
are mainly three choices: distributed by the data provider, distributed by
organizers, and distributed by NII.)
- Legal / Ethical issues
(If the data can cause legal or ethical problems, please describe how you
propose to address them. e.g., some medical data may need approval from an
ethical committee. e.g., some Web data may need filtering for excluding
discriminative messages.)
If you want NII to distribute your data to task participants on your
behalf, please email ntc-admin(a)nii.ac.jp before your task proposal
submission attaching the task proposal.
* REVIEW CRITERIA:
- Importance of the task to the information access community and the
society - Timeliness of the task
- Organizers’ commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)
- Soundness of the evaluation methodology
- Detailed description about the data to be used
- Language scope
* NTCIR-19 PROGRAM CO-CHAIRS:
Qingyao Ai (Tsinghua University, China)
Chung-Chi Chen (National Institute of Advanced Industrial Science and
Technology (AIST), Japan)
Shoko Wakamiya (Nara Institute of Science and Technology (NAIST), Japan)
* NTCIR-19 GENERAL CHAIRS:
Charles Clarke (University of Waterloo, Canada)
Noriko Kando (National Institute of Informatics, Japan)
Makoto P. Kato (University of Tsukuba, Japan)
Yiqun Liu (Tsinghua University, China)