*** Apologies for multiple copies *** ********************************************************** Call for Participation to NTCIR-6 Evaluation of Information Access Technologies: IR, QA and Cross-lingual Information Access Technologies http://research.nii.ac.jp/ntcir/ntcir-ws6/ http://research.nii.ac.jp/ntcir/ Final Meeting: 15-18 May 2007, NII, Tokyo, Japan ********************************************************** The NTCIR is a series of evaluation workshop to enhance the research in information access technologies, including text retrieval, question answering, cross-lingual information access, etc, by providing infrastructure of evaluation and research including test collections, evaluation metrics and methodologies, and a forum of researchers. NTCIR has used East Asian Language documents but attracted international participation. For example NTCIR-5 has participants from 15 different countries and areas including Australia, Canada, China PRC, Germany, Hong Kong, Ireland, Japan, Korea, Netherlands, Singapore, Spain, Switzerland, Taiwan ROC, UK, United States. TASKS: NTCIR-6 selected the following 4 areas of research as "Tasks" and 1 area as a "pilot workshop"; Other "pilot tasks" can be started any time during the process of NTCIR-6. 1. Cross-Lingual Information Retrieval Task (CLIR) Multi- and Bi-lingual CLIR, and Single language IR Languages: Traditional Chinese, Korean, and Japanese. Simplified Chinese can be added. To conclude the CLIR to news documents, 4 CLIR test collections, NTCIR-3 through -6 will be used and Cross-collection analysis will be done. New metrics for graded relevance judgments are used. Discussion about evaluation metrics and methodologies are also welcome. 2. Cross-Language Question Answering Task (CLQA) Focus on Named Entities, which are one of the problems in CL information access in Asian context. Traditional Chinese, English and Japanese. Korean language is under consideration. Volunteer for cooperation to organize Korean part is welcome. 3. Patent Retrieval Task (PATENT) Retrieval task: "Invalidity search" Using Japanese patents (10 year patent application fulltexts) and US patents. Use 4 patent collections of NTCIR-3 through -6. Classification task: Multi-viewpoint categorization. The purpose is to categorize target patent applications based on the F-term classification system. 4. Question Answering Task (QAC) Question answering beyond factoid questions and their evaluation method. Focusing to complicated questions like "WHY". 5. Pilot Tasks Any attempts to test the problems to be solved in short time, or feasibility studies for the future tasks. To be announced later. Opinion Extraction, Multilingual Multi- document Summarization, Evaluation of WEB Search engine are now under consideration, but not decided 6. Pilot Workshop: Multimodal Summarization of Trend Information (MuST) Focusing numerical information in the text. Extract numerical information representing the trends from multiple documents, analyze, summarize and display visully. In the last MuST Workshop, we constructed an annotated corpus and shared it. Research groups from various disciplinaries share the corpus and work various direction of research using the corpus. Currently using Japanese documents only. To be announced later. WHAT'S NEW to NTCIR-6: (1) Cross-Collection Evaluation: In IR related tasks, multiple test collections for the same document genres and the same users' tasks will be used and analyze the variability across the collections. (2) Submission Raw Data and Evaluation Results: All the submitted runs and their evaluated results using the metrics set by the task will be available for the active participants of the task. The purpose of this is that we invite all the task participants to discuss, analyze or examine the evaluation metrics and evaluation results of the task. Evaluation is very critical issue for all of the researchers. So please examine how the evaluation is done and how the metrics behave, or whether there are any methods to overcome the limitation of current practice of the evaluation. With your cooperation, we would like to obtain fruitful examination of the evaluation results and metrics. OPEN SUBMISSION SESSION: Conjunction with NTCIR-6 Final Meeting, a separate conference called "Open Submission Session" will be held. This is a refereed international conference on research in Information Access and their evaluation. Detailed information will be announced later. IMPORTANT DATES: Registration Deadline: May 31, 2006 Document Set Release: June 1, 2006 Dry Run: July - September, 2006 Formal Run： October - December, 2006 Evaluation Results Return: by February 1, 2007 Paper for the Proceedings Due : March 1, 2007 Final Meeting, Tokyo, Japan: May 15-18, 2007 ** ONLINE REGISTRATION IS AVAILABLE NOW !! INQUIRY: Noriko Kando at kando (at) nii.ac.jp Your participation is more than welcome. Please join and work together! Noriko. --- Noriko Kando ntcir project