(重複して受け取られた場合はご容赦下さい) 日本データベース学会の皆様： 国立情報学研究所の神門 典子と申します。 開催日が近づきましたので、改めて、Evaluation Design for Retrieval of Conversationsについての講演会について、ご案内申し上げます。 みなさまのご参加をお待ちしております。 また、周囲のご関心のある方々やメーリングリストなどに、適宜、ご紹介や転送 をしていただければ、幸いです。 よろしくお願いいたします 神門 典子 ---------------------------------------------------------------------- Title: Evaluation Design for Retrieval of Conversations Speaker: Douglas W. Oard, University of Maryland, USA Date: Tuesday, July 10th, at 13:00-14:30 Place: Room 2208, 22F, NII Abstract: Current approaches to the evaluation of information retrieval systems are based on several assumptions, including: (1) documents have a beginning and an end and they contain words, and (2) if we show the user the right documents they can recognize them. These are reasonable assumptions when searching documents such as news stories, books, and patents that were written originally with dissemination as a principal goal. But what about conversations? Is a conversation one "document" or many? Where does a conversation begin? Where does it end? If we can't unambiguously identify the unit of retrieval, many current approaches to evaluation would be ruled out. Even if we could somehow answer that question, ranked retrieval evaluation measures implicitly assume that the user can recognize relevant documents. That may work well when searching your own conversations, but try searching a few recorded conversations from the Apollo program, or President Kennedy's recorded meetings during the Cuban Missile Crisis, or President Clinton's email. You may need a "reader's guide" to figure out who the people are, and the words they are using mean. If such "sensemaking" capabilities are needed, can we devise some insightful, affordable and repeatable way of evaluating them? Perhaps our greatest challenge, however, is that we often have an expectation of privacy when engaging in conversations. Where, then, can we get the large collections of conversations that we would need to build realistic test collections? If we get public conversations (e.g., political debates), we risk not modeling private conversations well. Alternatively, we could use policy measures to protect sensitive collections used in our research, much as we presently do when working with copyrighted material. We have, however, little experience protecting privacy, so we might have much to learn as a community about how to actually do this. A third alternative would be to use an "algorithm deposit" model in which we send each retrieval algorithm to a secure data center and run it there, returning only the evaluation result. We have some experience with such an approach, and thus some basis for assessing its strengths and weaknesses. In this talk, I will step through each of these issues, fleshing each out with one or more examples. My goal will not be to answer any of these questions definitively, but rather to ask each of them clearly so that we can begin to work together on finding good answers. About the Speaker: Douglas Oard is a Professor at the University of Maryland, College Park, with joint appointments in the College of Information Studies and the Institute for Advanced Computer Studies, where he is the director of the Computational Linguistics and Information Processing Lab. He is a Visiting Professor at the National Institute of Informatics, and a General Co-Chair of NTCIR-10. Dr. Oard earned his Ph.D. in Electrical Engineering from the University of Maryland. His research interests center around the use of emerging technologies to support end user information seeking. Additional information is available at http://terpconnect.umd.edu/~oard/ Academic Host: Noriko Kando, Professor, NII Inquiry: Email to kando-secr@nii.ac.jp (ex. 2733) ------------------------------------------------------------------------