This Week | Coming Up | Past Talks | Contact Us
Weekly Colloquium
November 20, 2009
"Evaluating Natural Language Generation Systems in Online Virtual Environments"
Kristina Striegnitz, Ph.D.
Union College
The GIVE challenge (GIVE = Giving Instructions in Virtual Environments) is a recent shared task in which natural language generation systems are evaluated over the Internet. In the GIVE scenario, human users enter an online 3D virtual environment where they need to solve a treasure hunt with the help of automatically generated instructions. The challenge for the natural language generation systems is to produce, in real time, effective English instructions that guide the users to their goal.
In this talk, I will explain the rationale for the GIVE challenge, describe the software infrastructure that was developed to support it, and present the results of the first evaluation round and of a parallel laboratory experiment. The data collected over the Internet are consistent with the data collected in the more traditional laboratory setting, but the Internet approach offers the statistical power for more fine-grained evaluations because it is cheaper to collect large amounts of data.
For more information about the GIVE challenge see www.give-challenge.org/research.
=======================================
Kristina Striegnitz is an assistant professor at Union College. Before coming to Union, she received a joint PhD from Saarland University in Germany and the University Henri Poincare in France, and did a Postdoc at Northwestern University. Her research is in natural language generation, dialog systems, and embodied conversational agents.