An ecologically valid evaluation of speech summarization

  1. (PDF, 544 KB)
DOIResolve DOI:
AuthorSearch for: ; Search for: ; Search for: ; Search for:
Proceedings titleCHI 2012: Proceedings of the 2012 ACM annual conference extended abstracts on Human Factors in Computing Systems
ConferenceACM SIGCHI Conference on Human Factors in Computing Systems, Austin, Texas, USA, May 5-10, 2012
Pages22192224; # of pages: 6
SubjectSpeech summarization; user study; extrinsic evaluation
AbstractThe past decade has witnessed an explosion in the size and availability of online audio-visual repositories, such as entertainment, news, or lectures. Summarization systems have the potential to provide significant assistance with navigating such repositories. Unfortunately, automatically-generated summaries often fall short of delivering the information needed by users. This is due, in no small part, to the fact that the natural language heuristics used to generate summaries are often optimized with respect to currently-used evaluation metrics. Such metrics simply score automaticallygenerated summaries against subjectively-classified gold standards without taking into account the usefulness of a summary in assisting a user achieve a certain goal or even overall summary coherence. We have previously shown that an immediate consequence of this problem is that even the most linguistically-complex summarization systems perform no better than basic heuristics, such as picking the longest sentences from a general-topic, spontaneous dialog, or the first few sentences from a news recording. Our hypothesis is that complex systems are in fact better, if measured properly. What is thus needed instead are evaluation metrics (and consequently, automatic summarizers) that incorporate features such as user preferences and task-orientation. For this, we propose an ecologically valid evaluation metric that determines the value of a summary when embedded in a task, rather than how closely a summary matches a gold standard.
Publication date
AffiliationInformation and Communication Technologies; National Research Council Canada
Peer reviewedYes
NPARC number20255958
Export citationExport as RIS
Report a correctionReport a correction
Record identifierc3d41365-ca29-4a69-b939-bb6d93195ce6
Record created2012-07-07
Record modified2016-05-09
Bookmark and share
  • Share this page with Facebook (Opens in a new window)
  • Share this page with Twitter (Opens in a new window)
  • Share this page with Google+ (Opens in a new window)
  • Share this page with Delicious (Opens in a new window)