Automation of summarization evaluation methods and their application to the summarization process
View/ Open
Date
30/06/2011Author
Nahnsen, Thade
Metadata
Abstract
Summarization is the process of creating a more compact textual representation of a
document or a collection of documents. In view of the vast increase in electronically
available information sources in the last decade, filters such as automatically generated
summaries are becoming ever more important to facilitate the efficient acquisition
and use of required information. Different methods using natural language processing
(NLP) techniques are being used to this end. One of the shallowest approaches is the
clustering of available documents and the representation of the resulting clusters by
one of the documents; an example of this approach is the Google News website. It is
also possible to augment the clustering of documents with a summarization process,
which would result in a more balanced representation of the information in the cluster,
NewsBlaster being an example. However, while some systems are already available on
the web, summarization is still considered a difficult problem in the NLP community.
One of the major problems hampering the development of proficient summarization
systems is the evaluation of the (true) quality of system-generated summaries. This
is exemplified by the fact that the current state-of-the-art evaluation method to assess
the information content of summaries, the Pyramid evaluation scheme, is a manual
procedure.
In this light, this thesis has three main objectives.
1. The development of a fully automated evaluation method. The proposed scheme
is rooted in the ideas underlying the Pyramid evaluation scheme and makes use
of deep syntactic information and lexical semantics. Its performance improves
notably on previous automated evaluation methods.
2. The development of an automatic summarization system which draws on the
conceptual idea of the Pyramid evaluation scheme and the techniques developed
for the proposed evaluation system. The approach features the algorithm for
determining the pyramid and bases importance on the number of occurrences of
the variable-sized contributors of the pyramid as opposed to word-based methods
exploited elsewhere.
3. The development of a text coherence component that can be used for obtaining
the best ordering of the sentences in a summary.