From b95dfec4f40cf5e03d56501a2421b92bcc396325 Mon Sep 17 00:00:00 2001 From: Frederik Arnold Date: Wed, 29 Nov 2023 10:57:05 +0100 Subject: [PATCH] Improvements to key passages --- _drafts/2023-11-28-Key-Passages.markdown | 127 ++++++++++++----------- 1 file changed, 65 insertions(+), 62 deletions(-) diff --git a/_drafts/2023-11-28-Key-Passages.markdown b/_drafts/2023-11-28-Key-Passages.markdown index 122e7c5..bd0ce82 100644 --- a/_drafts/2023-11-28-Key-Passages.markdown +++ b/_drafts/2023-11-28-Key-Passages.markdown @@ -1,9 +1,9 @@ --- -title: "TBD" +title: "Working Title: Key Passages in Literary Works" layout: post author: [robert, frederik] comments: true -date: 2023-12-24 +date: 2023-11-28 --- # Context @@ -16,7 +16,7 @@ TBD # Automatic Identification of Quotations -Scholarly texts contain a number of different types of references. For example, verbatim quotes from short lengths of single words to longer quotations spanning multiple sentences, and indirect quotations in the form of summarizations or re-narrations. In the first phase of the project, we focused on the automatic identification of linking of direct quotations starting with quotations of a length of five or more words. In [Lotte and Annette: And Framwork for Finding and Exploring Key Passages in Literary Works](https://aclanthology.org/2021.nlp4dh-1.7.pdf)[^1], we outline the current landscape for text reuse detection and the development of our tool [Quid](https://hu.berlin/quid). Although there are a number of existing tools, we found that all had limitations for our specific use case. We evaluate Quid and compare it to the existing tools. +Scholarly texts contain a number of different types of quotations. For example, verbatim quotes from short lengths of single words to longer quotations spanning multiple sentences, and indirect quotations in the form of summarizations or re-narrations. In the first phase of the project, we focused on the automatic identification of linking of direct quotations starting with quotations of a length of five or more words. In [Lotte and Annette: And Framwork for Finding and Exploring Key Passages in Literary Works](https://aclanthology.org/2021.nlp4dh-1.7.pdf)[^1], we outline the current landscape for text reuse detection and the development of our tool [Quid](https://hu.berlin/quid). Although there are a number of existing tools, we found that all had limitations for our specific use case. We evaluated Quid and compared it to the existing tools. [^1]: Lotte and Annette have since been renamed to Quid and QuidEx, respectively. @@ -30,64 +30,65 @@ Scholarly texts contain a number of different types of references. For example, Precision Recall - F1 + F1 Precision Recall - F1 + F1 BLAST - 0 - 0 - 0 - 0 - 0 - 0 + 0.59 + 0.61 + 0.60 + 0.37 + 0.59 + 0.45 Copyfind - 0 - 0 - 0 - 0 - 0 - 0 + 0.85 + 0.75 + 0.79 + 0.76 + 0.79 + 0.78 SimT - 0 - 0 - 0 - 0 - 0 - 0 + 0.91 + 0.64 + 0.76 + 0.83 + 0.74 + 0.79 Textmatcher - 0 - 0 - 0 - 0 - 0 - 0 + 0.69 + 0.37 + 0.48 + 0.68 + 0.42 + 0.52 Quid - 0 - 0 - 0 - 0 - 0 - 0 + 0.82 + 0.90 + 0.86 + 0.70 + 0.90 + 0.78 -Considerably more difficult to identify are quotations which are shorter than 5 words. In A Novel Approach for Identification and Linking of Short Quotations in Scholarly Texts and Literary Works, we develop and compare two approaches to tackle this challenge, ProQuo and ProQuoLM. ProQuo is a pipeline consisting of three steps. +Considerably more difficult to identify are quotations which are shorter than 5 words. In A Novel Approach for Identification and Linking of Short Quotations in Scholarly Texts and Literary Works, we develop and compare two approaches to tackle this challenge, _ProQuo_ and _ProQuoLM_. >Our main idea behind ProQuo is to use the references corresponding to long quotations as examples to distinguish references corresponding to short quotations from other text in parentheses and other references, for example, Bible references or references to other literary works. We then extract relations between short quotations and references and use that information and the position of long quotations as anchors to link short quotations to the literary work. -The second approach is a more general, language model based approach where we fine-tune a German Bert for classification. For this second approach, we first extract candidates for short quotations and then use a fine-tuned language model to filter the candidates. +> +>The second approach is a more general, language model based approach where we fine-tune a German Bert for classification. For this second approach, we first extract candidates for short quotations and then use a fine-tuned language model to filter the candidates. @@ -98,40 +99,40 @@ The second approach is a more general, language model based approach where we fi - + + - + - - - - - - + + + + + + - - - - - - + + + + + + - - - - - - + + + + + +
PrecisionRecall - F1RecallF1 Precision RecallF1F1
Baseline0000000.650.780.710.590.750.66
ProQuo0000000.870.720.790.870.660.75
ProQuoML0000000.880.750.810.870.690.77
@@ -140,7 +141,9 @@ The second approach is a more general, language model based approach where we fi To allow for exploration of the results, we created [QuidEx](https://hu.berlin/quidex), a visualization and exploration website. >On the left, a heatmap of the complete literary text shows the distribution of quoted passages. The darker the text, the more often it has been quoted and thus the more important it is assumed to be. Next to the heatmap, the literary work is shown. The grayscale is determined by how many scholarly works quote some part of a key passage. That is, the color is always the same for the whole key passage. The font size is determined by how often a minimal segment is quoted. At the bottom, next to the literary text, a list of all scholarly works is shown. -> ->
- Key passages, website -
\ No newline at end of file + +
+ Key passages, website +
+ +In summary, TBD \ No newline at end of file