De siesta en siesta

hugo’s blog

String Manipulation example in IMPURE

Pre-processing a text file in Impure

Here is a simple way to pre-process a text file in Impure using the ReplaceSubstrings control.

I wanted to analyse the book “Alice in Wonderland” using Impure’s linguistic analysis tools. The text of the book is available form the Gutenberg project in .txt format at http://www.gutenberg.org/files/11/11.txt

Loading this file into Impure and visualising it is very easy with a FileLoader and a SimpleStringVisualizator:Impure screen shot

The SimpleStringVisualizator goes a little mad when the file to be visualized is too large, so for debugging purposes I descided to work with a small part of the text only, which I obtained using the getSubstring control:

Ok, now we can see the file. But if you look closely at the lines, you will notice that paragraphs are broken by line breaks. More specifically, in this file lines were separated by line breaks (actually using \r\n) as in the original printed book. Parargaphs were separated by double line breaks (using \r\n\r\n). Impure does a good job at converting all kinds of line breaks into impure line breaks, so we dont need to worry about the line break encodings. However, line breaks are problematic because one sentence can be broken into two or more lines, which may confuse the linguistic analyser.

What we need to do is to reconnect the sentences. This could be done simply by replacing line breaks by spaces using a replaceSubstring control. However, this would eliminate paragraphs. If we want to preserver paragraph information, we need to:

  1. replace paragraph breaks (double line breaks) into a temporary marker,</li>
  2. replace single line breaks by spaces (this puts sentences back together),</li>
  3. replace the temporary marker by a single line break, recovering the paragraphs.</li>

These three operations can be done using three replaceSubstring, or with a single replace Substrings, as done here:

We now have a string where each line break indicates a paragraph. It is now very easy to pick a few paragraphs for linguistic processing. To select a single paragraph, a splitString by linebreak and getElementFromList the desired paragraph:

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: