I use your code to scrape my own bank statements (via the python wrapper) and then feed them into a NER pipeline for analytics. As a result, whitespaces matter a bit to me for tokenisation.
Currently my credit card statements contain padding to line up some vendor details. These get removed entirely on the as-is package:
and i lose the word boundary.
if I up the removeRepeatedCharacters call (in TextChunk.groupByLines) from 3 to 20, I get the original padding that statement pdf intended.
I think going forward, it would be nice to either
- make this minRunLength a parameter of sorts so people can choose for themselves
- OR update TextChunk,squeeze to leave a single whitespace if all chars being squeezed are whitespaces just to not lose a word boundary as it's a little different to other spelling things.
I use your code to scrape my own bank statements (via the python wrapper) and then feed them into a NER pipeline for analytics. As a result, whitespaces matter a bit to me for tokenisation.
Currently my credit card statements contain padding to line up some vendor details. These get removed entirely on the as-is package:
and i lose the word boundary.
if I up the removeRepeatedCharacters call (in TextChunk.groupByLines) from 3 to 20, I get the original padding that statement pdf intended.
I think going forward, it would be nice to either