NAME

       Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer

SYNOPSIS

               # isa Plucene::Analysis::CharTokenizer

DESCRIPTION

       This is the standard tokenizer.

       This should be a good tokenizer for most European-language documents.

METHODS

   token_re
       The regular expression for tokenising.

   normalize
       Remove 's and .

perl v5.36.0                                       2022-12-04             Plucene::Analys...andardTokenizer(3pm)