Provided by: libdbix-fulltextsearch-perl_0.73-12_all bug

NAME

       DBIx::FullTextSearch - Indexing documents with MySQL as storage

SYNOPSIS

       DBIx::FullTextSearch uses a MySQL database backend to index files, web documents and database fields.
       Supports must include, can include, and cannot include words and phrases.  Support for boolean (AND/OR)
       queries, stop words and stemming.

           use DBIx::FullTextSearch;
           use DBI;
           # connect to database (regular DBI)
           my $dbh = DBI->connect('dbi:mysql:database', 'user', 'passwd');

           # create a new stoplist
           my $sl = DBIx::FullTextSearch::StopList->create_default($dbh, 'sl_en', 'English');

           # create a new index with default english stoplist and english stemmer
           my $fts = DBIx::FullTextSearch->create($dbh, 'fts_web_1',
                       frontend => 'string', backend => 'blob',
                       stoplist => 'sl_en', stemmer => 'en-us');
           # or open existing one
           # my $fts = DBIx::FullTextSearch->open($dbh, 'fts_web_1');

           # index documents
           $fts->index_document('krtek', 'krtek leze pod zemi');
           $fts->index_document('jezek', 'Jezek ma ostre bodliny.');

           # search for matches
           my @docs = $fts->contains('foo');
           my @docs = $fts->econtains('+foo', '-Bar');
           my @docs = $fts->search('+foo -Bar');
           my @docs = $fts->search('foo AND (bar OR baz)');

DESCRIPTION

       DBIx::FullTextSearch is a flexible solution for indexing contents of documents.  It uses the MySQL
       database to store the information about words and documents and provides Perl interface for indexing new
       documents, making changes and searching for matches.  For DBIx::FullTextSearch, a document is nearly
       anything -- Perl scalar, file, Web document, database field.

       The basic style of interface is shown above. What you need is a MySQL database and a DBI with DBD::mysql.
       Then you create a DBIx::FullTextSearch index -- a set of tables that maintain all necessary information.
       Once created it can be accessed many times, either for updating the index (adding documents) or
       searching.

       DBIx::FullTextSearch uses one basic table to store parameters of the index. Second table is used to store
       the actual information about documents and words, and depending on the type of the index (specified
       during index creation) there may be more tables to store additional information (like conversion from
       external string names (eg. URL's) to internal numeric form). For a user, these internal thingies and
       internal behaviour of the index are not important. The important part is the API, the methods to index
       document and ask questions about words in documents. However, certain understanding of how it all works
       may be usefull when you are deciding if this module is for you and what type of index will best suit your
       needs.

   Frontends
       From the user, application point of view, the DBIx::FullTextSearch index stores documents that are named
       in a certain way, allows adding new documents, and provides methods to ask: "give me list of names of
       documents that contain this list of words". The DBIx::FullTextSearch index doesn't store the documents
       itself. Instead, it stores information about words in the documents in such a structured way that it
       makes easy and fast to look up what documents contain certain words and return names of the documents.

       DBIx::FullTextSearch provides a couple of predefined frontend classes that specify various types of
       documents (and the way they relate to their names).

       default
           By default, user specifies the integer number of the document and the content (body) of the document.
           The code would for example read

                   $fts->index_document(53, 'zastavujeme vyplaty vkladu');

           and  DBIx::FullTextSearch  will remember that the document 53 contains three words.  When looking for
           all documents containing word (string) vklad, a call

                   my @docs = $fts->contains('vklad*');

           would return numbers of all documents containing words starting with 'vklad', 53 among them.

           So here it's user's responsibility to maintain a relation between  the  document  numbers  and  their
           content,  to  know  that  a  document  53  is  about vklady. Perhaps the documents are already stored
           somewhere and have unique numeric id.

           Note that the numeric id must be no larger than 2^"doc_id_bits".

       string
           Frontend string allows the user to specify the names of the documents as strings, instead of numbers.
           Still the user has to specify both the name of the document and the content:

                   $fts->index_document('foobar',
                                   'the quick brown fox jumped over lazy dog!');

           After that,

                   $fts->contains('dog')

           will return 'foobar' as one of the names of documents with word 'dog' in it.

       file
           To index files, use the frontend file. Here the content of the document is clearly the content of the
           file specified by the filename, so in a call to index_document,  only  the  name  is  needed  --  the
           content of the file is read by the DBIx::FullTextSearch transparently:

                   $fts->index_document('/usr/doc/FAQ/Linux-FAQ');
                   my @files = $fts->contains('penguin');

       url Web  document  can  be indexed by the frontend url. DBIx::FullTextSearch uses LWP to get the document
           and then parses it normally:

                   $fts->index_document('http://www.perl.com/');

           Note that the HTML tags themselves are indexed along with the text.

       table
           You can have a DBIx::FullTextSearch index that indexes char or blob  fields  in  MySQL  table.  Since
           MySQL  doesn't support triggers, you have to call the "index_document" method of DBIx::FullTextSearch
           any time something changes in the table. So the sequence probably will be

                   $dbh->do('insert into the_table (id, data, other_fields)
                           values (?, ?, ?)', {}, $name, $data, $date_or_something);
                   $fts->index_document($name);

           When calling "contains", the id (name) of the record will be returned. If  the  id  in  the_table  is
           numeric, it's directly used as the internal numeric id, otherwise a string's way of converting the id
           to numeric form is used.

           When  creating  this  index,  you'll  have  to  pass  it  three  additionial  options,  "table_name",
           "column_name", and "column_id_name".  You may use the optional column_process option  to  pre-process
           data in the specified columns.

       The  structure of DBIx::FullTextSearch is very flexible and adding new frontend (what will be indexed) is
       very easy.

   Backends
       While frontend specifies what is indexed and how the user sees the collection of  documents,  backend  is
       about  low  level  database  way  of  actually  storing  the  information  in the tables. Three types are
       available:

       blob
           For each word, a blob holding list of all documents containing that word is stored in the table, with
           the count (number of occurencies) associated with each  document  number.  That  makes  it  for  very
           compact  storage.  Since  the  document  names (for example URL) are internally converted to numbers,
           storing and fetching the data is  fast.  However,  updating  the  information  is  very  slow,  since
           information  concerning  one document is spread across all table, without any direct database access.
           Updating a document (or merely reindexing it) requires update of all blobs, which is slow.

           The list of documents is stored sorted by document name so  that  fetching  an  information  about  a
           document  for  one  word is relatively easy, still a need to update (or at least scan) all records in
           the table makes this storage unsuitable for collections of documents that often change.

       column
           The column backend stores a word/document pair in database fields, indexing both, thus allowing  both
           fast  retrieval and updates -- it's easy to delete all records describing one document and insert new
           ones.  However, the database indexes that have to be maintained are large.

           Both blob and column backends only store a count -- number of occurencies of the word in the document
           (and even this can be switched off, yielding just a yes/no information about  the  word's  presence).
           This allows questions like

                   all documents containing words 'voda' or 'Mattoni'
                           but not a word 'kyselka'

           but you cannot ask whether a document contains a phrase 'kyselka Mattoni' because such information is
           not maintained by these types of backends.

       phrase
           To  allow phrase matching, a phrase backend is available. For each word and document number it stores
           a blob of lists of positions of the word in the document. A query

                   $fts->contains('kyselk* Mattoni');

           then only returns those documents (document names/numbers) where word kyselka (or kyselky, or so)  is
           just before word Mattoni.

   Mixing frontends and backends
       Any  frontend can be used with any backend in one DBIx::FullTextSearch index. You can index Web documents
       with "url" frontend and "phrase" backend to be able to find phrases in the documents. And you can use the
       default, number based document scheme with "blob" backend  to  use  the  disk  space  as  efficiently  as
       possible  -- this is usefull for example for mailing-list archives, where we need to index huge number of
       documents that do not change at all.

       Finding optimal combination is very important and may require some analysis of  the  document  collection
       and  manipulation,  as well as the speed and storage requirements. Benchmarking on actual target platform
       is very useful during the design phase.

METHODS

       The following methods are available on the user side as DBIx::FullTextSearch API.

       create
                   my $fts = DBIx::FullTextSearch->create($dbh, $index_name, %opts);

           The class method "create" creates index of given name (the name of the index is the name of its basic
           parameter table) and all necessary tables, returns an object -- newly created index. The options that
           may be specified after the index name define the frontend and backend types, storage parameters  (how
           many bits for what values), etc. See below for list of create options and discussion of their use.

       open
                   my $fts = DBIx::FullTextSearch->open($dbh, $index_name);

           Opens  and  returns  object,  accessing  specifies  DBIx::FullTextSearch  index.  Since all the index
           parameters and information are stored in the $index_name table (including names of all  other  needed
           tables), the database handler and the name of the parameter table are the only needed arguments.

       index_document
                   $fts->index_document(45, 'Sleva pri nakupu stribra.');
                   $fts->index_document('http://www.mozilla.org/');
                   $fts->index_document('http://www.mozilla.org/','This is the mozilla web site');

           For  the  "default" and "string" frontends, two arguments are expected -- the name (number or string)
           of the document and its content. For "file", "url", and "table" frontends the  content  is  optional.
           Any content that you pass will be appended to the content from the file, URL, or database table.

       delete_document
                   $fts->delete_document('http://www.mozilla.org/');

           Removes  information  about  document  from the index. Note that for "blob" backend this is very time
           consuming process.

       contains
                   my @docs = $fts->contains('sleva', 'strib*');

           Returns list of names (numbers or strings, depending on the frontend) of documents that contain  some
           of specified words.

       econtains
                   my @docs = $fts->contains('foo', '+bar*', '-koo');

           Econtains  stands  for  extended  contains  and allows words to be prefixed by plus or minus signs to
           specify that the word must or mustn't be present in the document for it to match.

       search
            my @docs = $fts->search(qq{+"this is a phrase" -koo +bar foo});
            my @docs = $fts->search("(foo OR baz) AND (bar OR caz)");

           This is a wrapper to econtains which takes a user input string and parses it into can-include,  must-
           include, and must-not-include words and phrases.  It also can handle boolean (AND/OR) queries.

       contains_hashref, econtains_hashref, search_hashref
           Similar  to  "contains",  "econtains"  and  "search",  only  instead of list of document names, these
           methods return a hash reference to a hash where keys are the document names and values are the number
           of occurencies of the words.

       drop
           Removes all tables associated with the  index,  including  the  base  parameter  table.  Effectivelly
           destroying the index form the database.

            $fts->drop;

       empty
           Emptys the index so you can reindex the data.

            $fts->empty;

INDEX OPTIONS

       Here  we  list  the  options that may be passed to "create" method.  These allow to specify the style and
       storage parameters in great detail.

       backend
           The backend type, default "blob", possible values  "blob",  "column"  and  "phrase"  (see  above  for
           explanation).

       frontend
           The  frontend  type.  The  "default" frontend requires the user to specify numeric id of the document
           together with the content of the document, other possible values are "string", "file" and "url"  (see
           above for more info).

       word_length
           Maximum length of words that may be indexed, default 30.

       data_table
           Name  of the table where the actual data about word/document relation is stored. By default, the name
           of the index (of the base table) with _data suffix is used.

       name_length
           Any frontend that uses strings as names of documents needs to maintain a conversion table from  these
           names  to internal integer ids. This value specifies maximum length of these string names (URLs, file
           names, ...).

       blob_direct_fetch
           Only for "blob" backend. When looking for information about specific document in the list  stored  in
           the  blob, the blob backend uses division of interval to find the correct place in the blob. When the
           interval gets equal or shorter that this value, all values are fetched  from  the  database  and  the
           final search is done in Perl code sequentially.

       word_id_bits
           With  "column"  or  "phase"  backends,  DBIx::FullTextSearch  maintains a numeric id for each word to
           optimize the space requirements. The word_id_bits parameter specifies the number of bits  to  reserve
           for  this  conversion  and  thus effectively limits number of distinct words that may be indexed. The
           default is 16 bits and possible values are 8, 16, 24 or 32 bits.

       word_id_table
           Name of the table that holds conversion from words to their numeric id  (for  "column"  and  "phrase"
           backends). By default is the name of the index with _words suffix.

       doc_id_bits
           A  number  of  bits  to  hold a numeric id of the document (that is either provided by the user (with
           "default" frontend) or generated by the module to accomplish the conversion from the string  name  of
           the  document). This value limits the maximum number of documents to hold. The default is 16 bits and
           possible values are 8, 16 and 32 bits for "blob" backend and 8, 16, 24 and 32 bits for  "column"  and
           "phrase" backends.

       doc_id_table
           Name  of  the  table  that  holds  conversion  from string names of documents to their numeric id, by
           default the name of the index with _docid suffix.

       count_bits
           Number of bits reserved for storing number of occurencies of each word in the document.  The  default
           is 8 and possible values are the same as with doc_id_bits.

       position_bits
           With  "phrase  backend",  DBIx::FullTextSearch  stores  positions of each word of the documents. This
           value specifies how much space should be reserved for this  purpose.  The  default  is  32  bits  and
           possible  values are 8, 16 or 32 bits. This value limits the maximum number of words of each document
           that can be stored.

       index_splitter
           DBIx::FullTextSearch allows the user to provide any Perl code that will be used to split the  content
           of  the  document  to  words  when  indexing  documents.   The  code  will  be  evalled inside of the
           DBIx::FullTextSearch code. The default is

                   /(\w{2,$word_length})/g

           and shows that the input is stored in the variable $data and the code may access any  other  variable
           available  in  the  perl_and_index_data_*  methods  (see  source), especially $word_length to get the
           maximum length of words and $backend to get the backend object.

           The default value also shows that by default, the minimum length of words indexed is 2.

       search_splitter
           This is similar to the "index_splitter" method, except that it  is  used  in  the  "contains_hashref"
           method when searching for documents instead of when indexing documents.  The default is

                  /(\w{2,$word_length}\*?)/g

           Which, unlike the default "index_splitter", allows for the wild card character (*).

       filter
           The output words of splitter (and also any parameter of (e)contains* methods) are send to filter that
           may do further processing. Filter is again a Perl code, the default is

                   map { lc $_ }

           showing  that the filter operates on input list and by default does conversion to lowercase (yielding
           case insensitive index).

       init_env
           Because user defined splitter or filter may depend on other things  that  it  is  reasonable  to  set
           before  the  actual  procession  of  words,  you  can use yet another Perl hook to set things up. The
           default is no initialization hook.

       stoplist
           This is the name of a DBIx::FullTextSearch::StopList object that is used for stop words.

       stemmer
           If this option is set, then word stemming will be enabled in the indexing and searching.

           The value is the name of a Lingua::Stem recognized locale.  Currently, 'en', 'en-us' and 'en-uk'  are
           the only recognized locales.  All locale identifiers are converted to lowercase.

       table_name
           For "table" frontend; this is the name of the table that will be indexed.

       column_name
           For  "table"  frontend;  this is a reference to an array of columns in the "table_name" that contains
           the documents -- data to be indexed. It can also have a form table.column that will be  used  if  the
           "table_name" option is not specified.

       column_id_name
           For  "table"  frontend;  this  is the name of the field in "table_name" that holds names (ids) of the
           records. If not specified, a field that has primary key on it is used. If this field is numeric, it's
           values are directly used as identifiers, otherwise a conversion to numeric values is made.

NOTES

       To handle internationalization, it may help to use the following in your code  (for  example  Spanish  in
       Chile):

         use POSIX;
         my $loc = POSIX::setlocale( &POSIX::LC_ALL, "es_CL" );

       I haven't tested this, so I would be interested in hearing whether this works.

ERROR HANDLING

       The  create  and open methods return the DBIx::FullTextSearch object on success, upon failure they return
       undef and set error message in $DBIx::FullTextSearch::errstr variable.

       All other methods return reasonable (documented  above)  value  on  success,  failure  is  signalized  by
       unreasonable  (typically  undef  or  null)  return  value;  the  error  message  may then be retrieved by
       "$fts->errstr" method call.

VERSION

       This documentation describes DBIx::FullTextSearch module version 0.73.

BUGS

       Error handling needs more polishing.

       We do not check if the stored values are larger that specified by the *_bits parameters.

       No CGI administration tool at the moment.

       No scoring algorithm implemented.

DEVELOPMENT

       These  modules  are  under  active  development.   If  you  would  like  to  contribute,  please   e-mail
       tjmather@maxmind.com

       There  are  two  mailing lists for this module, one for users, and another for developers.  To subscribe,
       visit http://sourceforge.net/mail/?group_id=8645

AUTHOR

       (Original) Jan Pazdziora, adelton@fi.muni.cz, http://www.fi.muni.cz/~adelton/ at Faculty of  Informatics,
       Masaryk University in Brno, Czech Republic

       (Current  Maintainer)  T.J.  Mather,  tjmather@maxmind.com, http://www.maxmind.com/app/opensourceservices
       Princeton, NJ USA

       Paid  support  is  available  from  directly  from  the  maintainers  of  this   package.    Please   see
       <http://www.maxmind.com/app/opensourceservices> for more details.

CREDITS

       Fixes, Bug Reports, Docs have been generously provided by:

         Vladimir Bogdanov
         Ade Olonoh
         Kate Pugh
         Sven Paulus
         Andrew Turner
         Tom Bille
         Joern Reder
         Tarik Alkasab
         Dan Collis Puro
         Tony Bowden
         Mario Minati
         Miroslav Suchý
         Stephen Patterson
         Joern Reder
         Hans Poo

       Of  course,  big thanks to Jan Pazdziora, the original author of this module.  Especially for providing a
       clean, modular code base!

COPYRIGHT

       All rights reserved. This package is free software; you can redistribute it and/or modify  it  under  the
       same terms as Perl itself.

SEE ALSO

       DBIx::FullTextSearch::StopWord, Class::DBI::mysql::FullTextSearch

OTHER PRODUCTS and why I've written this module

       I'm  aware  of DBIx::TextIndex and DBIx::KwIndex modules and about UdmSearch utility, and about htdig and
       glimpse on the non-database side of the world.

       To me, using a database gives reasonable maintenance benefits. With products that use their own files  to
       store  the  information  (even  if  the storage algorithms are efficient and well thought of), you always
       struggle with permissions on files and directories for various users, with files that somebody accidently
       deleted or mungled, and making the index available remotely is not trivial.

       That's why I've wanted a module that will use a database as a storage backend. With MySQL, you get remote
       access and access control for free, and on many web servers MySQL is part of the standard  equipment.  So
       using it for text indexes seemed natural.

       However,  existing  DBIx::TextIndex  and  UdmSearch  are  too narrow-aimed to me. The first only supports
       indexing of data that is stored in the database, but you may  not  always  want  or  need  to  store  the
       documents  in  the database as well. The UdmSearch on the other hand is only for web documents, making it
       unsuitable for indexing mailing-list archives or local data.

       I believe that DBIx::FullTextSearch is reasonably flexible and still very efficient. It  doesn't  enforce
       its own idea of what is good for you -- the number of options is big and you can always extend the module
       with  your  own backend of frontend if you feel that those provided are not sufficient. Or you can extend
       existing by adding one or two parameters that will add  new  features.  Of  course,  patches  are  always
       welcome.   DBIx::FullTextSearch  is  a  tool  that  can be deployed in many projects. It's not a complete
       environment since different people have different needs. On the other hand, the methods that it  provides
       make it easy to build a complete solution on top of this in very short course of time.

POD ERRORS

       Hey! The above document had some coding errors, which are explained below:

       Around line 1188:
           Non-ASCII character seen before =encoding in 'Suchý'. Assuming CP1252

perl v5.26.1                                       2018-04-02                          DBIx::FullTextSearch(3pm)