NAME

Search::Fulltext::Tokenizer::MeCab - Provides Japanese fulltext search for Search::Fulltext module

SYNOPSIS

use Search::Fulltext;
use Search::Fulltext::Tokenizer::MeCab;


my $query = '猫';
my @docs = (
    '我輩は猫である',
    '犬も歩けば棒に当る',
    '実家でてんちゃんって猫を飼ってまして，ものすっごい可愛いんですよほんと',
);


my $fts = Search::Fulltext->new({
    docs      => \@docs,
    tokenizer => "perl 'Search::Fulltext::Tokenizer::MeCab::tokenizer'",
});
my $results = $fts->search($query);
is_deeply($results, [0, 2]);        # 1st & 3rd include '猫'
my $results = $fts->search('猫 AND 可愛い');
is_deeply($results, [2]);

DESCRIPTION

Search::Fulltext::Tokenizer::MeCab is a Japanse tokenizer working with fulltext search module Search::Fulltext. Only you have to do is specify perl 'Search::Fulltext::Tokenizer::MeCab::tokenizer' as a tokenizer of Search::Fulltext.

my $fts = Search::Fulltext->new({
    docs      => \@docs,
    tokenizer => "perl 'Search::Fulltext::Tokenizer::MeCab::tokenizer'",
});

You are supposed to use UTF-8 strings for docs.

Although various queries are available like "QUERIES" in Search::Fulltext, wildcard query (e.g. '我*') and phrase query (e.g. '"我輩は猫である"') are not supported.

User dictionary can be used to change the tokenizing behavior of internally-used Text::MeCab. See ENVIRONMENTAL VARIABLES section for detailes.

ENVIRONMENTAL VARIABLES

Some environmental variables are provided to customize the behavior of Search::Fulltext::Tokenizer::MeCab.

Typical usage:

$ ENV1=foobar ENV2=buz perl /path/to/your_script_using_this_module ARGS

• MECABDIC_USERDIC

  Specify path(s) to MeCab's user dictionary.

  See MeCab's manual to learn how to create user dictionary.

  Examples:

    MECABDIC_USERDIC="/path/to/yourdic1.dic"
    MECABDIC_USERDIC="/path/to/yourdic1.dic, /path/to/yourdic2.dic"
  

• MECABDIC_DEBUG

  When set to not 0, debug strings appear on STDERR.

  Especially, outputs below would help check how your docs are tokenized.

    string to be parsed: 我輩は猫である (7)
    token: 我輩 (2)
    token: は (1)
    token: 猫 (1)
    token: で (1)
    token: ある (2)
    ...
    string to be parsed: 猫 AND 可愛い (9)
    token: 猫 (1)
    string to be parsed:  可愛い (4)
    token: 可愛い (3)
  

  Note that not only docs but also queries are also tokenized.

SUPPORTS

Bug reports and pull requests are welcome at https://github.com/laysakura/Search-Fulltext-Tokenizer-MeCab !

To read this manual via perldoc, use -t option for correctly displaying UTF-8 caracters.

$ perldoc -t Search::Fulltext::Tokenizer::MeCab

VERSION

Version 1.05

AUTHOR

Sho Nakatani lay.sakura@gmail.com, a.k.a. @laysakura