ITokenizer instance is ExtendedWhitespaceTokenizer not ChineseTokenizerAdapter

[beneo]

CONTENTS DELETED

The author has deleted this message.

The fallback will only be used if the primary is not available
(because of a missing optional JAR for example). I've just checked and
it works fine given you have Lucene in path (because that's what being
used for tokenizing Chinese).

Can you use JIRA for submitting bug reports, please? And best: attach
a snippet of code that reproduces the buggy behavior. Thanks.

http://issues.carrot2.org/

Dawid

On Mon, Mar 26, 2012 at 11:46 AM, beneo <[hidden email]> wrote:

> the class org.carrot2.examples.clustering.ClusteringNonEnglishContent in the
> carrot2-example.
>
> and the LanguageCode is set to Chinese Simplified
>
> when invoke the DefaultTokenizerFactory#getTokenizer(LanguageCode), the
> instance is assign from FallbackFactory, when call
> FallbackFactory#createInstance, and the defaultFactory.createInstance() will
> throw exception, so the fallbackFactory.createInstance() will called.
>
> however, the defaultFactory is the class
> org.carrot2.text.linguistic.lucene.ChineseTokenizerAdapter, and the
> fallbackFactory  class is the
> org.carrot2.text.analysis.ExtendedWhitespaceTokenizer
>
> i think it's bug.
>
> --
> View this message in context: http://carrot2-users-and-developers-forum.607571.n2.nabble.com/ITokenizer-instance-is-ExtendedWhitespaceTokenizer-not-ChineseTokenizerAdapter-tp7405352p7405352.html
> Sent from the Carrot2 Users and Developers Forum mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> This SF email is sponsosred by:
> Try Windows Azure free for 90 days Click Here
> http://p.sf.net/sfu/sfd2d-msazure
> _______________________________________________
> Carrot2-developers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/carrot2-developers
>

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Carrot2-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/carrot2-developers