当前位置:网站首页>How to get a token from tokenstream based on Lucene 3.5.0
How to get a token from tokenstream based on Lucene 3.5.0
2022-07-05 11:27:00 【Full stack programmer webmaster】
Through the study Lucene3.5.0 Of doc file , To be different release Version number lucene Version number of API Modify and analyze . Finally, we found valuable modification information . LUCENE-2302: Deprecated TermAttribute and replaced by a new CharTermAttribute. The change is backwards compatible, so mixed new/old TokenStreams all work on the same char[] buffer independent of which interface they use. CharTermAttribute has shorter method names and implements CharSequence and Appendable. This allows usage like Java’s StringBuilder in addition to direct char[] access. Also terms can directly be used in places where CharSequence is allowed (e.g. regular expressions). (Uwe Schindler, Robert Muir) The above information can be known , The original passed method can no longer extract the response Token 了
StringReader reader = new StringReader(s);
TokenStream ts =analyzer.tokenStream(s, reader);
TermAttribute ta = ts.getAttribute(TermAttribute.class);
Through analysis Api document information You know ,CharTermAttribute Has become a replacement TermAttribute So I wrote a sample to better understand the interface from TokenStream Extract from Token
package com.segment;
import java.io.StringReader;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.Token;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.tokenattributes.TermAttribute;
import org.apache.lucene.util.AttributeImpl;
import org.wltea.analyzer.lucene.IKAnalyzer;
public class Segment {
public static String show(Analyzer a, String s) throws Exception {
StringReader reader = new StringReader(s);
TokenStream ts = a.tokenStream(s, reader);
String s1 = "", s2 = "";
boolean hasnext= ts.incrementToken();
//Token t = ts.next();
while (hasnext) {
//AttributeImpl ta = new AttributeImpl();
CharTermAttribute ta = ts.getAttribute(CharTermAttribute.class);
//TermAttribute ta = ts.getAttribute(TermAttribute.class);
s2 = ta.toString() + " ";
s1 += s2;
hasnext = ts.incrementToken();
}
return s1;
}
public String segment(String s) throws Exception {
Analyzer a = new IKAnalyzer();
return show(a, s);
}
public static void main(String args[])
{
String name = " I'm Junjie , I love programming. , My test case ";
Segment s = new Segment();
String test = "";
try {
System.out.println(test+s.segment(name));
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Publisher : Full stack programmer stack length , Reprint please indicate the source :https://javaforall.cn/109513.html Link to the original text :https://javaforall.cn
边栏推荐
- COMSOL -- three-dimensional graphics random drawing -- rotation
- SET XACT_ABORT ON
- msfconsole命令大全,以及使用说明
- Ddrx addressing principle
- DDR4的特性与电气参数
- Beego cross domain problem solution - successful trial
- Solve the problem of slow access to foreign public static resources
- 【爬虫】charles unknown错误
- 解决readObjectStart: expect { or n, but found N, error found in #1 byte of ...||..., bigger context ..
- AUTOCAD——遮罩命令、如何使用CAD对图纸进行局部放大
猜你喜欢
[office] eight usages of if function in Excel
[advertising system] parameter server distributed training
数据库三大范式
【爬虫】wasm遇到的bug
R3Live系列学习(四)R2Live源码阅读(2)
COMSOL -- 3D casual painting -- sweeping
7.2 daily study 4
Intelligent metal detector based on openharmony
COMSOL--三维图形的建立
[advertising system] incremental training & feature access / feature elimination
随机推荐
7 themes and 9 technology masters! Dragon Dragon lecture hall hard core live broadcast preview in July, see you tomorrow
【DNS】“Can‘t resolve host“ as non-root user, but works fine as root
pytorch训练进程被中断了
COMSOL -- establishment of 3D graphics
高校毕业求职难?“百日千万”网络招聘活动解决你的难题
居家办公那些事|社区征文
SET XACT_ABORT ON
Web API配置自定义路由
解决grpc连接问题Dial成功状态为TransientFailure
In the last process before the use of the risk control model, 80% of children's shoes are trampled here
Solve the grpc connection problem. Dial succeeds with transientfailure
Wechat nucleic acid detection appointment applet system graduation design completion (8) graduation design thesis template
基础篇——基础项目解析
C#实现WinForm DataGridView控件支持叠加数据绑定
Harbor镜像仓库搭建
Harbor image warehouse construction
Solve the problem of slow access to foreign public static resources
Wechat nucleic acid detection appointment applet system graduation design completion (7) Interim inspection report
[SWT component] content scrolledcomposite
Risc-v-qemu-virt in FreeRTOS_ Scheduling opportunity of GCC