正版 驾驭文本:文本的发现、组织和处理:how to find, organize,

目录
第1章　开始驾驭文本 ...............................................................................1
1.1　驾驭文本重要的原因 ...............................................................................................2
1.2　预览：一个基于事实的问答系统 ...........................................................................4
1.2.1　嗨，弗兰肯斯坦医生 ...................................................................................5
1.3　理解文本很困难 .......................................................................................................8
1.4　驾驭的文本 .............................................................................................................11
1.5　文本及智能应用：搜索及其他 .............................................................................13
1.5.1　搜索和匹配 .................................................................................................13
1.5.2　抽取信息 .....................................................................................................14
1.5.3　对信息分组 .................................................................................................15
1.5.4　一个智能应用 .............................................................................................15
1.6　小结 .........................................................................................................................15
1.7　相关资源 .................................................................................................................16
第2章　驾驭文本的基础 ..........................................................................17
2.1　语言基础知识 .........................................................................................................18
2.1.1　词语及其类别 .............................................................................................19
2.1.2　短语及子句 .................................................................................................20
2.1.3　词法 .............................................................................................................21
2.2　文本处理常见工具 .................................................................................................23
2.2.1　字符串处理工具 .........................................................................................23
2.2.2　词条及切词 .................................................................................................23
2.2.3　词性标注 .....................................................................................................25
2.2.4　词干还原 .....................................................................................................27
2.2.5　句子检测 .....................................................................................................29
2.2.6　句法分析和文法 .........................................................................................31
2.2.7　序列建模 .....................................................................................................33
2.3　从常见格式文件中抽取内容并做预处理 .............................................................34
2.3.1　预处理的重要性 .........................................................................................35
2.3.2　利用Apache Tika抽取内容 ........................................................................37
2.4　小结 .........................................................................................................................39
2.5　相关资源 .................................................................................................................40
第3章　搜索 ............................................................................................41
3.1　搜索和多面示例：Amazon.com ............................................................................42
3.2　搜索概念入门 .........................................................................................................44
3.2.1　索引内容 .....................................................................................................45
3.2.2　用户输入 .....................................................................................................47
3.2.3　利用向量空间模型对文档排名 .................................................................51
3.2.4　结果展示 .....................................................................................................54
3.3　Apache Solr搜索服务器介绍 .................................................................................57
3.3.1　首次运行Solr ..............................................................................................58
3.3.2　理解Solr中的概念 ......................................................................................59
3.4　利用Apache Solr对内容构建索引 .........................................................................63
3.4.1　使用XML构建索引 ....................................................................................64
3.4.2　利用Solr和Apache Tika对内容进行抽取和索引 ......................................66
3.5　利用Apache Solr来搜索内容 .................................................................................69
3.5.1　Solr查询输入参数 ......................................................................................71
3.5.2　抽取内容的多面展示 .................................................................................74
3.6　理解搜索性能因素 .................................................................................................77
3.6.1　数量判定 .....................................................................................................77
3.6.2　判断数量 .....................................................................................................81
3.7　提高搜索性能 .........................................................................................................82
3.7.1　硬件改进 .....................................................................................................82
3.7.2　分析的改进 .................................................................................................83
3.7.3　提高查询性能 .............................................................................................85
3.7.4　其他评分模型 .............................................................................................88
3.7.5　提升Solr性能的技术 ..................................................................................89
3.8　其他搜索工具 .........................................................................................................91
3.9　小结 .........................................................................................................................93
3.10　相关资源 ...............................................................................................................93
第4章　模糊字符串匹配 ..........................................................................94
4.1　模糊字符串匹配方法 .............................................................................................96
4.1.1　字符重合度度量方法 .................................................................................96
4.1.2　编辑距离 .....................................................................................................99
4.1.3　n元组编辑距离 .........................................................................................102
4.2　寻找模糊匹配串 ...................................................................................................105
4.2.1　在Solr中使用前缀来匹配 ........................................................................105
4.2.2　利用trie树进行前缀匹配 .........................................................................106
4.2.3　使用n元组进行匹配 ..................................................................................111
4.3　构建模糊串匹配应用 ...........................................................................................112
4.3.1　在搜索中加入提前输入功能 ...................................................................113
4.3.2　搜索中的查询拼写校正 ...........................................................................117
4.3.3　记录匹配 ...................................................................................................122
4.4　小结 .......................................................................................................................127
4.5　相关资源 ...............................................................................................................128
第5章　命名实体识别 ...........................................................................129
5.1　命名实体的识别方法 ...........................................................................................131
5.1.1　基于规则的实体识别 ...............................................................................131
5.1.2　基于统计分类器的实体识别 ...................................................................132
5.2　基于OpenNLP的基本实体识别 ...........................................................................133
5.2.1　利用OpenNLP寻找人名 ...........................................................................134
5.2.2　OpenNLP识别的实体解读 .......................................................................136
5.2.3　基于概率过滤实体 ...................................................................................137
5.3　利用OpenNLP进行深度命名实体识别 ...............................................................137
5.3.1　利用OpenNLP识别多种实体类型 ...........................................................138
5.3.2　OpenNLP识别实体的背后机理 ...............................................................141
5.4　OpenNLP的性能 ...................................................................................................143
5.4.1　结果的质量 ...............................................................................................144
5.4.2　运行性能 ...................................................................................................145
5.4.3　OpenNLP的内存使用 ...............................................................................146
5.5　对新领域定制OpenNLP实体识别 .......................................................................147
5.5.1　训练模型的原因和方法 ...........................................................................147
5.5.2　训练OpenNLP模型 ...................................................................................148
5.5.3　改变建模输入 ...........................................................................................150
5.5.4　对实体建模的新方法 ...............................................................................152
5.6　小结 .......................................................................................................................154
5.7　进一步阅读材料 ...................................................................................................155
第6章　文本聚类 ..................................................................................156
6.1　Google News中的文档聚类 .................................................................................157
6.2　聚类基础 ...............................................................................................................158
6.2.1　三种聚类的文本类型 ...............................................................................158
6.2.2　选择聚类算法 ...........................................................................................160
6.2.3　确定相似度 ...............................................................................................161
6.2.4　给聚类结果打标签 ...................................................................................162
6.2.5　聚类结果的评估 .......................................................................................163
6.3　搭建一个简单的聚类应用 ...................................................................................165
6.4　利用Carrot2对搜索结果聚类 ...............................................................................166
6.4.1　使用Carrot2API ........................................................................................166
6.4.2　使用Carrot2对Solr的搜索结果聚类 ........................................................168
6.5　利用Apache Mahout对文档集聚类 ......................................................................171
6.5.1　对聚类的数据进行预处理 .......................................................................172
6.5.2　K-means聚类 ............................................................................................175
6.6　利用Apache Mahout进行主题建模 ......................................................................180
6.7　考察聚类性能 .......................................................................................................183
6.7.1　特征选择与特征约简 ...............................................................................183
6.7.2　Carrot2的性能和质量 ...............................................................................186
6.7.3　Mahout基准聚类算法 ..............................................................................187
6.8　致谢 .......................................................................................................................192
6.9　小结 .......................................................................................................................192
6.10　参考文献 .............................................................................................................193
第7章　分类及标注 ...............................................................................195
7.1　分类及归类概述 ...................................................................................................197
7.2　分类过程 ...............................................................................................................200
7.2.1　选择分类机制 ...........................................................................................201
7.2.2　识别文本分类中的特征 ...........................................................................202
7.2.3　训练数据的重要性 ...................................................................................203
7.2.4　评估分类器性能 .......................................................................................206
7.2.5　将分类器部署到生产环境 .......................................................................208
7.3　利用Apache Lucene构建文档分类器 ..................................................................209
7.3.1　利用Lucene对文本进行分类 ...................................................................210
7.3.2　为MoreLikeThis分类器准备训练数据 ....................................................212
7.3.3　训练MoreLikeThis分类器 ........................................................................214
7.3.4　利用MoreLikeThis分类器对文档进行分类 ............................................217
7.3.5　测试MoreLikeThis分类器 ........................................................................220
7.3.6　将MoreLikeThis投入生产环境 ................................................................223
7.4　利用Apache Mahout训练朴素贝叶斯分类器 ......................................................223
7.4.1　利用朴素贝叶斯算法进行文本分类 .......................................................224
7.4.2　准备训练数据 ...........................................................................................225
7.4.3　留存测试数据 ...........................................................................................229
7.4.4　训练分类器 ...............................................................................................229
7.4.5　测试分类器 ...............................................................................................231
7.4.6　改进自举过程 ...........................................................................................232
7.4.7　将Mahout贝叶斯分类器集成到Solr ........................................................234
7.5　利用OpenNLP进行文档分类 ...............................................................................238
7.5.1　回归模型及最大熵文档分类 ...................................................................239
7.5.2　为最大熵文档分类器准备训练数据 .......................................................241
7.5.3　训练最大熵文档分类器 ...........................................................................242
7.5.4　测试最大熵文档分类器 ...........................................................................248
7.5.5　生产环境下的最大熵文档分类器 ...........................................................249
7.6　利用Apache Solr构建标签推荐系统 ...................................................................250
7.6.1　为标签推荐收集训练数据 .......................................................................253
7.6.2　准备训练数据 ...........................................................................................255
7.6.3　训练Solr标签推荐系统 ............................................................................256
7.6.4　构建推荐标签 ...........................................................................................258
7.6.5　对标签推荐系统进行评估 .......................................................................261
7.7　小结 .......................................................................................................................263
7.8　参考文献 ...............................................................................................................265
第8章　构建示例问答系统 ....................................................................266
8.1　问答系统基础知识 ...............................................................................................268
8.2　安装并运行QA代码 .............................................................................................270
8.3　一个示例问答系统的架构 ...................................................................................271
8.4　理解问题并产生答案 ...........................................................................................274
8.4.1　训练答案类型分类器 ...............................................................................275
8.4.2　对查询进行组块分析 ...............................................................................279
8.4.3　计算答案类型 ...........................................................................................280
8.4.4　生成查询 ...................................................................................................283
8.4.5　对候选段落排序 .......................................................................................285
8.5　改进系统的步骤 ...................................................................................................287
8.6　本章小结 ...............................................................................................................287
8.7　相关资源 ...............................................................................................................288
第9章　未驾驭的文本：探索未来前沿 ..................................................289
9.1　语义、篇章和语用：探索高级NLP ....................................................................290
9.1.1　语义 ...........................................................................................................291
9.1.2　篇章 ...........................................................................................................292
9.1.3　语用 ...........................................................................................................294
9.2　文档及文档集自动摘要 .......................................................................................295
9.3　关系抽取 ...............................................................................................................298
9.3.1　关系抽取方法综述 ...................................................................................299
9.3.2　评估 ...........................................................................................................302
9.3.3　关系抽取工具 ...........................................................................................303
9.4　识别重要内容和人物 ...........................................................................................303
9.4.1　全局重要性及权威度 ...............................................................................304
9.4.2　个人重要性 ...............................................................................................305
9.4.3　与重要性相关的资源及位置 ...................................................................306
9.5　通过情感分析来探测情感 ...................................................................................306
9.5.1　历史及综述 ...............................................................................................307
9.5.2　工具及数据需求 .......................................................................................308
9.5.3　一个基本的极性算法 ...............................................................................309
9.5.4　高级话题 ...................................................................................................311
9.5.5　用于情感分析的开源库 ...........................................................................312
9.6　跨语言检索 ...........................................................................................................313
9.7　本章小结 ...............................................................................................................315
9.8　相关资源 ...............................................................................................................315

Grant Ingersoll是一位工程师、讲师和培训师，也是Lucene代码的提交者已经机器学习项目Mahout的联合创始人。

Thomas Morton是OpenNLP和Maximum Entropy(优选熵)的主要开发者。

Drew Farris是一位技术顾问、软件开发人员及Mahout、Lucene和Solr的贡献者。

王斌，博士，中国科学院信息工程研究所研究员，博士生导师，研究方向为信息检索与自然语言处理。主持重量、省部级科研项目20余项，发表学术论文120余篇。现为中国中文信息学会理事、信息检索专委会、社会媒体处理专委会及语言与知识计算专业委员会委员，《中文信息学报》编委，中国计算机学会不错会员及中文信息处理专委会委员。

《驾驭文本：文本的发现、组织和处理》处理是目前互联网内容应用(如搜索引擎、推荐引擎)的关键技术。本书涵盖了文本处理概念和技术的多个方面，包括文本预处理、搜索、字符串匹配、信息抽取、命名实体识别、分类、聚类、标签生成、摘要、问答等。《驾驭文本：文本的发现、组织和处理》的特点在于通过实例来理解文本处理的这些概念和技术，读者利用现有的开源工具就可以自己实现这些实例。

《驾驭文本：文本的发现、组织和处理》适合于互联网文本内容处理领域的开发人员阅读，也适合有志于加入这一领域的学生、从业人员阅读。即使对于已经从事多年文本处理研究和开发工作的人员来说，《驾驭文本：文本的发现、组织和处理》也不失为一种有益的补充性读物。

自然语言文本的处理是搜索引擎、机器翻译、智能问答、舆情分析等重要系统的核心技术，本书从应用出发，抽丝剥茧般地介绍自然语言文本处理的各项技术，并通过组合这些技术利用开源工具构建了一个事实型的问答系统。