- 浏览: 1637597 次
- 性别:
- 来自: 北京
文章分类
- 全部博客 (405)
- C/C++ (16)
- Linux (60)
- Algorithm (41)
- ACM (8)
- Ruby (39)
- Ruby on Rails (6)
- FP (2)
- Java SE (39)
- Java EE (6)
- Spring (11)
- Hibernate (1)
- Struts (1)
- Ajax (5)
- php (2)
- Data/Web Mining (20)
- Search Engine (19)
- NLP (2)
- Machine Learning (23)
- R (0)
- Database (10)
- Data Structure (6)
- Design Pattern (16)
- Hadoop (2)
- Browser (0)
- Firefox plugin/XPCOM (8)
- Eclise development (5)
- Architecture (1)
- Server (1)
- Cache (6)
- Code Generation (3)
- Open Source Tool (5)
- Develope Tools (5)
- 读书笔记 (7)
- 备忘 (4)
- 情感 (4)
- Others (20)
- python (0)
最新评论
-
532870393:
请问下,这本书是基于Hadoop1还是Hadoop2?
Hadoop in Action简单笔记(一) -
dongbiying:
不懂呀。。
十大常用数据结构 -
bing_it:
...
使用Spring MVC HandlerExceptionResolver处理异常 -
一别梦心:
按照上面的执行,文件确实是更新了,但是还是找不到kernel, ...
virtualbox 4.08安装虚机Ubuntu11.04增强功能失败解决方法 -
dsjt:
楼主spring 什么版本,我的3.1 ,xml中配置 < ...
使用Spring MVC HandlerExceptionResolver处理异常
很好的一本书,介绍的非常全面,看了很久了,还没有看完,刚看完前十章,发现前面看的都忘的差不多了,还是回来记一下吧。
Boolean Retrieval
一、information retrieval定义:
学院派定义:
Information retrieval (IR) is finding material (usually documents) of
an unstructured nature (usually text) that satisfies an information need
from within large collections (usually stored on computers).
Category :
Category By Scale :
web search、domain-specific search、personal information retrieval
Basic need:
1、To process large document collections quickly.
2、To allow more flexible matching operations
3、To allow ranked retrieval
Simple idea:
term-document incidence matrix use binary logical OR AND NOT...:110100 AND 110111 AND 101111 = 100100
What is Boolean Retrival:
The Boolean retrieval model is a model for information BOOLEAN RETRIEVAL retrieval in which we
MODEL can pose any query which is in the form of a Boolean expression of terms,
that is, in which terms are combined with the operators AND, OR, and NOT.
Such queries effectively view each document as a set of words.
What's the boolean retrival query like:
(Calpurnia AND Brutus) AND Caesar
how to assess IR system
Precision
: What fraction of the returned results are relevant to the information
need?
Recall
: What fraction of the relevant documents in the collection were returned
by the system?
vector space model: Easy to rank
Term-document matrix: not scalable
Inverted index: dictionary and posting list.
How Build Inverted index :
1. Collect the documents to be indexed:
Friends, Romans, countrymen. So let it be with Caesar . . .
2. Tokenize the text, turning each document into a list of tokens:
Friends Romans countrymen So . . .
3. Do linguistic preprocessing, producing a list of normalized tokens, which
are the indexing terms: friend roman countryman so . . .
4. Index the documents that each term occurs in by creating an inverted index,
consisting of a dictionary and postings.
Processing Boolean queries:
AND operation:
intersect two posting list:
1 answer ← ()
2 while p1 != NIL and p2 != NIL
3 do if docID(p1) = docID(p2)
4 then ADD(answer, docID(p1))
5 p1 ← next(p1)
6 p2 ← next(p2)
7 else if docID(p1) < docID(p2)
8 then p1 ← next(p1)
9 else p2 ← next(p2)
10 return answer
mulitiple term AND operation:
Process terms in order of increasing document frequency:
if we start by intersecting the two smallest postings lists, then all intermediate resultsmust be no bigger than the smallest postings list, and we are therefore likely to do the least amount of total work
1 terms ← SORTBYINCREASINGFREQUENCY(ht1, . . . , tni)
2 result ← postings( f irst(terms))
3 terms ← rest(terms)
4 while terms != NIL and result != NIL
5 do result ← INTERSECT(result, postings( f irst(terms)))
6 terms ← rest(terms)
7 return result
OR operation:
The idea is 归并排序中的n路归并,similarily with AND operation。
The extended Boolean model versus ranked retrieval:
Proximity operator:
A proximity operator is a way of specifying that two terms in a query must occur in a document close to each other, where closeness may be measured
by limiting the allowed number of intervening words or by reference to a structural unit such as a sentence or paragraph.
Addition to do:
1. We would like to better determine the set of terms in the dictionary and
to provide retrieval that is tolerant to spelling mistakes and inconsistent
choice of words.
2. It is often useful to search for compounds or phrases that denote a concept
such as “operating system”. As the Westlaw examples show, we might also
wish to do proximity queries such as Gates NEAR Microsoft. To answer
such queries, the index has to be augmented to capture the proximities of
terms in documents.
3. A Boolean model only records term presence or absence, but often we
would like to accumulate evidence, givingmoreweight to documents that
have a term several times as opposed to ones that contain it only once. To
be able to do this we need the term frequency information TERM FREQUENCY (the number of
times a term occurs in a document) in postings lists.
4. Boolean queries just retrieve a set of matching documents, but commonly
we wish to have an effective method to order (or “rank”) the returned
results. This requires having a mechanism for determining a document
score which encapsulates how good a match a document is for a query.
发表评论
-
Lucene 索引格式
2013-06-25 20:11 0索引结构: 索引层次 ... -
计算广告学
2012-08-12 13:53 0计算广告学一: 1、核 ... -
《Lucene in Action》简单笔记
2011-12-22 09:19 0第一章 Meet Lucene -
Information Retrieval Resources
2011-04-07 16:40 1369Information Retrieval Resource ... -
使用Jsoup抽取数据
2011-03-20 19:22 4917Jsoup是一个Java的HTML解析器,提供了非常方便的抽取 ... -
常见文件类型识别
2010-09-22 20:09 11790根据文件的后缀名识别文件类型并不准确,可以使用文件的头信息进行 ... -
(zz)信息检索领域资料整理
2010-06-05 13:05 3135A Guide to Information Retrieva ... -
Introduce to Inforamtion Retrieval读书笔记(2)
2009-10-31 13:02 1892The term vocabulary and posting ... -
Query Log Mining notes
2009-10-02 18:08 1245Enhancing Efficiency of Search ... -
百度搜索的一些高级语法
2009-08-27 20:06 18941.title语法 就是在title ... -
Hadoop好书推荐:Hadoop The Definitive Guide
2009-08-16 22:49 3615第一本详细介绍Hadoop的书籍,从网上下来看了几章,作者是H ... -
Java开源搜索引擎[收藏]
2008-04-24 00:09 2880Egothor Egothor是一个用Java编写的开 ... -
分享一本斯坦福的信息检索的教材
2008-01-04 23:59 2433斯坦福的信息检索的教材,还没出版,先分享一下电子版原稿. 对于 ... -
分享一本搜索引擎的电子书
2007-12-29 19:42 2496还没有来得及看,但搜索引擎的书不是很好找,先放上,希望对大家能 ... -
分享一个Nutch入门学习的资料
2007-12-18 20:49 4241分享一个Nutch入门学习的资料,感觉写的还不错. -
搜索引擎Nutch源代码研究之一 网页抓取(4)
2007-12-17 22:37 8355今天来看看Nutch如何Parse网页的: Nutch使用了两 ... -
[转]MAP/REDUCE:Google和Nutch实现异同及其他
2007-12-15 19:21 2951设计要素 nutch包含以下几个部分: 辅助类 Log:记载运 ... -
Nutch源代码学习小小总结一下
2007-12-15 19:13 4422我现在看得源码主要是网页抓取部分,这部分相对比较容易。我首先定 ... -
搜索引擎Nutch源代码研究之一 网页抓取(3)
2007-12-15 16:39 4543今天我们看看Nutch网页抓取,所用的几种数据结构: 主要涉及 ... -
搜索引擎Nutch源代码研究之一 网页抓取(2)
2007-12-15 00:36 5527今天我们来看看Nutch的源代码中的protocol-h ...
相关推荐
教材introduce to java programming 9th英文版,pdf,欢迎下载
Introduce to Algorithms, A Creative Approach .英文版
introduce to linux.html
目前为止找到的最详细的NS2说明文档 比官网的ns manual 还要详细
包括 Introduce to Java Programming 8th的全部课后习题答案(偶数以及奇数习题),还包括课本讲述过程中的习题。欢迎下载。
龙书 9~15章 的代码,"" 需加 L"", d3dutility.cpp 文件中需加 winmm.lib
最详细的 MIT 线性代数 公开课笔记 结合 MIT线性代数+《Introduce to linear algebra》书籍的详细中文翻译
线性优化讲义 introduction to linear optimization ,
EIB的控制网络及协议介绍,通过此文档可以理解EIB控制网络及布局,理解EIB协议设计的基本知识
team introduce.key
computer book
中文翻译Introduction to Linear Algebra, 5th Edition 5.2节(仅供交流学习)
这是对设计模式的简单简绍,希望对大家有用
Introduction to Lens Design
This book is meant to be an overview of the Tornado web server, and will walk readers through the basics of the framework, some sample applications, and best practices for use in the real world....
Gilbert Strang's textbooks have changed the entire approach to learning linear algebra -- away from abstract vector spaces to specific examples of the four fundamental subspaces: the column space and ...
我们想快速地乘上傅里叶矩阵 F 与它的逆 F−1。这通过快速傅里叶变换完成。一个普通乘积 Fc 用到 n2 次乘法(F 具有 n2 项)。FFT 仅需要 n 乘以 12 log2 n 次乘法。我们将看到这是如何实现的。 FFT 彻底改变了信号...
LINUX INTRODUCE
Fuel_Gauge_introduce
计算机系统概论 英文版 作者: [美] Yale N. Patt'Introduction to Computing Systems: From bits & gates ... To understand the computer, the authors introduce the LC-3 and provide the LC-3 Simulator to give st