Installer le moteur de recherche (ht://Dig)


ht://Dig 3.1.5 -- 25 février 2000, (.tar.gz) 1,9 Mo
ht://Dig 3.2.0b3 -- 22 février 2001, (.tar.gz) 1,9 Mo
WWW Search Engine Software
The ht://Dig system is a complete world wide web indexing and searching system for a small domain or intranet. This system is not meant to replace the need for powerful internet-wide search systems like Lycos, Infoseek, Webcrawler and AltaVista. Instead it is meant to cover the search needs for a single company, campus, or even a particular sub section of a web site.
As opposed to some WAIS-based or web-server based search engines, ht://Dig can span several web servers at a site. The type of these different web servers doesn't matter as long as they understand the HTTP 1.0 protocol.

ht://Dig est composé de 5 utilitaires :
htdig : Htdig retrieves HTML documents using the HTTP protocol and gathers information from these documents which can later be used to search these documents. This program can be referred to as the search robot.
htmerge : Htmerge is used to create a document index and word database from the files that were created by htdig. These databases are then used by htsearch to perform the actual searched.
htnotify : Htnotify scans the document database created by htmerge and sends an email message for every page that is out of date. Look in the notification manual for instructions to set up this service.
htfuzzy : Htfuzzy creates indexes for different "fuzzy" search algorithms. These indexes can then be used by the htsearch program.
htsearch : Htsearch est le moteur de recherche actuel du système de recherche ht://Dig. C'est un programme CGI qui est conçu pour être invoqué à partir d'un formulaire HTML. Il accepte les méthodes GET et POST de passage de paramètresvers le programme CGI.

Dictionnaires de langues
htDig utilise les mêmes dictionnaires que ispell.
francais --

phpSearch 0.2 -- 5 janvier 1999
PHPSearch est un très simple script PHP3 qui génère une boîte de recherche dans n'importe quelle page web avec un champ de saisie afin que les utilisateurs puissent rechercher sur n'importe quels moteurs de recherche.


$ lynx
$ tar xvzf htdig-3.1.2.tar.gz
$ cd htdig-3.1.2
$ CFLAGS='-O3 -s -march=pentiumpro' CXXFLAGS='-O3 -s -march=pentiumpro' \
	./configure --prefix=/opt/htdig \
	--with-cgi-bin-dir=/usr/local/apache/cgi-bin \
	--with-image-dir=/opt/htdig/www \
$ make
$ make install
$ cd ..
$ rm -rf htdig-3.1.2
Vous pouvez éventuellement stripper les binaires obtenus :
$ strip /opt/htdig/bin/ht*
$ strip /usr/local/apache/cgi-bin/htsearch
Installez le dictionnaire franšais :
$ mkdir francais
$ cd francais
$ ftp
$ tar xvzf francais-IREQ.tar.gz
$ mkdir /opt/htdig/common/francais
$ cp francais.aff /opt/htdig/common/francais
$ cp francais.dico /opt/htdig/common/francais


htDig : /opt/htdig/conf/htdig.conf

# Example config file for ht://Dig.
# This configuration file is used by all the programs that make up ht://Dig.
# Please refer to the attribute reference manual for more details on what
# can be put into this file.  (
# Note that most attributes have very reasonable default values so you
# really only have to add attributes here if you want to change the defaults.
# What follows are some of the common attributes you might want to change.

# Specify where the database files need to go.  Make sure that there is
# plenty of free disk space available for the databases.  They can get
# pretty big.
database_dir:		/opt/htdig/db

# This specifies the URL where the robot (htdig) will start.  You can specify
# multiple URLs here.  Just separate them by some whitespace.
# The example here will cause the ht://Dig homepage and related pages to be
# indexed.
# You could also index all the URLs in a file like so:
# start_url:	       `${common_dir}/start.url`
start_url: \ \ \

# This attribute limits the scope of the indexing process.  The default is to
# set it to the same as the start_url above.  This way only pages that are on
# the sites specified in the start_url attribute will be indexed and it will
# reject any URLs that go outside of those sites.
# Keep in mind that the value for this attribute is just a list of string
# patterns. As long as URLs contain at least one of the patterns it will be
# seen as part of the scope of the index.
limit_urls_to:		${start_url}

# If there are particular pages that you definately do NOT want to index, you
# can use the exclude_urls attribute.  The value is a list of string patterns.
# If a URL matches any of the patterns, it will NOT be indexed.  This is
# useful to exclude things like virtual web trees or database accesses.  By
# default, all CGI URLs will be excluded.  (Note that the /cgi-bin/ convention
# may not work on your web server.  Check the  path prefix used on your web
# server.)
exclude_urls:		/cgi-bin/ .cgi

# The string htdig will send in every request to identify the robot.  Change
# this to your email address.

# The excerpts that are displayed in long results rely on stored information
# in the index databases.  The compiled default only stores 512 characters of
# text from each document (this excludes any HTML markup...)  If you plan on
# using the excerpts you probably want to make this larger.  The only concern
# here is that more disk space is going to be needed to store the additional
# information.  Since disk space is cheap (! :-)) you might want to set this
# to a value so that a large percentage of the documents that you are going
# to be indexing are stored completely in the database.  At SDSU we found
# that by setting this value to about 50k the index would get 97% of all
# documents completely and only 3% was cut off at 50k.  You probably want to
# experiment with this value.
# Note that if you want to set this value low, you probably want to set the
# excerpt_show_top attribute to false so that the top excerpt_length characters
# of the document are always shown.
max_head_length:	10000

# To limit network connections, ht://Dig will only pull up to a certain limit
# of bytes. This prevents the indexing from dying because the server keeps
# sending information. However, several FAQs happen because people have files
# bigger than the default limit of 100KB. This sets the default a bit higher.
# (see <"> for more)
max_doc_size:		200000

# Depending on your needs, you might want to enable some of the fuzzy search
# algorithms.  There are several to choose from and you can use them in any
# combination you feel comfortable with.  Each algorithm will get a weight
# assigned to it so that in combinations of algorithms, certain algorithms get
# preference over others.  Note that the weights only affect the ranking of
# the results, not the actual searching.
# The available algorithms are:
#	exact
#	endings
#	synonyms
#	soundex
#	metaphone
# By default only the "exact" algorithm is used with weight 1.
# Note that if you are going to use any of the algorithms other than "exact",
# you need to use the htfuzzy program to generate the databases that each
# algorithm requires.
search_algorithm:	exact:1 synonyms:0.5 endings:0.1

# The following are the templates used in the builtin search results
# The default is to use compiled versions of these files, which produces
# slightly faster results. However, uncommenting these lines makes it
# very easy to change the format of search results.
# See < for more details.
# template_map: Long long ${common_dir}/long.html \
#		Short short ${common_dir}/short.html
# template_name: long

# The following are used to change the text for the page index.
# The defaults are just boring text numbers.  These images spice
# up the result pages quite a bit.  (Feel free to do whatever, though)
next_page_text:		<img src=/htdig/buttonr.gif border=0 align=middle width=30 height=30 alt=next>
prev_page_text:		<img src=/htdig/buttonl.gif border=0 align=middle width=30 height=30 alt=prev>
page_number_text:	"<img src=/htdig/button1.gif border=0 align=middle width=30 height=30 alt=1>" \
			"<img src=/htdig/button2.gif border=0 align=middle width=30 height=30 alt=2>" \
			"<img src=/htdig/button3.gif border=0 align=middle width=30 height=30 alt=3>" \
			"<img src=/htdig/button4.gif border=0 align=middle width=30 height=30 alt=4>" \
			"<img src=/htdig/button5.gif border=0 align=middle width=30 height=30 alt=5>" \
			"<img src=/htdig/button6.gif border=0 align=middle width=30 height=30 alt=6>" \
			"<img src=/htdig/button7.gif border=0 align=middle width=30 height=30 alt=7>" \
			"<img src=/htdig/button8.gif border=0 align=middle width=30 height=30 alt=8>" \
			"<img src=/htdig/button9.gif border=0 align=middle width=30 height=30 alt=9>" \
			"<img src=/htdig/button10.gif border=0 align=middle width=30 height=30 alt=10>"
# To make the current page stand out, we will put a border arround the
# image for that page.
no_page_number_text:	"<img src=/htdig/button1.gif border=2 align=middle width=30 height=30 alt=1>" \
			"<img src=/htdig/button2.gif border=2 align=middle width=30 height=30 alt=2>" \
			"<img src=/htdig/button3.gif border=2 align=middle width=30 height=30 alt=3>" \
			"<img src=/htdig/button4.gif border=2 align=middle width=30 height=30 alt=4>" \
			"<img src=/htdig/button5.gif border=2 align=middle width=30 height=30 alt=5>" \
			"<img src=/htdig/button6.gif border=2 align=middle width=30 height=30 alt=6>" \
			"<img src=/htdig/button7.gif border=2 align=middle width=30 height=30 alt=7>" \
			"<img src=/htdig/button8.gif border=2 align=middle width=30 height=30 alt=8>" \
			"<img src=/htdig/button9.gif border=2 align=middle width=30 height=30 alt=9>" \
			"<img src=/htdig/button10.gif border=2 align=middle width=30 height=30 alt=10>"

# local variables:
# mode: text
# eval: (if (eq window-system 'x) (progn (setq font-lock-keywords (list '("^#.*" . font-lock-keyword-face) '("^[a-zA-Z][^ :]+" . font-lock-function-name-face) '("[+$]*:" . font-lock-comment-face) )) (font-lock-mode)))
# end:

locale:			fr_FR
lang_dir:		${common_dir}/francais
bad_word_list:		${lang_dir}/bad_words
endings_affix_file:	${lang_dir}/francais.aff
endings_dictionary:	${lang_dir}/francais.dico
endings_root2word_db:	${lang_dir}/root2word.db
endings_word2root_db:	${lang_dir}/word2root.db
Executer les commandes:
$ ./htfuzzy endings
$ ./rundig

Apache : /usr/local/apache/conf/httpd.conf

Alias /htdig/ /opt/htdig/www/
<Directory /opt/htdig/www>
    Options Indexes MultiViews
    AllowOverride None
    Order allow,deny
    Allow from all

crond :

Configurez la crontab pour lancer l'indexation des sites tous les jours ou toutes les semaines.




Christophe Merlet
©Tous droits réservés
11 juin 2001