installer: move base zebra config files
[koha.git] / catalogue / search.pl
blob40be2cd478288a12099bee0fce699241f3b92504
1 #!/usr/bin/perl
2 # Script to perform searching
3 # For documentation try 'perldoc /path/to/search'
5 # $Header$
7 # Copyright 2006 LibLime
9 # This file is part of Koha
11 # Koha is free software; you can redistribute it and/or modify it under the
12 # terms of the GNU General Public License as published by the Free Software
13 # Foundation; either version 2 of the License, or (at your option) any later
14 # version.
16 # Koha is distributed in the hope that it will be useful, but WITHOUT ANY
17 # WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
18 # A PARTICULAR PURPOSE. See the GNU General Public License for more details.
20 # You should have received a copy of the GNU General Public License along with
21 # Koha; if not, write to the Free Software Foundation, Inc., 59 Temple Place,
22 # Suite 330, Boston, MA 02111-1307 USA
24 =head1 NAME
26 search - a search script for finding records in a Koha system (Version 3.0)
28 =head1 OVERVIEW
30 This script contains a new search API for Koha 3.0. It is designed to be
31 simple to use and configure, yet capable of performing feats like stemming,
32 field weighting, relevance ranking, support for multiple query language
33 formats (CCL, CQL, PQF), full or nearly full support for the
34 bib1 attribute set, extended attribute sets defined in Zebra profiles, access
35 to the full range of Z39.50 query options, federated searches on Z39.50
36 targets, etc.
38 I believe the API as represented in this script is mostly sound, even if the
39 individual functions in Search.pm and Koha.pm need to be cleaned up. Of course,
40 you are free to disagree :-)
42 I will attempt to describe what is happening at each part of this script.
43 -- JF
45 =head2 INTRO
47 This script performs two functions:
49 =over
51 =item 1. interacts with Koha to retrieve and display the results of a search
53 =item 2. loads the advanced search page
55 =back
57 These two functions share many of the same variables and modules, so the first
58 task is to load what they have in common and determine which template to use.
59 Once determined, proceed to only load the variables and procedures necessary
60 for that function.
62 =head2 THE ADVANCED SEARCH PAGE
64 If we're loading the advanced search page this script will call a number of
65 display* routines which populate objects that are sent to the template for
66 display of things like search indexes, languages, search limits, branches,
67 etc. These are not stored in the template for two reasons:
69 =over
71 =item 1. Efficiency - we have more control over objects inside the script,
72 and it's possible to not duplicate things like indexes (if the search indexes
73 were stored in the template they would need to be repeated)
75 =item 2. Customization - if these elements were moved to the sql database it
76 would allow a simple librarian to determine which fields to display on the page
77 without editing any html (also how the fields should behave when being searched).
79 =back
81 However, they create one problem : the strings aren't translated. I have an idea
82 for how to do this that I will purusue soon.
84 =head2 PERFORMING A SEARCH
86 If we're performing a search, this script performs three primary
87 operations:
89 =over
91 =item 1. builds query strings (yes, plural)
93 =item 2. perform the search and return the results array
95 =item 3. build the HTML for output to the template
97 =back
99 There are several additional secondary functions performed that I will
100 not cover in detail.
102 =head3 1. Building Query Strings
104 There are several types of queries needed in the process of search and retrieve:
106 =over
108 =item 1 Koha query - the query that is passed to Zebra
110 This is the most complex query that needs to be built. The original design goal
111 was to use a custom CCL2PQF query parser to translate an incoming CCL query into
112 a multi-leaf query to pass to Zebra. It needs to be multi-leaf to allow field
113 weighting, koha-specific relevance ranking, and stemming. When I have a chance
114 I'll try to flesh out this section to better explain.
116 This query incorporates query profiles that aren't compatible with non-Zebra
117 Z39.50 targets to acomplish the field weighting and relevance ranking.
119 =item 2 Federated query - the query that is passed to other Z39.50 targets
121 This query is just the user's query expressed in CCL CQL, or PQF for passing to a
122 non-zebra Z39.50 target (one that doesn't support the extended profile that Zebra does).
124 =item 3 Search description - passed to the template / saved for future refinements of
125 the query (by user)
127 This is a simple string that completely expresses the query in a way that can be parsed
128 by Koha for future refinements of the query or as a part of a history feature. It differs
129 from the human search description:
131 1. it does not contain commas or = signs
133 =item 4 Human search description - what the user sees in the search_desc area
135 This is a simple string nearly identical to the Search description, but more human
136 readable. It will contain = signs or commas, etc.
138 =back
140 =head3 2. Perform the Search
142 This section takes the query strings and performs searches on the named servers, including
143 the Koha Zebra server, stores the results in a deeply nested object, builds 'faceted results',
144 and returns these objects.
146 =head3 3. Build HTML
148 The final major section of this script takes the objects collected thusfar and builds the
149 HTML for output to the template and user.
151 =head3 Additional Notes
153 Not yet completed...
155 =cut
157 use strict; # always use
159 ## STEP 1. Load things that are used in both search page and
160 # results page and decide which template to load, operations
161 # to perform, etc.
162 ## load Koha modules
163 use C4::Context;
164 use C4::Output;
165 use C4::Auth;
166 use C4::Search;
167 use C4::Languages; # getAllLanguages
168 use C4::Koha;
169 use POSIX qw(ceil floor);
170 use C4::Branch; # GetBranches
171 # create a new CGI object
172 # not sure undef_params option is working, need to test
173 use CGI qw('-no_undef_params');
174 my $cgi = new CGI;
176 my ($template,$borrowernumber,$cookie);
178 # decide which template to use
179 my $template_name;
180 my $template_type;
181 my @params = $cgi->param("limit");
182 if ((@params>=1) || ($cgi->param("q")) || ($cgi->param('multibranchlimit')) || ($cgi->param('limit-yr')) ) {
183 $template_name = 'catalogue/results.tmpl';
185 else {
186 $template_name = 'catalogue/advsearch.tmpl';
187 $template_type = 'advsearch';
189 # load the template
190 ($template, $borrowernumber, $cookie) = get_template_and_user({
191 template_name => $template_name,
192 query => $cgi,
193 type => "intranet",
194 authnotrequired => 0,
195 flagsrequired => { catalogue => 1 },
198 if (C4::Context->preference("marcflavour") eq "UNIMARC" ) {
199 $template->param('UNIMARC' => 1);
202 =head1 BUGS and FIXMEs
204 There are many, most are documented in the code. The one that
205 isn't fully documented, but referred to is the need for a full
206 query parser.
208 =cut
210 ## URI Re-Writing
211 # Deprecated, but preserved because it's interesting :-)
212 #my $rewrite_flag;
213 #my $uri = $cgi->url(-base => 1);
214 #my $relative_url = $cgi->url(-relative=>1);
215 #$uri.="/".$relative_url."?";
216 #warn "URI:$uri";
217 #my @cgi_params_list = $cgi->param();
218 #my $url_params = $cgi->Vars;
220 #for my $each_param_set (@cgi_params_list) {
221 # $uri.= join "", map "\&$each_param_set=".$_, split("\0",$url_params->{$each_param_set}) if $url_params->{$each_param_set};
223 #warn "New URI:$uri";
224 # Only re-write a URI if there are params or if it already hasn't been re-written
225 #unless (($cgi->param('r')) || (!$cgi->param()) ) {
226 # print $cgi->redirect( -uri=>$uri."&r=1",
227 # -cookie => $cookie);
228 # exit;
231 # load the branches
232 my $branches = GetBranches();
233 my @branch_loop;
234 #push @branch_loop, {value => "", branchname => "All Branches", };
235 for my $branch_hash (sort keys %$branches) {
236 push @branch_loop, {value => "$branch_hash" , branchname => $branches->{$branch_hash}->{'branchname'}, };
239 my $categories = GetBranchCategories(undef,'searchdomain');
241 $template->param(branchloop => \@branch_loop, searchdomainloop => $categories);
243 # load the itemtypes
244 my $itemtypes = GetItemTypes;
245 my @itemtypesloop;
246 my $selected=1;
247 my $cnt;
248 my $imgdir = getitemtypeimagesrc();
249 foreach my $thisitemtype ( sort {$itemtypes->{$a}->{'description'} cmp $itemtypes->{$b}->{'description'} } keys %$itemtypes ) {
250 my %row =( number=>$cnt++,
251 imageurl=> $itemtypes->{$thisitemtype}->{'imageurl'}?($imgdir."/".$itemtypes->{$thisitemtype}->{'imageurl'}):"",
252 code => $thisitemtype,
253 selected => $selected,
254 description => $itemtypes->{$thisitemtype}->{'description'},
255 count5 => $cnt % 4,
257 $selected = 0 if ($selected) ;
258 push @itemtypesloop, \%row;
260 $template->param(itemtypeloop => \@itemtypesloop);
262 # # load the itypes (Called item types in the template -- just authorized values for searching)
263 # my ($itypecount,@itype_loop) = GetCcodes();
264 # $template->param(itypeloop=>\@itype_loop,);
266 # load the languages ( for switching from one template to another )
267 $template->param(languages_loop => getTranslatedLanguages('intranet','prog'));
269 # The following should only be loaded if we're bringing up the advanced search template
270 if ( $template_type eq 'advsearch' ) {
271 # load the servers (used for searching -- to do federated searching, etc.)
272 my $primary_servers_loop;# = displayPrimaryServers();
273 $template->param(outer_servers_loop => $primary_servers_loop,);
275 my $secondary_servers_loop;# = displaySecondaryServers();
276 $template->param(outer_sup_servers_loop => $secondary_servers_loop,);
278 # determine what to display next to the search boxes (ie, boolean option
279 # shouldn't appear on the first one, scan indexes should, adding a new
280 # box should only appear on the last, etc.
281 my @search_boxes_array;
282 my $search_boxes_count = C4::Context->preference("OPACAdvSearchInputCount") | 3; # FIXME: should be a syspref
283 for (my $i=1;$i<=$search_boxes_count;$i++) {
284 # if it's the first one, don't display boolean option, but show scan indexes
285 if ($i==1) {
286 push @search_boxes_array,
288 scan_index => 1,
292 # if it's the last one, show the 'add field' box
293 elsif ($i==$search_boxes_count) {
294 push @search_boxes_array,
296 boolean => 1,
297 add_field => 1,
300 else {
301 push @search_boxes_array,
303 boolean => 1,
308 $template->param(uc(C4::Context->preference("marcflavour")) => 1,
309 search_boxes_loop => \@search_boxes_array);
311 # load the language limits (for search)
312 my $languages_limit_loop = getAllLanguages();
313 $template->param(search_languages_loop => $languages_limit_loop,);
315 # use the global setting by default
316 if ( C4::Context->preference("expandedSearchOption") == 1) {
317 $template->param( expanded_options => C4::Context->preference("expandedSearchOption") );
319 # but let the user override it
320 if ( ($cgi->param('expanded_options') == 0) || ($cgi->param('expanded_options') == 1 ) ) {
321 $template->param( expanded_options => $cgi->param('expanded_options'));
324 output_html_with_http_headers $cgi, $cookie, $template->output;
325 exit;
328 ### OK, if we're this far, we're performing an actual search
330 # Fetch the paramater list as a hash in scalar context:
331 # * returns paramater list as tied hash ref
332 # * we can edit the values by changing the key
333 # * multivalued CGI paramaters are returned as a packaged string separated by "\0" (null)
334 my $params = $cgi->Vars;
336 # Params that can have more than one value
337 # sort by is used to sort the query
338 # in theory can have more than one but generally there's just one
339 my @sort_by;
340 my $default_sort_by = C4::Context->preference('defaultSortField')."_".C4::Context->preference('defaultSortOrder')
341 if (C4::Context->preference('defaultSortField') && C4::Context->preference('defaultSortOrder'));
343 @sort_by = split("\0",$params->{'sort_by'}) if $params->{'sort_by'};
344 $sort_by[0] = $default_sort_by unless $sort_by[0];
345 foreach my $sort (@sort_by) {
346 $template->param($sort => 1);
348 $template->param('sort_by' => $sort_by[0]);
350 # Use the servers defined, or just search our local catalog(default)
351 my @servers;
352 @servers = split("\0",$params->{'server'}) if $params->{'server'};
353 unless (@servers) {
354 #FIXME: this should be handled using Context.pm
355 @servers = ("biblioserver");
356 # @servers = C4::Context->config("biblioserver");
359 # operators include boolean and proximity operators and are used
360 # to evaluate multiple operands
361 my @operators;
362 @operators = split("\0",$params->{'op'}) if $params->{'op'};
364 # indexes are query qualifiers, like 'title', 'author', etc. They
365 # can be single or multiple parameters separated by comma: kw,right-Truncation
366 my @indexes;
367 @indexes = split("\0",$params->{'idx'});
369 # an operand can be a single term, a phrase, or a complete ccl query
370 my @operands;
371 @operands = split("\0",$params->{'q'}) if $params->{'q'};
373 # limits are use to limit to results to a pre-defined category such as branch or language
374 my @limits;
375 @limits = split("\0",$params->{'limit'}) if $params->{'limit'};
377 if($params->{'multibranchlimit'}) {
378 push @limits, join(" or ", map { "branch: $_ "} @{GetBranchesInCategory($params->{'multibranchlimit'})}) ;
381 my $available;
382 foreach my $limit(@limits) {
383 if ($limit =~/available/) {
384 $available = 1;
387 $template->param(available => $available);
389 # append year limits if they exist
390 push @limits, map "yr:".$_, split("\0",$params->{'limit-yr'}) if $params->{'limit-yr'};
392 # Params that can only have one value
393 my $scan = $params->{'scan'};
394 my $count = C4::Context->preference('numSearchResults') || 20;
395 my $results_per_page = $params->{'count'} || $count;
396 my $offset = $params->{'offset'} || 0;
397 my $page = $cgi->param('page') || 1;
398 #my $offset = ($page-1)*$results_per_page;
399 my $hits;
400 my $expanded_facet = $params->{'expand'};
402 # Define some global variables
403 my ( $error,$query,$simple_query,$query_cgi,$query_desc,$limit,$limit_cgi,$limit_desc,$stopwords_removed,$query_type);
405 my @results;
407 ## I. BUILD THE QUERY
408 ( $error,$query,$simple_query,$query_cgi,$query_desc,$limit,$limit_cgi,$limit_desc,$stopwords_removed,$query_type) = buildQuery(\@operators,\@operands,\@indexes,\@limits,\@sort_by,$scan);
410 ## parse the query_cgi string and put it into a form suitable for <input>s
411 my @query_inputs;
412 for my $this_cgi ( split('&',$query_cgi) ) {
413 next unless $this_cgi;
414 $this_cgi =~ m/(.*=)(.*)/;
415 my $input_name = $1;
416 my $input_value = $2;
417 $input_name =~ s/=$//;
418 push @query_inputs, { input_name => $input_name, input_value => $input_value };
420 $template->param ( QUERY_INPUTS => \@query_inputs );
422 ## parse the limit_cgi string and put it into a form suitable for <input>s
423 my @limit_inputs;
424 for my $this_cgi ( split('&',$limit_cgi) ) {
425 next unless $this_cgi;
426 $this_cgi =~ m/(.*=)(.*)/;
427 my $input_name = $1;
428 my $input_value = $2;
429 $input_name =~ s/=$//;
430 push @limit_inputs, { input_name => $input_name, input_value => $input_value };
432 $template->param ( LIMIT_INPUTS => \@limit_inputs );
434 ## II. DO THE SEARCH AND GET THE RESULTS
435 my $total; # the total results for the whole set
436 my $facets; # this object stores the faceted results that display on the left-hand of the results page
437 my @results_array;
438 my $results_hashref;
440 if (C4::Context->preference('NoZebra')) {
441 $query=~s/yr(:|=)\s*([\d]{1,4})-([\d]{1,4})/(yr>=$2 and yr<=$3)/g;
442 $simple_query=~s/yr\s*(:|=)([\d]{1,4})-([\d]{1,4})/(yr>=$2 and yr<=$3)/g;
443 warn $query;
444 eval {
445 ($error, $results_hashref, $facets) = NZgetRecords($query,$simple_query,\@sort_by,\@servers,$results_per_page,$offset,$expanded_facet,$branches,$query_type,$scan);
447 } else {
448 eval {
449 ($error, $results_hashref, $facets) = getRecords($query,$simple_query,\@sort_by,\@servers,$results_per_page,$offset,$expanded_facet,$branches,$query_type,$scan);
452 if ($@ || $error) {
453 $template->param(query_error => $error.$@);
455 output_html_with_http_headers $cgi, $cookie, $template->output;
456 exit;
459 # FIXME: This belongs in tools/ not in the primary search results page
460 my $op=$cgi->param("operation");
461 if ($op eq "bulkedit"){
462 my ($countchanged,$listunchanged)=
463 ModBiblios($results_hashref->{'biblioserver'}->{"RECORDS"},
464 $params->{"tagsubfield"},
465 $params->{"inputvalue"},
466 $params->{"targetvalue"},
467 $params->{"test"}
469 $template->param(bulkeditresults=>1,
470 tagsubfield=>$params->{"tagsubfield"},
471 inputvalue=>$params->{"inputvalue"},
472 targetvalue=>$params->{"targetvalue"},
473 countchanged=>$countchanged,
474 countunchanged=>scalar(@$listunchanged),
475 listunchanged=>$listunchanged);
477 if (C4::Context->userenv->{'flags'}==1 ||(C4::Context->userenv->{'flags'} & ( 2**9 ) )){
478 #Edit Catalogue Permissions
479 $template->param(bulkedit => 1);
480 $template->param(tagsubfields=>GetManagedTagSubfields());
483 # At this point, each server has given us a result set
484 # now we build that set for template display
485 my @sup_results_array;
486 for (my $i=0;$i<=@servers;$i++) {
487 my $server = $servers[$i];
488 if ($server =~/biblioserver/) { # this is the local bibliographic server
489 $hits = $results_hashref->{$server}->{"hits"};
490 my $page = $cgi->param('page') || 0;
491 my @newresults = searchResults( $query_desc,$hits,$results_per_page,$offset,@{$results_hashref->{$server}->{"RECORDS"}});
492 $total = $total + $results_hashref->{$server}->{"hits"};
493 if ($hits) {
494 $template->param(total => $hits);
495 $template->param(limit_cgi => $limit_cgi);
496 $template->param(query_cgi => $query_cgi);
497 $template->param(query_desc => $query_desc);
498 $template->param(limit_desc => $limit_desc);
499 if ($query_desc || $limit_desc) {
500 $template->param(searchdesc => 1);
502 $template->param(stopwords_removed => "@$stopwords_removed") if $stopwords_removed;
503 $template->param(results_per_page => $results_per_page);
504 $template->param(SEARCH_RESULTS => \@newresults);
505 ## Build the page numbers on the bottom of the page
506 my @page_numbers;
507 # total number of pages there will be
508 my $pages = ceil($hits / $results_per_page);
509 # default page number
510 my $current_page_number = 1;
511 $current_page_number = ($offset / $results_per_page + 1) if $offset;
512 my $previous_page_offset = $offset - $results_per_page unless ($offset - $results_per_page <0);
513 my $next_page_offset = $offset + $results_per_page;
514 # If we're within the first 10 pages, keep it simple
515 #warn "current page:".$current_page_number;
516 if ($current_page_number < 10) {
517 # just show the first 10 pages
518 # Loop through the pages
519 my $pages_to_show = 10;
520 $pages_to_show = $pages if $pages<10;
521 for ($i=1; $i<=$pages_to_show;$i++) {
522 # the offset for this page
523 my $this_offset = (($i*$results_per_page)-$results_per_page);
524 # the page number for this page
525 my $this_page_number = $i;
526 # it should only be highlighted if it's the current page
527 my $highlight = 1 if ($this_page_number == $current_page_number);
528 # put it in the array
529 push @page_numbers, { offset => $this_offset, pg => $this_page_number, highlight => $highlight, sort_by => join " ",@sort_by };
534 # now, show twenty pages, with the current one smack in the middle
535 else {
536 for ($i=$current_page_number; $i<=($current_page_number + 20 );$i++) {
537 my $this_offset = ((($i-9)*$results_per_page)-$results_per_page);
538 my $this_page_number = $i-9;
539 my $highlight = 1 if ($this_page_number == $current_page_number);
540 if ($this_page_number <= $pages) {
541 push @page_numbers, { offset => $this_offset, pg => $this_page_number, highlight => $highlight, sort_by => join " ",@sort_by };
546 $template->param( PAGE_NUMBERS => \@page_numbers,
547 previous_page_offset => $previous_page_offset) unless $pages < 2;
548 $template->param(next_page_offset => $next_page_offset) unless $pages eq $current_page_number;
550 } # end of the if local
551 else {
552 # check if it's a z3950 or opensearch source
553 my $zed3950 = 0; # FIXME :: Hardcoded value.
554 if ($zed3950) {
555 my @inner_sup_results_array;
556 for my $sup_record ( @{$results_hashref->{$server}->{"RECORDS"}} ) {
557 my $marc_record_object = MARC::Record->new_from_usmarc($sup_record);
558 my $control_number = $marc_record_object->field('010')->subfield('a') if $marc_record_object->field('010');
559 $control_number =~ s/^ //g;
560 my $link = "http://catalog.loc.gov/cgi-bin/Pwebrecon.cgi?SAB1=".$control_number."&BOOL1=all+of+these&FLD1=LC+Control+Number+LCCN+%28K010%29+%28K010%29&GRP1=AND+with+next+set&SAB2=&BOOL2=all+of+these&FLD2=Keyword+Anywhere+%28GKEY%29+%28GKEY%29&PID=6211&SEQ=20060816121838&CNT=25&HIST=1";
561 my $title = $marc_record_object->title();
562 push @inner_sup_results_array, {
563 'title' => $title,
564 'link' => $link,
567 my $servername = $server;
568 push @sup_results_array, { servername => $servername, inner_sup_results_loop => \@inner_sup_results_array};
569 $template->param(outer_sup_results_loop => \@sup_results_array);
573 } #/end of the for loop
574 #$template->param(FEDERATED_RESULTS => \@results_array);
577 $template->param(
578 #classlist => $classlist,
579 total => $total,
580 opacfacets => 1,
581 facets_loop => $facets,
582 scan => $scan,
583 search_error => $error,
586 if ($query_desc || $limit_desc) {
587 $template->param(searchdesc => 1);
590 ## Now let's find out if we have any supplemental data to show the user
591 # and in the meantime, save the current query for statistical purposes, etc.
592 my $koha_spsuggest; # a flag to tell if we've got suggestions coming from Koha
593 my @koha_spsuggest; # place we store the suggestions to be returned to the template as LOOP
594 my $phrases = $query_desc;
595 my $ipaddress;
597 if ( C4::Context->preference("kohaspsuggest") ) {
598 my ($suggest_host, $suggest_dbname, $suggest_user, $suggest_pwd) = split(':', C4::Context->preference("kohaspsuggest"));
599 eval {
600 my $koha_spsuggest_dbh;
601 # FIXME: this needs to be moved to Context.pm
602 eval {
603 $koha_spsuggest_dbh=DBI->connect("DBI:mysql:$suggest_dbname:$suggest_host","$suggest_user","$suggest_pwd");
605 if ($@) {
606 warn "can't connect to spsuggest db";
608 else {
609 my $koha_spsuggest_insert = "INSERT INTO phrase_log(phr_phrase,phr_resultcount,phr_ip) VALUES(?,?,?)";
610 my $koha_spsuggest_query = "SELECT display FROM distincts WHERE strcmp(soundex(suggestion), soundex(?)) = 0 order by soundex(suggestion) limit 0,5";
611 my $koha_spsuggest_sth = $koha_spsuggest_dbh->prepare($koha_spsuggest_query);
612 $koha_spsuggest_sth->execute($phrases);
613 while (my $spsuggestion = $koha_spsuggest_sth->fetchrow_array) {
614 $spsuggestion =~ s/(:|\/)//g;
615 my %line;
616 $line{spsuggestion} = $spsuggestion;
617 push @koha_spsuggest,\%line;
618 $koha_spsuggest = 1;
621 # Now save the current query
622 $koha_spsuggest_sth=$koha_spsuggest_dbh->prepare($koha_spsuggest_insert);
623 #$koha_spsuggest_sth->execute($phrases,$results_per_page,$ipaddress);
624 $koha_spsuggest_sth->finish;
626 $template->param( koha_spsuggest => $koha_spsuggest ) unless $hits;
627 $template->param( SPELL_SUGGEST => \@koha_spsuggest,
631 if ($@) {
632 warn "Kohaspsuggest failure:".$@;
636 # VI. BUILD THE TEMPLATE
637 output_html_with_http_headers $cgi, $cookie, $template->output;