1 Instructions for hacking on Xapian
2 ==================================
4 .. contents:: Table of contents
6 This file is aimed to help developers get started with working on
7 Xapian. The documentation contains a section covering various internal
8 aspects of the library - this can also be found on the Xapian website
11 Extra options to give to configure
12 ==================================
14 Note: Non-developer configure options are described in INSTALL
16 You will probably want to use some of these if you're going to be developing
20 This enables compiling of assertion code which will throw
21 Xapian::AssertionError if the code detects violating of
22 preconditions, postconditions, or fails other consistency checks.
24 --enable-assertions=partial
25 This option enables a subset of the assertions enabled by
26 "--enable-assertions", but not the most expensive. The intention is
27 that it should be suitable for use in a real-world system for tracking
28 down problems without imposing too much of an overhead (but note that
29 we haven't yet performed timings to measure the overhead...)
32 This enables compiling code into the library which generates verbose
33 debugging messages. See "Debugging Messages", below.
36 In 1.2.0 and earlier, this used to use the debug logging macros to
37 report to stderr how long each method takes to execute. This feature
38 was removed in 1.2.1 - you are likely to get better results using
39 dedicated profiling tools - for more information see:
40 https://trac.xapian.org/wiki/ProfilingXapian
42 --enable-maintainer-mode
43 This tells configure to enable make dependencies for regenerating build
44 system files (such as configure, Makefile.in, and Makefile) and other
45 generated files (such as the stemmers and query parser) when required.
46 These are disabled by default as some make programs try to rebuild them
47 when it's not appropriate (e.g. BSD make doesn't handle VPATH except
48 for implicit rules). For this reason, we recommend GNU make if you
49 enable maintainer mode. You'll also need a non-cross-compiling C
50 compiler for compiling the Lemon parser generator and the Snowball
51 stemming algorithm compiler. The configure script will attempt to
52 locate one, but you can override this autodetection by passing
53 CC_FOR_BUILD on the command line like so::
55 ./configure CC_FOR_BUILD=/opt/bin/gcc
57 --enable-documentation
58 This tells configure to enable make dependencies for regenerating
59 documentation files. By default it uses the same setting as
60 --enable-maintainer-mode.
65 If you configure with --enable-log, lots of places in the code generate
66 debugging messages to tell us what they're up to - this information can be
67 very useful for debugging both the Xapian library and code which uses it. But
68 the quantity of information generated is potentially vast so there's a
69 mechanism to allow you to select where to store the log and which types of
70 message you're interested by setting environment variables. You can:
72 * set XAPIAN_DEBUG_LOG to be the path to a file that you would like debugging
73 output to be appended to, or to the special value ``-`` to indicate that you
74 would like debugging output to be sent to stderr. Unless XAPIAN_DEBUG_LOG
75 is set, no debug logging will be performed. Occurrences of %p in
76 XAPIAN_DEBUG_LOG will be replaced with the current process-id.
78 * set XAPIAN_DEBUG_FLAGS to a string of capital letters indicating the types
79 of debugging message you would like to display (the default is to log calls
80 to API functions and methods). These letters are shown in the first column
81 of the log output, and are also listed in ``common/debuglog.h``. If the
82 first character is ``-``, then the letters indicate those categories of
83 message *not* be shown instead. As a consequence of this, setting
84 ``XAPIAN_DEBUG_FLAGS=-`` will give you all debugging messages.
86 These environment variables only have any effect if you ran configure with the
91 <message type> <pid> [<this>] <message>
95 A 16747 [0x57ad1e0] void Xapian::Query::Internal::validate_query()
97 Each nested call adds another space before the ``[`` so you can easily see
98 which function call and return messages correspond.
100 Debugging memory allocations
101 ============================
103 The testsuite can make use of valgrind 3.3.0 or newer to check for memory
104 leaks, reads from uninitialised memory, and some other bugs during tests.
106 Valgrind doesn't support every platform, but Xapian contains very little
107 platform specific code (and most of what there is is Microsoft Windows
108 specific) so even just testing with valgrind on one platform gives good
111 If you have a new enough version of valgrind installed, it's automatically
112 detected by configure and used when running the testsuite. The testsuite runs
113 more slowly under valgrind, so if you wish to disable this auto-detection you
114 can run configure with:
116 ./configure VALGRIND=
118 Or you can disable use of valgrind during a particular run of "make check"
123 Or disable it while running a test directly (under sh or bash):
125 VALGRIND= ./runtest ./apitest
127 Running test programs
128 =====================
130 To run all tests, use ``make check``. You can also run just the subset of
131 tests which exercise the inmemory, remote progserver, remote TCP,
132 multi-database, glass or honey backends using ``make check-inmemory``,
133 ``make check-remoteprog``, ``make check-remotetcp``, ``make check-multi``,
134 ``make check-glass`` or ``make check-honey`` respectively.
136 Also, ``make check-remote`` will run the tests on both variants of the remote
137 backend, and ``make check-none`` will run those tests which don't use any
138 backend. These are handy shortcuts when doing development work on a particular
141 The runtest script (in the tests subdirectory) takes care of the details of
142 running the test programs (including setting up the environment so they work
143 when srcdir != builddir and handling libtool dynamically linked binaries). To
144 run a test program by hand (rather than via make) just use:
148 You can specify options and arguments. Individual test programs optionally
149 take one or more test names as arguments, and you can also pass ``-v`` to get
150 more verbose output from failing tests, e.g.:
152 ./runtest ./apitest -v deldoc1
154 If the number of the test is omitted, all tests with that basename are run,
155 so to run deldoc1, deldoc2, etc:
157 ./runtest ./apitest deldoc
159 You can also use runtest to run a test program under gdb (or most other tools):
161 ./runtest gdb ./apitest -v deldoc1
162 ./runtest valgrind ./apitest -v deldoc1
164 Some test programs take special arguments - for example, you can restrict
165 apitest to the glass backend using ``-bglass``.
167 There are a few environment variables which the testsuite harness checks for
168 which you might find useful:
170 XAPIAN_TESTSUITE_SIG_DFL:
171 By default, the testsuite harness catches signals and handles them
172 gracefully - the current test is failed, and the testsuite moves onto the
173 next test. If you want to suppress this (some debugging tools may work
174 better if the signal is not caught) set the environment variable
175 XAPIAN_TESTSUITE_SIG_DFL to any value to prevent the testsuite harness
176 from installing its own signal handling.
178 XAPIAN_TESTSUITE_OUTPUT:
179 By default, the testsuite harness uses ANSI escape sequences to give
180 colour output if stdout is a tty. You can disable this feature by setting
181 XAPIAN_TESTSUITE_OUTPUT=plain (alternatively, piping the output (e.g.
182 through ``cat`` or ``more``) will have the same effect). Auto-detection
183 can be explicitly specified with XAPIAN_TESTSUITE_OUTPUT=auto (or empty).
184 Any other value forces the use of colour. Colour output is always disabled
185 on Microsoft Windows, so XAPIAN_TESTSUITE_OUTPUT has no effect there.
187 XAPIAN_TESTSUITE_LD_PRELOAD:
188 The runtest script will add this to LD_PRELOAD if it is set, allowing you
189 to easily load LD_PRELOAD libraries when running the testsuite. The
190 original intended use was to allow use of libeatmydata
191 (https://www.flamingspork.com/projects/libeatmydata/) which makes fsync
192 and related calls no-ops, but configure now checks for the eatmydata
193 wrapper script and this is used automatically. However, there may be
194 other LD_PRELOAD libraries which are useful, so we've left the machinery
197 Speeding up the testsuite with eatmydata
198 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
200 The testsuite does a lot of small database operations, and the calls to fsync,
201 fdatasync, etc which Xapian makes by default can slow down testsuite runs
202 substantially. There's a handy LD_PRELOAD library called eatmydata
203 (https://www.flamingspork.com/projects/libeatmydata/), which can help here, by
204 turning fsync and related calls into no-ops.
206 You need a version of eatmydata with the eatmydata wrapper script (version 37
207 or newer), and then configure should auto-detect it and it'll get used when
208 running the testsuite (via runtest). If you wish to disable this
209 auto-detection for some reason, you can run configure with:
211 ./configure EATMYDATA=
213 Or you can disable use of eatmydata during a particular run of "make check"
216 make check EATMYDATA=
218 Or disable it while running a test directly (under sh or bash):
220 EATMYDATA= ./runtest ./apitest
222 Using various debugging, profiling, and leak-finding tools
223 ==========================================================
225 GCC's libstdc++ supports a debug mode, which checks for various misuses of
226 the STL - to enable this, define _GLIBCXX_DEBUG when building Xapian:
228 ./configure CPPFLAGS=-D_GLIBCXX_DEBUG
230 For documentation of this option, see:
231 https://gcc.gnu.org/onlinedocs/libstdc++/manual/debug_mode.html
233 Note: all C++ code must be compiled with this defined or you'll get problems.
234 Xapian's API headers include a check that the same setting is used when
235 building code using Xapian as was used to build Xapian.
237 To use valgrind (http://www.valgrind.org/), no special build options are
238 required, but make sure you compile with debugging information (on by default
239 for GCC) and the valgrind documentation recommends disabling optimisation (with
240 optimisation, line numbers in error messages can be confusing due to code
243 ./configure CXXFLAGS='-O0 -g'
245 To use gdb (https://www.gnu.org/software/gdb/), no special build options are
246 required, but make sure you compile with debugging information (on by default
247 for GCC). You'll probably find debugging easier if you compile without
248 optimisation (with optimisation, line numbers in error messages can be
249 confusing due to code inlining, etc, and the values of some variables can't be
250 printed because they've been eliminated from the code completely):
252 ./configure CXXFLAGS='-O0 -g'
254 To enable profiling for gprof:
256 ./configure CXXFLAGS=-pg LDFLAGS=-pg
258 To use Purify (a proprietary tool):
260 ./configure CXXLD='purify c++' --disable-shared
262 To use Insure (another proprietary tool):
264 ./configure CXX=insure
266 To use lcov (at least version 1.10) to generate a test coverage report (see
267 `lcov.xapian.org <http://lcov.xapian.org/>`_ for reports) there are three make
268 targets (all in the `xapian-core` directory):
270 * `make coverage-reconfigure`: reruns configure in the source tree. See
271 Makefile.am for details of the configure options used and why they
272 are needed. If you're using ccache, make sure it's at least version
273 3.0, and ideally at least 3.2.2.
275 * `make coverage-reconfigure-maintainer-mode`: does the same thing, except
276 the tree is configured in "maintainer mode", which is what you want if
277 generating coverage reports while working on the code.
279 * `make coverage-check`: runs `make check` and generates an HTML report in a
280 directory called `lcov`.
282 + You can specify extra arguments to pass to the ``genhtml`` tool using
283 `GENHTML_ARGS`, so for example if you plan to serve the generated HTML
284 coverage report from a webserver, you might use:
285 `make coverage-check GENHTML_ARGS=--html-gzip`
287 You ideally want lcov 1.11 or later, since 1.11 includes patches to reduce
288 memory usage significantly - lcov 1.10 would run out of memory in a 1GB VM.
290 If you have runes for using other tools, please add them above, or send them
296 If you want to try unreleased Xapian code, you can fetch it from our git
297 repository. For convenience, we also provide bootstrapped tarballs (much like
298 the sourcecode download for any release version) which get built every 20
299 minutes if there have been any changes checked in. These tarballs need to
300 pass "make distcheck" to be automatically uploaded, so using them will help
301 to assure that you don't pick a "bad" version. The snapshots are available
302 from the "Bleeding Edge" page of the Xapian website.
307 When building from a git checkout, we *strongly* recommend that you use
308 the ``bootstrap`` script in the top level directory to set up the tree ready
309 for building. This script will check which directories you have checked out,
310 so you can bootstrap a partial tree. You can also ``touch .nobootstrap`` in
311 a subdirectory to tell bootstrap to ignore it.
313 You will need the following tools installed to build from git:
315 * GNU m4 >= 4.6 (for autoconf)
316 * perl >= 5.6 (for automake; also for various maintainer scripts)
317 * python >= 2.3 (for generating the Python bindings)
318 * GNU make (or another make which support VPATH for explicit rules)
319 * GNU bison (for building SWIG, used for generating the bindings)
320 * Tcl (to generate unicode/unicode-data.cc)
322 For a recent version of Debian or Ubuntu, this command should ensure you have
323 all the necessary tools and libraries::
325 apt-get install build-essential m4 perl python zlib1g-dev uuid-dev wget bison tcl
327 If you want to build Omega, you'll also need::
329 apt-get install libpcre3-dev libmagic-dev
331 On Fedora, the uuid library can be installed by doing::
333 yum install libuuid-devel
335 On Mac OS X, if you're using macports you'll want the following:
337 * file (magic.h in configure)
339 If you're using homebrew you'll want the following::
341 brew install libmagic pcre
343 If you're doing much development work, you'll probably also want the following
346 * valgrind for better testsuite error finding
347 * ccache for faster rebuilds
348 * eatmydata for faster testsuite runs
350 The repository does not contain any automatically generated files
351 (such as configure, Makefile.in, Snowball-generated stemmers, Lemon-generated
352 parsers, SWIG-generated code, etc) because experience shows it's best to keep
353 these out of version control. To avoid requiring you to install the correct
354 versions of the tools required, we either include the source to these tools in
355 the repo directly (in the case of Snowball and Lemon), or the bootstrap script
356 will download them as tarballs (autoconf, automake, libtool) or
357 from git (SWIG), build them, and install them within the source tree.
359 To download source tarballs, bootstrap will use wget, curl or lwp-request if
360 installed. If not, it will give an error telling you the URL to download from
361 by hand and where to copy the file to.
363 Bootstrap will then run autoreconf on each of the checked-out subdirectories,
364 and generate a top-level configure script. This configure script allows you to
365 configure xapian-core and any other modules you've checked out with single
366 simple command, such that the other modules link against the uninstalled
367 xapian-core (which is very handy for development work and a bit fiddly to set
368 up by hand). It automatically passes --enable-maintainer-mode to the
369 subprojects so that the autotools will be rerun if configure.ac, Makefile.am,
372 The bootstrap script doesn't care what the current directory is. The top-level
373 configure script generated by it supports building in a separate directory to
374 the sources: simply create the directory you want to build in, and then run the
375 configure script from inside that directory. For example, to build in a
376 directory called "build" (starting in the top level source directory)::
383 When running bootstrap, if you need to add any extra macro directories to the
384 path searched by aclocal (which is part of automake), you can do this by
385 specifying these in the ACLOCAL_FLAGS environment variable, e.g.::
387 ACLOCAL_FLAGS=-I/extra/macro/directory ./bootstrap
389 If you wish to prevent bootstrap from downloading and building the autotools
390 pass the --without-autotools option. You can force it to delete the downloaded
391 and installed versions by passing --clean.
393 If you are tracking development in git, there will sometimes be changes
394 to the build system sources which require regeneration of the generated
395 makefiles and associated machinery. We aim to make the build system
396 automatically regenerate the necessary files, but in the event that a build
397 fails after an update, it may be worth re-running the bootstrap script to
398 regenerate the build system from scratch, before looking for the cause of the
401 Tools required to build documentation
402 -------------------------------------
404 If you want to be able to build distribution tarballs (with "make dist") then
405 you'll also need some further tools. If you don't want to have to install all
406 these tools, then pass --disable-documentation to configure to disable these
407 rules (the default state of this follows the setting of
408 --enable-maintainer-mode, so in a non-maintainer-mode tree, you can pass
409 --enable-documentation to enable these rules). Without the documentation,
410 "make dist" will fail (to prevent accidentally distributing tarballs without
411 documentation), but you can configure and build.
413 The documentation tools are:
415 * doxygen (v1.8.8 is used for 1.3.x snapshots and releases; 1.7.6.1 fails to
416 process trunk after PL2Weight was added).
417 * dot (part of Graphviz. Doxygen's DOT_MULTI_TARGETS option apparently needs
420 * rst2html or rst2html.py (in python-docutils on Debian/Ubuntu)
421 * pngcrush (optional - used to reduce the size of PNG files in the HTML
423 * sphinx-doc (in python-sphinx and python3-sphinx on Debian/Ubuntu, or as
424 sphinx via pip install)
426 For a recent version of Debian or Ubuntu, this command should install all the
427 required documentation tools::
429 apt-get install doxygen graphviz help2man python-docutils pngcrush python-sphinx python3-sphinx
431 Documentation builds on OS X
432 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
434 On Mac OS X, if you're using homebrew, you'll want the following::
436 brew install doxygen help2man graphviz pngcrush
438 (Ensure you're up to date with brew, as earlier packaging of graphviz
439 didn't properly install dot.)
441 You also need sphinx and docutils, which are python packages; you can
442 install them via pip::
444 pip install sphinx docutils
446 You may find it easier to use homebrew to install python first, so
447 these packages are separate from the system python::
451 If you install both python (v2) and python3 (v3) via homebrew, you
452 will be able to build bindings for both; you'll then need to install
460 As of 1.3.2, we no longer build PDF versions of the API docs by default, but
461 you can build them yourself with::
463 make -C docs apidoc.pdf
465 Additional tools are needed for these:
467 * gs (part of Ghostscript)
468 * pdflatex (in texlive-latex-base on Debian/Ubuntu)
469 * epstopdf (in texlive-extra-utils on Debian/Ubuntu)
470 * makeindex (in texlive-binaries on Debian/Ubuntu, or texlive-base-bin for older releases)
472 Note that pdflatex, epstopdf, gs, and makeindex must all currently be on your
473 path (as specified by the environment variable PATH), since doxygen will look
476 For a recent version of Debian or Ubuntu, this command should install these
479 apt-get install ghostscript texlive-latex-base texlive-extra-utils texlive-binaries texlive-fonts-extra texlive-fonts-recommended texlive-latex-extra texlive-latex-recommended
481 On Mac OS X, if you're using macports you'll want the following:
483 * texlive (pdflatex during build)
484 * texlive-basic (for makeindex in configure)
485 * texlive-latex-extra (latex style)
487 Alternatively, you can install MacTeX from https://www.tug.org/mactex/ instead
488 of texlive, texlive-basic and texlive-latex-extra.
490 The homebrew texlive package only supports 32 bit systems, so even if you're
491 using homebrew, you'll probably want to install MacTeX from
492 https://www.tug.org/mactex/ instead.
497 * autoconf 2.69 is used to generate snapshots and releases.
499 autoconf 2.64 is a hard minimum requirement.
501 autoconf 2.60 is required for docdir support and AC_TYPE_SSIZE_T.
503 autoconf 2.62 generates faster configure scripts and warns about unrecognised
504 options passed to configure.
506 autoconf 2.63 fixes a regression in AC_C_BIGENDIAN introduced in 2.62
507 (Omega uses this macro).
509 autoconf 2.64 generates smaller configure scripts by using shell functions.
511 * automake 1.16.1 is used to generate snapshots and releases.
513 automake 1.13 is a hard minimum requirement, needed for
514 `AC_CONFIG_MACRO_DIRS`.
516 * libtool 2.4.6 is used to generate snapshots and releases.
518 libtool 2.2.8 is the current hard minimum requirement.
520 libtool 2.2 is required for us to be able to override link_all_deplibs_CXX
521 and sys_lib_dlsearch_path_spec in configure. It also fixes some
522 long-standing issues and is significantly faster.
524 Please tell us if you find that newer versions of any of these tools work or
527 There is a good GNU autotools tutorial at
528 <http://www.lrde.epita.fr/~adl/autotools.html>.
530 Building from git on Windows with MSVC
531 --------------------------------------
533 Building using MSVC is now supported by the autotools build system. You need
534 to install a set of Unix-like tools first - we recommended MSYS2:
535 https://www.msys2.org/
537 For details of how to specify MSVC to ``configure`` see the "INSTALL" document.
539 When building from git, by default you'll need some extra tools to generate
540 Unicode tables (Tcl) and build documentation (doxygen, help2man, sphinx-doc).
541 We don't currently have detailed advice on how to do this (if you can provide
542 some then please send a patch).
544 You can avoid needing Tcl by copying ``xapian-core/unicode/unicode-data.cc``
545 from another platform or a release which uses the same Unicode version. You
546 can avoid needing most of the documentation tools by running configure with
547 the ``--disable-documentation`` option.
549 Using a Vagrant-driven Ubuntu virtual machine
550 ---------------------------------------------
552 Note: Vagrant support is experimental. Please report bugs in the
553 normal fashion, to https://trac.xapian.org/newticket, or ask for help
554 on the #xapian IRC channel on Freenode.
556 If you have Vagrant (http://www.vagrantup.com/, tested on version
557 1.5.2) and VirtualBox (https://www.virtualbox.org/, tested on version
558 4.3.10) installed, `vagrant up` will make a virtual machine suitable
559 for developing Xapian:
561 * Ubuntu 13.04 with all packages needed to build Xapian and its
564 * eatmydata (to speed up test runs) and valgrind (for debugging
565 memory allocations) both also installed
567 * source code from this checkout in /vagrant; edit it on your host
568 operating system and changes are reflected in the VM. The source
569 tree is bootstrapped automatically (ensuring that the right
570 versions of the build tools are available on the VM)
572 * build tree in /home/vagrant/build, configured to install into
573 /home/vagrant/install, with maintainer mode and documentation
576 Setting up can take a long time, as it downloads a minimal base box
577 and then installs all the required packages; once this is done you
578 don't have to wait so long if you need to reprovision the VM. (Once
579 Ubuntu 14.04 is released the plan is to build our own base box with
580 these packages already installed, which should make the process much
583 `vagrant ssh` will log you into the VM, and you can type `cd build &&
584 make` to build Xapian. `make check` will run the tests.
586 (As noted above, in maintainer mode most changes that require
587 reconfiguration will happen automatically. If you need to do it by
588 hand you can either run the configure command yourself, or you can run
589 `vagrant provision`, which also checks for any system package
592 The VM has a single 64 bit virtual processor, with 384M of memory; it
593 takes about 8G of disk space once up and running.
598 * As of Xapian 1.3.3, a compiler with decent support for C++11 is required to
599 build Xapian. We currently aim to allow users to use a non-C++11 compiler
600 to build code which uses Xapian.
602 There are now several compilers with good C++11 support, but there are a
603 few shortfalls in commonly deployed versions of most of them. Often we can
604 work around this, and we should do where the effort is low compared to the
605 gain (so a compiler version which is widely used is more worth supporting
606 than one which is hardly used by anyone).
608 However, we shouldn't have to jump through hoops to cater for compilers where
609 their authors aren't putting in the effort to keep up with the language
612 Please avoid the following C++11 features for the time being:
614 * ``std::to_string()`` - this is completely missing on current versions of
615 mingw and cygwin - in the library, you can ``#include "str.h"`` and then
616 use the ``str()`` function instead for most cases. This is also usually
617 faster than ``std::to_string()``.
619 * C++ features we currently assume:
621 * We assume <sstream> is available. GCC < 2.95.3 didn't have it but GCC
622 2.95.3 includes a backported version. We aren't aware of any other
623 compilers still in use which lack it.
625 * Non-".h" versions of standard ISO C++ headers (e.g. ``#include <list>``
626 rather than ``#include <list.h>``). We aren't aware of any compiler still
627 in use which lacks these, and GCC 4.3 no longer has the old versions. If
628 there are any, we could add a directory full of forwarding headers to work
631 * Standard header ``<limits>`` (for ``numeric_limits<>``) - for GCC, this was
634 * Standard header ``<streambuf>`` (GCC < 3.0 only has ``<streambuf.h>``).
636 * RTTI (dynamic_cast<>, typeid, etc): Needing to use RTTI features in the
637 library most likely indicates a design flaw, and you should avoid use
638 of these features. Where necessary, you can use a technique similar to
639 Database::as_networkdatabase() to replace dynamic_cast<>.
641 * Exceptions: In hindsight, throwing exceptions in the library seems to have
642 been a poor design decision. GCC on Solaris can't cope with exceptions in
643 shared libraries (though it appears this may have been fixed in more recent
644 versions), and we've also had test failures on other platforms which only
645 occur with shared libraries - possibly with a similar cause. Exceptions can
646 also be a pain to handle elegantly in the bindings. We intend to investigate
647 modifying the library to return error codes internally, and then offering the
648 user the choice of exception throwing or error code returning API methods
649 (with the exception being thrown by an inlined wrapper in the externally
650 visible header files). With this in mind, please don't complicate the
651 internal handling of exceptions...
653 * "using namespace std;" and "using std::XXX;" - it's OK to use these in
654 applications, library code, and internal library headers. But in externally
655 visible headers (such as anything included by "#include <xapian.h>") you MUST
656 use explicit "std::" qualifiers - it's not acceptable to pull anything from
657 namespace std into the namespace of an application which uses Xapian.
659 * Use C++ style casts (static_cast<>, reinterpret_cast<>, and const_cast<>)
660 or constructor-syntax (e.g. ``double(value)``) in preference to C style
661 casts. The syntax of the C++ casts is ugly, but they do make the
662 intent much clearer which is definitely a good thing, and they avoid issues
663 such as casting away const when you only meant to cast the type of a pointer.
665 * std::pair<> with an STL class as one (or both) of the members can produce
666 very long symbols (over 4KB!) after name mangling - long enough to overflow
667 the size limits of some vendor compilers or toolchains (so this can affect
668 GCC if it is using the system ld or as). Even where the compiler works, the
669 symbol bloat in an unstripped build is probably best avoided, so it's
670 preferable to use a simple two member struct instead. The code is probably
671 more readable anyway, and easier to extend if more members are needed later.
673 * We try to avoid putting the full definition of virtual methods in header
674 files. This is because current compilers can't (as far as we know) inline
675 virtual methods, so putting the definition in the header file simply slows
676 down compilation (and, because method definitions often require further
677 header files to be included, this can result in many more files needing
678 recompilation after a change to a header file than is really necessary).
679 Just put the declaration in the header file, and put the definition in a .cc
680 file with the same basename.
682 Include ordering for source files
683 ---------------------------------
685 To help us move towards a consistent ordering of #include lines in source
686 files, please follow the following policy when ordering them:
688 * #include <config.h> should be first, and use <> not "" (as recommended by the
689 autoconf manual). Always include config.h from C/C++ source files, but don't
690 include it from header files - the autoconf manual recommends that it should
691 be included first, so including it from headers is either redundant, or may
692 hide a missing config.h include in the source file the header was included
693 from (better to get an error in this case).
695 * The header corresponding to the source file should be next. This means that
696 compilation of the library ensures that each header with a corresponding
697 source file is "self supporting" (i.e. it implicitly or explicitly includes
698 all of the headers it requires).
700 * External xapian-core headers, alphabetically. When included from other
701 external headers, use <> to reduce problems with finding headers in the
702 user's source tree by mistake. In sources and internal headers, use "" (?) -
703 practically this makes no difference as we have -I for srcdir and builddir,
704 but <> suggests installed header files so "" seems more natural).
706 * Internal headers, alphabetically (using "").
708 * "Safe" versions of library headers (include these first to avoid issues if
709 other library headers include the ones we want to wrap). Use "" and order
712 * Library headers, alphabetically.
714 * Standard C++ headers, alphabetically. Use the modern (no .h suffix) names.
716 C++ Portability Issues
717 ======================
722 The "C++ Super-FAQ" covers many frequently asked C++ questions:
723 https://isocpp.org/faq
725 Header Portability Issues
726 -------------------------
731 Don't directly '#include <fcntl.h>' - instead '#include "safefcntl.h"'.
733 The main reason for this is that when using certain compilers on certain
734 versions of Solaris, fcntl.h does '#define open open64'. Sadly this breaks C++
735 code which has methods called open (as we do). There's a cunning workaround
736 for this problem in common/safefcntl.h.
738 Also, safefcntl.h ensures the O_BINARY is defined (to 0 if not required) so
739 calls to open() and creat() can specify O_BINARY unconditionally for the
740 benefit of platforms which discriminate between text and binary files.
745 Don't directly '#include <windows.h>' - instead '#include "safewindows.h"'
746 which reduces the bloat of header files included and prevents some of the
747 more egregious namespace pollution. It also defines any constants we need
748 which might be missing in older versions of the mingw headers.
753 Don't directly '#include <winsock2.h>' - instead '#include "safewinsock2.h"'.
754 This ensure that safewindows.h is included before <winsock2.h> to avoid
755 winsock2.h including windows.h without our namespace pollution reducing
761 Don't directly '#include <errno.h>' - instead '#include "safeerrno.h"' which
762 works around a problem with Compaq's C++ compiler.
767 Don't directly '#include <sys/select.h>' - instead '#include "safesysselect.h"'
768 which supports older UNIX platforms which predate POSIX 1003.1-2001 and works
769 around a problem on Solaris.
774 Don't directly '#include <sys/socket.h>' - instead '#include "safesyssocket.h"'
775 which supports older UNIX platforms which predate POSIX 1003.1-2001 and works
781 Don't directly '#include <sys/stat.h>' - instead '#include "safesysstat.h"'
782 which under MSVC enables stat to work on files > 2GB, defines the missing
783 POSIX macros S_ISDIR and S_ISREG, pulls in <direct.h> for mkdir() (which is
784 provided by sys/stat.h under UNIX) and provides a compatibility wrapper for
785 mkdir() which takes 2 arguments (so code using mkdir can always just pass
791 To get `WEXITSTATUS` or `WIFEXITED` defined, '#include "safesyswait.h"'.
792 Note that this won't provide `waitpid()`, etc on Microsoft Windows, since
793 these functions are only really useful to use when `fork()` is available.
798 Don't directly '#include <unistd.h>' - instead '#include "safeunistd.h"'
799 - MSVC doesn't even HAVE unistd.h!
801 The various "safe" headers are maintained in xapian-core/common, but also used
802 by Omega. Currently bootstrap sorts out setting up a copy of this subdirectory
803 via a secondary git checkout.
805 Warning-Free Compilation
806 ------------------------
808 Compiling without warnings on every platform is our goal, though it's not
809 always possible to achieve. For example, some GCC 3.x compilers produce the
810 occasional bogus warning (e.g. warning that a variable may be used
811 uninitialised, despite it being initialised at the point of declaration!)
813 You should consider configure-ing with:
815 ./configure CXXFLAGS=-Werror
817 when doing development work on Xapian. This promotes warnings to errors,
818 which should ensure you at least don't introduce new warnings for the compiler
821 If you configure with --enable-maintainer-mode, and are using GCC 4.1 or newer,
822 this is done for you automatically. This is intended to be an aid rather than
823 a form of automated punishment - it's all too easy to miss a new warning as
824 once a file is compiled, you don't see it unless you modify that file or one of
827 With Intel's C++ compiler, --enable-maintainer-mode also enables -Werror.
828 If you know the equivalent of -Werror for other compilers, please add a note
829 here, or tell us so that we can add a note.
831 Miscellaneous Portability Issues
832 --------------------------------
834 Make sure that the last line of any source file ends with a linefeed character
835 since it's undefined behaviour if it doesn't (most compilers accept it, though
836 at least GCC gives a warning).
838 Branch Prediction Hints
839 =======================
841 For compilers which support ``__builtin_expect()`` (GCC >= 3.0 and some others)
842 you can provide manual hints to assist branch prediction. We've wrapped these
843 in macros which evaluate to just their argument for compilers which don't
844 support ``__builtin_expect()__``.
846 Within the xapian-core library code, you can mark the expressions in ``if`` and
847 ``while`` statements as ``rare`` (if the condition is rarely true) or ``usual``
848 (if the condition is usually true).
852 if (rare(something_unusual())) deal_with_it();
854 while (usual(!end_condition()) keep_going();
856 It's easy to make incorrect assumptions about where hotspots are and which
857 branches are usually taken or not, so except for really obvious cases (such
858 as ``if (!consistency_check()) throw_exception();``) you should benchmark
859 that new ``rare`` and ``usual`` hints help rather than hinder before committing
860 them to the repository. It's also likely to be a waste of effort to add them
861 outside of areas of code which are executed very frequently.
863 Don't expect miracles - the first 15 uses added saved approximately 1%.
865 If you know how to implement the ``rare`` and ``usual`` macros for other
866 compilers, please let us know.
871 Especially for a library, compile-time options aren't a good solution for
872 how to integrate a new feature. An increasingly large number of users install
873 pre-built binary packages rather than building from source, and unless the
874 package is capable of being split into modules, the packager has to choose a
875 set of compile-time options to use. And they'll tend to choose either the
876 standard ones, or perhaps a broader set to try to keep everyone happy. For a
877 library, similar issues occur when installing from source as well - the
878 sysadmin must choose the options which will keep all users happy.
880 Another problem with compile-time options is that it's hard to ensure that
881 a change doesn't break compilation under some combination of options without
882 actually building and running the test-suite on all combinations. The fewer
883 compile-time options, the more likely the code will compile with every
886 So please think carefully before adding more compile-time options. They're
887 probably OK for experimental features (but should go away once a feature is no
888 longer experimental). Options to instrument a build for special purposes
889 (debug, profiling, etc) are also acceptable. Disabling whole features probably
890 isn't (e.g. the --disable-backend-XXX options we already have are dubious,
891 though being able to disable the remote backend can be useful when trying to
892 get Xapian going on a platform).
897 We don't want to force those building Xapian from the source distribution to
898 have to use GNU make. Requiring GNU make for "make dist" isn't such a problem
899 but it's probably better to use portable constructs everywhere to avoid
900 problems when people move or copy code between targets. If you do make use
901 of non-portable constructs where it's OK, add a comment noting the special
902 circumstances which justify doing so.
904 Here's an incomplete list of things to avoid:
906 * Don't use "$(RM)" - it's defined by GNU make, but using it actually harms
907 portability as other makes don't define it. Use plain "rm" instead.
909 * Don't use "%" pattern rules - these are GNU make specific. Use an
910 implicit rule (e.g. ".c.o:") if you can. Otherwise, write out each version
913 * Don't use "$<" except in implicit rules. This is an annoying restriction,
914 as using "$<" makes it much easier to make VPATH builds work. But it's only
915 portable in implicit rules. Tips for rewriting - if it's a source file,
920 If it's a generated object file or similar, just write the name as is. The
921 tricky case is a generated file which isn't in git but is shipped in the
922 distribution tarball, as such a file could be in either the source or build
923 tree. Use this trick to make sure it's found whichever directory it's in::
925 `test -f foo.ext || echo '$(srcdir)/'`foo.ext
927 * Don't use "exit 0" to make a rule fail. Use "false" instead. BSD make
928 doesn't like "exit 0" in a rule.
930 * Don't use make conditionals. Automake offers conditionals which may be
931 of use, and these are implemented to work with any make. See the automake
932 manual for details, and a few caveats.
934 * The list of portable utilities is:
936 cat cmp cp diff echo egrep expr false grep install-info
937 ln ls mkdir mv pwd rm rmdir sed sleep sort tar test touch true
939 Note that versions of these (GNU versions in particular) support switches
940 which aren't portable - notably, "test -r" isn't portable; neither is
941 "cp -a". And note that "mkdir -p" isn't portable - the semantics vary.
942 The autoconf manual has some useful information about writing portable
943 shell code (most of it not specific to autoconf)::
945 https://www.gnu.org/software/autoconf/manual/autoconf.html#Portable-Shell
947 * Don't use "include" - it's not present in BSD make (at least some versions
948 have ".include" instead, but that doesn't really seem to help...) Automake
949 provides a configure-time include, which may provide a replacement for some
952 * It appears that BSD make only supports VPATH for implicit rules (e.g.
953 ".c.o:") - there's certainly a restriction there which is not present in GNU
954 make. We used to try to work around this, but now we use AM_MAINTAINER_MODE
955 to disable rules which are only needed by those developing Xapian (these were
956 the rules which caused problems). And we recommend those developing Xapian
957 use GNU make to avoid problems.
959 * Rules with multiple targets can cause problems for parallel builds. These
960 rules are really just a shorthand for multiple rules with the same
961 prerequisites and commands, and it is fine to use them in this way. However,
962 a common temptation is to use them when a single invocation of a command
963 generates multiple output files, by adding each of the output files as a
964 target. Eg, if a swig language module generates xapian_wrap.cc and
965 xapian_wrap.h, it is tempting to add a single rule something like::
967 # This rule has a problem
968 xapian_wrap.cc xapian_wrap.h: xapian.i
971 This can result in SWIG_commands being run twice, in parallel. If
972 SWIG_commands generates any temporary files, the two invocations can
973 interfere causing one of them to fail.
975 Instead of this rule, one solution is to pick one of the output files as a
976 primary target, and add a dependency for the second output file on the first
979 # This rule also has a problem
980 xapian_wrap.h: xapian_wrap.cc
981 xapian_wrap.cc: xapian.i
984 This ensures that make knows that only one invocation of SWIG_commands is
985 necessary, but could result in problems if the invocation of SWIG_commands
986 failed after creating xapian_wrap.cc, but before creating xapian_wrap.h.
987 Instead, we recommend creating an intermediate target::
989 # This rule works in most cases
990 xapian_wrap.cc xapian_wrap.h: xapian_wrap.stamp
991 xapian_wrap.stamp: xapian.i
995 Because the intermediate target is only touched after the commands have
996 executed successfully, subsequent builds will always retry the commands if an
997 error occurs. Note that the intermediate target cannot be a "phony" target
998 because this would result in the commands being re-run for every build.
1000 However, this rule still has a problem - if the xapian_wrap.cc and
1001 xapian_wrap.h files are removed, but the xapian_wrap.stamp file is not, the
1002 .cc and .h files will not be regenerated. There is no simple solution to
1003 this, but the following is a recipe taken from the automake manual which
1004 works. For details of *why* it works, see the section in the automake manual
1005 titled "Multiple Outputs"::
1007 # This rule works even if some of the output files were removed
1008 xapian_wrap.cc xapian_wrap.h: xapian_wrap.stamp
1009 ## Recover from the removal of $@. A full explanation of these rules is in
1010 ## the automake manual under the heading "Multiple Outputs".
1011 @if test -f $@; then :; else \
1012 trap 'rm -rf xapian_wrap.lock xapian_wrap.stamp' 1 2 13 15; \
1013 if mkdir xapian_wrap.lock 2>/dev/null; then \
1014 rm -f xapian_wrap.stamp; \
1015 $(MAKE) $(AM_MAKEFLAGS) xapian_wrap.stamp; \
1016 rmdir xapian_wrap.lock; \
1018 while test -d xapian_wrap.lock; do sleep 1; done; \
1019 test -f xapian_wrap.stamp; exit $$?; \
1022 xapian_wrap.stamp: xapian.i
1026 * This is actually a robustness point, not portability per se. Rules which
1027 generate files should be careful not to leave a partial file in place if
1028 there's an error as it will have a timestamp which leads make to believe it's
1029 up-to-date. So this is bad:
1032 $PERL script.pl > foo.cc
1037 $PERL script.pl > foo.tmp
1040 Alternatively, pass the output filename to the script and make sure you
1041 delete the output on error or a signal (although this approach can leave
1042 a partial file in place if the power fails). All used Makefile.am-s and
1043 scripts have been checked (and fixed if required) as of 2003-07-10 (didn't
1044 check xapian-bindings).
1046 * Another robustness point - if you add a non-file target to a makefile, you
1047 should also list it in ".PHONY". Otherwise your target won't get remade
1048 reliably if someone creates a file with the same name in their tree. For
1051 .PHONY: hello goodbye
1059 And lastly a style point - using "@" to suppress echoing of commands being
1060 executed removes choice from the user - they may want to see what commands
1061 are being executed. And if they don't want to, many versions of make support
1062 the use "make -s" to suppress the echoing of commands.
1064 Using @echo on a message sent to stdout or stderr is acceptable (since it
1065 avoids showing the message twice). Otherwise don't use "@" - it makes it
1066 harder to track down problems in the makefiles.
1071 Scripts generally should *not* have an extension indicating the language they
1072 are currently implemented in (e.g. ``runtest`` rather than ``runtest.sh`` or
1073 ``runtest.pl``). The problem with such an extension is that if we decide
1074 to reimplement the script in a different language, we either have to rename
1075 the script (which is annoying as people will be used to the name, and may
1076 have embedded it in their own scripts), or we have a script with a confusing
1077 name (e.g. a Python script with extension ``.pl``).
1079 The above reasoning doesn't apply to scripts which have to be in a particular
1080 language for some reason, though for consistency they probably shouldn't get
1081 an extension either, unless there's a good reason to have one.
1086 Use Assert to perform internal consistency checks, and to check for invalid
1087 arguments to functions and methods (e.g. passing a NULL pointer when this isn't
1088 permitted). It should *NOT* be used to check for error conditions such as
1089 file read errors, memory allocation failing, etc (since we want to perform such
1090 checks in non-debug builds too).
1092 File format errors should also not be tested with Assert - we want to catch
1093 a corrupted database or a malformed input file in a non-debug build too.
1095 There are several variants of Assert:
1097 - Assert(P) -- asserts that expression P is true.
1099 - AssertRel(a,rel,b) -- asserts that (a rel b) is true - rel can be a boolean
1100 relational operator, i.e. one of ``==``, ``!=``, ``>``, ``>=``, ``<``,
1101 ``<=``. The message given if the assertion fails reports the values of
1102 a and b, so ``AssertRel(a,<,b);`` is more helpful than ``Assert(a < b);``
1104 - AssertEq(a,b) -- shorthand for AssertRel(a,==,b).
1106 - AssertEqDouble(a,b) -- asserts a and b differ by less than DBL_EPSILON
1108 - AssertParanoid(P) -- a particularly expensive assertion. If you want a build
1109 with Asserts enabled, but without a great performance overhead, then
1110 passing --enable-assertions=partial to configure and AssertParanoids
1111 won't be checked, but Asserts will. You can also use AssertRelParanoid
1112 and AssertEqParanoid.
1114 - CompileTimeAssert(P) -- this has now been removed, since we require C++11
1115 support from the compiler, and C++11 added ``static_assert``.
1117 Marking Features as Deprecated
1118 ==============================
1120 In the API headers, a feature (a class, method, function, enum, typedef, etc)
1121 can be marked as deprecated by using the XAPIAN_DEPRECATED() or
1122 XAPIAN_DEPRECATED_CLASS macros. Note that you can't deprecate a preprocessor
1125 For compilers with a suitable mechanism (such as GCC, clang and MSVC) this
1126 causes compile-time warning messages to be emitted for any use of the
1127 deprecated feature. For compilers without support, the macro just expands to
1130 Sometimes a deprecated feature will also be removed from the library itself
1131 (particularly something like a typedef), but if the feature is still used
1132 inside the library (for example, so we can define class methods), then use
1133 XAPIAN_DEPRECATED_EX() or XAPIAN_DEPRECATED_CLASS_EX instead, which will only
1134 issue a warning in user code (this relies on user code including xapian.h
1135 and library code including individual headers)
1137 You must add this line to any API header which uses XAPIAN_DEPRECATED() or
1138 XAPIAN_DEPRECATED_CLASS::
1140 #include <xapian/deprecated.h>
1142 When marking a feature as deprecated, document the deprecation in
1143 docs/deprecation.rst. When actually removing deprecated features, please tidy
1144 up by removing the inclusion of <xapian/deprecated.h> from any file which no
1145 longer marks any features as deprecated.
1147 The XAPIAN_DEPRECATED() macro should wrap the whole declaration except for the
1148 semicolon and any "definition" part, for example::
1150 XAPIAN_DEPRECATED(int old_function(double arg));
1154 XAPIAN_DEPRECATED(int old_method());
1156 XAPIAN_DEPRECATED(int old_const_method() const);
1158 XAPIAN_DEPRECATED(virtual int old_virt_method()) = 0;
1160 XAPIAN_DEPRECATED(static int old_static_method());
1162 XAPIAN_DEPRECATED(static const int OLD_CONSTANT) = 42;
1165 Mark a class as deprecated by inserting ``XAPIAN_DEPRECATED_CLASS`` after the
1166 class keyword like so::
1168 class XAPIAN_DEPRECATED_CLASS Foo {
1175 With recent versions of GCC (4.4.7 allows this, 3.3.5 doesn't), you can
1176 simply mark a method defined inline in a class with ``XAPIAN_DEPRECATED()``
1181 // This failed to compile with GCC 3.3.5.
1182 XAPIAN_DEPRECATED(int old_inline_method()) { return 42; }
1185 Xapian 1.3.x and later require at least GCC 4.7, so you can now just use the
1191 If you have a patch to fix a problem in Xapian, or to add a new feature,
1192 please send it to us for inclusion. Any major changes should be discussed
1193 on the xapian-devel mailing list first:
1194 <https://xapian.org/lists>
1196 Also, please read the following section on licensing of patches before
1199 We find patches in unified diff format easiest to read. If you're using
1200 git, then "git diff" is good (or "git format-patch" for a patch series). If
1201 you're working from a tarball, you can unpack a second clean copy of the files
1202 and compare the two versions with "diff -pruN" (-p reports the function name
1203 for each chunk, -r acts recursively, -u does a unified diff, and -N shows
1204 new files in the diff). Alternatively "ptardiff" (which comes with perl, at
1205 least on Debian and Ubuntu) can diff against the original tarball, unpacking
1208 Please set the width of a tab character in your editor to 8 spaces, and use
1209 Unix line endings (i.e. LF, not CR+LF). Failing to do so will make it much
1210 harder for us to merge in your changes.
1212 We don't currently have a formal coding standards document, but please try
1213 to follow the style of the existing code. In particular:
1215 * Indent C++ code by 4 spaces for a new indentation level, and set your editor
1216 to tab-fill indentation (with a tab being 8 spaces wide).
1218 As an exception, "public", "protected" and "private" declarations in classes
1219 and structs should be indented by 2 spaces, and the following code should be
1220 indented by 2 more spaces::
1227 The rationale for this exception is that class definitions in header files
1228 often have fairly long lines, so losing an indent level to the access
1229 specifier tends to make class definitions less readable.
1231 The default access for a class is always "private", so there's no need
1232 to specify that explicitly - in other words, write this::
1235 int internal_method();
1238 int external_method();
1245 int internal_method();
1248 int external_method();
1251 If a class only contains public methods and data, consider declaring it as a
1252 "struct" (the only difference in C++ is that the default access for a
1253 struct is "public").
1255 * Put a space before the "(" after control flow constructs like "for", "if",
1256 "while", etc. Don't put a space before the "(" in function calls. So
1257 write "if (strlen(p) > 10)" not "if(strlen (p) > 10)".
1259 * When "if", "else", "for", "while", "do," "switch", "case", "default", "try",
1260 or "catch" is followed by a block enclosed in braces, the opening brace
1261 should be on the same line, like so::
1270 The rationale for this is that it conserves vertical space (allowing more
1271 code to fit on screen) without reducing readability.
1273 * If you have an empty loop body, use `{ }` rather than `;` as the former
1274 stands out more clearly to the reader (but also consider if the code might be
1275 clearer written a different way).
1277 * Prefer "++i;" to "i++;", "i += 1;", or "i = i + 1". For simple integer
1278 variables these should generate equivalent (if not identical) code, but if i
1279 is an iterator object then the pre-increment form can be more efficient in
1280 some cases with some compilers. It's simpler and more consistent to always
1281 use the pre-increment form (unless you make use of the old value which the
1282 post-increment form returns). For the same reasons, prefer "--i;" to "i--;",
1283 "i -= 1;", or "i = i - 1;".
1285 * Prefer "container.empty()" to "container.size() == 0" (and
1286 "!container.empty()" to "container.size() != 0" or "container.size() > 0").
1287 Finding the size of a container may not be a constant time operation for
1288 all containers (e.g. std::list may not be, and indeed isn't for GCC - see
1289 https://gcc.gnu.org/onlinedocs/libstdc++/manual/containers.html#sequences.list.size).
1290 Also the "empty()" form makes the intent of the test more explicit.
1292 * Prefer not to use "else" when the control flow is diverted elsewhere at the
1293 end of the "if" block (e.g. by "return", "continue", "break", "throw"). This
1294 eliminates a level of indentation from the code in the "else" block, and
1295 typically makes the control flow logic clearer. For example::
1317 * For standard ISO C headers, prefer the C++ form for ISO C headers (e.g.
1318 "#include <cstdlib>" rather than "#include <stdlib.h>") unless there's a good
1319 reason (e.g. portability) to do otherwise. Be sure to document such
1320 exceptions to avoid another developer changing them to the standard form.
1321 Global exceptions: <signal.h> (lots of POSIX stuff which e.g. Sun's compiler
1322 doesn't provide in <csignal>).
1324 * For standard ISO C++ headers, *always* use the ISO C++ form '#include <list>'
1325 (pre-ISO compilers used '#include <list.h>', but GCC has generated a warning
1326 for this form for years, and GCC 4.3 dropped support entirely).
1328 * Some guidelines for efficient use of std::string:
1330 + When passing an empty string to a method expecting ``const std::string &``
1331 prefer ``std::string()`` to ``""`` or ``std::string("")`` as the first form
1332 is more likely to directly use a special "empty string representation" (it
1333 does with GCC at least).
1335 + To make a string object empty, ``s.resize(0)`` (if you want to keep the
1336 current reserved space) or ``s = string()`` (if you don't) seem the best
1339 + Use ``std::string::assign()`` rather than building a temporary string
1340 object and assigning that. For example, ``foo = std::string(ptr, len);``
1341 is better written as ``foo.assign(ptr, len);``.
1343 + It's generally better to build up strings using ``+=`` rather than
1344 combining series of components with ``+``. So ``foo = a + " and " + c`` is
1345 better written as ``foo = a; foo += " and "; foo += c;``. It's possible
1346 for compilers to handle the former without a lot of temporary string
1347 objects by returning a proxy object to allow the concatenation to happen
1348 lazily, but not all compilers do this, and it's likely to still have some
1349 overhead. Note that GCC 4.1 seems to produce larger code in some cases for
1350 the latter approach, but it's a definite win with GCC 4.4.
1352 * ``std::string(1, '\0')`` seems to be slightly more efficient than
1353 ``std::string("", 1)`` for constructing a std::string containing a single
1354 ASCII nul character.
1356 * Prefer ``new SomeClass`` to ``new SomeClass()``, since the latter tends to
1357 lead one to write ``SomeClass foo();` which is a function prototype, and not
1358 equivalent to the variable definition ``SomeClass foo``. However, note that
1359 ``new SomePODType()`` is *not* the same as ``new SomePODType`` (if
1360 SomePODType is a POD (Plain Old Data) type) - the former will zero-initialise
1361 scalar members of SomePODType.
1363 * When catching an exception which is an object, do it by const reference, so
1368 } catch (const ErrorClass &e) {
1372 Catching by value is bad because it "slices" the object if an object of a
1373 derived type is thrown. Even if derived types aren't a worry, it also causes
1374 the copy constructor to be called needlessly.
1376 See also: http://www.parashift.com/c++-faq-lite/exceptions.html#faq-17.7
1378 A const reference is preferable to a non-const reference as it stops the
1379 object being inadvertently modified. In the rare cases when you want to
1380 modify the caught object, a non-const reference is OK.
1382 We will do our best to give credit where credit is due - if we have used
1383 patches from you, or received helpful reports or advice, we will add your name
1384 to the AUTHORS file (unless you specifically request us not to). If you see we
1385 have forgotten to do this, please draw it to our attention so that we can
1386 address the omission.
1388 Licensing of patches
1389 ====================
1391 If you want a patch to be considered for inclusion in the Xapian sources, you
1392 must own the copyright on this patch. Employers often claim copyright on code
1393 written by their employees (even if the code is written in their spare time),
1394 so please check with your employer if this applies. Be aware that even if you
1395 are a student your university may try and claim some rights on code which you
1398 Patches which are submitted to Xapian will only be included if the copyright
1399 holder(s) dual-license them under each of the following licences:
1401 - GPL version 2 and all later versions (see the file "COPYING" for details).
1404 Copyright (c) <year> <copyright holders>
1406 Permission is hereby granted, free of charge, to any person obtaining a copy
1407 of this software and associated documentation files (the "Software"), to
1408 deal in the Software without restriction, including without limitation the
1409 rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
1410 sell copies of the Software, and to permit persons to whom the Software is
1411 furnished to do so, subject to the following conditions:
1413 The above copyright notice and this permission notice shall be included in
1414 all copies or substantial portions of the Software.
1416 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
1417 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
1418 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
1419 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
1420 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
1421 FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
1424 The current distribution of Xapian contains many files which are only licensed
1425 under the GPL, but we are working towards being able to distribute Xapian under
1426 a more permissive license, and are not willing to accept patches which we will
1427 have to rewrite before this can happen.
1429 Tips for Submitting a Good Patch
1430 ================================
1432 1) Make sure that the documentation is updated
1433 ----------------------------------------------
1435 * API classes, methods, functions, and types must be documented by
1436 documentation comments alongside the declaration in ``include/xapian/*.h``.
1437 These are collated by doxygen - see doxygen's documentation for details
1438 of the supported syntax. We've decided to prefer to use @ rather than \
1439 to introduce doxygen commands (the choice is essentially arbitrary, but
1440 \ introduces C/C++ escape sequences so @ is likely to make for easier to
1441 read mark up for C/C++ coders).
1443 * The documentation comments don't give users a good overview, so we also
1444 need documentation which gives a good overview of how to achieve particular
1445 tasks. In particularly, major new functionality should have its own "topic"
1446 document, or extend an existing topic document if more appropriate.
1448 * Internal classes, etc should also be documented by documentation comments
1449 where they are declared.
1451 2) Make sure the tests are right
1452 --------------------------------
1454 * If you're adding a feature, also add feature tests for it. These both
1455 ensure that the feature isn't broken to start with and detect if later
1456 changes stop it working as intended.
1458 * If you've fixed a bug, make sure there's a regression test which
1459 fails on the existing code and succeeds after your changes.
1461 * If you're adding a new testcase to demonstrate an existing bug, and not
1462 checking a fix in at the same time, mark the testcase as a known failure (by
1463 calling ``XFAIL("explanatory message")`` at the start of your testcase (if
1464 necessary this can be conditional on backend or other factors - the backend
1465 case has explicit support via ``XFAIL_FOR_BACKEND("backend", "message")``).
1467 This will mean that this testcase failing will be reported as "XFAIL" which
1468 won't cause the test run to fail. If such a testcase in fact passes, that
1469 gets reported as "XPASS" and *will* cause the test run to fail. A testcase
1470 should not be flagged as "XFAIL" for a long time, but it can be useful to be
1471 able to add such testcases during development. It also allows a patch
1472 series which fixes a bug to first demonstrate the bug via a new testcase
1473 marked as "XFAIL", then fix the bug and remove the "XFAIL" - this makes it
1474 clear that the regression test actually failed before the fix.
1476 Note that failures which are due to valgrind errors or leaked fds are not
1477 affected by this macro - such errors are inherently not suitable for "XFAIL"
1478 as they go away when the testsuite is run without valgrind or on a platform
1479 where our fd leak detector code isn't supported.
1481 * Make sure all existing tests continue to pass.
1483 If you don't know how to write tests using the Xapian test rig, then
1484 ask. It's reasonably simple once you've done it once. There is a brief
1485 introduction to the Xapian test system in ``docs/tests.html``.
1487 3) Make sure the attributions are right
1488 ---------------------------------------
1490 * If necessary, modify the copyright statement at the top of any
1491 files you've altered. If there is no copyright statement, you may
1492 add one (there are a couple of Makefile.am's and similar that don't
1493 have copyright statements; anything that small doesn't really need
1494 one anyway, so it's a judgement call). If you've added files which
1495 you've written from scratch, they should include the GPL boilerplate
1496 with your name only.
1498 * If you're not in there, add yourself to the AUTHORS file.
1505 + If there's a trac ticket or other reference for the bug, mention it in the
1506 commit message - it's a great help to future developers trying to work out
1507 why a change was made.
1509 5) Consider backporting
1510 -----------------------
1512 * If there's an active release branch, check if the bug is present in that
1513 branch, and if the fix is appropriate to backport - if the fix breaks ABI
1514 compatibility or is very invasive, you need to fix it in a different way
1515 for the release branch, or decide not to backport the fix.
1520 * If there's a related trac ticket, update it (if the issue is completely
1521 addressed by the changes you've made, then close it).
1523 * Update the release notes for the most recent release with a copy of the
1524 patch. If the commit from git applies cleanly, you can just link to
1525 it. If it fails to apply, please attach an adjusted patch which does.
1526 If there are conflicts in test cases which aren't easy to resolve, it is
1527 acceptable to just drop those changes from the patch if we can still be
1528 confident that the issue is actually fixed by the patch.
1533 We use reference counted pointers for most API classes. These are implemented
1534 using Xapian::Internal::intrusive_ptr, the implementation of which is exposed
1535 for efficiency, and because it's unlikely we'll need to change it frequently,
1538 For the reference counted classes, the API class (e.g. Xapian::Enquire) is
1539 really just a wrapper around a reference counted pointer. This points to an
1540 internal class (e.g. Xapian::Enquire::Internal). The reference counted
1541 pointer is a member variable of the API class called internal. Conceptually
1542 this member is private, though it typically isn't declared as private (this
1543 is to avoid littering the external headers with friend declarations for
1546 There are a few exceptions to the reference counted structure, such as
1547 MSetIterator and ESetIterator which have an exposed implementation. Tests show
1548 this makes a substantial difference to speed (it's ~20% faster) in typical
1549 cases of iterator use.
1551 The postfix operator++ for iterators should be implemented inline in terms
1552 of the prefix form as described by Joe Buck on the gcc mailing list
1553 - excerpt from http://article.gmane.org/gmane.comp.gcc.devel/50201 ::
1555 class some_iterator {
1558 some_iterator& operator++();
1560 some_iterator operator++(int) {
1561 some_iterator tmp = *this;
1567 The compiler is allowed to assume that the copy constructor only does
1568 a copy, and to optimize away unneeded copy operations. The result
1569 in this case should be that, for some_iterator above, using the
1570 postfix operator without using the result should give code equivalent
1571 to using the prefix operator.
1573 Now, for [GCC 3.4], you'll find that the dead uses of tmp are only
1574 completely optimized away if tmp has only one data member that can fit in a
1575 register. [GCC 4.0 will do] better, and you should find that this style
1576 comes very close to eliminating any penalty from "incorrect" use of the
1579 Xapian's PostingIterator, TermIterator, PositionIterator, and ValueIterator all
1580 have only one data member which fits in a register.
1582 Handy tips for aiding development
1583 =================================
1585 If you are find you are repeatedly changing the API headers (in include/)
1586 during development, then you may become annoyed that the docs/ subdirectory
1587 will rebuild the doxygen documentation every time you run "make" since this
1588 takes a while. You can disable this temporarily (if you're using GNU make),
1589 by creating a file "docs/GNUmakefile" containing these two lines::
1592 @echo "Skipping 'make $@' in docs"
1594 Note that the whitespace at the start of the second line needs to be a
1595 single "tab" character!
1597 Don't forget to remove (or rename) this and check the documentation builds
1598 before committing or generating a patch though!
1600 If you are using an editor or other tool capable of running syntax checks as you
1601 work there you can use the `make` target 'check-syntax'. For 'emacs' users this
1602 works well with 'flymake'. Usage from a shell::
1604 make check-syntax check_sources=api/omdatabase.cc
1607 How to make a release
1608 =====================
1610 This is a (hopefully complete) list of the jobs which need doing:
1612 * Email Fabrice Colin and Tim Brody so they can check RPM packaging.
1614 * Check if `config/config.guess` and `config/config.sub` need updating to
1615 more recent versions from http://git.savannah.gnu.org/gitweb/?p=config.git
1617 * Check the revision currently specified in the bootstrap for the common
1618 subdirectory. Unless there's a good reason, we should release
1619 xapian-core and omega with synchronised versions of the shared files.
1621 * Make sure that any new/changed/removed API methods in xapian-core have been
1622 wrapped/updated/removed in xapian-bindings.
1624 * Update the lists of deprecated/removed API methods in docs/deprecation.rst
1626 * Update the NEWS files using information from the git logs
1628 * Update the version in configure.ac for each module (xapian-core, omega, and
1629 xapian-bindings), and the library version info in xapian-core's configure.ac
1631 * Make sure the submitters of fixed bugs are mentioned in the "thanks" list in
1632 xapian-core/AUTHORS. Check the list for the appropriate milestone::
1634 https://trac.xapian.org/query?col=id&col=summary&col=reporter&milestone=1.4.4
1636 * Check for any unfixed bugs on the milestone for the new release, and if they
1637 aren't blockers, retarget them:
1639 https://trac.xapian.org/roadmap
1641 * Tag the source trees for the new revision - use the git-tag-release script,
1642 running it with the new version number, for example:
1644 xapian-maintainer-tools/git-tag-release 1.4.4
1646 This script also generates tarballs for the new release and copies them
1647 across to the website.
1651 Create a new page https://trac.xapian.org/wiki/ReleaseNotes/X.Y.Z and link it
1652 into https://trac.xapian.org/wiki/ReleaseNotes in place of the old current
1653 release link, which should be moved to the archived section.
1655 Also update the roadmap at https://trac.xapian.org/wiki/RoadMap by recording
1656 the date of this release and adding an entry for the next release with an
1657 estimated release date.
1659 * Update the website: `generate` in the `www.xapian.org` git repo contains the
1660 latest version and the date it was released.
1662 * Run /home/olly/tmp/xapian-website-update/update_website.sh
1664 * Announce the new version on xapian-discuss
1666 * Have a nice cup of tea!
1668 How to make Debian packages for a new release
1669 =============================================
1671 Debian control files are stored in separate git repositories:
1673 * https://anonscm.debian.org/cgit/collab-maint/xapian-bindings.git
1674 * https://anonscm.debian.org/cgit/collab-maint/xapian-core.git
1675 * https://anonscm.debian.org/cgit/collab-maint/xapian-omega.git
1677 To package a new upstream release, these should be updated as follows:
1679 * If there are any patch files in "debian/patches", check if these have been
1680 incorporated into the new release, and if so remove them and update
1681 "debian/patches/series".
1683 * Update the debian/changelog file, being sure to keep it in the
1684 standard Debian format (the easiest way is to use the dch utility
1685 like so: "dch -v 1.2.19-1". The new version number should be the
1686 version number of the release followed by "-1" (i.e., a debian
1687 patch number of 1). The changelog message should indicate that
1688 there is a new upstream release, and should mention any significant
1689 changes in the new release.
1691 * Tag using: ``git tag -s -m 1.2.19-1 1.2.19-1``
1693 * FIXME: Document how to make source packages, or update
1694 ``make-source-packages``.
1696 * FIXME: Document how to build binary packages, or update ``build-packages``.
1698 * Test the packages.
1700 * Run ``debsign build/*_amd64.changes`` to GPG sign the packages.
1702 * Run ``dput build/*_amd64.changes`` to upload them to Debian.
1704 * For the Ubuntu backports::
1706 ./backport-source-packages xapian-core 1.2.19-1 ubuntu
1707 ./backport-source-packages xapian-omega 1.2.19-1 ubuntu
1708 ./backport-source-packages xapian-bindings 1.2.19-1 ubuntu
1710 And once libsearch-xapian-perl is uploaded to Debian unstable::
1712 ./backport-source-packages libsearch-xapian-perl 1.2.19.0-1 ubuntu
1716 debsign build/*99*_source.changes
1720 dput xapian-backports build/xapian-core*99*_source.changes
1722 Wait for that to have a chance to build, and then::
1724 dput xapian-backports build/xapian-[bo]*99*_source.changes
1725 dput xapian-backports build/libsearch-xapian-perl*_source.changes