Overhaul the snippet implementation
The approach from the paper Mihai based his implementation on doesn't
directly consider the query, but instead looks at the top few documents
matched and builds a document language model. In theory this is a nice
approach - it has a sound theoretical basis, and it will consider
interesting terms outside of the query. But in practice, this turns out
to have a serious drawback - it sometimes selects a snippet which
doesn't contain any of the query terms, and users find that surprising
(quite reasonably I think). It's also slower than is ideal.
We also had a patch for generating snippets from fastmail, but that has
different drawbacks - for example, its segmenting of text doesn't
exactly match what TermGenerator produces, so it fails to highlight in
some cases; also it considers each term in turn, so doesn't prefer a
snippet containing more terms from the query.
So I've taken the best ideas from each, and implemented a new snippet
generating algorithm. A key design choice is that it makes a single
pass over the text we're generating the snippet from (with scope to
terminate early). It prefers occurrences of the query terms in contexts
containing "interesting" non-query terms. And it also handles exact
phrases and wildcards (both selecting snippets based on them, and
highlighting them).
Fixes #211.