Merge branch 'mob'

[arxana.git] / latex / arxana-reboot.tex

%;; arxana.tex                   -*- mode: Emacs-Lisp; -*-
%;; Copyright (C) 2005-2009 Joe Corneli <holtzermann17@gmail.com>

%;; This program is free software: you can redistribute it and/or modify
%;; it under the terms of the GNU Affero General Public License as published by
%;; the Free Software Foundation, either version 3 of the License, or
%;; (at your option) any later version.
%;; 
%;; This program is distributed in the hope that it will be useful,
%;; but WITHOUT ANY WARRANTY; without even the implied warranty of
%;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
%;; GNU Affero General Public License for more details.
%;; 
%;; You should have received a copy of the GNU Affero General Public License
%;; along with this program.  If not, see <http://www.gnu.org/licenses/>.

% (progn
%   (find-file "~/arxana.tex")
%   (save-excursion
%     (goto-char (point-max))
%     (let ((beg (progn (search-backward "\\begin{verbatim}")
%                       (match-end 0)))
%           (end (progn (search-forward "\\end{verbatim}")
%                       (match-beginning 0))))
%       (eval-region beg end)
%       (lit-process))))

%%% Commentary:

%% To load: remove %'s above and evaluate with C-x C-e.

%% Alternatively, run this:
% head -n 13 arxana.tex | sed -e "/%/s///" > arxana-loader.el
%% on the command line to produce something you can use
%% to load Arxana when you start Emacs:
% emacs -l arxana-loader.el

%% Or put the expression in your ~/.emacs (perhaps wrapped
%% in function like `eval-arxana').

%% Or search for a similar form below and evaluate there!

%% Q.  Where exactly are we supposed to store the most
%% up-to-date Arxana files when they are ready to go?

%% A.  Copy them into /usr/lib/sbcl/site-systems/arxana/
%% and that should be enough.  Make sure that arxana.asd
%% is in that directory and that you have a symbolic link,
%% made via

%% ln -s ./arxana/arxana.asd .

%% in the directory /usr/lib/sbcl/site-systems/
%% -- Make sure to load once as root to generate new fasls.

%% Q. How to run the remote slime after that?

%% A. Make sure that Emacs `slime-protocol-version' matches
%% Common Lisp's `swank::*swank-wire-protocol-version*', then,
%% like this:

%% ssh -L 4005:127.0.0.1:4005 joe@li23-125.members.linode.com
%% linode$ sbcl
%% M-x slime-connect RET RET

%%% Code:

\documentclass{article}

\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{verbatim}

\newcommand{\meta}[1]{$\langle${\it #1}$\rangle$}

\theoremstyle{definition}
\newtheorem{nota}{Note}[section]

\parindent = 1.2em

\newenvironment{notate}[1]
  {\begin{nota}[{\bf {\em #1}}]}%
  {\end{nota}}

\makeatletter
\newenvironment{elisp}
  {\let\ORGverbatim@font\verbatim@font
   \def\verbatim@font{\ttfamily\scshape}%
   \verbatim}
  {\endverbatim
  \let\verbatim@font\ORGverbatim@font}
\makeatother

\makeatletter
\newenvironment{common}[1]
  {\let\ORGverbatim@font\verbatim@font
   \def\verbatim@font{\ttfamily\scshape}%
   \verbatim}
  {\endverbatim
  \let\verbatim@font\ORGverbatim@font}
\makeatother

\makeatletter
\newenvironment{idea}
  {\let\ORGverbatim@font\verbatim@font
   \def\verbatim@font{\ttfamily\slshape}%
   \verbatim}
  {\endverbatim
  \let\verbatim@font\ORGverbatim@font}
\makeatother

\begin{document}

\title{\emph{Arxana}}

\author{Joseph Corneli\thanks{Copyright (C) 2005-2010
    Joseph Corneli {\tt <holtzermann17@gmail.com>}\newline
    $\longrightarrow$ transferred to the public domain.}}
\date{Last revised: \today}

\maketitle

\abstract{A tool for building hackable semantic hypertext
  platforms.  Source code and mailing lists are at {\tt
    http://common-lisp.net/project/arxana}.}

\tableofcontents

\section{Introduction}

\begin{notate}{What is ``Arxana''?} \label{arxana}
\emph{Arxana} is the name of a ``next generation''
hypertext system that emphasizes annotation.  Every object
in this system is annotatable.  Because of this, I
sometimes call Arxana's core ``the scholium system'', but
the name ``Arxana'' better reflects our aim: to explore
the mysterious world of links, attachments,
correspondences, and side-effects.
\end{notate}

\begin{notate}{The idea} \label{theoretical-context}
A scholia-based document model for commons-based peer
production will inform the development of our
system.\footnote{{\tt
http://www.metascholar.org/events/2005/freeculture/viewabstract.php?id=19
% alternate:
% http://br.endernet.org/~akrowne/planetmath/papers/corneli\_fcdl/corneli-krowne.pdf
\label{corneli-krowne}
}}
In this model, texts are made up of smaller texts until
you get to atomic texts; user actions are built in the
same way.  Multiple users should interact with a shared
persistent data-store, through functional annotation, not
destructive modification.  We should pursue the
asynchronous interaction model until we arrive at live,
synchronous, settings, where we facilitate real-time
computer-mediated interactions between users, and between
users and running hackable programs.
\end{notate}

\begin{notate}{The data model} \label{data-model}
Start by storing a collection of \emph{strings}.  Now add
in \emph{pairs} and \emph{triples} which point at 2 and 3
objects respectively.  (We can extend to n-tuples if that
turns out to be convenient.)  Finally, we will maintain a
collection of \emph{lists}, each of which points at an
unlimited number of objects.
\end{notate}

\begin{notate}{History}
Thinking about how to improve existing systems for
peer-based collaboration in 2004, I designed a simple
version of the scholium system that treated textual
commentary and markup as scholia.\footnote{{\tt
    http://wiki.planetmath.org/AsteroidMeta/old\_draft\_of\_scholium\_system}}
In 2006, I put together a single-user version of this
system that ran exclusively under Emacs.\footnote{{\tt
    http://metameso.org/files/sbdm4cbpp.tex} \label{old-version}}
The current system is an almost-completely rewritten
variant, bringing in a shared database and various other
enhancements to support multi-user interaction.
\end{notate}

\begin{notate}{A brisk review of the programming literature} \label{prog-lit-review}
Many years before I started working on this project, there
was something called the Emacs HyperText
System.\footnote{{\tt
    http://www.aue.aau.dk/\~{}kock/Publications/HyperBase/}}
What we're doing here updates for modern database methods,
uses a more interesting data storage format, and also
considers multiple front-ends to the same database (for
example, a web interface).

Contemporary Emacs-based hypertext creation systems
include Muse and Emacs Wiki.\footnote{{\tt
    http://mwolson.org/projects/EmacsMuse.html}}$^,$\footnote{{\tt
    http://mwolson.org/projects/EmacsWiki.html}} The
browsing side features old standbys, Info and
Emacs/w3m\footnote{Not to be confused with Emacs-w3m,
  which is not entirely ``Emacs-based''.}.  These packages
provide ways to author or view what what we should now
call ``traditional'' hypertext documents.

An another legacy tool worth mentioning is
HyperCard\footnote{{\tt
    http://en.wikipedia.org/wiki/HyperCard}}.  This system
was oriented around the idea of using hypertext to create
software, a vision we share, but like just about everyone
else working in the field at the time, it used
uni-directional links.

Hypertext \emph{nouveau} is based on semantic triples.
The Semantic Web standard provides one specification of
the features we can expect from triples.\footnote{{\tt
    http://www.w3.org/TR/2004/REC-rdf-primer-20040210/}}
Triples provide a framework for knowledge representation
with more depth and flexibility than the popular
``tagging'' methodology.  For example, suitable
collections of triples implement AI-style ``frames''.  The
idea of using triples to organize archival material is
generating some interest as Semantic Web ideas
spread.\footnote{Cf. recent museum and library
  conferences}$^,$\footnote{Even among academic computer
  scientists! (Josh Grochow, p.c.)}

An abstractly similar project to Arxana with some grand
goals is being developed by Chris Hanson at MIT under the
name ``Web-scale Environments for Deduction
Systems''.\footnote{{\tt
    http://publications.csail.mit.edu/abstracts/abstracts07/cph2/cph2.html}}

Another technically similar project is Freebase, a hand
rolled database of open content, organized on frame-based,
triple driven, principles.  The developer of the Freebase
graphd database has some interesting things to say about
old and new ways of handling triples.\footnote{{\tt
    http://blog.freebase.com/2008/04/09/a-brief-tour-of-graphd/}}
\end{notate}

\begin{notate}{Fitting in}
My current development goal is to use this system to
create a more flexible multiuser interaction platform than
those currently available to web-based collaborative
projects (such as PlanetMath\footnote{{\tt
    http://planetmath.org}}).  As an intermediate stage,
I'm using Arxana to help organize material for a book I'm
writing.  Arxana's theoretical generality, active
development status, detailed documentation, and
superlatively liberal terms of use may make it an
attractive option for you to try as well!
\end{notate}

\begin{notate}{What you get}
Arxana has an Emacs frontend, a Common Lisp middle-end,
and a SQL backend.  If you want to do some work, any one
of these components can be swapped out and replaced with
the engine of your choice.  I've released all of the
implementation work on this system into the public domain,
and it runs on an entirely free/libre/open source software
platform.
\end{notate}

\begin{notate}{Acknowledgements}
Ted Nelson's ``Literary Machines'' and Marvin Minsky's
``Society of Mind'' are cornerstones in the historical and
social contextualization of this work.  Alfred Korzybski's
``Science and Sanity'' and Gilles Deleuze's ``The Logic of
Sense'' provided grounding and encouragement.  \TeX\ and
GNU Emacs have been useful not just in prototyping this
system, but also as exemplary projects in the genre I'm
aiming for.  John McCarthy's Elephant 2000 was an
inspiring thing to look at and think about\footnote{{\tt
    http://www-formal.stanford.edu/jmc/elephant/elephant.html}}, and of course Lisp has been a vital ingredient.

Thanks also to everyone who's talked about this project
with me!
\end{notate}

\section{Using the program}

\begin{notate}{Dependencies} \label{dependencies}
Our interface is embedded in Emacs.  Backend processing is
done with Common Lisp.  We are currently using the
PostgreSQL database.  These packages should be available
to you through the usual channels.  (I've been using SBCL,
but any Lisp should do; please make sure you are using a
contemporary Emacs version.)

We will connect Emacs to Lisp via Slime\footnote{{\tt
    http://common-lisp.net/project/slime/}}, and Lisp to
PostgreSQL via CLSQL.\footnote{{\tt http://clsql.b9.com/}}
CLSQL also talks directly to the Sphinx search engine,
which we use for text-based search.\footnote{{\tt
    http://www.sphinxsearch.com/}} Once all of these
things are installed and working together, you should be
able to begin to use Arxana.

Setting up all of these packages can be a somewhat
time-consuming and confusing task, especially if you
haven't done it before!  See Appendix \ref{appendix-setup}
for help.
\end{notate}

\begin{notate}{Export code and set up the interface}
If you are looking at the source version of this document
in Emacs, evaluate the following s-expression (type
\emph{C-x C-e} with the cursor positioned just after its
final parenthesis).  This exports the Common Lisp
components of the program to suitable files for subsequent
use, and prepares the Emacs environment.  (The code that
does this is in Appendix \ref{appendix-lit}.)
\end{notate}

\begin{idea}
(save-excursion
  (let ((beg (search-forward "\\begin{verbatim}"))
        (end (progn (search-forward "\\end{verbatim}")
                    (match-beginning 0))))
    (eval-region beg end)
    (lit-process)))
\end{idea}

\begin{notate}{To load Common Lisp components at run-time} \label{load-at-runtime}
Link {\tt arxana.asd} somewhere where Lisp can find it.
Then run commands like these in your Lisp; if you like,
you can place all of this stuff in your config file to
automatically load Arxana when Lisp starts.  The final
form is only necessary if you plan to use CLSQL's special
syntax on the Lisp command-line.
\end{notate}

\begin{idea}
(asdf:operate 'asdf:load-op 'clsql)
(asdf:operate 'asdf:load-op 'arxana)
(in-package arxana)
(connect-to-database)
(locally-enable-sql-reader-syntax)
\end{idea}

\begin{notate}{To connect Emacs to Lisp}
Either run {\tt M-x slime RET} to start and connect to
Lisp locally, or {\tt M-x slime-connect RET RET} after you
have opened a remote connection to your remote server with
a command like this: {\tt ssh -L 4005:127.0.0.1:4005
  <username>@<host>} and started Lisp and the Swank server
on the remote machine.  To have Swank start automatically
when you start Lisp, put commands like this in your config
file.
\end{notate}

\begin{idea}
(asdf:operate 'asdf:load-op 'swank)
(setf swank:*use-dedicated-output-stream* nil)
(setf swank:*communication-style* :fd-handler)
(swank:create-server :dont-close t)
\end{idea}

\begin{notate}{To define database structures}
If you haven't yet defined the basic database structures,
make sure to load them now!  (Using {\tt tabledefs.lisp},
or the SQL code in Section \ref{sql-code})
\end{notate}

\begin{notate}{Importing this document into system}
You can browse this document inside Arxana: after loading
the code, run \emph{M-x autoimport-arxana}.
\end{notate}

%%%% START OF BACKEND
\input{arxana-reboot-backend}
%%% END OF BACKEND

\section{Emacs-side} \label{emacs-side}

\subsection{The interface to Common Lisp}

\begin{notate}{On `Defun'} \label{defun-interface}
A way to define Elisp functions whose bodies are evaluated
by Common Lisp.  Trust me, this is a good idea.  Besides,
it exhibits some facinating backquote and comma tricks.
But be careful: this definition of `Defun' did not work on
Emacs version 21.

If we want to be able to feed in a standard arglist to
Common Lisp (with optional elements and so forth), we'd
have define how these arguments are handled here!
\end{notate}

\begin{elisp}
(defmacro Defun (name arglist &rest body)
  (declare (indent defun))
  `(defun ,name ,arglist
     (let* ((outbound-string
             (translate-emacs-syntax-to-common-syntax
              (format "%S"
                      (append
                       (list
                        (append (list 'lambda ',arglist)
                                ',body))
                       (mapcar
                        (lambda (arg) `',arg)
                        (list
                         ,@(remove-if
                                 (lambda (testelt)
                                   (eq testelt
                                 '&optional))
                                 arglist)))))))
            (returned-string
             (second
              ;; we now specify the right package!
              (slime-eval
               (list 'swank:eval-and-grab-output
                     outbound-string)
               :arxana))))
       (process-slime-output returned-string))))
\end{elisp}

\begin{notate}{On `process-slime-output'}
This should downcase all constituent symbols, but for
expediency I'm just downcasing `NIL' at the moment.  Will
come back for more testing and downcasing shortly.  (I
suspect the general case is just about as easy as what
happens here.)
\end{notate}

\begin{elisp}
(defun process-slime-output (str)
  (condition-case nil
      (let ((read-value (read str)))
        (if (symbolp read-value)
            (read (downcase str)))
        (nsubst nil 'NIL read-value))
    (error str)))
\end{elisp}

\begin{elisp}
(defun translate-emacs-syntax-to-common-syntax (str)
  (with-temp-buffer
    (insert str)
    (dolist (swap '(("(\\` " "`")
                    ("(\\\, " ",")))
      (goto-char (point-min))
      (while (search-forward (first swap) nil t)
        (goto-char (match-beginning 0))
        (forward-sexp)
        (delete-char -1)
        (goto-char (match-beginning 0))
        (delete-region (match-beginning 0)
                       (match-end 0))
        (insert (second swap))))
    (buffer-substring-no-properties (point-min)
                                    (point-max))))
\end{elisp}

\begin{notate}{Interactive `Defun'}
Note, an improved version of this macro would allow me to
specify that some Defuns are interactive and some are not.
This could be done by examining the submitted body, and
adjusting the defun if its car is an `interactive' form.
Most of the Defuns will be things that people will want to
use interactively, so making this change would probably be
a good idea.  What I'm doing in the mean time is just
writing 2 functions each time I need to make an
interactive function that accesses Common Lisp data!
\end{notate}

\begin{notate}{Common Lisp evaluation of code chunks}
Another potentially beneficial and simple approach is to
write a form like `progn' that evaluates its contents on
Common Lisp.  This saves us from having to rewrite all of
the `defun' facilities into `Defun' (e.g. interactivity).
But... the problem with \emph{this} is that Common Lisp
doesn't know the names of all the variables that are
defined in Emacs!  I'm not sure how to get all of the
values of these variable substituted \emph{first}, before
the call to Common Lisp is made.
\end{notate}

\begin{notate}{Debugging `Defun'}
In order to make debugging go easier, it might be nice to
have an option to make the code that is supposed to be
evaluated by Defun actually \emph{print} on the REPL
instead of being processed through an invisible back-end.
There could be a couple of different ways to do that, one
would be to simulate just what a user might do, the other
would be a happy medium between that and what we're doing
now: just put our computery auto-generated code on the
REPL and evaluate it.  (To some extent, I think the
*slime-events* buffer captures this information, but it is
not particularly easy to read.)
\end{notate}

\begin{notate}{Interactive Common Lisp?}
Suppose we set up some kind of interactive environment in
Common Lisp; how would we go about passing this
environment along to a user interacting via Emacs?  (Note
that SLIME's presentation of the debugging loop is one
good example.)
\end{notate}

\subsection{Database interaction} \label{interaction}

\begin{notate}{The `article' function} \label{the-article-function}
You can use this function to create an article with a
given name and contents.  If you like you can put it in a
list.
\end{notate}

\begin{elisp}
(Defun article (name contents &optional heading)
  (let ((coordinates (add-triple name
                                 "has content"
                                 contents)))
    (when theory (add-triple coordinates "in" heading))
    (when place (if (numberp place)
                    (put-in-place coordinates place)
                  (put-in-place coordinates)))
    coordinates))
\end{elisp}

\begin{notate}{The `scholium' function} \label{the-scholium-function}
You can use this function to link annotations to objects.
As with the `article' function, you can optionally
categorize the connection on a given list (cf. Note
\ref{the-article-function}).
\end{notate}

\begin{elisp}
(Defun scholium (beginning link end &optional heading)
  (let ((coordinates (add-triple beginning
                                 link
                                 end)))
    (when list (add-triple coordinates "in" heading))
    (when place (if (numberp place)
                    (put-in-place coordinates place)
                  (put-in-place coordinates)))
    coordinates))
\end{elisp}

\begin{notate}{Uses of coordinates}
Note that, if desired, you can feed input of the form
'(\meta{code} \meta{ref}) into `article' and `scholium'.
It's convenient to do further any processing of the object
we've created, while we still have ahold of the coordinates
returned by `add-triple' (cf. Note
\ref{import-code-continuations} for an example).
\end{notate}

\begin{notate}{Finding all the members of a list by type?}
We just narrow according to type.
\end{notate}

\begin{notate}{On `get-article'} \label{get-article}
Get the contents of the article named `name'.  Optional
argument `list' lets us find and use the position on the
given list that holds the name, and use that instead of
the name itself.

We do not yet deal well with the ambiguous case in which
there are several positions that correspond to the given
name that appear on the same list.

Note also that out of the data returned by
`triples-given-beginning-and-middle', we should pick the
(hopefully just) ONE that corresponds to the given list.

This means we need to pick over the list of triples
returned here, and test each one to see if it is in our
heading.  As to WHY there might be more than one ``has
content'' for a place that we know to be in our
heading... I'm not sure.  I guess we can go with the
assumption that there is just one, for now.
\end{notate}

\begin{elisp}
(Defun get-article (name &optional heading)
  (let* ((place-pseudonyms
          (if heading
              (get-places-subject-to-constraint
               name `(nil "in" ,heading))
            (get-places name)))
         (goes-by (cond
                    ((eq (length place-pseudonyms) 1)
                     `(1 ,(car place-pseudonyms)))
                    ((triple-exact-match
                      name "in" heading)
                     name)
                    ((not heading) name)
                    (t nil))))
    (when goes-by
      ;; it might be nice to also return `goes-by'
      ;; so we can access the appropriate place again.
      (third (print-triple
              (resolve-ambiguity
               (triples-given-beginning-and-middle
                goes-by "has content"))
              t)))))
\end{elisp}

\begin{notate}{On `get-names'} \label{get-names}
This function simply gets the names of articles that have
names -- in other words, every triple built around the
``has content'' relation.
\end{notate}

\begin{elisp}
(Defun get-names (&optional heading)
  (let ((conditions (list (list nil "has content" t))))
    (when heading
      (setq conditions
            (append conditions
                    (list (list nil "in" heading)))))
    (mapcar
     (lambda (place-or-string)
       (cond
         ;; place case
         ((eq (first place-or-string) 1)
          (print-system-object
           (place-lookup (second place-or-string))))
         ;; string case
         ((eq (first place-or-string) 0)
          (print-system-object place-or-string))))
     (mapcar
      (lambda (triple)
        (isolate-beginning triple))
      (satisfy-conditions conditions)))))
\end{elisp}

\begin{notate}{Contrasting cases} \label{contrasting-cases}
Consider the difference between
\begin{quote}
(? ``has author'' ``Arthur C. Clarke'') \\
(? ``has genre'' ``fiction'')
\end{quote}
and
\begin{quote}
(\emph{name} ``has content'' *) \\
(\emph{name} ``in'' ``heading'')
\end{quote}
where, in the latter case, we know \emph{who} we're
talking about, and we just want to limit the list of items
generated by the ``*'' by the second condition.  This
should help illustrate the difference between `get-names'
(which is making a general query) and `get-article' (which
already knows the name of a specific article), and the
logic that they use.
\end{notate}

\begin{notate}{Placing items from Emacs} \label{place-item}
We periodically need to place items from within Emacs.
The function `place-item' is a wrapper for `put-in-place'
that makes this possible (it also provides the user with
an extra option, namely to put the place itself under a
given heading).

Notice that when the symbol is placed in some pre-existing
place (which can only happen when `id' is not nil), that
place may already be under some other heading.  We will ignore
this case for now (since it seems that putting objects
into \emph{new} places will be the preferred action), but
later we will have to look at what to do in this other
case.
\end{notate}

\begin{elisp}
(Defun place-item (symbol &optional id heading)
  (let ((coordinates (put-in-place symbol id)))
    (when heading (add-triple coordinates "in" heading))
    coordinates))
\end{elisp}

\begin{notate}{Automatic classifications} \label{classifications}
It will presumably make sense to offer increasingly
``automatic'' classifications for new objects.  At this
point, we've set things up so that the user can optionally
supply the name of \emph{one} heading that their new object
is a part of.

It may make more sense to allow an `\&rest theories'
argument, and add the triple to all of the specified
theories.  This would require modifying `Defun' to
accommodate the `\&rest' idiom; see Note
\ref{defun-interface}.
\end{notate}

\begin{notate}{Postconditions and provenance}
After adding something to the database, we may want to do
something extra; perhaps generating provenance
information, perhaps checking or enforcing database
consistency, or perhaps running a hook that causes some
update in the frontend (cf. Note \ref{provenance}).
Provisions of this sort will come later, as will
short-hand convenience functions for making particularly
common complex entries.
\end{notate}

\subsection{Importing \LaTeX\ documents} \label{importing}

\begin{notate}{Importing sketch} \label{importing-sketch}
The code in this section imports a document as a
collection of (sub-)sections and notes.  It gathers the
sections, sub-sections, and notes recursively and records
their content in a tree whose nodes are places (Note
\ref{places}) and whose links express the ``component-of''
relation described in Note \ref{order-of-order}.

This representation lets us see the geometric,
hierarchical, structure of the document we've imported.
It exemplifies a general principle, that geometric data
should be represented by relationships between places, not
direct relationships between strings.  This is because
``the same'' string often appears in ``different'' places
in any given document (e.g. a paper's many sub-sections
titled ``Introduction'' will not all have the same
content).

What goes into the places is in some sense arbitrary.  The
key is that whatever is \emph{in} or \emph{attached} to
these places must tell us everything we need to know about
the part of the document associated with that place
(e.g. in the case of a note, its title and contents).
That's over and above the \emph{structural} links which
say how the places relate to one another.  Finally, all of
these places and structural links will be added to a
heading that represents the document as a whole.

A natural convention we'll use will be to put the name
of any document component that's associated with a given
place into that place, and add all other information as
annotations.
\end{notate}

\begin{notate}{Ordered versus unordered data} \label{ordered-vs-unordered}
The code in this section is an example of one way to work
with ordered data (i.e. \LaTeX\ documents are not just
hierarchical, but the elements at each level of the
hierarchy are also ordered).

Since \emph{many} artifacts are hierachical (e.g. Lisp
code), we should try to be compatible with \emph{native}
methods for working with order (in the case of Lisp, feed
the code into a Lisp processor and use CDR and CAR, etc.).

We \emph{can} use triples such as (``rank'' ``1''
``Fred'') and (``rank'' ``2'' ``Barney'') to talk about
order.  There may be some SQL techniques that would help.
(FYI, order can be handled very explicitly in Elephant!)

In order to account for \emph{different} orderings, we
need one more piece of data -- some explicit treatment of
where the order \emph{is}; in other words, theories.
(This table illustrates the fact that a heading is not so
different from ``an additional triple''; indeed, the only
reason to make them different is to have the extra
convenience of having their elements be numbered.)

\begin{center}
\begin{tabular}{|lll|l|}
\hline
rank & 1 & Fred & Friday \\
rank & 2 & Barney & Friday \\
rank & 1 & Barney & Saturday \\
rank & 2 & Fred & Saturday \\
\hline
\end{tabular}
\end{center}
\end{notate}

\begin{notate}{The order of order} \label{order-of-order}
The triples (``rank'' ``1'' ``Fred'') and (``rank'' ``2''
``Barney'') mentioned in Note \ref{ordered-vs-unordered}
are easy enough to read and understand; it might be more
natural in some ways for us to say (``Fred'' ``rank''
``1'') -- Fred has rank 1.  In this section, we're
concerned with talking about the ordered parts of a
document, and ($A$ $n$ $B$) seems like an intuitive way to
say ``$A$'s $n$th component is $B$''.
\end{notate}

\begin{notate}{It's not overdoing it, right?}
When importing \emph{this} document, we see links like the
following.  I hope that's not ``overdoing it''.  (Take a
look at Note \ref{get-article} and Note \ref{get-names} to
see how we go about getting information out of the
database.)  We could get rid of one link if theories were
database objects (cf. Note
\ref{theories-as-database-objects}).
\end{notate}

\begin{idea}
"T557[P135|Web interface|.in.arxana.tex]"
"T558[Future plans.9.P135|Web interface|]"
"T559[T558[Future plans.9.P135|Web interface|].in.arxana.tex]"
\end{idea}

\begin{notate}{Importing in general} \label{importing-generally}
We will eventually have a collection of parsers to get
various kinds of documents into the system in various
different ways (Note \ref{parsing}).  For now, this
section gives a simple way to get some sorts of
\LaTeX\ documents into the system, namely documents
structured along the same lines as the document you're
reading now!

An interesting approach to parsing \emph{math} documents
has been undertaken in the \LaTeX ML
project.\footnote{{\tt http://dlmf.nist.gov/LaTeXML/}}
Eventually it would be nice to get that level of detail
here, too!  Emacsspeak is another example of a
\LaTeX\ parser that deals with large-scale textual
structures as well as smaller bits and
pieces.\footnote{{\tt
    http://www.cs.cornell.edu/home/raman/aster/aster-thesis.ps}}

It would probably be useful to put together some parsers
for HTML and wiki code soon.
\end{notate}

\begin{notate}{On `import-buffer'}
This function imports \LaTeX\ documents, taking care of
the non-recursive aspects of this operation.  It imports
frontmatter (everything up to the first
\verb+\begin{section}+), but assumes ``backmatter'' is
trivial, and does not import it.  The imported material is
classified as a ``document'' with the same name as the
imported buffer.
\end{notate}

\begin{elisp}
(defun import-buffer (&optional buffername)
  (save-excursion
    (set-buffer (get-buffer (or buffername
                                (current-buffer))))
    (goto-char (point-min))
    (search-forward-regexp "\\\\begin{document}")
    (search-forward-regexp "\\\\section")
    (goto-char (match-beginning 0))
    ;; other links will be made in the "heading of this
    ;; document", but here we make a broader assertion.
    (scholium buffername "is a" "document")
    (scholium buffername
              "has frontmatter"
              (buffer-substring-no-properties
               (point-min)
               (point))
              buffername)
    ;;; These should maybe be scholia attached to
    ;; root-coords (below), but for some reason that
    ;; wasn't working so well -- investigate later --
    ;; maybe it just wasn't good to run after running
    ;; `import-within'.
    (let* ((root-coords (place-item buffername nil
                                    buffername))
           (levels
            '("section" "subsection" "subsubsection"))
           (current-parent buffername)
           (level-end nil)
           (sections (import-within levels))
           (index 0))
      (while sections
        (let ((coords (car sections)))
          (setq index (1+ index))
          (scholium root-coords
                    index
                    coords
                    buffername))
        (setq sections (cdr sections))))))
\end{elisp}

\begin{notate}{On `import-within'}
Recurse through levels of sectioning to import
\LaTeX\ code.

It would be good if we could do something about sections
that contain neither subsections nor notes (for example, a
preface), or, more generally, about text that is not
contained in any environment (possibly that appears before
any section).  We'll save things like this for another
editing round!

For the moment, we've decided to build the document
hierarchy with links that are blind to whether the $k$th
component of a section is a note or a subsection.
Children that are notes are attached in the subroutine
`import-notes' and those that are sections are attached in
`import-within'.  Users can find out what type of object
they are looking at based on whether or not it ``has
content''.

Incidentally, when looking for the end of an importing
level, `nil' is an OK result -- if this is the \emph{last}
section at this level \emph{and} there is no subsequent
section at a higher level.
\end{notate}

\begin{elisp}
(defun import-within (levels)
  (let ((this-level (car levels))
        (next-level (car (cdr levels))) answer)
    (while (re-search-forward
            (concat
             "^\\\\" this-level "{\\([^}\n]*\\)}"
             "\\( +\\\\label{\\)?"
             "\\([^}\n]*\\)?")
            level-end t)
      (let* ((name (match-string-no-properties 1))
             (at (place-item name nil buffername))
             (level-end
              (or (save-excursion
                    (search-forward-regexp
                     (concat "^\\\\" this-level "{.*")
                     level-end t))
                  level-end))
             (notes-end
              (if next-level
                  (or (progn (point)
                             (save-excursion
                               (search-forward-regexp
                                (concat "^\\\\"
                                        next-level "{.*")
                                level-end t)))
                      level-end)
                level-end))
             (index (let ((current-parent at))
                      (import-notes notes-end)))
             (subsections (let ((current-parent at))
                            (import-within (cdr levels)))))
        (while subsections
          (let ((coords (car subsections)))
            (setq index (1+ index))
            (scholium at
                      index
                      coords
                      buffername)
            (setq subsections (cdr subsections))))
        (setq answer (cons at answer))))
    (reverse answer)))
\end{elisp}

\begin{notate}{On `import-notes'} \label{import-notes}
We're going to make the daring assumption that the
``textual'' portions of incoming \LaTeX\ documents are
contained in ``Notes''.  That assumption is true, at
least, for the current document.  The function returns the
count of the number of notes imported, so that
`import-within' knows where to start counting this
section's non-note children.

Would this same function work to import all notes from a
buffer without examining its sectioning structure?  Not
quite, but close! (Could be a fun exercise to fix this.)
\end{notate}

\begin{elisp}
(defun import-notes (end)
  (let ((index 0))
    (while (re-search-forward (concat "\\\\begin{notate}"
                                      "{\\([^}\n]*\\)}"
                                      "\\( +\\\\label{\\)?"
                                      "\\([^}\n]*\\)?")
                              end t)
      (let* ((name
              (match-string-no-properties 1))
             (tag (match-string-no-properties 3))
             (beg
              (progn (next-line 1)
                     (line-beginning-position)))
             (end
              (progn (search-forward-regexp
                      "\\\\end{notate}")
                     (match-beginning 0)))
             (coords (place-item name nil buffername)))
        (setq index (1+ index))
        (scholium current-parent
                  index
                  coords
                  buffername)
        ;; not in the heading
        (scholium coords
                  "has content"
                  (buffer-substring-no-properties
                   beg end))
        (import-code-continuations coords)))
    index))
\end{elisp}

\begin{notate}{On `import-code-continuations'} \label{import-code-continuations}
This runs within the scope of `import-notes', to turn the
series of Lisp chunks or other code snippets that follow a
given note into a scholium attached to that note.  Each
separate snippet becomes its own annotation.

The ``conditional regexps'' used here only work with Emacs
version 23 or higher.

I'm noticing a problem with the way the `looking-at'
form behaves.  It matches the expression in question,
but then the match-end is reported as one character
less than it supposed to be.  Maybe `looking-at' is
just not as good as `re-search-forward'?  But it's
what seems easiest to use.
\end{notate}

\begin{elisp}
(defun import-code-continuations (coords)
  (let ((possible-environments
         "\\(1?:lisp\\|idea\\|common\\)"))
    (while (looking-at
            (concat "\n*?\\\\begin{"
                    possible-environments
                    "}"))
      (let* ((beg (match-end 0))
             (environment (match-string 1))
             (end (progn (search-forward-regexp
                          (concat "\\\\end{"
                                  environment
                                  "}"))
                         (match-beginning 0)))
             (content (buffer-substring-no-properties
                       beg
                       end)))
        (scholium (scholium coords
                            "has attachment"
                            content)
                  "has type"
                  environment)))))
\end{elisp}

\begin{notate}{On `autoimport-arxana'} \label{autoimport-arxana}
This just calls `import-buffer', and imports this document
into the system.
\end{notate}

\begin{elisp}
(defun autoimport-arxana ()
  (interactive)
  (import-buffer "arxana.tex"))
\end{elisp}

\begin{notate}{Importing textual links}
Of course, it would be good to import the links that users
make between articles, since then we can quickly navigate
from an article to the various articles that cite that
article, as well as follow the usual forward-directional
links.  Indeed, we should be able to browse each article
within a ``neighborhood'' of other related articles.
(We'll need to import labels as well, of course.)
\end{notate}

\subsection{Browsing database contents} \label{browsing}

\begin{notate}{Browsing sketch} \label{browsing-sketch}
This section facilitates browsing of documents represented
with structures like those created in Section
\ref{importing}, and sets the ground for browsing other
sorts of contents (e.g. collections of tasks, as in
Section \ref{managing-tasks}).

In order to facilitate general browsing, it is not enough
to simply use `get-article' (Note \ref{get-article}) and
`get-names' (Note \ref{get-names}), although these
functions provide our defaults.  We must provide the means
to find and display different things differently -- for
example, a section's  table of contents will typically
be displayed differently from its actual contents.

Indeed, the ability to display and select elements of
document sections (Note \ref{display-section}) is
basically the core browsing deliverable.  In the process
we develop a re-usable article selector (Note
\ref{selector}; cf. Note \ref{browsing-tasks}).  This in
turn relies on a flexible function for displaying
different kinds of articles (Note \ref{display-article}).
\end{notate}

\begin{notate}{On `display-article'} \label{display-article}
This function takes in the name of the article to display.
Furthermore, it takes optional arguments `retriever' and
`formatter', which tell it how to look up and/or format
the information for display, respectively.

Thus, either we make some statement up front (choosing our
`formatter' based on what we already know about the
article), or we decide what to display after making some
investigation of information attached to the article, some
of which may be retrieved and displayed (this requires
that we specify a suitable `retriever' and a complementary
`formatter').

For example, the major mode in which to display the
article's contents could be stored as a scholium attached
to the article; or we might maintain some information
about ``areas'' of the database that would tell us up
front what which mode is associated with the current area.
(The default is to simply insert the data with no markup
whatsoever.)

Observe that this works when no heading argument is given,
because in that case `get-article' looks for \emph{all}
place pseudonyms.  (But of course that won't work well
when we have multiple theories containing things with the
same names, so we should get used to using the heading
argument.)

(The business about requiring the data to be a sequence
before engaging in further formatting is, of course, just
a matter of expediency for making things work with the
current dataset.)
\end{notate}

\begin{elisp}
(defun display-article
  (name &optional heading retriever formatter)
  (interactive "Mname: ")
  (let* ((data (if retriever
                   (funcall retriever name heading)
                 (get-article name heading))))
    (when (and data (sequencep data))
      (save-excursion
        (if formatter
            (funcall formatter data heading)
          (pop-to-buffer (get-buffer-create
                          "*Arxana Display*"))
          (delete-region (point-min) (point-max))
          (insert "NAME: " name "\n\n")
          (insert data)
          (goto-char (point-min)))))))
\end{elisp}

\begin{notate}{An interactive article selector} \label{selector}
The function `get-names' (Note \ref{get-names}) and
similar functions can give us a collection of articles.
The next few functions provide an interactive
functionality for moving through this collection to find
the article we want to look at.

We define a ``display style'' that the article selector
uses to determine how to display various articles.  These
display styles are specified by text properties attached
to each option the selector provides.  Similarly, when
we're working within a given heading, the relevant heading
is also specified as a text property.

At selection time, these text properties are checked to
determine which information to pass along to
`display-article'.
\end{notate}

\begin{elisp}
(defvar display-style '((nil . (nil nil))))

(defun thing-name-at-point ()
  (buffer-substring-no-properties
   (line-beginning-position)
   (line-end-position)))

(defun get-display-type ()
  (get-text-property (line-beginning-position)
                     'arxana-display-type))

(defun get-relevant-heading ()
  (get-text-property (line-beginning-position)
                     'arxana-relevant-heading))

(defun arxana-list-select ()
  (interactive)
  (apply 'display-article
         (thing-name-at-point)
         (get-relevant-heading)
         (cdr (assoc (get-display-type)
                     display-style))))

(define-derived-mode arxana-list-mode fundamental-mode
  "arxana-list" "Arxana List Mode.

\\{arxana-list-mode-map}")

(define-key arxana-list-mode-map (kbd "RET")
            'arxana-list-select)
\end{elisp}

\begin{notate}{On `pick-a-name'} \label{pick-a-name}
Here `generate' is the name of a function to call to
generate a list of items to display, and `format' is a
function to put these items (including any mark-up) into
the buffer from which individiual items can then be
selected.

One simple way to get a list of names to display would be
to reuse a list that we had already produced (this would
save querying the database each time).  We could, in fact,
store a history list of lists of names that had been
displayed previously (cf. Note \ref{local-storage}).

We'll eventually want versions of `generate' that provide
various useful views into the data, e.g., listing all of
the elements of a given section (Note
\ref{display-section}).

Finding all the elements that match a given search term,
whether that's just normal text search or some kind of
structured search would be worthwhile too.  Upgrading the
display to e.g. color-code listed elements according to
their type would be another nice feature to add.
\end{notate}

\begin{elisp}
(defun pick-a-name (&optional generate format heading)
  (interactive)
  (let ((items (if generate
                   (funcall generate)
                 (get-names heading))))
    (when items
      (set-buffer (get-buffer-create "*Arxana Articles*"))
      (toggle-read-only -1)
      (delete-region (point-min)
                     (point-max))
      (if format
          (funcall format items)
        (mapc (lambda (item) (insert item "\n")) items))
      (toggle-read-only t)
      (arxana-list-mode)
      (goto-char (point-min))
      (pop-to-buffer (get-buffer "*Arxana Articles*")))))
\end{elisp}

\begin{notate}{On `display-section'} \label{display-section}
When browsing a document, if you select a section, you
should display a list of that section's constituent
elements, be they notes or subsections.  The question
comes up: when you go to display something, how do you
know whether you're looking at the name of a section, or
the name of an article?

When you get the section's contents out of the database
(Note \ref{get-section-contents})
\end{notate}

\begin{elisp}
(defun display-section (name heading)
  (interactive (list (read-string
                      (concat
                       "name (default "
                       (buffer-name) "): ")
                      nil nil (buffer-name))))
  ;; should this pop to the Articles window?
  (pick-a-name `(lambda ()
                  (get-section-contents
                   ,name ,heading))
               `(lambda (items)
                  (format-section-contents
                   items ,heading))))

(add-to-list 'display-style
             '(section . (display-section
                          nil)))
\end{elisp}

\begin{notate}{On `get-section-contents'} \label{get-section-contents}
Sent by `display-section' (Note \ref{display-section})
to `pick-a-name' as a generator for the table of contents
of the section with the given name in the given heading.

This function first finds the triples that begin with the
(placed) name of the section, then checks to see which of
these are in the heading of the document we're examinining
(in other words, which of these links represent structural
information about that document).  It also looks at the
items found at the end of these links to see if they are
sections or notes (``noteness'' is determined by them
having content).  The links are then sorted by their
middles (which show the order in which these components
have in the section we're examining).  After this ordering
information has been used for sorting, it is deleted, and
we're left with just a list of names in the apropriate
order together with an indication of their noteness.
\end{notate}

\begin{elisp}
(Defun get-section-contents (name heading)
  (let (contents)
    (dolist (triple (triples-given-beginning
                     `(1 ,(resolve-ambiguity
                           (get-places name)))))
      (when (triple-exact-match
             `(2 ,(car triple)) "in" heading)
        (let* ((number (print-middle triple))
               (site (isolate-end triple))
               (noteness
                (when (triples-given-beginning-and-middle
                       site "has content")
                  t)))
        (setq contents
              (cons (list number
                          (print-system-object
                           (place-contents site))
                          noteness)
                    contents)))))
    (mapcar 'cdr
            (sort contents
                  (lambda (component1 component2)
                    (< (parse-integer (car component1))
                       (parse-integer (car component2))))))))
\end{elisp}

\begin{notate}{On `format-section-contents'} \label{format-section-contents}
A formatter for document contents, used by
`display-document' (Note \ref{display-document}) as input
for `pick-a-name' (Note \ref{pick-a-name}).

Instead of just printing the items one by one,
like the default formatter in `pick-a-name'  does,
this version adds appropriate text properties, which
we determine based the second component of
of `items' to format.
\end{notate}

\begin{elisp}
(defun format-section-contents (items heading)
  ;; just replicating the default and building on that.
  (mapc (lambda (item)
          (insert (car item))
          (let* ((beg (line-beginning-position))
                 (end (1+ beg)))
            (unless (second item)
              (put-text-property beg end
                                 'arxana-display-type
                                 'section))
            (put-text-property beg end
                               'arxana-relevant-heading
                               heading))
          (insert "\n"))
        items))
\end{elisp}

\begin{notate}{On `display-document'} \label{display-document}
When browsing a document, you should first display its
top-level table of contents.  (Most typically, a list of
all of that document's major sections.)  In order to do
this, we must find the triples that are begin at the node
representing this document \emph{and} that are in the
heading of this document.  This boils down to treating the
document's root as if it was a section and using the
function `display-section' (Note \ref{display-section}).
\end{notate}

\begin{elisp}
(defun display-document (name)
  (interactive (list (read-string
                      (concat
                       "name (default "
                       (buffer-name) "): ")
                      nil nil (buffer-name))))
  (display-section name name))
\end{elisp}

\begin{notate}{Work with `heading' argument}
We should make sure that if we know the heading we're
working with (e.g. the name of the document we're
browsing) that this information gets communicated in the
background of the user interaction with the article
selector.
\end{notate}

\begin{notate}{Selecting from a hierarchical display} \label{hierarchical-display}
A fancier ``article selector'' would be able to display
several sections with nice indenting to show their
hierarchical order.
\end{notate}

\begin{notate}{Browser history tricks} \label{history-tricks}
I want to put together (or put back together) something
similar to the multihistoried browser that I had going in
the previous version of Arxana and my Emacs/Lynx-based web
browser, Nero\footnote{{\tt http://metameso.org/~joe/nero.el}}.
The basic features are:
(1) forward, back, and up inside the structure of a given
document; (2) switch between tabs.  More advanced features
might include: (3) forward and back globally across all
tabs; (4) explicit understanding of paths that loop.

These sorts of features are independent of the exact
details of what's printed to the screen each time
something is displayed.  So, for instance, you could flip
between section manifests a la Note \ref{display-section},
or between hierarchical displays a la Note
\ref{hierarchical-display}, or some combination; the key
thing is just to keep track in some sensible way of
whatever's been displayed!
\end{notate}

\begin{notate}{Local storage for browsing purposes} \label{local-storage}
Right now, in order to browse the contents of the
database, you need to query the database every time.  It
might be handy to offer the option to cache names of
things locally, and only sync with the database from time
to time.  Indeed, the same principle could apply in
various places; however, it may also be somewhat
complicated to set up.  Using two systems for storage, one
local and one permanent, is certainly more heavy-duty than
just using one permanent storage system and the local
temporary display.  However, one thing in favor of local
storage systems is that that's what I used in the the
previous prototype of Arxana -- so some code already
exists for local storage!  (Caching the list of
\emph{names} we just made a selection from would be one
simple expedient, see Note \ref{pick-a-name}.)
\end{notate}

\begin{notate}{Hang onto absolute references}
Since `get-article' (Note \ref{get-article}) translates
strings into their ``place pseudonyms'', we may want to
hang onto those pseudonyms, because they are, in fact, the
absolute references to the objects we end up working with.
In particular, they should probably go into the
text-property background of the article selector, so it
will know right away what to select!
\end{notate}

\subsection{Exporting \LaTeX\ documents$^*$}

\begin{notate}{Roundtripping}
The easiest test is: can we import a document into the
system and then export it again, and find it unchanged?
\end{notate}

\begin{notate}{Data format}
We should be able to \emph{stably} import and export a
document, as well as export any modifications to the
document that were generated within Arxana.  This means
that the exporting functions will have to read the data
format that the importing functions use, \emph{and} that
any functions that edit document contents (or structure)
will also have to use the same format.  Furthermore,
\emph{browsing} functions will have to be somewhat aware
of this format.  So, this is a good time to ask -- did we
use a good format?
\end{notate}

\subsection{Editing database contents$^*$} \label{editing}

\begin{notate}{Roundtripping, with changes}
Here, we should import a document into the system and then
make some simple changes, and after exporting, check with
diff to make sure the changes are correct.
\end{notate}

\begin{notate}{Re-importing}
One nice feature would be a function to ``re-import'' a
document that has changed outside of the system, and make
changes in the system's version whereever changes appeared
in the source version.
\end{notate}

\begin{notate}{Editing document structure}
The way we have things set up currently, it is one thing
to make a change to a document's textual components, and
another to change its structure.  Both types of changes
must, of course, be supported.
\end{notate}

\section{Applications}

\subsection{Managing tasks} \label{managing-tasks}

\begin{notate}{What are tasks?}
Each task tends to have a \emph{name}, a
\emph{description}, a collection of \emph{prerequisite
  tasks}, a description of other \emph{material
  dependencies}, a \emph{status}, some \emph{justification
  of that status}, a \emph{creation date}, and an
\emph{estimated time of completion}.  There might actually
be several ``estimated times of completion'', since the
estimate would tend to improve over time.  To really
understand a task, one should keep track of revisions like
this.
\end{notate}

\begin{notate}{On `store-task-data'} \label{store-task-data}
Here, we're just filling in a frame.  Since ``filling in a
frame'' seems like the sort of operation that might happen
over and over again in different contexts, to save space,
it would probably be nice to have a macro (or similar)
that would do a more general version of what this function
does.
\end{notate}

\begin{elisp}
(Defun store-task-data
  (name description prereqs materials status
        justification submitted eta)
  (add-triple name "is a" "task")
  (add-triple name "description" description)
  (add-triple name "prereqs" prereqs)
  (add-triple name "materials" materials)
  (add-triple name "status" status)
  (add-triple name "status justification" justification)
  (add-triple name "date submitted" submitted)
  (add-triple name "estimated time of completion" eta))
\end{elisp}

\begin{notate}{On `generate-task-data'} \label{generate-task-data}
This is a simple function to create a new task matching
the description above.
\end{notate}

\begin{elisp}
(defun generate-task-data ()
  (interactive)
  (let ((name (read-string "Name: "))
        (description (read-string "Description: "))
        (prereqs (read-string
                  "Task(s) this task depends on: "))
        (materials (read-string "Material dependencies: "))
        (status (completing-read
                 "Status (tabled, in progress, completed):
                 " '("tabled" "in progress" "completed")))
        (justification (read-string "Why this status? "))
        (submitted
         (read-string
          (concat "Date submitted (default "
                  (substring (current-time-string) 0 10)
                  "): ")
          nil nil (substring (current-time-string) 0 10)))
        (eta
         (read-string "Estimated date of completion:")))
    (store-task-data name description prereqs materials
                     status
                     justification submitted eta)))
\end{elisp}

\begin{notate}{Possible enhancements to `generate-task-data'}
In order to make this function very nice, it would be good
to allow ``completing read'' over known tasks when filling
in the prerequisites.  Indeed, it might be especially nice
to offer a type of completing read that is similar in some
sense to the tab-completion you get when completing a file
name, i.e., quickly completing certain sub-strings of the
final string (in this case, these substrings would
correspond to task areas we are progressively zooming down
into).

As for the task description, rather than forcing the user
to type the description into the minibuffer, it might be
nice to pop up a separate buffer instead (a la the
Emacs/w3m textarea).  If we had a list of all the known
tasks, we could offer completing-read over the names of
existing tasks to generate the list of `prereqs'.  It
might be nice to systematize date data, so we could more
easily e.g. sort and display task info ``by date''.
(Perhaps we should be working with predefined database
types for dates and so on; but see Note
\ref{choice-of-database}.)

Also, before storing the task, it might be nice to offer
the user the chance to review the data they entered.
\end{notate}

\begin{notate}{On `get-filler'} \label{get-filler}
Just a wrapper for `triples-given-beginning-and-middle'.
(Maybe add `heading' as an option here.)
\end{notate}

\begin{elisp}
(Defun get-filler (frame slot)
  (third (first
          (print-triples
           (triples-given-beginning-and-middle frame
                                               slot)))))
\end{elisp}

\begin{notate}{On `get-task'} \label{get-task}
Uses `get-filler' (Note \ref{get-filler}) to assemble the
elements of a task's frame.
\end{notate}

\begin{elisp}
(Defun get-task (name)
  (when (triple-exact-match name "is a" "task")
    (list (get-filler name "description")
          (get-filler name "prereqs")
          (get-filler name "materials")
          (get-filler name "status")
          (get-filler name "status justification")
          (get-filler name "date submitted")
          (get-filler name
                      "estimated time of completion"))))
\end{elisp}

\begin{notate}{On `review-task'} \label{review-task}
This is a function to review a task by name.
\end{notate}

\begin{elisp}
(defun review-task (name)
  (interactive "MName: ")
  (let ((task-data (get-task name)))
    (if task-data
        (display-task task-data)
      (message "No data."))))

(defun display-task (data)
  (save-excursion
    (pop-to-buffer (get-buffer-create
                    "*Arxana Display*"))
    (delete-region (point-min) (point-max))
    (insert "NAME: " name "\n\n")
    (insert "DESCRIPTION: " (first data) "\n\n")
    (insert "TASKS THIS TASK DEPENDS ON: "
            (second data) "\n\n")
    (insert "MATERIAL DEPENDENCIES: "
            (third data) "\n\n")
    (insert "STATUS: " (fourth data) "\n\n")
    (insert "WHY THIS STATUS?: " (fifth data) "\n\n")
    (insert "DATE SUBMITTED:" (sixth data) "\n\n")
    (insert "ESTIMATED TIME OF COMPLETION: "
            (seventh data) "\n\n")
    (goto-char (point-min))
    (fill-individual-paragraphs (point-min) (point-max))))
\end{elisp}

\begin{notate}{Possible enhancements to `review-task'}
Breaking this down into a function to select the task and
another function to display the task would be nice.  Maybe
we should have a generic function for selecting any object
``by name'', and then special-purpose functions for
displaying objects with different properties.

Using text properties, we could set up a ``field-editing
mode'' that would enable you to select a particular field
and edit it independently of the others.  Another more
complex editing mode would \emph{know} which fields the
user had edited, and would store all edits back to the
database properly.  See Section \ref{editing} for more on
editing.
\end{notate}

\begin{notate}{Browsing tasks} \label{browsing-tasks}
The function `pick-a-name' (Note \ref{pick-a-name}) takes
two functions, one that finds the names to choose from,
and the other that says how to present these names.  We
can therefore build `pick-a-task' on top of `pick-a-name'.
\end{notate}

\begin{elisp}
(Defun get-tasks ()
  (mapcar #'first
          (print-triples
           (triples-given-middle-and-end "is a" "task")
           t)))

(defun pick-a-task ()
  (interactive)
  (pick-a-name
   'get-tasks
   (lambda (items)
     (mapc (lambda (item)
             (let ((pos (line-beginning-position)))
               (insert item)
               (put-text-property pos (1+ pos)
                                  'arxana-display-type
                                  'task)
               (insert "\n"))) items))))

(add-to-list 'display-style
             '(task . (get-task display-task)))
\end{elisp}

\begin{notate}{Working with theories}
Presumably, like other related functions, `get-tasks'
should take a heading argument.
\end{notate}

\begin{notate}{Check display style}
Check if this works, and make style consistent between
this usage and earlier usage.
\end{notate}

\begin{notate}{Example tasks}
It might be fun to add some tasks associated with
improving Arxana, just to show that it can be done...
maybe along with a small importer to show how importing
something without a whole lot of structure can be easy.
\end{notate}

\subsection{Other ideas$^*$}

\begin{notate}{A browser within a browser} \label{browser-within}
All the stuff we're doing with triples can be superimposed
over the existing web and existing web interfaces, by, for
example, writing a web browser as a web app, and in this
``browser within a browser'' offer the ability to annotate
and rewrite other people's web pages, produce 3rd-party
redirects, and so forth, sharing these mods with other
subscribers to the service.  (Already websites such as the
short-lived scrum.diddlyumptio.us have offered limited
versions of ``web annotation'', but, so far, what one can
do with such services seems quite weak compared with
what's possible.)
\end{notate}

\begin{notate}{Improvements to the PlanetMath backend}
From one point of view, the SQL tables are the main thing
in Noosphere.  We could say that getting the things out of
SQL and storing new things there is what Noosphere mainly
does.  Following this line of thought, anything that
adjusts these tables will do just as well, e.g., it
shouldn't be terribly hard to develop an email-based
front-end.  But rather than making Arxana work with the
Noosphere relational table system, it is probably
advantageous to translate the data from these tables into
the scholium system.
\end{notate}

\begin{notate}{A new communication platform}
One of the premier applications I have in mind is a new
way to handle communications in an online-forum.  I have
previously called this ``subchanneling'', but really,
joining channels is just as important.
\end{notate}

\begin{notate}{Some tutorials}
It would be interesting to write a tutorial for Common
Lisp or just about any other topic with this system.  For
example, some little ``worksheets'' or ``gymnasia'' that
will help solidify user knowledge in topics on which
questions keep appearing.
\end{notate}

\section{Topics of philosophical interest}

\begin{notate}{Research and development}
In Note \ref{theoretical-context}, I mentioned a model
that could apply in many contexts; it is an essentially
metaphysical conception.  I'm pretty sure that the data
model of Note \ref{data-model} provides a general-enough
framework to represent anything we might find ``out
there''.  However, even if this is the case, questions as
to \emph{efficient} means of working with such data still
abound (cf. Note \ref{models-of-theories}, Note
\ref{use-of-views}).

I propose that along with \emph{development} of Arxana as
a useful system for \emph{doing} ``commons-based peer
production'' should come a \emph{research} programme for
understanding in much greater detail what ``commons-based
peer production'' \emph{is}.  Eventually we may want to
change the name of the subject of study to reflect still
more general ideas of resource use.

While the ``frontend'' of this research project is
anthropological, the ``backend'' is much closer to
artificial intelligence.  On this level, the project is
about understanding \emph{effective} means for solving
human problems.  Often this will involve decomposing
events and processes into constituent elements, making
increasingly detailed treatments along the lines described
in Note \ref{arxana}.
\end{notate}

\begin{notate}{The relationship between text and commentary}
Text under revision might be marked up by a copyeditor: in
cases like these, the interpretation is clear.  However,
what about marginalia with looser interpretations?  These
seem to become part of the copy of the text they are
attached to.  What about steering processes applied to a
given course of action?  How about the relationship of
thoughts or words to perception and action?  How can we
lower the barrier between conception and action, while
still maintaining some purchase on wisdom?

You see, a lot of issues in life have to do with overlays,
multi-tracking, interchange between different systems; and
in these terms, a lot of philosophy reduces to ``media
awareness'' which extends into more and more immediate
contexts (Note \ref{theoretical-context}).
\end{notate}

\begin{notate}{Heuristic flow}
Continuing the notion above: one does not need a
fully-developed ``heading'' of work in order to do work --
instead, one wants some straightforward heuristics that
will enable the desired work to get done.  So, even
supposing the work is ``heading building'', it can progress
without becoming overwhelmed in abstractions -- because
theories and heuristics are different things.
\end{notate}

\begin{notate}{Limits of simple languages} \label{simple-languages}
Triples are frequently ``subject, verb, object''
statements, although with the annotation features, we can
modify any part of any such statement; for example, we
can apply an adverb to a given verb.

``Tags'', of course, already provide ``subject,
predicate'' relationships.  It will be interesting to
examine the degree to which human languages can be mapped
down into these sorts of simple languages.  What features
are needed to make such languages \emph{useful}?  (Lisp's
`car' and `cdr' seem related to the idea of making
predicates useful.)

How are triples and predicates ``enough''?  What, if
anything, do they lack?  The difference between triples
and predicates illustrates the issue.  How should we
characterize Arxana's additions to Lisp?
\end{notate}

\begin{notate}{Higher dimensions}
Why stop with three components?  Why not have $(A, B, C,
D, T)$ represent a semantic relationship between all of
$A$, $B$, $C$, and $D$ (in heading $T$, of course)?
Actually, there is no reason to stop apart from the fact
that I want to explore simple languages (Note
\ref{simple-languages}).  In real life, things are not as
simple, and we should be ready to deal with the
complexities! (Cf., for example, Note \ref{pointing}).
\end{notate}

\section{Future plans}

\begin{notate}{Development pathways}
To the extent that it's possible, I'd like to maintain a
succinct non-linear roadmap in which tasks are outlined
and prioritized, and some procedural details are made
concrete.  Whenever relevant this map should point into
the current document.  I'll begin by revising the plans
I've used so far!\footnote{{\tt
    http://metameso.org/files/plan-arxana.pdf}} Over the
next several months, I'd like to see these plans develop
into a genuine production machine, and see the machine
begin to stabilize its operations.
\end{notate}

\begin{notate}{Theories as database objects} \label{theories-as-database-objects}
We're just beginning to treat theories as database
objects; I expect there will be more work to do to make
this work really well.  We'll want to make some test
cases, like building a ``theory of chess'', or even just
describing a particular chess board; cf. Note
\ref{partial-image}.
\end{notate}

\begin{notate}{Search engine/elements} \label{search-engine}
One of the features that came very easy in the Emacs-only
prototype was textual search.  With the strings stored in
a database, Sphinx seems to be the most suitable search
engine to use.  It is tempting to try to make our own
inverted index using triples, so that text-based search
can be even more directly integrated with semantic search.
(Since the latest version(s) of Sphinx can act to some
extent like a MySQL database, we almost have a direct
connection in the backend, but since Sphinx is not
\emph{the same} database, one would at least need some
glue code to effect joins and so forth.)

More to the point, it is important for this project that
the scholia-based document model be transparently extended
down to the level of words and characters.  It may be
helpful to think about text as \emph{always being}
hypertext; a document as a heading; and a word in the
inverted index as a frame.
\end{notate}

\begin{notate}{Pointing at database elements and other things} \label{pointing}
We will want to be able to point at other tables and at
other sorts of objects and make use of their contents.
The plan is that our triples will provide a sort of guide
or backbone superimposed over a much larger data system.
\end{notate}

\begin{notate}{Feature-chase}
There are lots of different features that could be
explored, for example: multi-dimensional history lists; a
useful treatment of ``clusions''; MS Word-like colorful
annotations; etc.  Many of these features are already
prototyped.\footnote{See footnote \ref{old-version}.}
\end{notate}

\begin{notate}{Regression testing}
Along with any major feature chase, we should provide
and maintain a regression testing suite.
\end{notate}

\begin{notate}{Deleting and changing things}
How will we deal with unlinking, disassociating,
forgetting, entropy, and the like?  Changes can perhaps
be modeled by an insertion following a deletion, and,
as noted, we'll need effective ways to represent and
manage change (Note \ref{change}).
\end{notate}

\begin{notate}{Tutorial}
Right now the system is simple enough to be pretty much
self-explanatory, but if it becomes much more complicated,
it might be helpful to put together a simple guide to some
likely-to-be-interesting features.
\end{notate}

\begin{notate}{Computing possible paths and connections}
If we can find all the \emph{direct} paths from one node
to another using `triples-given-beginning-and-end', can we
inject some algorthms for finding longer, indirect paths
into the system, and find ways to make them useful?

Similarly, we can satisfy local conditions (Note
\ref{satisfy-conditions}), but we'll want to deal with
increasingly ``non-local'' conditions (even just using the
logical operator ``or'', instead of ``and'', for example).
\end{notate}

\begin{notate}{Monster Mountain}
In Summer 2007, we checked out the Monster Mountain MUD
server\footnote{{\tt http://code.google.com/p/mmtn/}},
which would enable several users to interact with one
LISP, instead of just one database.  This would have a
number of advantages, particularly for exploring
``scholiumific programming'', but also towards fulfilling
the user-to-user interaction objective stated in Note
\ref{theoretical-context}. I plan to explore this after
the primary goal of multi-user interaction with the
database has been solidly completed.
\end{notate}

\begin{notate}{Web interface}
A finished web interface may take a considerable amount of
work (if the complexity of an interesting Emacs interface
is any indication), but the basics shouldn't be hard to
put together soon.
\end{notate}

\begin{notate}{Parsing input} \label{parsing}
Complicated objects specified in long-hand (e.g. triples
pointing to triples) can be read by a relatively simple
parser -- which we'll have to write!  The simplest goal
for the parser would be to be able to distinguish between
a triple and a string -- presumably that much isn't hard.
And of course, building complexes of triples that
represent statements from natural language is a good
long-term goal. (Right now, our granularity level is set
much higher.)
\end{notate}

\begin{notate}{Choice of database} \label{choice-of-database}
I expect Elephant\footnote{{\tt
    http://common-lisp.net/project/elephant/}} may become
our preferred database at some point in the future; we are
currently awaiting changes to Elephant that make nested
queries possible and efficient.  Some core queries related
to managing a database of semantic links with the current
Elephant were constructed by Ian Eslick, Elephant's
maintainer.\footnote{{\tt
    http://planetx.cc.vt.edu/\~{}jcorneli/arxana/variant-4.lisp}}

On the other hand, it might be reasonable to use an Emacs
database and redo the whole thing to work in Emacs
(again), e.g. for single-user applications or users who
want to work offline a lot of the time.
\end{notate}

\begin{notate}{Different kinds of theories}
Theories or variants thereof are of course already popular
in other knowledge representation contexts.\footnote{{\tt
    http://www.cyc.com/cycdoc/vocab/mt-expansion-vocab.html}}$^{,}$\footnote{{\tt
    http://www.stanford.edu/\~{}kdevlin/HHL\_SituationTheory.pdf}}
We'll want to adopt some useful techniques for knowledge
management as soon as the core systems are ready.

Various notions of a mathematical theory
exist.\footnote{{\tt
    http://planetmath.org/encyclopedia/Theory.html}} It
would be nice to be able to assign specific logic to
theories in Arxana, following the ``little theories''
design of e.g. IMPS.\footnote{{\tt
    http://imps.mcmaster.ca/manual/node13.html}}
\end{notate}

\section{Conclusion} \label{conclusion}

\begin{notate}{Ending and beginning again}
This is the end of the Arxana system itself; the
appendices provide some ancillary tools, and some further
discussion.  Contributions that support the development of
the Arxana project are welcome.
\end{notate}

\appendix

\section{Appendix: Auto-setup} \label{appendix-setup}

\begin{notate}{Setting up auto-setup}
This section provides code for satifying dependencies and
setting up the program.  This code assumes that you are
using a Debian/APT-based system (but things are not so
different using say, Fedora or Fink; writing a
multi-package-manager-friendly installer shouldn't be
hard).  Of course, feel free to set things up differently
if you have something else in mind!
\end{notate}

\begin{elisp}
(defalias 'set-up 'shell-command)

(defun alternative-set-up (string)
  (save-excursion
    (pop-to-buffer (get-buffer-create "*Arxana Help*"))
    (goto-char (point-max))
    (insert string "\n")))

(defun set-up-arxana-environment ()
  (interactive)
  (if (y-or-n-p
       "Run commands (y) (or just show instructions)? ")
      (fset 'set-up 'shell-command)
    (fset 'set-up 'alternative-set-up))
  (when (y-or-n-p "Install dependencies? ")
    (set-up "mkdir ~/arxana")
    (set-up "cd arxana"))

  (when (y-or-n-p "Download latest Arxana? ")
    (set-up "wget http://metameso.org/files/arxana.tex"))

  (unless (y-or-n-p "Is your emacs good enough?... ")
    (set-up
     (concat "cvs -z3 -d"
             ":pserver:anonymous@cvs.savannah.gnu.org:"
             "/sources/emacs co emacs"))
    (set-up "mv emacs ~")
    (set-up "cd ~/emacs")
    (set-up "./configure && make bootstrap")
    (set-up "cd ~/arxana"))

  (defvar pac-man nil)

  (cond ((y-or-n-p
          "Do you use an apt-based package manager? ")
         (setq pac-man "apt-get"))
        (t (message
            "OK, get Lisp and SQL on your own, then!")))

  (when pac-man
    (when (y-or-n-p "Install Common Lisp? ")
      (set-up (concat pac-man " install sbcl")))

    (when (y-or-n-p "Install Postgresql? ")
      (set-up (concat pac-man " install postgresql"))
      (when (y-or-n-p "Help setting up PostgreSQL? ")
        (save-excursion
          (pop-to-buffer (get-buffer-create "*Arxana Help*"))
          (insert "As superuser (root),
edit /etc/postgresql/7.4/main/pg_hba.conf
make sure it says this:
host all all 127.0.0.1 255.255.255.255 trust
then edit /etc/postgresql/7.4/main/postgresql.conf
and make it say
tcpip_socket = true
then restart:
/etc/init.d/postgresql-7.4 restart
su postgres
createuser username
exit
as username, run
createdb -U username\n")))))

  (when (y-or-n-p "Install SLIME...? ")
    (set-up (concat "cvs -d :pserver:anonymous"
                           ":anonymous@common-lisp.net:"
                           "/project/slime/cvsroot co slime"))
    (set-up
     (concat "echo \";; Added to ~/.emacs for Arxana:\n\n"
             "(add-to-list 'load-path \"~/slime/\")\n"
             "(setq inferior-lisp-program \"/usr/bin/sbcl\")\n"
             "(require 'slime)\n"
             "(slime-setup '(slime-repl))\n\n\""
             "| cat - ~/.emacs > ~/updated.emacs &&"
             "mv ~/updated.emacs ~/.emacs")))

  (when (y-or-n-p "Set up Common Lisp environment? ")
    (set-up "mkdir ~/.sbcl")
    (set-up "mkdir ~/.sbcl/site")
    (set-up "mkdir ~/.sbcl/systems")
    (set-up "cd ~/.sbcl/site")
    (set-up (concat "wget http://files.b9.com/"
                    "clsql/clsql-latest.tar.gz"))
    (set-up "tar -zxf clsql-4.0.3.tar.gz")
    (set-up (concat "wget http://files.b9.com/"
                           "uffi/uffi-latest.tar.gz"))
    (set-up "tar -zxf uffi-1.6.0.tar.gz")
    (set-up (concat "wget http://files.b9.com/"
                           "md5/md5-1.8.5.tar.gz"))
    (set-up "tar -zxf md5-1.8.5.tar.gz")
    (set-up "cd ~/.sbcl/systems")
    (set-up "ln -s ../site/md5-1.8.5/md5.asd .")
    (set-up "ln -s ../site/uffi-1.6.0/uffi.asd .")
    (set-up "ln -s ../site/clsql-4.0.3/clsql.asd .")
    (set-up "ln -s ../site/clsql-4.0.3/clsql-uffi.asd .")
    (set-up (concat "ln -s ../site/clsql-4.0.3/"
                           "clsql-postgresql-socket.asd ."))
    (set-up "ln -s ~/arxana/arxana.asd ."))

  (when (y-or-n-p "Modify ~/.sbclrc so CL always starts Arxana? ")
    (set-up
     (concat "echo \";; Added to ~/.sbclrc for Arxana:\n\n"
             "(require 'asdf)\n\n"
             "(asdf:operate 'asdf:load-op 'swank)\n"
             "(setf swank:*use-dedicated-output-stream* nil)\n"
             "(setf swank:*communication-style* :fd-handler)\n"
             "(swank:create-server :port 4006 :dont-close t)\n\n"
             "(asdf:operate 'asdf:load-op 'clsql)\n"
             "(asdf:operate 'asdf:load-op 'arxana)\n"
             "(in-package arxana)\n"
             "(connect-to-database)\n"
             "(locally-enable-sql-reader-syntax)\n\n\""
             "| cat ~/.sbclrc - > ~/updated.sbclrc &&"
             "mv ~/updated.sbclrc ~/.sbclrc")))

  (when (y-or-n-p "Install Monster Mountain? ")
    (set-up "cd ~/.sbcl/systems")
    (set-up (concat
                    "darcs get http://common-lisp.net/project/"
                    "bordeaux-threads/darcs/bordeaux-threads/"))
    (set-up (concat
                    "svn checkout svn://common-lisp.net/project/"
                    "usocket/svn/usocket/trunk usocket-svn"))
    ;; I've had problems with this approach to setting cclan
    ;; mirror...
    (set-up
     (concat
      "wget \"http://ww.telent.net/cclan-choose-mirror"
      "?M=http%3A%2F%2Fthingamy.com%2Fcclan%2F\""))
    (set-up (concat "wget http://ww.telent.net/cclan/"
                           "split-sequence.tar.gz"))
    (set-up "tar -zxf split-sequence.tar.gz")
    (set-up
     (concat "svn checkout http://mmtn.googlecode.com/"
             "svn/trunk/ mmtn-read-only"))
    (set-up
     "ln -s ~/bordeaux-threads/bordeaux-threads.asd .")
    (set-up "ln -s ~/usocket-svn/usocket.asd .")
    (set-up "ln -s ~/split-sequence/split-sequence.asd .")
    (set-up "ln -s ~/mmtn/src/mmtn.asd .")))
\end{elisp}

\begin{notate}{Postgresql on Fedora}
There are some slightly different instructions for
installing postgresql on Fedora; the above will be
changed to include them, but for now, check them
out on the
web.\footnote{{\tt http://www.flmnh.ufl.edu/linux/install\_postgresql.htm}}
\end{notate}

\begin{notate}{Using MySQL and CLISP instead} \label{backend-variant}
Since my OS X box seems to have a variety of confusing
PostgreSQL systems already installed (which I'm not sure
how to configure), and CLISP is easy to install with fink,
I thought I'd try a different set up for simplicity and
variety.

In order to make it work, I enabled root user on Mac OS X
per instructions on web, and installed and configured
mysql; used a slight modification of the strings table
described previously; download and installed
cffi\footnote{{\tt
    http://common-lisp.net/project/cffi/releases/cffi\_latest.tar.gz}};
changed the definition of `connect-to-database' in
Arxana's utilities.lisp; doctored up my ~/.clisprc.lisp;
and changed how I started Lisp.  Details below.
\end{notate}

\begin{idea}
;; on the shell prompt
sudo apt-get install mysql
sudo mysqld_safe --user=mysql &
sudo daemonic enable mysql
sudo mysqladmin -u root password root
mysql --user=root --password=root -D test
create database joe; grant all on joe.* to joe@localhost
identified by 'joe'

;; in tabledefs.lisp
(execute-command "CREATE TABLE strings (
   id SERIAL PRIMARY KEY,
   text TEXT,
   UNIQUE INDEX (text(255))
);")

;; in ~/asdf-registry/ or whatever you've designated as
;; your asdf:*central-registry*
ln -s ~/cffi_0.10.4/cffi-uffi-compat.asd .
ln -s ~/cffi_0.10.4/cffi.asd .

;; In utilities.lisp
(defun connect-to-database ()
   (connect `("localhost" "joe" "joe" "joe")
            :database-type :mysql))

;; In ~/.clisprc.lisp
(asdf:operate 'asdf:load-op 'clsql)
(push "/sw/lib/mysql/"
CLSQL-SYS:*FOREIGN-LIBRARY-SEARCH-PATHS*)

;; From SLIME prompt, and not in ~/.clisprc.lisp
(in-package #:arxana)
(connect-to-database)
(locally-enable-sql-reader-syntax)
\end{idea}

\begin{notate}{Installing Sphinx}
Here are some tips on how to install and configure
Sphinx.
\end{notate}

\begin{idea}
;; Fedora/Postgresql flavor
yum install postgresql-devel
./configure --without-mysql
  --with-pgsql
  --with-pgsql-libs=/usr/lib/pgsql/
  --with-pgsql-includes=/usr/include/pgsql

;; Fink/MySQL flavor
./configure --with-mysql
  --with-mysql-includes=/sw/include/mysql
  --with-mysql-libs=/sw/lib/mysql
\end{idea}

\begin{notate}{Getting Sphinx set up} \label{sphinx-setup}
Here are some instructions I've used to get Sphinx set
up.
\end{notate}

\begin{notate}{Create a sphinx.conf}
I want a very minimal sphinx.conf, this seems to work.
(We should probably set this up so that it gets written
to a file when the Arxana is set up.)
\end{notate}

\begin{idea}
## Copy this to /usr/local/etc/sphinx.conf when you want
## to use it.

source strings
{
 type            = mysql
 sql_host        = localhost
 sql_user        = joe
 sql_pass        = joe
 sql_db          = joe
 sql_query       = SELECT id, text FROM strings
}

## index definition

index strings
{
 source          = strings
 path            = /Users/planetmath/sphinx/search-testing
 morphology      = none
}

## indexer settings

indexer
{
 mem_limit       = 32M
}

## searchd settings

searchd
{
 listen          = 3312
 listen          = localhost:3307:mysql41
 log             = /Users/planetmath/sphinx/searchd.log
 query_log       = /Users/planetmath/sphinx/searchd_query.log
 read_timeout    = 5
 max_children    = 30
 pid_file        = /Users/planetmath/sphinx/searchd.pid
 max_matches     = 1000
}
\end{idea}

\begin{notate}{Working from the command line}
Then you can run commands like these.
\end{notate}

\begin{idea}
/usr/local/bin/indexer strings
/usr/local/bin/search "but, then"

% mysql -h 127.0.0.1 -P 3307
mysql> SELECT * FROM strings WHERE MATCH('but, then');
\end{idea}

\begin{notate}{Integrating this with Lisp}
Since we can talk to Sphinx via Mysql
protocol, it seems reasonable that we should be able to talk to
it from CLSQL, too.  With a little fussing to get the format
right, I found something that works!
\end{notate}

\begin{idea}
(connect `("127.0.0.1" "" "" "" "3307") :database-type :mysql)
(mapcar (lambda (elt) (floor (car elt)))
  (query "select * from strings where match('text')"))
\end{idea}

\begin{notate}{Some added difficulty with Postgresql}
When I try to index things on the server, I get an
error, as below.  The question is a good one... I'm
not sure \emph{how} postgresql is set up on the server,
actually...
\end{notate}

\begin{idea}
ERROR: index 'strings': sql_connect: could not connect to server:
Connection refused
Is the server running on host "localhost" and accepting
TCP/IP connections on port 5432?
\end{idea}

\section{Appendix: A simple literate programming system} \label{appendix-lit}

\begin{notate}{The literate programming system used in this paper}
This code defines functions that grab all the Lisp
portions of this document, evaluate the Emacs Lisp
sections in Emacs, and save the Common Lisp sections in
suitable files.\footnote{{\tt
    Cf. http://mmm-mode.sourceforge.net/}} It requires
that the \LaTeX\ be written in a certain consistent way.
The function assumes that this document is the current
buffer.

\begin{verbatim}
(defvar lit-code-beginning-regexp
  "^\\\\begin{elisp}\\|^\\\\begin{common}{\\([^}\n]*\\)}")

(defvar lit-code-end-regexp
  "^\\\\end{elisp}\\|^\\\\end{common}")

(defun lit-process ()
  (interactive)
  (save-excursion
    (let ((to-buffer "*Lit Code*")
          (from-buffer (buffer-name (current-buffer)))
          (start-buffers (buffer-list)))
      (set-buffer (get-buffer-create to-buffer))
      (erase-buffer)
      (set-buffer (get-buffer-create from-buffer))
      (goto-char (point-min))
      (while (re-search-forward
              lit-code-beginning-regexp nil t)
        (let* ((file (match-string 1))
               (beg (match-end 0))
               (end (save-excursion
                      (search-forward-regexp
                       lit-code-end-regexp nil t)
                      (match-beginning 0)))
               (match (buffer-substring-no-properties
                       beg end)))
          (let ((to-buffer
                 (if file
                     (concat "*Lit Code*: " file)
                   "*Lit Code*")))
            (save-excursion
              (set-buffer (get-buffer-create
                           to-buffer))
              (insert match)))))
      (dolist
          (buffer (set-difference (buffer-list)
                                  start-buffers))
        (save-excursion
          (set-buffer buffer)
          (if (string= (buffer-name buffer)
                       "*Lit Code*")
              (eval-buffer)
            (write-region (point-min)
                          (point-max)
                          (concat "~/arxana/"
                                  (substring
                                   (buffer-name
                                    buffer)
                                   12)))))
        (kill-buffer buffer)))))
\end{verbatim}
\end{notate}

\begin{notate}{Emacs-export?}
It wouldn't be hard to export the Elisp sections so
that those who wanted to could ditch the literate
wrapper.
\end{notate}

\begin{notate}{Bidirectional updating}
Eventually it would be nice to have a code repository set
up, and make it so that changes to the code can get
snarfed up here.
\end{notate}

\begin{notate}{A literate style}
Ideally, each function will have its own Note to introduce
it, and will not be called before it has been defined.  I
sometimes make an exception to this rule, for example,
functions used to form recursions may appear with no
further introduction, and may be called before they are
defined.
\end{notate}

\section{Appendix: Hypertext platforms} \label{appendix-hyper}

\begin{notate}{The hypertextual canon} \label{canon}
There is a core library of texts that come up in
discussions of hypertext.
\begin{itemize}
% \item (Plato)
\item The Rosetta stone
\item The Talmud (Judah haNasi, Rav Ashi, and many others)
\item Monadology (Wilhelm Leibniz)
\item The Life and Opinions of Tristam Shandy, Gentleman
  (Lawrence Sterne)
\item Middlemarch (George Eliot)
% \item The Gay Science (Freidrich Nietzsche)
% \item (Wittgenstein)
% \item (Alan Turing)
\item The Nova Trilogy (William S. Burroughs)
\item The Logic of Sense (Gilles Deleuze)
% \item Open Creation and its Enemies (Asger Jorn)
\item Labyrinths (Jorge Luis Borges)
\item Literary Machines (Ted Nelson)
% \item Simulation and Simulacra (Jean Baudrillard)
\item Lila (Robert M. Pirsig)
% \item \TeX: the program (Donald Knuth)
\item Dirk Gently's Holistic Detective Agency
  (Douglas Adams)
\item Pussy, King of the Pirates (Kathy Acker)
% \item Rachel Blau DuPlessis,
% \item Emily Dickinson
% \item Gertrude Stein
% \item Zora Neale Hurston
\end{itemize}
At the same time, it is somewhat ironic that none of the
items on this list are themselves hypertexts in the
contemporary sense of the word.  It's also a bit funny
that certain other works (even some by the same authors)
aren't on this list.  Perhaps we begin to get a sense of
what's going on in this quote from Kathleen
Burnett:\footnote{{\tt http://www.iath.virginia.edu/pmc/text-only/issue.193/burnett.193}}
\begin{quote}
``Multiplicity, as a hypertextual principle, recognizes a
  multiplicity of relationships beyond the canonical
  (hierarchical).  Thus, the traditional concept of
  literary authorship comes under attack from two
  quarters--as connectivity blurs the boundary between
  author and reader, multiplicity problematizes the
  hierarchy that is canonicity.''
\end{quote}
It seems quite telling that non-hypertextual canons remain
mostly-non-hypertextual even today, despite the existence
of catalogs, indexes, and online access.\footnote{{\tt
    http://www.gutenberg.org/wiki/Category:Bookshelf}}
\end{notate}

\begin{notate}{A geek's guide to literature}
This title is a riff on Slasov \v{Z}i\v{z}ek's ``A
pervert's guide to cinema''.  Taking Note \ref{canon} as a
jumping-off point, why don't we make a survey of
historical texts from the point of view of an aficionado
of hypertext!  Just what does one have to do to ``get on
the list''?  Just what is ``the hypertextual
perspective''?  And, if \v{Z}i\v{z}ek is correct and we're
to look for the hyperreal in the world of cinematic
fictions -- what's left over for the world of literature?
(Or mathematics?)
\end{notate}

\begin{notate}{The number 3}
This is the number of things present if we count carefully
the items $A$, $B$, and a connection $C$ between them.
[Picture of $A\xrightarrow{C} B$.]

(Or even: given $A$ and $B$, we use Wittgenstein counting,
and \emph{intuit} that $C$ exists as the collection $\{A,
B\}$; after all,
  some connection must exist precisely because we were
  presented with $A$ and $B$ together -- and lest the
  connections proliferate infinitely, we lump them all
  together as one.  [Picture of $A$, $B$,
    with the \emph{frame} labeled $C$.])
\end{notate}

\begin{notate}{Surfaces}
Deleuze talks about a theory of surfaces associated with
verbs and events.  His surfaces represent the evanescence
of events in time, and of their descriptions in language.
An event is seen as a vanishingly-thin boundary between
one state of being and another.

Certainly, a statement that is true \emph{now} may not be
true five minutes from now.  It is easier to think and
talk about things that are coming up and things that have
already happened.  ``Living in the moment'' is regarded as
special or even ``Zen''.

We can begin to put these musings on a more solid
mathematical basis.  We first examine two types of
\emph{interfaces}:
\begin{enumerate}
\item $A\xrightarrow{C} B$, $A\xrightarrow{D} B$,
  $A\xrightarrow{E} B$
  (the interface of $A$ and $B$ across $C$, $D$, and $E$);
\item $A\xrightarrow{C} B$, $D\xrightarrow{C} E$,
  $F\xrightarrow{C} G$
  (the interface of various terms across $C$).
\end{enumerate}
\end{notate}

\begin{notate}{Comic books}
No geek's guide to literature would be complete without
putting comics in a hallowed place.  [Framed picture of
  $A$, $B$ next to framed
  picture of $A$, $B$, $a$.]  What happened?
  $\ddot{\smile}$
\end{notate}

\begin{notate}{Intersecting triples}
Diagrammatically, it is tempting to portray
$(ACB)_{\mathrm{mid}}DE$ as if it was closely related to
$A(CDE)_{\mathrm{beg}}B$, despite the fact that they are
notationally very different.  I'll have to think more
about what this means.
\end{notate}

\section{Appendix: Computational Linguistics} \label{appendix-linguistics}

\begin{notate}{What is this?}
It might be reasonable to make annotating sentences part
of our writeup on hypertext platforms -- but I'm putting
it here for now.  If hypertext is what deals with language
artifacts on the ``bulky'' level (saying, for example,
that a subsection is part of a section, and so on), then
computational linguistics is what deals with the finer
levels.  However, the distinction is in some ways
arbitrary, and many of the techniques should be at least
vaguely similar.
\end{notate}

\begin{notate}{Annotation sensibilities}\label{sense}
We will want to be able to make at least two different
kinds of annotations of verbs.  For example, given the
statement
\begin{itemize}
\item[$S$.] (``Who'' ``is on'' ``first''),
\end{itemize}
I'd like to be able to say
\begin{itemize}
\item[I.](``is on'' ``means'' ``the position of a base runner in baseball'').
\end{itemize}
However, I'd also like to be able to say
\begin{itemize}
\item[II.] (``is on'' ``because'' ``he was walked'').
\end{itemize}
Annotation I is meant to apply to the term ``is on''
itself (in a context that might be more general than just
this one sentence).  If Who is also on steroids, that's
another matter -- as this type of annotation helps make
clear!

Annotation II is meant to apply to the term ``is on''
\emph{as it
  appears in sentence $S$}.  In particular, Annotation II
seems to work best in a context in which we've already
accepted the ontological status of the verb-phrase ``is
on first''.

Whereas Annotation I should presumably exist before
statement $S$ is ever made (and it certainly helps make
that statement make sense), Annotation II is most properly
understood with reference to the fully-formed statement
$S$.  However, Annotation II is different from a statement
like ($S$ ``has truth value'' $F$) in that it looks into
the guts of $S$.
\end{notate}

\begin{notate}{Comparison of places and ontological status} \label{places-and-onto-status}
The difference between (I) a ``global'' annotation, and
(II) the annotation of a specific sentence is analogous to
the difference between (a) relationships between objects
without a place, and (b) relationships between objects in
specific places.  (Cf. Note \ref{sense}: ``global''
statements are of course made ``local'' by the theories
that scope them.)

For example, in a descriptive ontology of research
documents, I might make the ``placeless'' statement,
\begin{itemize}
\item[a.] (``Introduction'' ``names'' ``a section'')
\end{itemize}
On the other hand, the statement
\begin{itemize}
\item[b.] (``Introduction'' ``has subject'' ``American
  History''),
\end{itemize}
seems likely to be about a specific Introduction.  (And
somewhere in the backend, this triple should be expressed
in terms of places!)
\end{notate}

\begin{notate}{Semantics}
In a sentence like
\begin{quote}
(((``I'' ``saw'' ``myself'')$_{\mathrm{mid}}$ ``as if''
  ``through a glass'')$_{\mathrm{beg}}$ ``but'' ``darkly'')
\end{quote}
first of all, there may be different parenthesizations,
and second of all, the semantics of links like ``as if''
and ``but'' may shape, to some extent, the ways in
which we parethesize.
\end{notate}

\section{Appendix: Resource use} \label{appendix-resources}

\begin{notate}{Free culture in action}
I thought it worthwhile to include this quote from
a joint paper with Aaron Krowne:\footnote{See Footnote
\ref{corneli-krowne}.}
\begin{quote}
``[F]ree content typically
  manifests aspects of a common resource as well as an
  open access resource; while anyone can do essentially
  whatever they wish with the content offline, in its
  online life, the content is managed in a
  socially-mediated way.  In particular, rights to
  \emph{in situ} modification tend to be strictly
  controlled. [...]  By finding new ways to support
  freedom of speech within CBPP documents, we embrace
  subjectivity as a way to enhance the content of an
  intersubjectively valued corpus.  In the context of
  ``hackable'' media and maintenance protocols, the
  semantics with which scholia are handled can be improved
  upon indefinitely on a user-by-user basis and a
  resource-wide basis.  This is free culture in action.''
\end{quote}
\end{notate}

\begin{notate}{Learning}
The learner, confronted with a learning resource, or the
consumer of any other information resource (or indeed,
practically any resource whatsoever) may want a chance to
respond to the questions ``was this what you were looking
for?'' and ``did you find this helpful?''.  In some cases,
an independent answer to that question could be generated
(e.g. if a student is seen to come up with a correct
answer, or not).
\end{notate}

\begin{notate}{Connections}
A useful communication goal is to expose some of the
connections between disparate resources.  Some existing
connections may be far more explicit than others.  It's
important to facilitate the making and explicating of
connections by ``third parties'' (Note
\ref{browser-within}).  The search for connections between
ostensibly unrelated things is a key part of both
creativity and learning.  In addition, connecting with
what others are doing is an important part of being a
social animal.
\end{notate}

\begin{notate}{Boundaries}
Notice that the departmentalization of knowledge is
similar to any regime that oversees and administers
boundaries.  In addition to bridging different areas,
learning often involves pushing one's boundaries and
getting out of one's comfort zone.  The ``sociological
imagination'' involves seeing oneself as part of something
bigger; this goes along with the idea of a discourse that
lowers or transcends the boundaries between participants.
Imagination of any form can challenge myopic patterns of
resource use, although there are also myopic fictions
which neglect to look at what's going on in reality!
\end{notate}

\end{document}