DOC: moving lilydev build

[lilypond/patrick.git] / Documentation / contributor / administration.itexi

@c -*- coding: utf-8; mode: texinfo; -*-
@node Administrative policies
@chapter Administrative policies

This chapter discusses miscellaneous administrative issues which
don't fit anywhere else.

@menu
* Meta-policy for this document::
* Meisters::
* Administrative mailing list::
* Grand Organization Project (GOP)::
* Grand LilyPond Input Syntax Standardization (GLISS)::
* Unsorted policies::
@end menu

@node Meta-policy for this document
@section Meta-policy for this document

The Contributor's Guide as a whole is still a work in progress,
but some chapters are much more complete than others.  Chapters
which are @qq{almost finished} should not have major changes
without a discussion on @code{-devel}; in other chapters, a
disorganized @qq{wiki-style dump} of information is encouraged.

Do not change (other than spelling mistakes) without discussion:

@itemize

@item
@ref{Introduction to contributing}

@item
@ref{Working with source code}

@end itemize

Please dump info in an appropriate @@section within these manuals,
but discuss any large-scale reorganization:

@itemize

@item
@ref{Compiling}

@item
@ref{Documentation work}

@item
@ref{Issues}

@item
@ref{Regression tests}

@item
@ref{Programming work}


@end itemize

Totally disorganized; do whatever the mao you want:

@itemize

@item
@ref{Website work}

@item
@ref{LSR work}

@item
@ref{Release work}

@item
@ref{Administrative policies}

@end itemize



@node Meisters
@section Meisters

We have four jobs for organizing a team of contributors:

@itemize

@item
Bug Meister: trains new Bug Squad volunteers, organizes who works
on which part of their job, checks to make sure that everything is
running smoothly, and has final say on our policy for Issues.

Currently: Phil

@item
Doc Meister: trains new doc editors/writers, organizes who works
on which part of the job, checks to make sure that everything is
running smoothly, and has final say on our policy for
Documentation.  Also includes LSR work.

Currently: Graham

@item
Translation Meister: trains new translators, updates the
translation priority list, and handles merging branches (in both
directions).

Currently: Francisco

@item
Frog Meister: is responsible for code patches from (relatively)
inexperienced contributors.  Keeps track of patches, does initial
reviewing of those patches, sends them to @code{-devel} when
they've had some initial review on the Frog list, pesters the
@code{-devel} community into actually reviewing said patches, and
finally pushes the patches once they're accepted.  This person is
@emph{not} responsible for training new programmers, because that
would be far too much work -- he job is @qq{only} to guide
completed patches through our process.

Currently: Carl

@end itemize

@node Administrative mailing list
@section Administrative mailing list

An mailing list for administrative issues is maintained at
@code{lilypond-hackers@@gnu.org}.

This list is intended to be used for discussions that should be kept
private. Therefore, the archives are closed to the public.

Subscription to this list is limited to certain senior developers.

At the present time, the list is dormant.

Details about the criteria for membership, the types of discussion
to take place on the list, and other policies for the hackers list
will be finalized during the
@ref{Grand Organization Project (GOP)}.



@node Grand Organization Project (GOP)
@section Grand Organization Project (GOP)

GOP has two goals:

@itemize
@item
Clarify the various development tasks by writing down the polices
and techniques and/or simplifying the tasks directly.

@item
Get more people involved in development: specifically, find people
to do easy tasks to allow advanced developers to concentrate on
difficult tasks.

@end itemize

@menu
* Motivation::
* Ongoing jobs::
* Policy decisions::
@end menu

@node Motivation
@subsection Motivation

Most readers are probably familiar with the LilyPond Grand
Documentation Project, which ran from Aug 2007 to Aug 2008. This
project involved over 20 people and resulted in an almost complete
rewrite of the documentation. Most of those contributors were
normal users who decided to volunteer their time and effort to
improve lilypond for everybody. By any measure, it was a great
success.

The Grand Organization Project aims to do the same thing with a
larger scope -- instead of focusing purely on documentation, the
project aims to improve all parts of LilyPond and its community.
Just as with GDP, the main goal is to encourage and train users to
become more involved.

If you have never contributed to an open-source project before --
especially if you use Windows or OSX and do not know how to
program or compile programs -- you may be wondering if there's
anything you can do. Rest assured that you @emph{can} help.

@subheading "Trickle-up" development

One of the reasons I'm organizing GOP is "trickle-up"
development.  The idea is this: doing easy tasks frees up advanced
developers to do harder tasks.  Don't ask "am I the @emph{best}
person for this job"; instead, ask "am I @emph{capable} of doing
this job, so that the current person can do stuff I @emph{can't}
do?".

For example, consider lilypond's poor handling of grace notes in
conjunction with clef and tempo changes. Fixing this will require
a fair amount of code rewriting, and would take an advanced
developer a few weeks to do. It's clearly beyond the scope of a
normal user, so we might as well sit back and do nothing, right?

No; we @emph{can} help, indirectly. Suppose that our normal user
starts answering more emails on lilypond-user. This in turn means
that documentation writers don't need to answer those emails, so
they can spend more time improving the docs. I've noticed that all
doc writers tackle harder and harder subjects, and when they start
writing docs on scheme programming and advanced tweaks, they start
contributing bug fixes to lilypond. Having people performing these
easy-to-moderate bug fixes frees up the advanced developers to
work on the really hard stuff... like rewriting the grace note
code.

Having 1 more normal user answering emails on lilypond-user won't
have a dramatic trick-up affect all by himself, of course. But if
we had 8 users volunteering to answer emails, 6 users starting to
write documentation, and 2 users editing LSR... well, that would
free up a lot of current bug-fixing-capable contributors to focus
on that, and we could start to make a real dent in the number of
bugs in lilypond. Quite apart from the eased workload, having that
many new helpers will provide a great moral boost!

@node Ongoing jobs
@subsection Ongoing jobs

Although GOP is a short-term project, the main goal is to train
more people to handle ongoing jobs. The more people doing these
jobs, the ligher the work will be, and the more we can get done
with lilypond!

Also, it would be nice if we had at least one "replacement" /
"understudy" for each role -- too many tasks are only being done
by one person, so if that person goes on vacation or gets very
busy with other matters, work in that area grinds to a halt.

@subheading Jobs for normal users

@itemize
@item Consultant:
LilyPond is sometimes critized for not listening to users, but
whenever we ask for opinions about specific issues, we never get
enough feedback. This is somewhat aggravating.
We need a group of users to make a dedicated effort to test and
give feedback. If there's new documentation, read it. If there's
an experimental binary, download it and try compiling a score with
it. If we're trying to name a new command, think about it and give
serious suggestions.

@item lilypond-user support:
I think it would be nice if we had an official team of users
helping other users.

@item LilyPond Report:
Keeping a monthly newsletter running is a non-trivial task.  A lot
of work is needed to organize it; it would be great if we could
split up the work. One person could write the Snippet of the
Month, another person could do Quotes of the Month, another person
could do interviews, etc.

@item Documentation:
Although GDP (the Grand Documentation Project) did great work,
there's still many tasks remaining.

@item Translations:
Keeping the documentation translations is a monumental task; we
need all the help we can get!

@end itemize

@subheading Jobs for advanced users for developers

@itemize
@item Git help for writers:
We often receive reports of typos and minor text updates to the
documentation. It would be great if somebody could create
properly-formatted patches for these corrections.

Technical requirements: ability to run @ref{Lilydev}.

@item LSR editor:
LSR contains many useful examples of lilypond, but some snippets
are out of date and need updating. Other snippets need to be
advertized, and new snippets need to be sorted. We could use
another person to handle LSR.

Technical requirements: use of a web browser. LilyPond
requirements: you should be familiar with most of Notation
chapters 1 and 2 (or be willing to read the docs to find out).

@item Join the Frogs:
"Frogs" are a team of bug-fixers (because frogs eat bugs, and you
often find them in Ponds of Lilies) and new feature implementors.

Technical requirements: development environment (such as
@ref{Lilydev}), ability to read+write scheme and/or C++ code.

@end itemize


@node Policy decisions
@subsection Policy decisions

There are a number of policy decisions -- some of them fairly
important -- which we have been postponing for a few years.  When
GOP begins, we will start discussing them.

@warning{The fact that we are not arguing about them right now is
not, I repeat @strong{not}, an indication that we do not feel that
these issues are not important.  It is simply that if we began
talking about them now, it would postpone the 2.14 release for a
few months.}

Note that the presence of an item on this list does @emph{not}
mean that everybody thinks that something needs to be done.
Inclusion in this simply means that one developer thinks that we
should discuss it.  We are not going to filter this list; if any
developer thinks we should discuss something, just add it to the
bottom of the list.  (the list is unsorted)

Once GOP starts, the list will be sorted into a rough agenda.  We
will probably introduce one topic each week -- yes, it will
therefore take months to get through everything, but we must
balance productive work vs. policy administration.  If we find
that we settle questions faster (or slower) than predicted, we
will of course change the speed of new topic introductions.

There are some item(s) not displayed here; these are questions
that were posed to me privately, and I do not feel justified in
discussing them publicly without the consent of the person(s) that
brought them up. They will initially be discussed privately on the
lilypond-hackers mailing list -- but the first question will be
"do we absolutely need to do this privately", and if not, the
discussion will take place on lilypond-devel like the other items.

In most policy discussions in lilypond over the past few years,
the first half (or more) is wasted arguing on the basis of
incorrect or incomplete data; once all the relevant facts are
brought to light, the argument is generally resolved fairly
quickly.  In order to keep the GOP discussions focused, each topic
will be introduced with a collection of relevant facts and/or
proposals.  It is, of course, impossible to predict exactly which
facts will be relevant to the discussion -- but spending an hour
or two collecting information could still save hours of
discussion.

@warning{The estimated time required for "prep work", and the
following discussion, has been added to each item.  At the moment,
there is an estimated 30 hours of prep work and 135 hours of
discussion.}

@itemize
@item @strong{Patch reviewing}:
At the time of this writing, we have 23 (known) patches waiting
for review.  Some from main developers; some from new developers.
We desperately need more people helping with lilypond, but
ignoring patches is the best way to drive potential contributors
away.  This is not good.

(prep: 2 hours.  discuss: 10 hours)

@item @strong{Lessons from the 2.14 release; future release policy}:
What went well; what went badly? (how) should we change any
policies pertaining to releases? Should an undocumented new
feature count as release-blocking?

(prep: 1 hour.  discuss: 15 hours)

@item @strong{lilypond-hackers mailing list}:
Should we have a private mailing list for senior developers?  If
so, who should be on it?

(prep: 2 hours+3 weeks.  discuss: 10 hours)

@item @strong{Hackers B}:


@item @strong{Code style}:
New contributors sometimes struggle to follow our indentation and
code style -- this is especially difficult when parts of our
existing source code doesn't have a consistent style. This is
problematic... we want new contributors to be struggling with the
lilypond architecture, not playing games in their text editors!
(ok, we don't actually want them to be struggling with lilypond
internals... but given the current state of the CG, it's
understandable, and at any rate it's still better than struggling
with code style)
Speaking academically, C++ code style is a "solved problem". Let's
pick one of the existing solutions (probably either astyle,
uncrustify, or emacs), and let a computer deal with this.

(prep: 5 hours.  discuss: 15 hours)

@item @strong{Git repository(s)}:
We currently have a web/ branch in our main repo; this seems
misleading to new developers. More generally, should we have
branches that aren't related to the master? i.e. should we
restrict a git branch to code which is an actual "branch" of
development? Also, some of our code (notably the windows and osx
lilypad) isn't in a git repository at all.
We can add new repositories very easily; should make repositories
like
@example
git://git.sv.gnu.org/lilypond/gub.git
git://git.sv.gnu.org/lilypond/lilypad.git
git://git.sv.gnu.org/lilypond/misc.git
@end example
? More information here:
@uref{http://code.google.com/p/lilypond/issues/detail?id=980}

(prep: 2 hours.  discuss: 10 hours)

@item @strong{Roadmap of future development}:
Many projects have a roadmap of planned (or desired) future work.
Should we use one? If so, what should go on it, bearing in mind
our volunteer status? Is there any way of having a roadmap that
isn't vaporware?

(prep: 1 hour.  discuss: 5 hours)

@item @strong{Official links to other organizations?}:
There's something called the "software freedom conservancy", and
in general, there's a bunch of "umbrella organizations". Joining
some of these might give us more visibility, possibly leading to
more users, more developers, maybe even financial grants or use in
schools, etc.

(prep: 2 hours.  discuss: 5 hours)

@item @strong{Mailing lists}:
We currently have a mix of official GNU mailing lists and lilynet
lists. Is there a strong rationale for having separate mailing
list servers? Why not pick one place, and put all our lists there?
(or at least, all "permanent" lists?)

(prep: 1 hour.  discuss: 5 hours)

@item @strong{Issue tracking with google code}:
We use the google issue tracker, but this means that we are
relying on a commercial entity for a large part of our
development. Would it be better (safer in the long run) to use the
savannah bug tracker?

(prep: 1 hour.  discuss: 5 hours)

@item @strong{Patch review tool}:
Reitveld is inconvenient in some respects: it requires a google
account, and there's no way to see all patches relating to
lilypond. Should we switch to something like gerritt?
@uref{http://code.google.com/p/lilypond/issues/detail?id=1184}

(prep: 5 hours.  discuss: 15 hours)

@item @strong{Subdomains of *.lilypond.org}:
Unless Jan has a really weird DNS hosting setup, there are no
technical barriers to having names like lsr.lilypond.org,
frog.lilypond.org, or news.lilypond.org. Is this something that we
want to do?

(prep: 1 hours+2 weeks.  discuss: 5 hours)

@item @strong{Authorship in source files}:
Our documentation currently does not attempt to track individual
authors of each file, while our source code makes a confused and
jumbled attempt to track this. A number of guidelines for F/OSS
projects explicitly recommends _not_ tracking this in individual
files, since the code repository will track that for you.

(prep: 2 hours.  discuss: 15 hours)

@item @strong{Clarity for sponsorships}:
We currently do not advertize bounties and sponsorships on the
webpage.  How much advertising do we want, and what type?
Should we change the "structure" / "framework" for bounties?

(prep: 2 hours.  discuss: 10 hours)

@item @strong{Separate branches for active development}:
it might be good to have @emph{everybody} working on separate
branches.  This complicates the git setup, but with sufficient
logic in lily-git.tcl, we can probably make it transparent to
newbies.  However, we'd need a reliable person to handle all the
required merging and stuff.

(prep: 2 hours.  discuss: 10 hours)

@item @strong{Precise definition of Critical issues}:
at the moment, a stable release is entirely dependent on the
number of Critical issues, but there's some questions about
precisely what a "Critical issue" should be.  We should clarify
this, in conjunction with a general discussion about how often we
want to have stable releases, how permissive we want to be about
patches, etc etc.

(prep: 1 hour.  discuss: 5 hours)

@end itemize



@node Grand LilyPond Input Syntax Standardization (GLISS)
@section Grand LilyPond Input Syntax Standardization (GLISS)

@subheading Summary

@itemize
@item
Start: sortly after 2.14 comes out, which is currently estimated
to happen in January 2011.

@item
Length: 6-12 months.  We're not going to rush this.

@item
Goal: define an input which we commit to being
machine-updateable for the forseeable future.  Any future patches
which change the syntax in a non-convert-ly-able format will be
rejected.  (subject to the limitations, below)
Once this is finished, we will release lilypond 3.0.

@end itemize


@subheading The Problem

One of the biggest complaints people have with lilypond -- other
than silly thing like "there's no gui" -- is the changing syntax.
Now, inventing a language or standards is difficult.  If you set
it in stone too soon, you risk being stuck with decisions which
may limit matters.  If you keep on updating the syntax,
interaction with older data (and other programs!) becomes complex.

@subheading Scope and Limitations

@itemize
@item
tweaks will not be included.  Anything with \override, \set,
\overrideProperty, \tweak, \revert, \unset... including even those
command names themselves... is still fair game for NOT_SMART
convert-ly updates.

@item
other than that, everything is on the table.  Is it a problem to
have the tagline inside \header?  What should the default behavior
of \include be?  When we abolish \times, do we move to \tuplet 3:2
or \tuplet 2/3 or what (for typical triplets in 4/4 time)?

@item
we need to get standards for command names.  This will help users
remember them, and reduce the options for future names (and
potential renamings later on).  \commandOn and \commandOff seem to
work well (should we *always* have an Off command?), but what
about the "command" part?  Should it be \nounVerbOn, or
\verbNounOn ?  Or \verbNotesWithExtraInformationOn ?

@item
we need standards for the location of commands.  Ligature
brackets, I'm looking at you.  (non-postfix notation must die!)

@item
this Grand Project doesn't affect whether we have a 2.16 or not.
The main problem will be deciding what to do (with a bit of
messiness anticipated for \tuplet); we should definitely release a
2.16 before merging _any_ of these changes.

@item
we obviously can't /guarantee/ that we'll /never/ make any
non-convert-ly changes in the basic format.  But we *can*
guarantee that such changes would force lilypond 4.0, and that we
would only do so for overwhelmingly good reasons.

@end itemize

@subheading Workflow

@itemize
@item
We're going to have lots and lots of emails flying around.  The
vast majority won't really fit into either -devel or -user, so
we'll use a list devoted to syntax issues.

@item
Once we have a serious proposal that gained general acceptance
from the separate syntax mailing list, I'll bring it to -devel.
We're not going to make any changes without discussing it on
-devel, but if we're going to have huge threads about English
grammar and silly ideas, and I don't want to clutter up -devel.
Once whatever chaotic silliness on the syntax list is settled
down, I'll bring the ideas to -devel.

@item
as with GDP, I'll moderate the discussion.  Not as with mailist
moderation, but rather by introducing issues at specific times.
We don't want a free-for-all discussion of all parts of the syntax
at once; nothing will get resolved.

@item
Whenever possible, we'll decide on policies at the highest level
of abstraction.  For example, consider \numericTimeSignature,
\slurUp, \xNotesOn, \startTextSpan, and \verylongfermata.  One of
them starts with the name of the notation first (slur).  One has
an abbreviation (x instead of cross).  One has the verb at the end
(On), another has it at the beginning (start).  The adjective can
come at the beginning (numeric, x) or end (Up).  Most are in
camelCase, but one isn't (verylongfermata).

@item
Instead of arguing about each individual command, we'll decide on
abstract questions.  Should each command begin the notation-noun,
or the verb?  Should all commands be in camelCase, or should we
make everything other than articulations in camelCase but make
articulations all lower-case?  Are abbreviations allowed?

@item
Once we've answered such fundamental questions, most of the syntax
should fall into place pretty easily.  There might be a few odd
questions left ("is it a span, or a spanner?"), but those can be
settled fairly quickly.

@end itemize

@subheading Implementation

Nothing until the project is finished, then we declare the next
stable release (2.16.0 or 2.18.0 ?) to be the final 2.x version,
release it, then apply all the GLISS syntax changes and start
testing a beta for 3.0 a week or two later.

@subheading Discussion

Don't respond to any of the specifics yet.  Yes, we all have our
pet irritations (like "what's up with \paper and \layout?!").
There will be plenty of time to discuss them once GLISS starts.

That said, we have a list of specific items that people really
wanted to have written down.  See @ref{Specific GLISS issues}.

@menu
* Specific GLISS issues::
@end menu


@node Specific GLISS issues
@subsection Specific GLISS issues

@itemize
@item
add regtests for every piece of syntax (not one-command-per-file,
but making a few files which, between them, use every single piece
of syntax.)  This is a great test for convert-ly.

@item
should GLISS cover suggested conventions?  (indentation,
one-bar-per-line, etc -- the kind of stuff we list for the
lilypond formatting in the docs ?)

@item
how much (if any) syntactic sugar should we add?  i.e.
@example
  \instrumentName #'foo
% instead of
  \set Staff.instrumentName
@end example
?  Carl: maybe yes, Neil: no.  (for example, it fails for
pianostaff)

@item
the values that are used as arguments to common used overrides.
Sometimes they are a symbol (e.g. #'around), sometimes a
predefined variable referring to a Scheme value or object (e.g.
#LEFT, #all-visible ). The main trouble is that for novice users
it is not clear when there should be an apostrophe and when not.

@item
When do we need -\command and when is it just \command ?


@item
Command-line options to the lilypond binary.  -dfoo counts as a
tweak; we won't be trying to pin those down.

@item
@verbatim
\layout {
  \context { \Score
% vs.
\layout {
  \context {
    \Score
@end verbatim

@item
If would be pedagogically simpler to realize this difference if
the syntax was separate if you define a context from scratch (as
is the case with \RemoveEmptyStaffContext) or if it's defined by
adding onto an existing context. For example, a syntax like

@verbatim
\context{
 % Copy the current settings of the Staff context:
 \use Staff
 % do whatever additional settings
}
%%% could be used to distinguish from
\context{
 % Take settings from a variable:
 \Variable
 % do whatever additional settings
}

%%% and

\context{
 % Start from scratch:
 \type ...
 \name ...
 \consists ...
 ...
}
@end verbatim

@item
Capitalization of identifiers: \VoiceOne ?

@item
@verbatim
%%% Allow
{ music expression } * 4
%%% instead of
\repeat unfold 4 { music expression }
@end verbatim

?  patch here:
@uref{http://lists.gnu.org/archive/html/lilypond-devel/2010-04/msg00467.html}

@item
Personally, I find it easier to understand when there's a repeated
8 in the half-bar position; it's much easier to see that you have
two groups of 4:

@example
c8 c c c c8 c c c
%%% instead of one group of eight:
c8 c c c c c c c
@end example

@item
trivially simple bar-lines:

c1 | c1 |

encourage, allow, or discourage, or disallow?

@item
indentation of \\ inside a @{@} construct.


@item
barline checks at the end of line should be preceded by at least 2
spaces?  barline checks should line up if possible (i.e.  if you
can use less than 4, 8, X empty spaces before a barline check to
make them line up?)

@item
Why doesn't \transpose respect \relative mode?


@item
on \score vs. \new Score

But in the light of a consistent syntax and semantic, I see no
reason (from the users POV) to disallow it.  After all, the real
top-level context is a \book @{@}, isn't it, and I don't see a point
in disallowing a \new Score construct just like \new Staff.

From a syntactical POV, I see the following pros for \new Score:
- You can write \with @{ ... @} for every other context but \Score,
which (for consistency) should also work with \new Score.
- When there's a \new Foo Bar, there's also a \context Foo Bar,
  which makes the same as a parallel instantiation of all Bar's.
- [Quoting Rune from
@uref{http://www.mail-archive.com/lilypond-devel@@gnu.org/msg14713.html}
  "I know that the \score-statement is a syntactical construct,
but I think it would be nice to hide this fact from the users.  I
think we could make the use of score-block much more intuitive if
changing the syntax to \new \Score and adding an implicit
sequential-statement to the score."


@item
Discussion on
http://code.google.com/p/lilypond/issues/detail?id=1322
about \new vs. \context.


@item
Let users add their own items to the parser?  comment 11 on:
http://code.google.com/p/lilypond/issues/detail?id=1322

@item
should engravers be pluralized (note_heads_engraver) or not
(note_head_engraver) ?

@end itemize


@node Unsorted policies
@section Unsorted policies

@subsubheading Language-specific mailing lists

A translator can ask for an official lilypond-xy mailing list once
they've finished all @qq{priority 1} translation items.

@subsubheading Performing yearly copyright update (@qq{grand-replace})

At the start of each year, copyright notices for all source files
should be refreshed by running the following command from the top of
the source tree:

@example
make grand-replace
@end example

Internally, this invokes the script @file{scripts/build/grand-replace.py},
which performs a regular expression substitution for old-year -> new-year
wherever it finds a valid copyright notice.

Note that snapshots of third party files such as @file{texinfo.tex} should
not be included in the automatic update; @file{grand-replace.py} ignores these
files if they are listed in the variable @code{copied_files}.


@subsubheading Push git access

Git access is given out when a contributor has a significant
record of patches being accepted without problems.  If existing
developers are tired of pushing patches for a contributor, we'll
discuss giving them push access.  Unsolicited requests from
contributors for access will almost always be turned down.