gitweb/lib - Use locking to avoid 'cache miss stampede' problem

[git/jnareb-git.git] / gitweb / README

GIT web Interface
=================

The one working on:
  http://git.kernel.org/

From the git version 1.4.0 gitweb is bundled with git.


How to configure gitweb for your local system
---------------------------------------------

See also the "Build time configuration" section in the INSTALL
file for gitweb (in gitweb/INSTALL).

You can specify the following configuration variables when building GIT:
 * GIT_BINDIR
   Points where to find the git executable.  You should set it up to
   the place where the git binary was installed (usually /usr/bin) if you
   don't install git from sources together with gitweb.  [Default: $(bindir)]
 * GITWEB_SITENAME
   Shown in the title of all generated pages, defaults to the server name
   (SERVER_NAME CGI environment variable) if not set. [No default]
 * GITWEB_PROJECTROOT
   The root directory for all projects shown by gitweb. Must be set
   correctly for gitweb to find repositories to display.  See also
   "Gitweb repositories" in the INSTALL file for gitweb.  [Default: /pub/git]
 * GITWEB_PROJECT_MAXDEPTH
   The filesystem traversing limit for getting the project list; the number
   is taken as depth relative to the projectroot.  It is used when
   GITWEB_LIST is a directory (or is not set; then project root is used).
   Is is meant to speed up project listing on large work trees by limiting
   search depth.  [Default: 2007]
 * GITWEB_LIST
   Points to a directory to scan for projects (defaults to project root
   if not set / if empty) or to a file with explicit listing of projects
   (together with projects' ownership). See "Generating projects list
   using gitweb" in INSTALL file for gitweb to find out how to generate
   such file from scan of a directory. [No default, which means use root
   directory for projects]
 * GITWEB_EXPORT_OK
   Show repository only if this file exists (in repository).  Only
   effective if this variable evaluates to true.  [No default / Not set]
 * GITWEB_STRICT_EXPORT
   Only allow viewing of repositories also shown on the overview page.
   This for example makes GITWEB_EXPORT_OK to decide if repository is
   available and not only if it is shown.  If GITWEB_LIST points to
   file with list of project, only those repositories listed would be
   available for gitweb.  [No default]
 * GITWEB_HOMETEXT
   Points to an .html file which is included on the gitweb project
   overview page ('projects_list' view), if it exists.  Relative to
   gitweb.cgi script.  [Default: indextext.html]
 * GITWEB_SITE_HEADER
   Filename of html text to include at top of each page.  Relative to
   gitweb.cgi script.  [No default]
 * GITWEB_SITE_FOOTER
   Filename of html text to include at bottom of each page.  Relative to
   gitweb.cgi script.  [No default]
 * GITWEB_HOME_LINK_STR
   String of the home link on top of all pages, leading to $home_link
   (usually main gitweb page, which means projects list).  Used as first
   part of gitweb view "breadcrumb trail": <home> / <project> / <view>.
   [Default: projects]
 * GITWEB_SITENAME
   Name of your site or organization to appear in page titles.  Set it
   to something descriptive for clearer bookmarks etc.  If not set
   (if empty) gitweb uses "$SERVER_NAME Git", or "Untitled Git" if
   SERVER_NAME CGI environment variable is not set (e.g. if running
   gitweb as standalone script).  [No default]
 * GITWEB_BASE_URL
   Git base URLs used for URL to where fetch project from, i.e. full
   URL is "$git_base_url/$project".  Shown on projects summary page.
   Repository URL for project can be also configured per repository; this
   takes precedence over URLs composed from base URL and a project name.
   Note that you can setup multiple base URLs (for example one for
   git:// protocol access, another for http:// access) from the gitweb
   config file.  [No default]
 * GITWEB_CSS
   Points to the location where you put gitweb.css on your web server
   (or to be more generic, the URI of gitweb stylesheet).  Relative to the
   base URI of gitweb.  Note that you can setup multiple stylesheets from
   the gitweb config file.  [Default: static/gitweb.css (or
   static/gitweb.min.css if the CSSMIN variable is defined / CSS minifier
   is used)]
 * GITWEB_LOGO
   Points to the location where you put git-logo.png on your web server
   (or to be more generic URI of logo, 72x27 size, displayed in top right
   corner of each gitweb page, and used as logo for Atom feed).  Relative
   to base URI of gitweb.  [Default: static/git-logo.png]
 * GITWEB_FAVICON
   Points to the location where you put git-favicon.png on your web server
   (or to be more generic URI of favicon, assumed to be image/png type;
   web browsers that support favicons (website icons) may display them
   in the browser's URL bar and next to site name in bookmarks).  Relative
   to base URI of gitweb.  [Default: static/git-favicon.png]
 * GITWEB_JS
   Points to the location where you put gitweb.js on your web server
   (or to be more generic URI of JavaScript code used by gitweb).
   Relative to base URI of gitweb.  [Default: static/gitweb.js (or
   static/gitweb.min.js if JSMIN build variable is defined / JavaScript
   minifier is used)]
 * GITWEB_CONFIG
   This Perl file will be loaded using 'do' and can be used to override any
   of the options above as well as some other options -- see the "Runtime
   gitweb configuration" section below, and top of 'gitweb.cgi' for their
   full list and description.  If the environment variable GITWEB_CONFIG
   is set when gitweb.cgi is executed, then the file specified in the
   environment variable will be loaded instead of the file specified
   when gitweb.cgi was created.  [Default: gitweb_config.perl]
 * GITWEB_CONFIG_SYSTEM
   This Perl file will be loaded using 'do' as a fallback if GITWEB_CONFIG
   does not exist.  If the environment variable GITWEB_CONFIG_SYSTEM is set
   when gitweb.cgi is executed, then the file specified in the environment
   variable will be loaded instead of the file specified when gitweb.cgi was
   created.  [Default: /etc/gitweb.conf]
 * HIGHLIGHT_BIN
   Path to the highlight executable to use (must be the one from
   http://www.andre-simon.de due to assumptions about parameters and output).
   Useful if highlight is not installed on your webserver's PATH.
   [Default: highlight]


Runtime gitweb configuration
----------------------------

You can adjust gitweb behaviour using the file specified in `GITWEB_CONFIG`
(defaults to 'gitweb_config.perl' in the same directory as the CGI), and
as a fallback `GITWEB_CONFIG_SYSTEM` (defaults to /etc/gitweb.conf).
The most notable thing that is not configurable at compile time are the
optional features, stored in the '%features' variable.

Ultimate description on how to reconfigure the default features setting
in your `GITWEB_CONFIG` or per-project in `project.git/config` can be found
as comments inside 'gitweb.cgi'.

See also the "Gitweb config file" (with an example of config file), and
the "Gitweb repositories" sections in INSTALL file for gitweb.


The gitweb config file is a fragment of perl code. You can set variables
using "our $variable = value"; text from "#" character until the end
of a line is ignored. See perlsyn(1) man page for details.

Below is the list of variables which you might want to set in gitweb config.
See the top of 'gitweb.cgi' for the full list of variables and their
descriptions.

Gitweb config file variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can set, among others, the following variables in gitweb config files
(with the exception of $projectroot and $projects_list this list does
not include variables usually directly set during build):
 * $GIT
   Core git executable to use.  By default set to "$GIT_BINDIR/git", which
   in turn is by default set to "$(bindir)/git".  If you use git from binary
   package, set this to "/usr/bin/git".  This can just be "git" if your
   webserver has a sensible PATH.  If you have multiple git versions
   installed it can be used to choose which one to use.
 * $version
   Gitweb version, set automatically when creating gitweb.cgi from
   gitweb.perl. You might want to modify it if you are running modified
   gitweb.
 * $projectroot
   Absolute filesystem path which will be prepended to project path;
   the path to repository is $projectroot/$project.  Set to
   $GITWEB_PROJECTROOT during installation.  This variable have to be
   set correctly for gitweb to find repositories.
 * $projects_list
   Source of projects list, either directory to scan, or text file
   with list of repositories (in the "<URI-encoded repository path> SP
   <URI-encoded repository owner>" line format; actually there can be
   any sequence of whitespace in place of space (SP)).  Set to
   $GITWEB_LIST during installation.  If empty, $projectroot is used
   to scan for repositories.
 * $my_url, $my_uri
   Full URL and absolute URL of gitweb script;
   in earlier versions of gitweb you might have need to set those
   variables, now there should be no need to do it.
 * $base_url
   Base URL for relative URLs in pages generated by gitweb,
   (e.g. $logo, $favicon, @stylesheets if they are relative URLs),
   needed and used only for URLs with nonempty PATH_INFO via
   <base href="$base_url">.  Usually gitweb sets its value correctly,
   and there is no need to set this variable, e.g. to $my_uri or "/".
 * $home_link
   Target of the home link on top of all pages (the first part of view
   "breadcrumbs").  By default set to absolute URI of a page ($my_uri).
 * @stylesheets
   List of URIs of stylesheets (relative to base URI of a page). You
   might specify more than one stylesheet, for example use gitweb.css
   as base, with site specific modifications in separate stylesheet
   to make it easier to upgrade gitweb. You can add 'site' stylesheet
   for example by using
      push @stylesheets, "gitweb-site.css";
   in the gitweb config file.
 * $logo_url, $logo_label
   URI and label (title) of GIT logo link (or your site logo, if you choose
   to use different logo image). By default they point to git homepage;
   in the past they pointed to git documentation at www.kernel.org.
 * $projects_list_description_width
   The width (in characters) of the projects list "Description" column.
   Longer descriptions will be cut (trying to cut at word boundary);
   full description is available as 'title' attribute (usually shown on
   mouseover).  By default set to 25, which might be too small if you
   use long project descriptions.
 * @git_base_url_list
   List of git base URLs used for URL to where fetch project from, shown
   in project summary page.  Full URL is "$git_base_url/$project".
   You can setup multiple base URLs (for example one for  git:// protocol
   access, and one for http:// "dumb" protocol access).  Note that per
   repository configuration in 'cloneurl' file, or as values of gitweb.url
   project config.
 * $default_blob_plain_mimetype
   Default mimetype for blob_plain (raw) view, if mimetype checking
   doesn't result in some other type; by default 'text/plain'.
 * $default_text_plain_charset
   Default charset for text files. If not set, web server configuration
   would be used.
 * $mimetypes_file
   File to use for (filename extension based) guessing of MIME types before
   trying /etc/mime.types. Path, if relative, is taken currently as
   relative to the current git repository.
 * $fallback_encoding
   Gitweb assumes this charset if line contains non-UTF-8 characters.
   Fallback decoding is used without error checking, so it can be even
   'utf-8'. Value must be valid encoding; see Encoding::Supported(3pm) man
   page for a list.   By default 'latin1', aka. 'iso-8859-1'.
 * @diff_opts
   Rename detection options for git-diff and git-diff-tree. By default
   ('-M'); set it to ('-C') or ('-C', '-C') to also detect copies, or
   set it to () if you don't want to have renames detection.
 * $prevent_xss
   If true, some gitweb features are disabled to prevent content in
   repositories from launching cross-site scripting (XSS) attacks.  Set this
   to true if you don't trust the content of your repositories. The default
   is false.
 * $maxload
   Used to set the maximum load that we will still respond to gitweb queries.
   If server load exceed this value then return "503 Service Unavailable" error.
   Server load is taken to be 0 if gitweb cannot determine its value.  Set it to
   undefined value to turn it off.  The default is 300.
 * $highlight_bin
   Path to the highlight executable to use (must be the one from
   http://www.andre-simon.de due to assumptions about parameters and output).
   Useful if highlight is not installed on your webserver's PATH.
   [Default: highlight]
 * $caching_enabled
   If true, gitweb would use caching to speed up generating response.
   Currently supported is only output (response) caching.  See "Gitweb caching"
   section below for details on how to configure and customize caching.
   The default is false (caching is disabled).


Projects list file format
~~~~~~~~~~~~~~~~~~~~~~~~~

Instead of having gitweb find repositories by scanning filesystem starting
from $projectroot (or $projects_list, if it points to directory), you can
provide list of projects by setting $projects_list to a text file with list
of projects (and some additional info).  This file uses the following
format:

One record (for project / repository) per line, whitespace separated fields;
does not support (at least for now) lines continuation (newline escaping).
Leading and trailing whitespace are ignored, any run of whitespace can be
used as field separator (rules for Perl's "split(' ', $line)").  Keyed by
the first field, which is project name, i.e. path to repository GIT_DIR
relative to $projectroot.  Fields use modified URI encoding, defined in
RFC 3986, section 2.1 (Percent-Encoding), or rather "Query string encoding"
(see http://en.wikipedia.org/wiki/Query_string#URL_encoding), the difference
being that SP (' ') can be encoded as '+' (and therefore '+' has to be also
percent-encoded).  Reserved characters are: '%' (used for encoding), '+'
(can be used to encode SPACE), all whitespace characters as defined in Perl,
including SP, TAB and LF, (used to separate fields in a record).

Currently list of fields is
 * <repository path>  - path to repository GIT_DIR, relative to $projectroot
 * <repository owner> - displayed as repository owner, preferably full name,
                        or email, or both

You can additionally use $projects_list file to limit which repositories
are visible, and together with $strict_export to limit access to
repositories (see "Gitweb repositories" section in gitweb/INSTALL).


Per-repository gitweb configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can also configure individual repositories shown in gitweb by creating
file in the GIT_DIR of git repository, or by setting some repo configuration
variable (in GIT_DIR/config).

You can use the following files in repository:
 * README.html
   A .html file (HTML fragment) which is included on the gitweb project
   summary page inside <div> block element. You can use it for longer
   description of a project, to provide links (for example to project's
   homepage), etc. This is recognized only if XSS prevention is off
   ($prevent_xss is false); a way to include a readme safely when XSS
   prevention is on may be worked out in the future.
 * description (or gitweb.description)
   Short (shortened by default to 25 characters in the projects list page)
   single line description of a project (of a repository). Plain text file;
   HTML will be escaped. By default set to
     Unnamed repository; edit this file to name it for gitweb.
   from the template during repository creation. You can use the
   gitweb.description repo configuration variable, but the file takes
   precedence.
 * cloneurl (or multiple-valued gitweb.url)
   File with repository URL (used for clone and fetch), one per line.
   Displayed in the project summary page. You can use multiple-valued
   gitweb.url repository configuration variable for that, but the file
   takes precedence.
 * gitweb.owner
   You can use the gitweb.owner repository configuration variable to set
   repository's owner. It is displayed in the project list and summary
   page. If it's not set, filesystem directory's owner is used
   (via GECOS field / real name field from getpwiud(3)).
 * various gitweb.* config variables (in config)
   Read description of %feature hash for detailed list, and some
   descriptions.


Gitweb caching
~~~~~~~~~~~~~~

Currently gitweb supports only output (HTTP response) caching, similar
to the one used on http://git.kernel.org.  To turn it on, set 
$caching_enabled variable to true value in gitweb config file, i.e.:

   our $caching_enabled = 1;

You can choose which caching engine should gitweb use by setting $cache
variable to _initialized_ instance of cache interface, e.g.:

   use CHI;
   our $cache = CHI->new( driver => 'Memcached',
        servers => [ "10.0.0.15:11211", "10.0.0.15:11212" ],
        l1_cache => { driver => 'FastMmap', root_dir => '/var/cache/gitweb' }
   );

Alternatively you can set $cache variable to the name of cache class,
e.g.:

   our $cache = 'Cache::FileCache';

In this case caching engine should support Cache::Cache or CHI names for
cache config (see below), and ignore unrecognized options.  Such caching
engine should also implement (at least) ->get($key) and ->set($key, $data)
methods (Cache::Cache and CHI compatible interface).

You can set $cache to 'GitwebCache::SimpleFileCache' if you don't want
to use locking, but then some advanced features, like generating data in
background, wouldn't work because they require locking.

If $cache is left unset (if it is left undefined), then gitweb would use
GitwebCache::FileCacheWithLocking as caching engine.  This engine is 'dumb' (but
fast) file based caching layer, currently without any support for cache size
limiting, or even removing expired / grossly expired entries.  It has
therefore the downside of requiring a huge amount of disk space if there are
a number of repositories involved.  It is not uncommon for git.kernel.org to
have on the order of 80G - 120G accumulate over the course of a few months.
It is therefore recommended that the cache directory be periodically
completely deleted; this operation is safe to perform.  Suggested mechanism
(substitute $cachedir for actual path to gitweb cache):

   # mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush

Site-wide cache options are defined in %cache_options hash.  Those options
apply only when $cache is unset (GitwebCache::SimpleFileCache is used), or
if $cache is name of cache class (e.g. $cache = 'Cache::FileCache').  You
can override cache options in gitweb config, e.g.:

   $cache_options{'expires_in'} = 60; # 60 seconds = 1 minute

Please read comments for %cache_options entries in gitweb/gitweb.perl for
description of available cache options.


Webserver configuration
-----------------------

If you want to have one URL for both gitweb and your http://
repositories, you can configure apache like this:

<VirtualHost *:80>
    ServerName          git.example.org
    DocumentRoot        /pub/git
    SetEnv                      GITWEB_CONFIG   /etc/gitweb.conf

    # turning on mod rewrite
    RewriteEngine on

    # make the front page an internal rewrite to the gitweb script
    RewriteRule ^/$  /cgi-bin/gitweb.cgi

    # make access for "dumb clients" work
    RewriteRule ^/(.*\.git/(?!/?(HEAD|info|objects|refs)).*)?$ /cgi-bin/gitweb.cgi%{REQUEST_URI}  [L,PT]
</VirtualHost>

The above configuration expects your public repositories to live under
/pub/git and will serve them as http://git.domain.org/dir-under-pub-git,
both as cloneable GIT URL and as browseable gitweb interface.
If you then start your git-daemon with --base-path=/pub/git --export-all
then you can even use the git:// URL with exactly the same path.

Setting the environment variable GITWEB_CONFIG will tell gitweb to use
the named file (i.e. in this example /etc/gitweb.conf) as a
configuration for gitweb.  Perl variables defined in here will
override the defaults given at the head of the gitweb.perl (or
gitweb.cgi).  Look at the comments in that file for information on
which variables and what they mean.

If you use the rewrite rules from the example you'll likely also need
something like the following in your gitweb.conf (or gitweb_config.perl) file:

  @stylesheets = ("/some/absolute/path/gitweb.css");
  $my_uri = "/";
  $home_link = "/";


Webserver configuration with multiple projects' root
----------------------------------------------------

If you want to use gitweb with several project roots you can edit your apache
virtual host and gitweb.conf configuration files like this :

virtual host configuration :

<VirtualHost *:80>
    ServerName                  git.example.org
    DocumentRoot                /pub/git
    SetEnv                              GITWEB_CONFIG   /etc/gitweb.conf

    # turning on mod rewrite
    RewriteEngine on

    # make the front page an internal rewrite to the gitweb script
    RewriteRule ^/$             /cgi-bin/gitweb.cgi [QSA,L,PT]

    # look for a public_git folder in unix users' home
    # http://git.example.org/~<user>/
    RewriteRule ^/\~([^\/]+)(/|/gitweb.cgi)?$   /cgi-bin/gitweb.cgi [QSA,E=GITWEB_PROJECTROOT:/home/$1/public_git/,L,PT]

    # http://git.example.org/+<user>/
    #RewriteRule ^/\+([^\/]+)(/|/gitweb.cgi)?$  /cgi-bin/gitweb.cgi [QSA,E=GITWEB_PROJECTROOT:/home/$1/public_git/,L,PT]

    # http://git.example.org/user/<user>/
    #RewriteRule ^/user/([^\/]+)/(gitweb.cgi)?$ /cgi-bin/gitweb.cgi [QSA,E=GITWEB_PROJECTROOT:/home/$1/public_git/,L,PT]

    # defined list of project roots
    RewriteRule ^/scm(/|/gitweb.cgi)?$          /cgi-bin/gitweb.cgi [QSA,E=GITWEB_PROJECTROOT:/pub/scm/,L,PT]
    RewriteRule ^/var(/|/gitweb.cgi)?$          /cgi-bin/gitweb.cgi [QSA,E=GITWEB_PROJECTROOT:/var/git/,L,PT]

    # make access for "dumb clients" work
    RewriteRule ^/(.*\.git/(?!/?(HEAD|info|objects|refs)).*)?$ /cgi-bin/gitweb.cgi%{REQUEST_URI}  [L,PT]
</VirtualHost>

gitweb.conf configuration :

$projectroot = $ENV{'GITWEB_PROJECTROOT'} || "/pub/git";

These configurations enable two things. First, each unix user (<user>) of the
server will be able to browse through gitweb git repositories found in
~/public_git/ with the following url : http://git.example.org/~<user>/

If you do not want this feature on your server just remove the second rewrite rule.

If you already use mod_userdir in your virtual host or you don't want to use
the '~' as first character just comment or remove the second rewrite rule and
uncomment one of the following according to what you want.

Second, repositories found in /pub/scm/ and /var/git/ will be accesible
through http://git.example.org/scm/ and http://git.example.org/var/.
You can add as many project roots as you want by adding rewrite rules like the
third and the fourth.


PATH_INFO usage
-----------------------
If you enable PATH_INFO usage in gitweb by putting

   $feature{'pathinfo'}{'default'} = [1];

in your gitweb.conf, it is possible to set up your server so that it
consumes and produces URLs in the form

http://git.example.com/project.git/shortlog/sometag

by using a configuration such as the following, that assumes that
/var/www/gitweb is the DocumentRoot of your webserver, and that it
contains the gitweb.cgi script and complementary static files
(stylesheet, favicon):

<VirtualHost *:80>
        ServerAlias git.example.com

        DocumentRoot /var/www/gitweb

        <Directory /var/www/gitweb>
                Options ExecCGI
                AddHandler cgi-script cgi

                DirectoryIndex gitweb.cgi

                RewriteEngine On
                RewriteCond %{REQUEST_FILENAME} !-f
                RewriteCond %{REQUEST_FILENAME} !-d
                RewriteRule ^.* /gitweb.cgi/$0 [L,PT]
        </Directory>
</VirtualHost>

The rewrite rule guarantees that existing static files will be properly
served, whereas any other URL will be passed to gitweb as PATH_INFO
parameter.

Notice that in this case you don't need special settings for
@stylesheets, $my_uri and $home_link, but you lose "dumb client" access
to your project .git dirs. A possible workaround for the latter is the
following: in your project root dir (e.g. /pub/git) have the projects
named without a .git extension (e.g. /pub/git/project instead of
/pub/git/project.git) and configure Apache as follows:

<VirtualHost *:80>
        ServerAlias git.example.com

        DocumentRoot /var/www/gitweb

        AliasMatch ^(/.*?)(\.git)(/.*)?$ /pub/git$1$3
        <Directory /var/www/gitweb>
                Options ExecCGI
                AddHandler cgi-script cgi

                DirectoryIndex gitweb.cgi

                RewriteEngine On
                RewriteCond %{REQUEST_FILENAME} !-f
                RewriteCond %{REQUEST_FILENAME} !-d
                RewriteRule ^.* /gitweb.cgi/$0 [L,PT]
        </Directory>
</VirtualHost>

The additional AliasMatch makes it so that

http://git.example.com/project.git

will give raw access to the project's git dir (so that the project can
be cloned), while

http://git.example.com/project

will provide human-friendly gitweb access.

This solution is not 100% bulletproof, in the sense that if some project
has a named ref (branch, tag) starting with 'git/', then paths such as

http://git.example.com/project/command/abranch..git/abranch

will fail with a 404 error.



Originally written by:
  Kay Sievers <kay.sievers@vrfy.org>

Any comment/question/concern to:
  Git mailing list <git@vger.kernel.org>