Add simple echo CGI script.

[newgopher.git] / public_gopher / newgopher.txt

# New Gopher

## Prerequisites

This  proposal  will  use  many terms as used in the original Gopher RFC
1436.

## Encoding

All  strings  sent  have  to be in UTF‐8, except for »binary« or »data«.
These are binary data streams and should be handled like that. No Byte-
Order-Mark (BOM) must be sent.

### URI

URIs  do not allow full UTF‐8 encoding. If for example punycode is used,
then the client has to convert them into UTF‐8, so  new  gopher  servers
will understand it. UTF-8 should be enforced there.

Because  of  the  different type behaviour do the URIs operate without a
item type. See the section on how to determine the file type  below  for
further details.

## Request

Requests  can be sent over every transport protocol, which can guarantee
the data stream to be correct. No error detection is done in this proto‐
col.  The main intention for this protocol is it to be used over TCP/IP,
using mandatory TLS encryption. TLS should be used over other  transport
protocols too.

### Simple

        C->S: selector\r\n
        S->C: $size\t$mime-type\r\n
              [data]

The  $size  line  is defining the number of bytes that will follow after
the »$size\t$mime‐type\r\n« line. If $size  is  »‐1«,  then  the  client
should listen until the connection is closed.

The $mime-type is defining what data will follow.

### Complex

        C->S: selector\tsearch\trange\r\n
              $size\t$mime-type\r\n
              [data]
        S->C: $size\t$mime-type\r\n
              [menu/binary/metadata]

In a complex request the »\t« in the first retrieve line is seen as flag
to retrieve a further line that will tell the server how much  data  the
client   will send and what kind of data this will be ($mime‐type).  The
server may apply rules or selector‐specific rules how much data  is  al‐
lowed to be retrieved by a client.

The  $size  line  is defining the number of bytes that will follow after
the »$size\t$mime‐type\r\n« line. If $size  is  »‐1«,  then  the  client
should  listen  until the connection is closed. When clients are sending
data, then they are required to specify a valid size.

Adding  another »\t« to the request allows the client to specify a range
of bytes to be retrieved. See the section about ranges for  further  de‐
tails.

### Pipelining

Every  connection  to  a server could be used for many requests. Servers
should implement this, but can apply rules upon how  many  requests  and
what requests are allowed when pipelining multiple requests.

It is possible for simple servers to close the connection after each re‐
quest. Simple clients can close the  connection  after  simple  requests
too.  This  may  disable  certain interactive sessions for these kind of
clients.

One exception in pipelining is when a selector is has an undefined size.
Then the connection must be closed after all data has been sent  to  the
client.  Both  sides  of the connection can close the connection, when a
certain rule maximum of bytes is reached.

### Selectors

Selectors   are  UTF‐8  strings.  They must not contain the »\0« or »\t«
charac‐ ters, to avoid parsing errors. Servers need »\t«  for  determin‐
ing,   if   the request is a complex request. See the sections below for
clarification.

Selectors  must  use  »/« as path delimiter. This definition is required
for automagic file type recognition.

Relative  selectors  (containing a »..« or ».« path element) must not be
handled by a server and answered with an error.

### Search

If  a  »search«  part is given in a request, then this is used when some
form of magic script will handle the answer. See the CGI section for how
to use it further.

### Range

The default value of if »range« part is »‐«, which means to retrieve the
selector from its beginning to the end. If the »range« part has  a  zero
length, then the default value is assumed.

        $beginning-$end

As seen, $beginning and $end can be of zero length and will be replaced
by »0« ($beginning) or the size of the selector data ($end).

Both, $beginning and $end must be positive natural numbers.

        -$end
        $beginning-

These  are  possible  formats and mean, that the client is requesting in
the first case the byte range from the first byte of the data  (byte  0)
to  byte  $end; including byte $end. The second example will send to the
client byte $beginning to the end of the data.

Multiple byte ranges are best served using pipelining and multiple range
requests. There is no support for them in one request.

If a range cannot be satisfied, see the section about Error Handling how
to handle this.

A range of zero bytes can be used to determine the size and mime‐type of
a selector.

### Mime-Type

There are some special MIME types required for this document.

        text/x-menu – menu
        application/x-interactive – interactive 

The  short forms of menu item types must not be used when sending or re‐
questing a selector.

## Menu

Each  menu  item  consists of one line. The line must be valid UTF‐8 and
consists of the following form.

        $mime-type\t$name\t$selector\t$searchstring\t$host\t$port\r\n

$mime‐type  and  $name are mandatory for every line. The other parts can
be left out, if unspecified and will result in the below  specified  de‐
faults  to  be  applied. If a menu type does not have any use for a menu
item part, then it can be left out and ignored.

For $mime-type see the section about them below.

$name is referring to the text that should be displayed at this position
in the menu.

$selector specifies the selector that should be sent to the server, then
this is a line pointing back to a  server  implementing  this  protocol.
See the different menu item types below for exceptions.

$searchstring is what should be sent to the server in a complex query as
the »search« part. This implies the  client  to  implement  complex  re‐
quests.

$host  and  $port  are used when this item type is referring to a server
implementing this protocol. If they are empty, then  the  host  or  port
used to retrieve this menu is used as a replacement.

### Menu Item MIME types

The  used  $mime‐type fields in this protocol can refer not just to MIME
types. The history of gopher has shown short item types to be more prac‐
tical,  but  insufficent to handle modern needs. This proposal here is a
mix out of both worlds.

If  the  length  of  the $mime‐type is just one char, then the following
item types are assumed:

        i – informational
        m – menu
        s – interactive
        u – uri
        e – error
        b – binary

If the $mime‐type is longer than one char, then this is a true MIME type
and will be used like that.

#### Informational

This type is describing text in the $name field that should be displayed
at that position in the menu.

#### Menu 

The selector of this item will lead to another menu.

When requesting this selector, then the $mime‐type of »text/x‐menu« must
be used.

#### Interactive 

See the section about search / interactive sessions how to handle these.

When requesting this selector, then the $mime‐type of »application/x‐in‐
teractive« must be used.


#### Uri 

When  displaying  this  menu item issuing/clicking it should lead to the
URI mentioned in the $selector field and the text  in  the  $name  field
should  be displayed as a description. Handling URIs must be done on the
client side.

#### Error 

This  menu  item  is  describing  an error message. The item must be the
first item in the menu. See the section on error handling for  more  de‐
tails.

#### Binary

Implementing this menu item can be done on server or client side. If the
server supports this, then it should parse the menus before sending them
and  try to determine the real MIME type of the file the selector is re‐
ferring to. If this is not possible, then the client has to decide.

If  the  client  is  supporting this menu type and the »b« type appears,
then it should assume this to be of type »application/octet‐stream«  and
simply store the file.

### Search / Interactive

The client is retrieving the selector where the interactive sessions be‐
gins:

        C->S: /somemenu\r\n
        S->C: [...]
              s\tPlease Register!\t/register\r\n

A  client  not  willing to interpret this as an interactive session will
just retrieve the raw data. Clients willing to  interact  interpret  the
following in a new request:

        C->S: /register\r\n
        S->C: $size\ts\r\n
              s\tNext register step\t/register2\t$metadata\r\n
              Username: \n
              Password: \n
              E-Mail: \n
        [The user is now shown its favourite text editor, which shows
         all the given back text, except for the first line. ]

The  first  line  of  the output is a menu item, which the client should
follow after the user applied its changed to the data below it.

Data  following this first line can be binary and application‐dependent,
but it must be valid UTF‐8, if it is text. See the  section  about  sim‐
plicity below for how to handle this.

When  showing  the  data  to the user and it is a text interaction, then
filters could apply on the application side or the client side,  to  add
some  more descriptive text, that tells the user, that it is possible to
abort everything by changing nothing and just leaving the editor.  Other
binary applications may have different rules.

Base  on  the  example, another sub standard could be used to assume the
form of »Field: Value« to be a form entry, which  is  shown  in  a  more
graphical way, so users won’t change the key text.

Clients  should  apply  filters on redirects to menu items away from the
server the interactive session started. It is adviced to not follow them
by default.

The answer could be:

        C->S: /register2\t$metadata\r\n
              $size\ts\r\n
              Username: fee\n
              Password: bar\n
              E-Mail: me@somewhere.com\n
        S->C: $size\tm\r\n
              i\tWelcome to this service!\r\n
              m\tNext >>\t/\r\n

When  sending the answer, the client is not sending a line for the redi‐
rect. It is sending back the changed data from the user.

Interactive  sessions may end from the client or server side. The server
side must guide the user to a sane end whenever possible.

### Simplicity

Authors   creating interactive sessions are encouraged to keep them sim‐
ple.  It should be possible to access them using commandline tools.  The
best way to ensure this is to keep all interaction in valid UTF‐8 and an
understandable form. Binary formats make things hard to  parse  and  use
without specialized tools.

## Error Handling

When  an  error  occurs, then the server is sending back a menu with the
first element being of type »e«.

Upon errors the server is not required to close the connection, if it is
not a fatal error that would prevent further serving.

Error  messages  must  not be localized and should contain at least some
standard messages. The parsing is done using simple string operations.

Some predefined error messages that should be used are:

        Selector not found
        Permission denied
        Range cannot be satisfied
        Unhandled CGI error
        Malformed request

If the server encounters a relative selector:

        Relative selectors are not allowed

When the client did not send enough data:

        Wrong size

The chameleon of error messages is

        Internal server error

Sending  out  this error message is allowed but should be avoided at all
costs.

Example:

        C->S: /someselector\r\n
        S->C: $size\tm\r\n
              e\tFile not found\r\n
              m\tBack to the index\t/\r\n

## File Types

If  just  a server and selector is given to a client and it tries to de‐
termine how to handle the data behind this selector, following algorithm
should be used.

1. See if the mime-type is cached somewhere.
2. If the selector is the root of the server, just fetch it and abort,
   if this is not a type of »m«.
3. Use a request with the range of »0-0« to determine the MIME type
   and size.
4. If the MIME type is still »b«, then assume »application/octet-stream«.


## CGI

If  scripts  are called within a server or a request the following envi‐
ronment variables should be provided in the  operating  system  specific
way.

By  default  all  variables  are  unset. Only if relevant information is
available they should be set.

The  value  of  these variables should be in UTF‐8, unless the operating
system would make using this default impractical. Then the possible  de‐
velopers must be informed about this new assumption.

### CONTENT_LENGTH

The size of the data sent by the client.

### CONTENT_TYPE

The  mime‐type the client sent for the request. The default mediatype is
UTF‐8. This incompatibility to HTTP CGI is intentional.

### GATEWAY_INTERFACE

»CGI/1.1«. This is not really used.

### PATH_INFO

This is the selector given by the client.

### PATH_TRANSLATED

This  si the PATH_INFO mapped to the storage where the server is getting
the script from. It does not need to be a valid physical path.

### QUERY_STRING

This should be the »search« part of a complex request.

### REMOTE_ADDR

This  must be the address of the client requesting an answer in the com‐
mon used representation for such an address. If the protocol has no  no‐
tion of an address, it can be left blank.

### REMOTE_HOST

REMOTE_HOST  can  contain  the fually qualified host of the REMOTE_ADDR.
The limitations of the used protocol are applied,  so  it  can  be  left
blank.

### REQUEST_METHOD

If  the client is providing data, then this is seen as a »POST« request.
If no data is provided, then it is a »GET« request.


### SCRIPT_NAME

This is the filename or leading path component of the PATH_INFO.

### SERVER_NAME

SERVER_NAME  is  the  hostname or address of the server. If the protocol
this protocol is used on has no notion of address, it must be set  to  a
funny name.

### SERVER_PORT

This  is  the  port on which the request was received. It should be left
blank, if no port notion is known in the underlying protocol.

### SERVER_PROTOCOL

»GOPHER/3.0«

### SERVER_SOFTWARE

This  variable describes the server and must include the server software
name and version. If no version is known, a funny name must be added.