Merge /pub/main

[educational.data.git] / AI01 / rbp / readme

                     Basis of AI Backprop\r
                   Code from April 10, 1996\r
               Documentation from April 10, 1996\r
\r
Copyright (c) 1990-96 by Donald R. Tveter\r
\r
CONTENTS\r
--------\r
1. Introduction\r
2. Making the Simulators\r
3. A Simple Example\r
4. Basic Facilities\r
5. The Format Command\r
6. Taking Training and Testing Patterns from a File\r
7. Saving and Restoring Weights\r
8. Initializing Weights\r
9. The Seed Values\r
10. The Algorithm Command\r
11. The Delta-Bar-Delta Method\r
12. Quickprop\r
13. Making a Network\r
14. Recurrent Networks\r
15. Miscellaneous Commands\r
16. Limitations\r
17. The Pro Version Additions\r
\r
1. Introduction\r
---------------\r
   This manual describes the free version of my Basis of AI Backprop\r
designed to accompany my not yet published (sigh) textbook, _The Basis\r
of AI_.  This program contains enough features for students in an\r
ordinary AI or Neural Networking course.  More serious users will\r
probably need the professional version of this software, see:\r
\r
http://www.mcs.com/~drt/probp.html\r
\r
or send me email at: drt@mcs.com.  Other free NN software for the\r
textbook is also available at:\r
\r
http://www.mcs.com/~drt/svbp.html\r
\r
   For more on backprop see my "Backpropagator's Review" at:\r
\r
http://www.mcs.com/~drt/bprefs.html\r
\r
   Notice: this is use at your own risk software.  There is no guarantee\r
that it is bug-free.  Use of this software constitutes acceptance for\r
use in an as is condition.  There are no warranties with regard to this\r
software. In no event shall the author be liable for any damages\r
whatsoever arising out of or in connection with the use or performance\r
of this software.\r
\r
   There are four simulators that can be constructed from the included\r
files.  The program, bp, does back-propagation using real weights and\r
arithmetic.  The program, ibp, does back-propagation using 16-bit\r
integer weights, 16 and 32-bit integer arithmetic and some floating\r
point arithmetic.  The program, sbp, uses symmetric floating point\r
weights and its sole purpose is to produce weights for two-layer\r
networks for use with the Hopfield and Boltzman relaxation algorithms\r
(included in another package).  The program sibp does the same using\r
16-bit integer weights.  The integer versions are faster on systems\r
without floating point hardware however sometimes these versions don't\r
have enough range or precision and then using the floating point\r
versions is necessary.  DOS binaries are included here for systems with\r
floating point hardware.  If you need other versions write me.\r
\r
2. Making the Simulators\r
------------------------\r
   This code has been written to use either 32-bit floating point\r
(float) or 64-bit floating point (double) arithmetic.  On System V\r
machines the standard seems to be that all floating point arithmetic is\r
done with double precision arithmetic so double arithmetic is faster\r
than float and therefore this is the default.  Other versions of C (e.g.\r
ANSI C) will do single precision real arithmetic which will ordinarily\r
be faster on most machines (I think).  To get 32-bit floating point set\r
the compiler flag FLOAT in the makefile.  The function, exp, defined in\r
real.c is double since System V specifies it as double.  If your C uses\r
float, change this definition as well.\r
\r
   For UNIX systems, use either makefile.unx or makereal.unx.\r
The makefile.unx will make any of the programs and makefile will keep\r
the bp object code files around while makereal.unx will only make bp\r
but it keeps the bp object code files around.  Also for DOS systems\r
there are two makefiles to choose from, makefile and makereal.  Makefile\r
is designed to make all four programs but it only leaves around the\r
object files for ibp while erasing object files for sibp, sbp and bp.\r
On the other hand, makereal only makes bp and it leaves its object\r
files around.  For 16-bit DOS you need to set the flag, -DDOS16 and for\r
32-bit DOS, you need to set the flag -DDOS32.  The flags I have in the\r
DOS makefiles are what I use with Zortech C 3.1.  The code is known not\r
to compile with at least one version of Turbo C because of an oddity\r
(or bug?) in the compiler.\r
\r
   There was a problem found with the previous free student version\r
where it crashed on a Sun when the program hits a call to free in the\r
file bp.c.  This can be solved by removing the two calls to free and the\r
amount of space you waste is minimal.  I haven't had a report of such a\r
problem with this version yet but if it happens, let me know, in all\r
probability removing a call or two to free in the file io.c will solve\r
the problem.\r
\r
   This code will work with basic C compilers however the libraries\r
sometimes vary from system to system.  DOS systems seem to use the\r
function getch in the conio library for hot key capability.  For a\r
System V UNIX system the code uses a home-made function called getch for\r
hot key capability.  This is the default setting for a UNIX system and\r
it also works with Suns.  If you use BSD UNIX then you need to define\r
the compiler variable BSD either in the cc command by adding the\r
parameter, -DBSD.  To get the hotkey feature to work with a NeXT use the\r
parameter -DNEXT.  At this point I don't know what other variations of\r
UNIX use so you may need to adapt the ioctl function call in the file\r
io.c and the files rbp.h and ibp.h to make them fit some other version.\r
If your system uses some other standard then if you can send me the\r
documentation I should be able to make it work as well.  If necessary\r
the hot key option can be removed by removing or commenting out the\r
line:\r
\r
#define HOTKEYS\r
\r
in the rbp.h and ibp.h files.\r
\r
   There are some other more minor options that can be compiled in or\r
left out but these are mentioned at other points in the documentation.\r
\r
   To make a particular executable file use the makefile given with the\r
data files and make any or all of them like so:\r
\r
        UNIX                        DOS\r
\r
    make -f makereal.unx bp       make -f makereal bp\r
    make -f makefile.unx bp       make bp\r
    make -f makefile.unx ibp      make ibp\r
    make -f makefile.unx sibp     make sibp\r
    make -f makefile.unx sbp      make sbp\r
\r
If you do get bugs on an odd system and you can let me telnet in to\r
your system (preferably on a separate login, rather than your personal\r
login) I will try and fix the problem for you.\r
\r
\r
3. A Simple Example\r
-------------------\r
   Each version would normally be called with the name of a file to read\r
commands from, as in:\r
\r
bp xor\r
\r
After the data file is read commands are then taken from the keyboard.\r
When no file name is specified bp will take commands from the keyboard\r
(stdin file).  Normally you will find it convenient to put the commands\r
you need to set up the network in a short file however it is possible to\r
type them all in to the program from the keyboard.  If you have more\r
than a tiny amount of data you should have the data ready in a training\r
file and a testing file if you have test data.\r
\r
   The commands are one or two or three letter commands and most of them\r
have optional parameters.  The `a', `d', `f' and 'q' commands allow a\r
number of sub-commands on a line.  The maximum length of any line is 256\r
characters.  An `*' is a comment and it can be used to make the\r
remainder of the line a comment.  In addition ctrl-R will run the\r
training.\r
\r
   Here is an example of a data file to do the xor problem:\r
           \r
* input file for the xor problem\r
           \r
m 2 1 1 x     * make a 2-1-1 network with extra input-output connections\r
s 7           * seed the random number function\r
ci            * clear and initialize the network with random weights\r
\r
rt {          * read training patterns into memory\r
1 0 1\r
0 0 0\r
0 1 1\r
1 1 0}\r
\r
e 0.5         * set eta, the learning rate to 0.5 (and eta2 to 0.5)\r
a 0.9         * set alpha, the momentum to 0.9\r
\r
First in this example, the m command will make a network with 2 units in\r
the input layer, 1 unit in the second layer and 1 unit in the third\r
layer.  Much of the time a three layer network where the connections are\r
only between adjacent layers is as complex as a network needs to be\r
however there are problems where having additional connections between\r
the input units and output units will greatly speed-up the learning\r
process.  The xor problem is one of those problems where the extra\r
connections help so the 'x' at the end of the command will add these\r
two extra connections.  The `s' (seed) command sets the seed for the\r
random number function.  The "ci" command (clear and initialize) clears\r
the existing network weights and initializes the weights to random\r
values between -1 and +1.  The rt (read training set) command gives four\r
new patterns to be read into the program.  All of them are listed\r
between the curly brackets ({}).  The input pattern comes first followed\r
by the output pattern.  The command "e 0.5" sets eta, the learning\r
rate for the upper layer to 0.5 and eta2 for the lower layers to 0.5 as\r
well.  The last line sets alpha, the momentum parameter, to 0.9.\r
\r
   After these commands are executed the following messages and prompt\r
appears:\r
\r
Basis of AI Backprop (c) 1990-96 by Donald R. Tveter\r
   drt@mcs.com - http://www.mcs.com/~drt/home.html\r
               April 10, 1996 version.\r
taking commands from stdin now\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? q\r
\r
The characters within the square brackets are a list of the possible\r
commands.  To run 100 iterations of back-propagation and print out the\r
status of the learning every 10 iterations type "r 100 10" at the\r
prompt:\r
\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? r 100 10\r
\r
This gives:\r
 \r
running . . .\r
   10      0.00 % 0.49947   \r
   20      0.00 % 0.49798   \r
   30      0.00 % 0.48713   \r
   40      0.00 % 0.37061   \r
   50      0.00 % 0.15681   \r
   59    100.00 % 0.07121    DONE\r
\r
The program immediately prints out the "running .  .  ." message.  After\r
each 10 iterations a summary of the learning process is printed giving\r
the percentage of patterns that are right and the average value of the\r
absolute values of the errors of the output units.  The program stops\r
when the each output for each pattern has been learned to within the\r
required tolerance, in this case the default value of 0.1.  Sometimes\r
the integer versions will do a few extra iterations before declaring the\r
problem done because of truncation errors in the arithmetic done to\r
check for convergence.  Unlike the previous student version the default\r
for these values is to be "up-to-date" however this can be over-ridden\r
to save a little on CPU time.\r
\r
   There are many factors that affect the number of iterations needed\r
for a network to converge.  For instance if your random number function\r
doesn't generate the same values as the one with the Zortech 3.1\r
compiler (which is the same one used by most UNIX C compilers) the\r
number of iterations it takes will be different.  The integer versions\r
produce slightly different results that the floating point versions.\r
\r
Listing Patterns\r
\r
   To get a listing of the status of each pattern use the `p' command\r
to give:\r
\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? p\r
    1  0.903  e 0.097 ok\r
    2  0.050  e 0.050 ok\r
    3  0.935  e 0.065 ok\r
    4  0.072  e 0.072 ok\r
   59    (TOL) 100.00 % (4 right  0 wrong)  0.07121 err/unit\r
\r
The number folloing the e (for error) is the sum of the absolute values\r
of the output errors for each pattern.  An `ok' is given to every\r
pattern that has been learned to within the required tolerance.  To get\r
the status of one pattern, say, the fourth pattern, type "p 4" to give:\r
\r
 0.07  (0.072) ok\r
\r
To get a summary without the complete listing use "p 0".  To get the\r
output targets for a given pattern, say pattern 3, use "o 3".\r
\r
   A particular test pattern can be input to the network by giving the\r
pattern at the prompt:\r
\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? 1 0\r
       0.903 \r
\r
Examining Weights\r
\r
   It is often interesting to see the values of some particular weights\r
in the network.  To see a listing of all the weights in a network you\r
can use the save weights command described later on and then list the\r
file containing the weights, however, to see the weights leading into a\r
particular node, say the node in row 3, node 1 use the w command as in:\r
\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? w 3 1\r
\r
layer unit  inuse  unit value    weight   inuse   input from unit\r
  1     1     1      1.00000     5.38258     1        5.38258\r
  1     2     1      0.00000    -4.86238     1        0.00000\r
  2     1     1      1.00000   -10.86713     1      -10.86710\r
  3     b     1      1.00000     7.71563     2        7.71563\r
                                              sum =   2.23111\r
\r
This listing also gives data on how the current activation value of the\r
node is computed using the weights and the activations values of the\r
nodes feeding into unit 1 of layer 3.  The `b' unit is the bias (also\r
called the threshold) unit.  The inuse column to the right of the unit\r
column is 1 when the unit is in use and 0 if it is not in use.  In this\r
free version there are no commands to take weights out of use.  A 1\r
indicates a regular weight in use and a 2 indicates a bias weight in\r
use.\r
\r
   Besides saving weights you can save all the parameters to a file\r
with the save everything command as in:\r
\r
   se saved\r
\r
At the same time the weights will be written to the current weights\r
file.  The file saved is virtually the same as the one you get with the\r
'?' command.  To start over from where you left off you can use:\r
\r
   bp saved\r
\r
and this also reads in the patterns and weights.  This command DOES NOT\r
save training and testing patterns since normally you would have them\r
in a file of their own.\r
\r
   To get a short online tutorial on how to use the program you can type\r
T at the command prompt and get the listing:\r
\r
   A Tutorial\r
\r
   The following topics are designed to be read in the order listed.\r
\r
   To get help on a topic type the code on the right at the prompt.\r
\r
   Understanding the Menus                                         h1\r
   Formatting Data for a Classification Problem                    h2\r
   Formatting Data for Function Approximation                      h3\r
   Formatting Data for a Recurrent Problem                         h4\r
   Making a Network for Classification or Function Approximation   h5\r
   Making a Recurrent Network                                      h6\r
   Reading the Data                                                h7\r
   Setting Algorithms and Parameters                               h8\r
   Running the Program                                             h9\r
   Saving Almost Everything                                        h10\r
   To Quit the Program                                             h11\r
\r
   To end the program the `q' (for quit) command is entered:\r
\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? q\r
\r
\r
4. Basic Facilities\r
-------------------\r
   There are a very large number of parameters that can be set for\r
various algorithms in these programs.  Typing a `?'  will get a compact\r
listing of them all however they are packed rather tight.  To get a\r
better view of the parameters there are now many upper-case letter\r
commands that give a listing of parameters in a less compact form.\r
These screens list parameters, generally on the left of the screen in\r
the form of the commands you would need to set them.  The center of the\r
screen gives a short description of the parameter.  Sometimes one or two\r
lines are inadequate to describe the command so at the far right there\r
may be a sequence you can type to get more help with the command.\r
\r
   The most important screen you can look at is the C for commands\r
screen that summarizes what each menu screen will show:\r
\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? C\r
\r
Screen     Includes Information and Parameters on:\r
\r
  A        algorithm parameters and tolerance\r
  C        this listing of major command groups\r
  D        delta-bar-delta parameters\r
  F        formats: patterns, output, paging, copying screen i/o\r
  G        gradient descent (plain backpropagation)\r
  M        miscellaneous commands: shell escape, seed values, clear,\r
           clear and initialize, quit, kick a network, run command,\r
           save almost everything\r
  N        network building: making a network, initializing\r
           a network, kicking a network\r
  P        pattern commands: reading patterns, testing patterns,\r
  Q        quickprop parameters\r
  T        a short tutorial\r
  W        weight commands: listing, saving, restoring\r
  ?        a compact listing of everything\r
\r
One typical menu screen is the A screen that lists the main algorithm\r
parameters:\r
\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? A\r
\r
Algorithm Parameters\r
\r
a a <char>     sets all act. functions to <char>; {ls}             h aa\r
a ah s         hidden layer(s) act. function; {ls}                 h aa\r
a ao s         output layer act. function; {ls}                    h aa\r
a d d          the output layer derivative term; {cdf}             h ad\r
a i -          initializes units before using the training set; {+-}\r
a u p          the weight update algorithm; {Ccdpq}                h au\r
t 0.100        tolerance/unit for successful learning; (0..1)\r
\r
f O -          allows out-of-date statistics to print; {+-}\r
f u -          compute up-to-date statistics; {+-}\r
\r
The first of these listings is the line:\r
\r
a a <char>     sets all act. functions to <char>; {ls}             h aa\r
\r
which doesn't give a parameter value but instead it gives the pattern of\r
a command designed to set the activation function for the entire\r
network.  The first sequence is:\r
\r
a a <char>\r
\r
and this sequence will change the activation function but when you type\r
it in you will have to substitute a character code for the activation\r
function instead of the string <char>.  One other activation function is\r
the linear activation function denoted by the character l, so to get\r
this function you can type in the line:\r
\r
a a l\r
\r
The first `a' codes for the "algorithm" command, the second `a' codes\r
for the "activation function" and s is the letter for the function.  The\r
idea of putting the variable portion of the command within the angle\r
brackets (<>) is a notation devised by Computer Scientists to describe\r
computer languages.  The word inside these brackets describes the kind\r
of thing that is the variable portion of the command.\r
\r
   The middle part of the line:\r
\r
a a <char>     sets all act. functions to <char>; {ls}             h aa\r
\r
gives a short description of the meaning of the command and within the\r
curly brackets there is a listing all the values for all the activation\r
functions, {ls}.  To get a more detailed explanation of the options type\r
the sequence on the right: `h aa', this gives:\r
\r
and the following comes up:\r
\r
   a a <char> sets every activation function to <char>.\r
   a ah <char> sets the hidden layer activation function to <char>.\r
   a ao <char> sets the output layer activation function to <char>,\r
   <char> can be any of the following:\r
\r
   <char>                  Function                           Range\r
\r
      l    linear function, x                             (-inf..+inf)\r
      s    standard sigmoid, 1 / (1 + exp(-x))               (0..+1)\r
\r
Here you get the code for the function, the function and the range of\r
values the function can take on.  This range portion following another\r
standard of notation used by Mathematicians.  A ( or ) next to a number,\r
say 0, means the range runs very close to 0 but never exactly to 0, in\r
other cases (not shown above) a [ or ] next to a number means value can\r
range up to exactly the number.  Thus the range:\r
\r
(0..+1]\r
\r
meaning that the range can run from ALMOST EXACTLY 0 up to exactly +1.\r
\r
   If we now return to the A screen, the second line was:\r
\r
a ah s         hidden layer(s) act. function; {ls}                 h aa\r
\r
Here the idea is to indicate that the activation function for the hidden\r
layer (or layers) of the network IS NOW the s (standard sigmoid\r
function).  Again there is a short explanation of this, the set of codes\r
for functions and information about how to get more help.  This line can\r
also be taken as a direction as to how to set the hidden layer\r
activation function as well.  To change it to l you can type in:\r
\r
a ah l\r
\r
(Note: normally you would only use the linear activation function in the\r
output layer of a network.)\r
\r
   The third line:\r
\r
a ao l         output layer act. function; {ls}                    h aa\r
\r
is similar except it states that the activation function for the output\r
layer is l.\r
\r
\r
Paging\r
\r
   In the student version the paging was a simple version of the System\r
V utility, pg.  Now the paging is more like the common UNIX more\r
command.  The default page size is 24 lines and it can be reset to\r
another value with the format command's paging size sub-command.  For\r
instance to get 12 lines / page instead of 24, use:\r
\r
f P 12\r
\r
To get no paging at all use:\r
\r
f P 0\r
\r
When the page is full you get the prompt:\r
\r
More?\r
\r
At this point you can type:\r
\r
   q                  to quit viewing the text if you are in a loop,\r
   a blank            to get another page,\r
   ^D                 to get another half a page,\r
   a carriage return  to get one more line and\r
   c                  to continue without paging.\r
\r
Mostly paging is needed for loops within the program, like running a\r
large number of iterations and printing the results, listing the values\r
of all the patterns or listing weights leading into a particular unit.\r
Typing the q quits these loops, however paging can also occur with some\r
of the longer screen menus that are generating lines of output without\r
running a loop.  For these cases the q does not work.\r
\r
   Every new command entered from the keyboard sets the page counting\r
variable to 0 however if input is being taken from a file other than\r
stdin the counter is NOT reset.  Most of the time this doesn't matter\r
since the little data files like the xor example used to set up\r
parameters don't produce any output anyway, however if they do paging is\r
in effect.  Having paging here is helpful in case there is a problem\r
with reading the files.\r
\r
Interrupts\r
\r
   In UNIX entering an interrupt will stop the current command and the\r
program will give the user another prompt.  With DOS entering a ctrl-C\r
will generate a similar kind of interrupt however DOS only checks for\r
this condition when it has to do i/o.  However when the DOS version is\r
in a training loop the program also checks to see if a key has been hit\r
and if that key is the the escape key, the program will break the\r
training loop.\r
\r
Control Command\r
\r
   One control-key command is available in this version, hitting ctrl-R\r
will run the training algorithm, it is a shorthand for typing r followed\r
by a carriage return.\r
\r
Passing Commands to the Operating System\r
\r
   By using the '!' command you can pass commands to the operating\r
system from within the program.  The kind of typical things you might\r
want to do are to list the contents of a directory, list a file or after\r
saving weights to a file you might want to list them or even edit them\r
and read them back in.  Here is what you can say for DOS to list the\r
little data file xor:\r
\r
! type xor\r
\r
Once a string has been defined with a ! command it can be re-run simply\r
by typing the ! followed immediately by a carriage return.\r
\r
Making a Copy of Your Session\r
\r
   Sometimes you may want to make of copy of everything that you type in\r
and the program prints out.  For instance you may get exceptionally good\r
or bad results using a certain training sequence and an exact record of\r
what you and the program did could be worth having.  Or you may need an\r
exact copy of the training or testing set values.  Or you may need lots\r
of runs where you average the results using another program.  To turn on\r
the making of a copy use the format command to turn on the copying\r
process:\r
\r
f c+\r
\r
and to turn it off use:\r
\r
f c-\r
\r
The text is written to the file copy.\r
\r
An Alphabetical Listing of the Commands\r
\r
   The following listing is designed to give you an idea of the set of\r
commands available.  Details are given in later sections.\r
\r
a <number>       sets the momentum parameter, alpha\r
a <options>      the algorithm command\r
c                clear the network\r
ci               clear and initialize the network\r
d <options>      set delta-bar-delta parameters\r
e <number(s)>    set the learning rate eta\r
f <options>      lots of formatting options\r
h <string>       gives more help with certain options\r
l <layer>        list the values of units on that layer\r
m <numbers>      make a network\r
o <number>       list the output targets of the training set pattern\r
p <options>      list information about training patterns\r
q                quit\r
qp               set quickprop parameters\r
r <options>      run the training algorithm\r
rt <options>     read the training set patterns\r
rw <filename>    read the weights\r
rx <filename>    read the extra training set patterns\r
s <seeds>        seed value\r
sb <real>        set bias weights\r
se <filename>    save almost everything\r
sw <filename>    save weights\r
t <options>      list testing file statistics of various sorts\r
t <real>         tolerance per output unit that must be met\r
tf <filename>    gives the file name with testing patterns\r
tr <int>         special test for a recurrent network\r
trp <int>        special test for a recurrent network\r
w <layer> <unit> list weights leading into unit\r
\r
The Summary Line\r
\r
   The default setting for the summaries you get produce up to\r
date statistics on the error and on how many patterns are correct.\r
Here are several lines of summaries from a problem that has training\r
data and test data for a classification problem:\r
\r
   10      0.00 %  49.04 % 0.47087       0.00 %  62.50 % 0.38234 \r
   20      0.00 %  73.08 % 0.38584       0.00 %  77.88 % 0.38108 \r
   30      2.88 %  76.92 % 0.35043       4.81 %  79.81 % 0.33285 \r
\r
The first column is of course the number of iterations, the next column\r
gives the percentage of training patterns that are correct based on the\r
tolerance.  The next column gives the percentage of correct training\r
patterns based on the maximum value of the output units.  The next\r
column is the average absolute value of the error per output unit.  Note\r
that many other programs will report the RMS error.  The columns on the\r
right list the percentage of test set patterns that are correct based on\r
tolerance, the percentage correct based on maximum value and finally the\r
average error on the test set.\r
\r
   Some CPU time can be saved by altering certain parameter settings\r
that skip some of the forward passes used to determine the current set\r
of statistics.  The more often you print the statistics the more time\r
you can save by altering these parameter settings.  The penalty is that\r
the statistics will be out of date by one iteration with the quickprop\r
delta-bar-delta, supersab and regular periodic update methods and only\r
approximate for both the right and wrong continuous update methods.\r
\r
   In all the update methods the training set statistics are computed\r
when the program passes back the error.  However then an update of the\r
weights is done and these numbers are out of date.  So if 100 iterations\r
have been done the program only has the statistics on iteration 99, the\r
values that were true before the weights were changed.  When it is time\r
to print out the program statistics the default is to do another forward\r
pass through the training set to get up to date statistics.  This can be\r
stopped by setting the off by 1 option in the format command like so:\r
\r
f O+\r
\r
The results that print out will now look like this:\r
\r
   10 -1   0.00 %  64.42 % 0.42463       0.00 %  62.50 % 0.38234 \r
   20 -1   0.00 %  73.08 % 0.39058       0.00 %  77.88 % 0.38108 \r
   30 -1   8.65 %  75.00 % 0.35052       4.81 %  79.81 % 0.33285 \r
\r
where the string "-1" comes right after the number of iterations done\r
and of course it means the numbers shown are for the previous iteration.\r
For the test set patterns one pass through the training set has to be\r
made to get up to date statistics on them so they are always up to date.\r
Most of the time you will probably be more interested in the test set\r
results than in the training set results so setting the off by 1 option\r
saves a little time and getting off by 1 results on the training set\r
is not important.\r
\r
   The situation when you use the "right" and "wrong" continuous update\r
methods is even more complicated.  After the forward pass for one\r
pattern is done it is checked to see if it is right and the error is\r
added in to a sum of errors.  Then weights are changed.  Then another\r
pattern is processed in the same way.  When the weight changes are done\r
for this second pattern they may well ruin the right/wrong decision for\r
all the previous patterns.  Thus the number of right and wrong patterns\r
and the average error can be off by quite a lot.  With the off by 1\r
option off ("f O-") the program still does a forward pass to get the\r
up to date statistics on the training set.  However when the off by 1\r
option is on the statistics look like this:\r
\r
   10 -1   1.92 %  66.35 % 0.42093 ?    31.73 %  40.38 % 0.46316 \r
   20 -1  12.50 %  64.42 % 0.38829 ?    40.38 %  44.23 % 0.53761 \r
   30 -1  34.62 %  75.00 % 0.30939 ?    50.96 %  63.46 % 0.37728 \r
\r
where the ? after the training set error flags the fact that the numbers\r
are very suspect.\r
\r
   Another option in the program is to do an extra forward pass through\r
the training set even when there are no statistics to print out.  The\r
option to give you up to date statistics is:\r
\r
f u+\r
\r
If you are using the periodic update method, quickprop or dbd you don't\r
need "f u+" as the program will report the correct values anyway.\r
\r
   The one line form of the summary is the default but it can be turned\r
off using:\r
\r
f s-\r
\r
With this you get nothing whatsoever and normally you won't want this\r
unless perhaps you are producing your own customized output.\r
\r
5. The Format Command (f)\r
-------------------------\r
   There are several ways to input and output patterns, numbers and\r
other values and there is one format command, `f', that is used to set\r
these options.  In the format command a number of options can be given\r
on a single line as for example in:\r
\r
f b+ ir oc wB\r
\r
Input Patterns\r
\r
   The programs are able to read pattern values in two different\r
formats.  Real numbers follow the C language notation and must be\r
separated by a space.  The letters `H' used in recurrent networks is\r
also allowed.  The letter `x' with a default value of 0.5 is also\r
allowed.  The `x' character has a default value of 0.5.  The value of\r
`x' can be changed, for example to make `x' -1 use:\r
\r
f x -1\r
\r
Real input format is now the default but if you use the other format\r
(a compressed binary format) you can re-set the format to real with:\r
\r
f ir\r
\r
   The other format is the compressed format, a format consisting of 1s,\r
0s and the letters `x' and `H'.  In compressed format each value is one\r
character and it is not necessary to have blanks between the characters.\r
For example, in compressed format the patterns for xor could be written\r
out in either of the following ways:\r
\r
101      10 1\r
000      00 0\r
011      01 1\r
110      11 0\r
\r
The second example is preferable because it makes it easier to see the\r
input and the output patterns.\r
\r
To change to compressed format use:\r
\r
f ic\r
\r
Output of Patterns\r
\r
   Output format is controlled with the `f' command as in:\r
\r
f or   * output node values using real (the C %f) format\r
f oc   * output node values using compressed format\r
f oa   * output node values using analog compressed format\r
f oe   * output values with e notation\r
\r
The first sets the output to real numbers.  The second sets the output\r
to be compressed mode where the value printed will be a `1' when the\r
unit value is greater than 1.0 - tolerance, a `^' when the value is\r
above 0.5 but less than 1.0 - tolerance, a `v' when the value is less\r
than 0.5 but greater than the tolerance.  Below the tolerance value a\r
`0' is printed.  The tolerance can be changed using the `t' command (not\r
a part of the format command).  For example, to make all values greater\r
than 0.8 print as `1' and all values less than 0.2 print as `0' use:\r
\r
t 0.2\r
\r
Of course this same tolerance value is also used to check to see if all\r
the patterns have converged.  The third output format is meant to give\r
"analog compressed" output.  In this format a `c' is printed when a\r
value is close enough to its target value.  Otherwise, if the answer is\r
close to 1, a `1' is printed, if the answer is close to 0, a `0' is\r
printed, if the answer is above the target but not close to 1, a `^' is\r
printed and if the answer is below the target but not close to 0, a `v'\r
is printed.  This output format is designed for problems where the\r
output is a real number, as for instance, when the problem is to make a\r
network learn sin(x).  The format "e" writes out node values using\r
exponential notation with four places to the right of the decimal point.\r
\r
Breaking up the Output Values\r
\r
   In the compressed formats the default is to print a blank after every\r
10 values.  This can be altered using the `B' (for inserting breaks)\r
option within the format ('f') command.  The use for this command is to\r
separate output values into logical groups to make the output more\r
readable.  For instance, you may have 24 output units where it makes\r
sense to insert blanks after the 4th, 7th and 19th positions.  To do\r
this, specify:\r
\r
f B 4 7 19\r
\r
Then for example the output will look like:\r
\r
  1 10^0 10^ ^000v00000v0 01000 e 0.17577\r
  2 1010 01v 0^0000v00000 ^1000 e 0.16341\r
  3 0101 10^ 00^00v00000v 00001 e 0.16887\r
  4 0100 0^0 000^00000v00 00^00 e 0.19880\r
\r
The break option allows up to 20 break positions to be specified.  The\r
default output format is the real format with 10 numbers per line.  For\r
the output of real values the option specifies when to print a carriage\r
return rather than when to print a blank.\r
\r
Pattern Formats\r
\r
   There are two different types of problems that back-propagation can\r
handle, the general type of problem where every output unit can take on\r
an arbitrary value and the classification type of problem where the goal\r
is to turn on output unit i and turn off all the other output units when\r
the pattern is of class i.  The xor problem is an example of the general\r
type of problem.  For an example of a classification problem, suppose\r
you have a number of data points scattered about through two-dimensional\r
space and you have to classify the points as either class 1, class 2 or\r
class 3.  For a pattern of class 1 you can always set up the output:\r
"1 0 0", for class 2: "0 1 0" and for class 3: "0 0 1", however doing\r
the translation to bit patterns can be annoying so another notation can\r
be used.  Instead of specifying the bit patterns you can set the pattern\r
format option to classification (as opposed to the default value of\r
general) like so:\r
\r
f pc\r
\r
and then the program will read data in the form:\r
\r
   1.33   3.61   1   *  shorthand for 1 0 0\r
   0.42  -2.30   2   *  shorthand for 0 1 0\r
  -0.31   4.30   3   *  shorthand for 0 0 1\r
\r
and translate it to the bit string form.  To switch to the general form\r
use "f pg".  Another benefit of the classification format is that when\r
the program outputs a status line it will also include the percentage of\r
correct patterns based on the maximum value rather than just on\r
tolerance.\r
\r
\r
Controlling Summaries\r
\r
   When the program is learning patterns you normally want to have it\r
print out the status of the learning process at regular intervals.  The\r
default is to print out a one-line summary of how learning is going\r
and this is set by using "f s+".  However if you want to customize\r
exactly what is printed out and you don't want the standard summary, use\r
"f s-".\r
\r
Skipping the "running . . ." Message\r
\r
   Normally whenever you run more training iterations the message,\r
"running . . ." prints out to reassure you that something is in fact\r
being done, however this can also be annoying at times.  To get rid of\r
this message use "f R-" and to bring it back use "f R+".\r
\r
Ringing the Bell\r
\r
   To ring the bell when the learning has been completed use "f b+" and\r
to turn off the bell use "f b-".\r
\r
Echoing Input\r
\r
   When you are reading commands from a file it is sometimes worthwhile\r
to see those commands echoed on the screen.  To do this, use "f e+" and\r
to turn off the echoing, use "f e-".\r
\r
Paging\r
\r
   To set the page size to some value, say, 25, use "f P 25" or to skip\r
paging use "f P 0".\r
\r
Making a Copy of Your Session\r
\r
   To make a copy of what appears on the screen use "f c+" to start\r
writing to the file "copy" and "f c-" to stop writing to this file.\r
Ending the session automatically closes this file as well.\r
\r
Up-To-Date Statistics\r
\r
   During the ith pass thru the network the program will collect\r
statistics on how many patterns are correct and how much error there is.\r
It does this so that it will know when to stop the training.  But it\r
gets these numbers BEFORE the weights are changed in the ith pass.  In\r
the case of periodic update methods (the periodic, delta-bar-delta,\r
quickprop and supersab) this is not much of a problem.  If the off by 1\r
flag is off ("f O-") there is another forward pass done whenever the\r
statistics are printed out so you get up to date statistics anyway.  If\r
the off by 1 flag is on ("f O+") you get the string "-1" after the\r
number of iterations is printed on the summary line.  Getting the\r
statistics in the off by 1 form is harmless and it saves a little CPU\r
time.  When the network converges the "-1" flag will not be shown.\r
\r
   However with the continuous update methods the weights are changed\r
after each pattern and this skews the statistics gathered by the\r
training process by quite a lot.  To get an accurate assessment of how\r
well the training is going when results are printed on the summary line\r
you either need to have the off by 1 flag set to "f O+" or you need to\r
set the up to date statistics flag by: "f u+".  The default is to leave\r
this flag off: "f u-".  Furthermore, if you are training to get an\r
accurate assessment of how many iterations it takes to learn the\r
training set you need to set "f u+" (NOT JUST "f O+"!).  The "u+"\r
setting makes a check after every complete pass through the training\r
set.  The "f O+" setting only makes a check when it is time to print\r
the status line.\r
\r
\r
6. Taking Training and Testing Patterns from Files (rt,rx,tf)\r
-------------------------------------------------------------\r
   In the xor example given above the four patterns were part of the\r
data file and to read them in the following lines were used:\r
\r
rt {\r
1 0 1\r
0 0 0\r
0 1 1\r
1 1 0 }\r
\r
However it is also convenient to take patterns from a file that contains\r
nothing but a list of patterns (and possibly comments).  To read a new\r
set of patterns from some file, patterns, use:\r
\r
rt patterns\r
\r
To add an extra group of patterns to the current set you can use:\r
\r
rx patterns\r
\r
To read in test patterns from say the file, xtest, do the following:\r
\r
tf xtest\r
\r
To evaluate all the test patterns without listing them do "t0".  To list\r
them, use "t".  To list one particular test pattern, say pattern 3, do\r
"t 3".\r
\r
\r
7. Saving and Restoring Weights and Related Values (sw,rw,sw+,swe,swem)\r
--------------------------------------------------------------\r
   Sometimes the amount of time and effort needed to produce a set of\r
weights to solve a problem is so great that it is more convenient to\r
save the weights rather than constantly recalculate them.  To save the\r
weights to the current weights file use "sw".  The weights are then\r
written on a file called "weights" or to the last file name you have\r
specified.  The weights file looks like:\r
\r
59r  m 2 1 1 x aahs aos bh 1.000000 bo 1.000000 Dh 1.000000 Do 1.000000  file = ../xor3.new\r
 8.926291e+000  1  1 1 to 2 1\r
-7.945858e+000  1  1 2 to 2 1\r
 3.898432e+000  2  2 b to 2 1\r
 5.382575e+000  1  1 1 to 3 1\r
-4.862383e+000  1  1 2 to 3 1\r
-1.086713e+001  1  2 1 to 3 1\r
 7.715632e+000  2  3 b to 3 1\r
\r
To write the weights the program starts with the second layer, writes\r
out the weights leading into these units in order with the threshold\r
weight last, then it moves on to the third layer, and so on.  In\r
addition to writing out the weights the second column lists whether or\r
not the weights are in use.  If the weight is in use it is marked with a\r
1, if it is a bias unit weight it is marked as 2 and if it is not in use\r
it is marked with a 0.  This is not used in this free version.  The last\r
4 numbers on each line tell which units the weights run between.  The\r
first weight listed runs from layer 1 unit 1 to layer 2 unit 1.  The\r
letter b indicates the weight is a bias unit.  These last 4 values on a\r
line are ignored when the file is read so in fact if you want to make up\r
your own weights file you don't need to type them in.  These last four\r
values are just here for human convenience.  However the inuse values\r
must be present if you write your own weights file.  And you must use\r
only one weight per line.\r
\r
   To restore these weights type `rw' for restore weights.  At this time\r
the program reads the header line and sets the total number of\r
iterations the program has gone through to be the first number it finds\r
on the header line.  It then reads the character immediately after the\r
number.  The `r' indicates that the weights will be real numbers\r
represented as character strings.\r
\r
   The remaining text on the first line of a weight file is not used by\r
the restore weights command at this time and it is there to give you a\r
record of what size and type the network was.  The fact that the rest of\r
this line is not read by the restore weights program means that before\r
you read in weights you have to make the proper size network with the\r
"m" command.  The "m 2 1 1 x" of course means there are 2 units in the\r
first layer, one in the second, one in the third and the x means there\r
are extra connections from the input units to the output unit.\r
Following that the initial command file that was read in is given.\r
\r
   To save weights to a file other than "weights" you can say: "sw\r
<filename>", where, of course, <filename> is the file you want to save\r
to.  To continue saving to the same file you can just do "sw".  If you\r
type "rw" to restore weights they will come from this current weights\r
file as well.  You can restore weights from another file by using: "rw\r
<filename>".  Of course this also sets the name of the file to write to\r
so if you're not careful you could lose your original weights file.\r
\r
\r
8. Initializing Weights (c,ci)\r
------------------------------\r
   All the weights in the network initially start out at 0 and they are\r
also set to 0 by using the clear (c) command.  In some problems where\r
all the weights are 0 the weight changes may cancel themselves out so\r
that no learning takes place.  Moreover, in most problems the training\r
process will usually converge faster if the weights start out with small\r
random values.  To do this use the clear and initialize command as in:\r
\r
ci 0.5\r
\r
where the random initial weights will run from -0.5 to +0.5.  If the\r
value is omitted the last range specified will be used.  The initial\r
value is 1.\r
\r
\r
9. The Seed Value (s)\r
---------------------\r
   The initial seed value is set to 0 and this value is as good as any\r
other value however networks often do not converge quickly or at all\r
with some sets of initial weights.  To get some other initial random\r
weights use the seed command as in:\r
\r
s 7\r
\r
where the seed is set to 7.  The seed value is of type unsigned.\r
\r
\r
10. The Algorithm Command (a)\r
-----------------------------\r
   A number of different variations on the original back-propagation\r
algorithm have been proposed in order to speed up convergence and some\r
of these have been built into these simulators.  These options are set\r
using the `a' command and a number of options can go on the one line.\r
\r
Activation Functions\r
\r
   To set the activation functions use:\r
\r
a a <char>  * to set the activation function for all layers to <char>.\r
a ah <char> * to set the hidden layer(s) function to <char>.\r
a ao <char> * to set the output layer function to <char>.\r
\r
where <char> can be:\r
\r
   l  for the linear activation function:  x\r
   s  for the traditional smooth activation function:\r
      1.0 / (1.0 + exp(x))\r
\r
   The s function is the standard smooth activation function originally\r
used by researchers and it is still the most commonly used one.  In the\r
bp program it is implemented by a table look-up (default) or if the\r
compiler variable LOOKUP is undefined in the file ibp.h the regular\r
time-consuming real valued calculations are done.\r
\r
   The linear activation function gives networks only a very limited\r
ability to learn patterns and it is therefore hardly ever used by itself\r
in a network however it is often used in the output layer of networks\r
with 3 or more layers so that the network can give output values beyond\r
the range of the other activation functions.  For instance, suppose you\r
need to train a network to compute some non-linear function but you need\r
to produce outputs in the range -10 to 10.  The usual activation\r
functions are restricted to the range 0 to 1 or -1 to 1 but you can\r
choose a non-linear function for the network's hidden layers and with \r
linear neurons in the output layer the network can produce values\r
in the range -10 to 10.\r
\r
\r
The Derivatives\r
\r
   The correct derivative for the standard activation function is s(1-s)\r
where s is the activation value of a unit however when s is near 0 or 1\r
this term will give only very small weight changes during the learning\r
process.  To counter this problem Fahlman proposed the following one\r
for the output layer:\r
\r
0.1 + s(1-s)\r
\r
(For the original description of this method see "Faster Learning\r
Variations of Back-Propagation:  An Empirical Study", by Scott E.\r
Fahlman, in Proceedings of the 1988 Connectionist Models Summer School,\r
Morgan Kaufmann, 1989.)\r
\r
   Besides Fahlman's derivative and the original one the differential\r
step size method (see "Stepsize Variation Methods for Accelerating the\r
Back-Propagation Algorithm", by Chen and Mars, in IJCNN-90-WASH-DC,\r
Lawrence Erlbaum, 1990) takes the derivative to be 1 in the layer going\r
into the output units and uses the correct derivative term for all other\r
layers.  The learning rate for the inner layers is normally set to some\r
smaller value.  To set a value for eta2 give two values in the `e'\r
command as in:\r
\r
e 0.1 0.01\r
\r
To set the derivative use the `a' command as in:\r
\r
a dc   * use the correct derivative for whatever function\r
a dd   * use the differential step size derivative (default)\r
a df   * use Fahlman's derivative in only the output layer\r
a do   * use the original derivative (same as `c' above)\r
\r
Update Methods\r
\r
   The choices are the periodic (batch) method, the continuous (online)\r
method, delta-bar-delta and quickprop.  The following commands set the\r
update methods:\r
\r
a uC   * for the "right" continuous update method\r
a uc   * for the "wrong" continuous update method\r
a ud   * for the delta-bar-delta method\r
a up   * for the original periodic update method (default)\r
a uq   * for the quickprop algorithm\r
\r
\r
11. The Delta-Bar-Delta Method (d)\r
----------------------------------\r
   The delta-bar-delta method attempts to find a learning rate, eta, for\r
each individual weight.  The parameters are the initial value for the\r
etas, the amount by which to increase an eta that seems to be too small,\r
the rate at which to decrease an eta that is too large, a maximum value\r
for each eta and a parameter used in keeping a running average of the\r
slopes.  Here are examples of setting these parameters:\r
\r
d d 0.5    * sets the decay rate to 0.5\r
d e 0.1    * sets the initial etas to 0.1\r
d k 0.25   * sets the amount to increase etas by (kappa) to 0.25\r
d m 10     * sets the maximum eta to 10\r
d n 0.005  * an experimental noise parameter\r
d t 0.7    * sets the history parameter, theta, to 0.7\r
\r
These settings can all be placed on one line:\r
\r
d d 0.5  e 0.1  k 0.25  m 10  t 0.7\r
\r
The version implemented here does not use momentum.  The symmetric\r
versions sbp and srbp do not implement delta-bar-delta.\r
\r
   The idea behind the delta-bar-delta method is to let the program find\r
its own learning rate for each weight.  The `e' sub-command sets the\r
initial value for each of these learning rates.  When the program sees\r
that the slope of the error surface averages out to be in the same\r
direction for several iterations for a particular weight the program\r
increases the eta value by an amount, kappa, given by the `k' parameter.\r
The network will then move down this slope faster.  When the program\r
finds the slope changes signs the assumption is that the program has\r
stepped over to the other side of the minimum and so it cuts down the\r
learning rate by the decay factor given by the `d' parameter.  For\r
instance, a d value of 0.5 cuts the learning rate for the weight in\r
half.  The `m' parameter specifies the maximum allowable value for an\r
eta.  The `t' parameter (theta) is used to compute a running average of\r
the slope of the weight and must be in the range 0 <= t < 1.  The\r
running average at iteration i, a[i], is defined as:\r
\r
a[i] = (1 - t) * slope[i] + t * a[i-1],\r
\r
so small values for t make the most recent slope more important than the\r
previous average of the slope.  Determining the learning rate for\r
back-propagation automatically is, of course, very desirable and this\r
method often speeds up convergence by quite a lot.  Unfortunately, bad\r
choices for the delta-bar-delta parameters give bad results and a lot of\r
experimentation may be necessary.  If you have n patterns in the\r
training set try starting e and k around 1/n.  The n parameter is an\r
experimental noise term that is only used in the integer version.  It\r
changes a weight in the wrong direction by the amount indicated when the\r
previous weight change was 0 and the new weight change would be 0 and\r
the slope is non-zero.  (I found this to be effective in an integer\r
version of quickprop so I tossed it into delta-bar-delta as well.  If\r
you find this helps please let me know.)  For more on delta-bar-delta\r
see "Increased Rates of Convergence" by Robert A. Jacobs, in Neural\r
Networks, Volume 1, Number 4, 1988.\r
\r
\r
12. Quickprop (qp)\r
------------------\r
    Quickprop (see "Faster-Learning Variations on Back-Propagation: An\r
Empirical Study", by Scott E. Fahlman, in Proceedings of the 1988\r
Connectionist Models Summer School", Morgan Kaufmann, 1989 or ftp to\r
archive.cis.ohio-state.edu, look in the directory pub/neuroprose for the\r
file, fahlman.quickprop-tr.ps.Z.) may be one of the fastest network\r
training algorithms.  It is loosely based on Newton's method.\r
\r
   The parameter mu is used to limit the size of the weight change to\r
less than or equal to mu times the previous weight change.  Fahlman\r
suggests mu = 1.75 is generally quite good so this is the initial value\r
for mu but slightly larger or slightly smaller values are sometimes\r
better.\r
\r
   To get the process started quickprop makes the typical backprop\r
weight change of - eta * slope.  I have found that a good value for the\r
quickprop eta value is around 1 / n or 2 / n where n is the number of\r
patterns in the training set.  Other sources often use much larger\r
values.  In addition Fahlman uses this term at other times.  I had to\r
wonder if this was a good idea so in this code I've included a\r
capability to add it in or not add it in.  So far it seems to me that\r
sometimes adding in this extra term helps and sometimes it doesn't.  The\r
default is to use the extra term.\r
\r
   Another factor involved in quickprop comes about from the fact that\r
the weights often grow very large very quickly.  To minimize this\r
problem there is a decay factor designed to keep the weights small.\r
The weight decay is implemented by decreasing the value of the slope\r
and it is different from the general weight decay that people use and\r
which is also implemented in this software.  Fahlman recently mentioned\r
that now he does not use does not use this unless the weights get very\r
large.  I've found that too large a decay factor can stall\r
out the learning process so that if your network isn't learning fast\r
enough or isn't learning at all one possible fix is to decrease the\r
decay factor.  Note:  in the old free version the value of the weight\r
decay constant is the value you enter / 1000 in order to allow small\r
weight decay values in the integer version however in this version the\r
problem is handled differently so that what you enter is exactly what\r
you get, not the value divided by 1000.\r
\r
   I built in one additional feature for the integer version.  I found\r
that by adding small amounts of noise the time to convergence can be\r
brought down and the number of failures can be decreased somewhat.  This\r
seems to be especially true when the weight changes get very small.  The\r
noise consists of moving uphill in terms of error by a small amount when\r
the previous weight change was zero.  Good values for the noise seem to\r
be around 0.005.\r
\r
   The parameters for quickprop are all set in the `qp' command like\r
so:\r
\r
qp d <value>  * set the weight decay factor for all layers to <value>\r
qp d h 0      * the default weight decay for hidden layer units\r
qp d o 0.0001 * the default weight decay for output layer units\r
qp e 0.5      * the default value for eta\r
qp m 1.75     * the default value for mu\r
qp n 0        * the default value for noise\r
qp s+         * the default value is to always include the slope\r
\r
or a whole series can go on one line:\r
\r
qp d 0.1 e 0.5 m 1.75 n 0 s+\r
\r
\r
13. Making a Network (m)\r
------------------------\r
   In the simplest form of the make a network command you type an `m'\r
followed by the number of units in each layer as in:\r
\r
m 8 4 4 2\r
\r
Most of the time this type of network is all you will ever need but\r
there are others that can be tried and which may sometimes will work\r
better.  One innovation that often speeds up learning is to include\r
extra connections between the input and output layers.  To get this\r
type of network you add an x to the end of the m command as in:\r
\r
m 8 4 2 x\r
\r
These extra connections are said to be important when the problem to\r
be solved is almost linear and then the hidden layer units provide some\r
extra corrections to the output neurons to distort the results from a\r
purely linear model.\r
\r
   In the student version every time you made a network all the training\r
and testing patterns were thrown out because they were attached to the\r
network.  (Not true in the pro version.)\r
\r
   To make a recurrent network with 25 regular input units, twenty\r
hidden layer units (that are copied to the input layer) and 25 output\r
units use:\r
\r
m 25+20 20 25\r
\r
This means that the first layer will have 45 inputs and the first 25 are\r
regular input values but the next 20 come from the first hidden layer.\r
These 20 units are called the short term memory units.  Then there are\r
20 units in the hidden layer.  This value should match the number of\r
units given for the short term memory units.  At the moment there is no\r
check to see that it does.  Finally there are 25 units in the output\r
layer.  This recurrent network notation also requires a change in the\r
way training and testing patterns are written down for input into the\r
program.  For more on this see the next section.\r
\r
14. Recurrent Networks\r
----------------------\r
   Recurrent back-propagation networks take values from hidden layer\r
and/or output layer units and copy them down to the input layer for use\r
with the next input.  These values that are copied down are a kind of\r
coded record of what the recent inputs to the network have been and this\r
gives a network a simple kind of short-term memory, possibly a little\r
like human short-term memory.  For instance, suppose you want a network\r
to memorize the two short sequences, "acb" and "bcd".  In the middle of\r
both of these sequences is the letter, `c'.  In the first case you want\r
a network to take in `a' and output `c', then take in `c' and output\r
`b'.  In the second case you want a network to take in `b' and output\r
`c', then take in `c' and output `d'.  To do this a network needs a\r
simple memory of what came before the `c'.\r
\r
   Let the network be an 7-3-4 network where input units 1-4 and output\r
units 1-4 stand for the letters a-d and the `h' stands for the value of\r
a hidden layer unit.  So the codes are:\r
\r
a: 1000\r
b: 0100\r
c: 0010\r
d: 0001\r
\r
In action, the networks need to do the following.  When `a' is input,\r
`c' must be output:\r
\r
   0010     <- output layer\r
\r
   hhh      <- hidden layer\r
\r
1000 stm    <- input layer\r
\r
In this context, when `c' is input, `b' should be output:\r
\r
   0100\r
\r
   hhh\r
\r
0010 stm\r
\r
For the other string, when `b' is input, `c' is output:\r
\r
   0010\r
\r
   hhh\r
\r
0100 stm\r
\r
and when `c' in input, `d' is output:\r
\r
   0001\r
\r
   hhh\r
\r
0010 stm\r
\r
This is easy to do if the network keeps a short-term memory of what its\r
most recent inputs have been.  Suppose we input a and the output is c:\r
\r
   0010     <- output layer\r
\r
   hhh      <- hidden layer\r
\r
1000 stm    <- input layer\r
\r
Placing `a' on the input layer generates some kind of code (like a hash\r
code) on the 3 units in the hidden layer.  On the other hand, placing\r
`b' on the input units will generate a different code on the hidden\r
units.  All we need to do is save these hidden unit codes and input them\r
with a `c'.  In one case the network will output `b' and in the other\r
case it will output `d'.  In one particular run inputting `a' produced:\r
\r
     0  0  1  0\r
\r
  0.993 0.973 0.020\r
\r
 1  0  0  0  0  0  0\r
\r
When `c' is input the hidden layer units are copied down to input to\r
give:\r
\r
        0  1  0  0\r
\r
    0.006 0.999 0.461\r
\r
0  0  1  0  0.993 0.973 0.020\r
\r
For the other pattern, inputting `b' gave:\r
\r
    0  0  1  0\r
\r
  0.986 0.870 0.020\r
\r
0  1  0  0  0  0  0\r
\r
Then the input of `c' gave:\r
\r
          0  0  0  1\r
\r
      0.005 0.999 0.264\r
\r
0  0  1  0  0.986 0.870 0.020\r
\r
   This particular problem can be set up as follows:\r
\r
m 7 3 4\r
s 7\r
ci\r
t 0.2\r
rt {\r
1000 H   0010\r
0010 H   0100\r
\r
0100 H   0010\r
0010 H   0001\r
}\r
\r
where the first four values on each line are the normal input.  The H\r
codes for however many hidden layer units there are.  The last four\r
values are the desired outputs.\r
\r
   By the way, this simple problem does not converge particularly fast\r
and you may need to do a number of runs before you hit on initial values\r
that will work quickly.  It will work more reliably with more hidden\r
units.\r
\r
   Rather than using recurrent networks to memorize sequences of letters\r
they are probably more useful at predicting the value of some variable\r
at time t+1 given its value at t, t-1, t-2, ... .  A very simple of this\r
is to give the value of sin(t+1) given a recent history of inputs to the\r
net.  Given a value of sin(t) the curve may be going up or down and the\r
net needs to keep track of this in order to correctly predict the next\r
value.  The following setup will do this:\r
\r
m 1+5 5 1\r
f ir\r
a aol dd uq\r
qp e 0.02\r
ci\r
rt {\r
   0.00000  H   0.15636\r
   0.15636  H   0.30887\r
   0.30887  H   0.45378\r
\r
   . . .\r
\r
  -0.15950  H  -0.00319\r
  -0.00319  H   0.15321\r
}\r
\r
and in fact it converges rather rapidly.  The complete set of data can\r
be found in the example file rsin.bp.\r
\r
   Another recurrent network included in the examples is one designed to\r
memorize two lines of poetry.  The two lines were:\r
\r
   I the heir of all the ages in the foremost files of time\r
\r
   For I doubt not through all the ages ones increasing purpose runs\r
\r
but for the sake of making the problem simpler each word was shortened\r
to 5 characters giving:\r
\r
   i the heir of all the ages in the frmst files of\r
\r
   time for i doubt not thru the ages one incre purpo runs\r
\r
The letters were coded by taking the last 5 bits of their ASCII codes.\r
See the file poetry.bp.  \r
\r
   Once upon a time I was wondering what would happen if the poetry\r
network learned its verses and then the program was given several words\r
in the middle of the verses.  Would it pick up the sequence and be able\r
to complete it given 1 or 2 or 3 or n words?  So given for example, the\r
short sequence "for i doubt" will it be able to "get on track" and\r
finish the verse?  To test for this there are an extra pair of commands,\r
tr and trp.  Given a test set (which should be the training set) they\r
start at every possible place in the test set, input n words and then\r
check to see if the net produces the right answer.  For this example I\r
tried n = 3, 4, 5, 6 and 7 with the following results:\r
\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? tr 3\r
 TOL:  81.82 %  ERROR: 0.022967\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? tr 4\r
 TOL:  90.48 %  ERROR: 0.005672\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? tr 5\r
 TOL:  90.00 %  ERROR: 0.005974\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? tr 6\r
 TOL: 100.00 %  ERROR: 0.004256\r
[ACDFGMNPQTW?!acdefhlmopqrstw]? tr 7\r
 TOL: 100.00 %  ERROR: 0.004513\r
\r
So after getting just 3 words the program was 81.82% right in predicting\r
the next word to within the desired tolerance.  Given 6 or 7 words it\r
was getting them all right.  The trp command does the same thing except\r
it also prints the final output value for each of the tests made.\r
\r
\r
15. Miscellaneous Commands\r
--------------------------\r
   Below is a list of some miscellaneous commands, a short example of\r
each and a short description of the command.\r
\r
\r
!   Example: ! ls\r
\r
Anything after `!' will be passed on to the OS as a command to execute.\r
An ! followed immediately by a carriage-return will repeat the last\r
command sent to the OS.\r
\r
l   Example: l 2\r
\r
Entering "l 2" will print the values of the units on layer 2, or\r
whatever layer is specified.\r
\r
sb  Example: sb -3\r
\r
Entering "sb -3" will set the bias unit weight to -3.  In the symmetric\r
versions the weight will be frozen at this value while in the regular\r
versions it will only be the initial value and should be set after the\r
other weights are initialized.\r
\r
\r
16. Limitations\r
---------------\r
   Weights in the ibp and sibp programs are 16-bit integer weights where\r
the real value of the weight has been multiplied by 1024.  The integer\r
versions cannot handle weights less than -32 or greater than 31.999.\r
The weight changes are all checked for overflow but there are other\r
places in these programs where calculations can possibly overflow as\r
well and none of these places are checked.  Input values for the integer\r
versions can run from -31.992 to 31.999.  Due to the method used to\r
implement recurrent connections, input values in the real version are\r
limited to -31992.0 and above.\r
\r
\r
17. The Pro Version Additions\r
-----------------------------\r
   This section lists the additions to the pro version at this time.\r
For a more detailed and more up-to-date description see the online pro\r
version manual at:\r
\r
http://www.mcs.com/~drt/probp.html\r
\r
The additional commands are:\r
\r
ac <units>      add a weight connection between the units\r
ah <layer>      add a hidden unit to <layer>\r
b               benchmarking\r
i <filename>    read input from the file\r
k <numbers>     give the network a kick\r
n <options>     dynamic network building parameters\r
ofu <unit>      turn off a unit\r
onu <unit>      turn on a unit\r
ofw <weight>    turn off a weight\r
onw <weight>    turn on a weight\r
pw <number>     prune weights\r
rp              set rprop parameters\r
s <seeds>       set multiple seed values\r
ss <options>    set SuperSAB parameters\r
swem <option>   save weights every minimum flag\r
sw+             increment the weight file suffix\r
to              overall tolerance to be met (not per pattern, as with t)\r
u               the same as p but for recurrent classification problems\r
v               the same as t but for recurrent classification problems\r
\r
Benchmarking allows you to make multiple runs of a problem and find the\r
mean, standard deviation and average CPU time to converge.  You can also\r
use it to average the outputs of multiple runs and thereby possibly get\r
a better overall answer.\r
\r
You can make networks in a cascade type of architecture.  You can make\r
a new network with a different number of hidden layer units without\r
losing the training and testing patterns.  You can add hidden layer\r
units as the network is trained.  You can turn on and off individual\r
units.\r
\r
\r
The additonal options:\r
\r
a bh <value>     set the hidden layer bias unit value\r
a bo <value>     set the output layer bias unit value\r
a Dh <value>     set the hidden layer sharpness/gain\r
a Do <value>     set the output layer sharpness/gain\r
a wd <value>     weight decay\r
f t <reals>      set target values for classification problems\r
f wR             saves all weight parameters\r
f wb             saves weights as binary\r
f wB             saves all weight parameters as binary\r
pm               print confusion matrix for training set\r
tm               print confusion matrix for test set\r
\r
The activation functions available are:\r
\r
<char>                  Function                           Range\r
\r
   a    an efficient approximation of t            [-0.96016..0.96016]\r
   g    Gaussian function, exp(-(D*x)**2)                 (0..+1]\r
   l    linear function, D*x                            (-inf..+inf)\r
   p    piecewise linear version of s                     [0..+1]\r
   s    standard sigmoid, 1 / (1 + exp(-D*x))             (0..+1)\r
   t    tanh(D*x)                                         (-1..+1)\r
   x    D * x / (1 + |D * x|)                             (-1..+1)\r
   y    (D * x / 2) / (1 + |D * x|) + 0.5                 (0..+1)\r
   z    (D*x)**2 for x >= 0 and -(D*x)**2 for x < 0     (-inf..+inf)\r