Fix --all option to clearpart.

[pykickstart.git] / docs / programmers-guide

                        pykickstart Programmer's Guide

                               by Chris Lumens

                              October  28, 2005


Introduction
============
pykickstart is a Python library for manipulating kickstart files.  It
contains a common data representation, a parser, and a writer.  This
library aims to be useful for all Python programs that need to work with
kickstart files.  The two most obvious examples are anaconda and
system-config-kickstart.  It is recommended that all other tools that need
to use kickstart files use this library so that we can maintain equivalent
levels of support across all tools.

The kickstart file format itself has only been defined in a rather ad-hoc
manner.  Various documents describe the format, commands, and their
effects.  However, each kickstart-related program implemented its own
parser.  As anaconda added support for new commands and options, other
programs drifted farther and farther out of sync with the "official"
format.  This leads to the problem that valid kickstart files are not
accepted by all programs, or that programs will strip out options it
doesn't understand so that the input and output files do not match.

pykickstart is an effort to correct this.  It was originally designed to
be a common code base for anaconda and system-config-kickstart, so making
the code generic and easily extendable were top priorities.  Another
priority was to formalize the currently recognized grammar in an easily
understood parser so that files that used to work would continue to.  I
believe these goals have been met.

This document will cover how to use pykickstart in your programs and how to
extend the basic parser to get customized behavior.  It includes a
description of the important classes and several working examples.


Getting Started
===============
Before diving into the full documentation, it is useful to see an example
of how simple it is to use the default pykickstart in your programs.  Here
is a code snippet that imports the required classes, parses a kickstart
file, and leaves the results in the common data format.

        #!/usr/bin/python
        from pykickstart.data import *
        from pykickstart.parser import *

        ksdata = KickstartData()
        kshandlers = KickstartHandlers(ksdata)
        ksparser = KickstartParser(ksdata, kshandlers)
        ksparser.readKickstart("ks.cfg")

The call to readKickstart() reads in the kickstart file and sets values in
ksdata.  After this call, you don't need kshandlers or ksparser anymore
but you will want to hold on to ksdata if you are interested in the
contents of the file.  If you want to do some manipulations and then write
out the results to a new file, that's as simple as:

        from pykickstart.writer import KickstartWriter
        kswriter = KickstartWriter(ksdata)
        outfile = open("out.cfg", 'w")
        outfile.write(kswriter.write())
        outfile.close()


Classes
=======
The important classes that make up pykickstart are spread across a handful
of files.

constants.py
------------
This file includes no classes, though it does include several important
constants representing various things in the KickstartData.  You should
import its contents like so:

        from pykickstart.constants import *

data.py
-------
This file contains the classes that make up the common data
representation.  The center of the data format is the KickstartData class,
which contains many attributes representing the values from the input
file.  When a new KickstartData is instantiated, these attributes are set
to appropriate default values.  Some attributes are a simple string or
boolean, while some are a list of other classes or dictionaries.  For the
most part, each lines up with a single kickstart command.

The KickstartLogVolData, KickstartNetworkData, KickstartPartData,
KickstartRaidData, and KickstartVolGroupData classes are contained as list
elements in KickstartData.  Each corresponds to a command that may appear
multiple times.  They exist as separate classes so that attributes are
guaranteed to exist.

There are three different types of scripts - pre, post, and traceback -
but all are represented in the kickstart data by a single list.  The
Script class is contained in parser.py and contains an attribute that may
be used to discriminate between classes.

The install package list, excluded package list, and install groups list
are kept as separate.  Excluded packages have the leading "-" stripped
off, and groups have the leading "@" stripped.

See the class reference at the end of this documentation for a brief
explanation of useful functions in each class.

parser.py
---------
This file represents the bulk of pykickstart code.  At its core is the
KickstartParser class, which is essentially a state machine.  There is one
state for each of the sections in a kickstart file, plus some specialized
ones to make the parser work.  The readKickstart method is the entry point
into this class and is designed to be as generic as possible.  You should
never have to override this method in a subclass.  All the other methods
handle a change into a specific state and may be overridden in a
superclass.

If the KickstartParser encounters an error while reading your input file,
it will raise a KickstartParseError with the line in question.  Examples
of errors include bad options given to section headers, include files not
existing, or headers given for sections that are not recognized (for
instance, typos).

Error messages should call the formatErrorMsg function to be properly
formatted before being sent into an exception.  A properly formatted error
message includes the line number in the kickstart file where the problem
occurred and optionally, a more descriptive message.

The other major class within parser.py is KickstartHeaders.  This makes up
the largest amount of the code and also does the most work as it deals
with processing all the options on all the kickstart commands.
KickstartHandlers.handlers is a dictionary mapping command names to
handling methods.  KickstartParser takes the current command and
dispatches the correct method based on this mapping.  If the command name
maps to None, no method is called and no error is issued, a handy feature
which will be discussed later in this documentation.

Each of the handlers makes use of Python's OptionParser module to parse
command arguments and set values in the KickstartData object.  For this
reason, any subclass of KickstartHandlers must call the superclass's
handler to ensure that the KickstartData is correct.  Our option parsing
code allows commands to be marked as deprecated, which causes a message to
be generated and logged by anaconda.  It also allows marking options as
deprecated or required.

There are a few other minor points to note about KickstartParser and
KickstartHandlers.  When creating a KickstartParser object, you can set
the followIncludes attribute to False if you do not wish for include files
to be looked up and parsed as well.  There are several instances when this
is handy.  The order of instantiation for these objects is fixed, as each
object requires certain ones created before it.  KickstartData must be
first, as KickstartHandlers and KickstartParser require it.
KickstartHandlers must come second, and then finally KickstartParser.
Note that you can pass None in for kshandlers in the special case if you
do not care about handling any commands at all.  As we will see in the
next section, this is useful in one special case.

writer.py
---------
This file contains the class that makes up the Kickstart writer.  The job
of this class is to take a KickstartData object and convert it into a
string.  This string should be a valid kickstart file that can then be
used in any program.  Ideally, it should be the same as the input file
though the order of options may have been shifted, as well as other
cosmetic differences.

It is important to note that KickstartWriter.write returns a string, but
does no file output.  KickstartWriter is laid out similarly to
KickstartParser.  It consists of one handler per command plus special ones
for scripts and packages, plus a list specifying the order these handlers
should be called in.  It is possible to add your own handlers to
KickstartWriter if you are extending kickstart with special commands, as
will be discussed in the next section.

See the class reference at the end of this documentation for a brief
explanation of useful functions in each class.


Extending pykickstart
=====================
By default, pykickstart reads in a kickstart file and sets values in a
KickstartData object.  This is useful for some applications, but not all.
anaconda in particular has some odd requirements for kickstart so it will
be our basis for examples on extending pykickstart.

Only paying attention to one command
------------------------------------
Sometimes, you only want to take action on a single kickstart command and
don't care about any of the others.  anaconda, for instance, supports a
vnc command that needs to be processed well before any of the other
commands.  Remember from earlier that any command mapping to None in the
handlers is skipped.  That makes this fairly easy, then.  All we need to
do is make a specialized KickstartHandlers subclass and hook this into a
regular parser:

        from pykickstart.data import *
        from pykickstart.parser import *

        class VNCHandlers(KickstartHandlers):
                def __init__ (self, ksdata):
                        KickstartHandlers.__init__(self, ksdata)
                        self.resetHandlers()
                        self.handlers["vnc"] = self.doVnc

        ksdata = KickstartData()
        kshandlers = VNCHandlers(ksdata)
        ksparser = KickstartParser(ksdata, kshandlers)
        ksparser.readKickstart("ks.cfg")

Here, we make use of the KickstartHandlers.resetHandlers method.  This
method blanks out every entry in the handlers dictionary.  We then set the
vnc command handler to the default value, which you can easily get from
checking out the pykickstart source.  Then, make an instance of our new
class and pass this to KickstartParser().  When readKickstart is called,
the file will be parsed as usual but only the vnc command will ever be
handled.

You can then check the results by examining ksdata.vnc.

Customized handlers
-------------------
In other cases, you may want to include some customized behavior in your
kickstart handlers.  In this case, you'll still want to create a subclass
of KickstartHandlers, though you will need to make sure your handler calls
the superclass still.

        from pykickstart.data import *
        from pykickstart.parser import *

        class KSHandlers(KickstartHandlers):
                def doBootloader (self, args):
                        KickstartHandlers.doBootloader(self, args)
                        print "bootloader location = %s" % self.ksdata.bootloader["location"]

        ksdata = KickstartData()
        kshandlers = VNCHandlers(ksdata)
        ksparser = KickstartParser(ksdata, kshandlers)
        ksparser.readKickstart("ks.cfg")

This example is very simple, but you can still see what would be required
for complex cases.  Your handler needs to accept an args argument, which
is a list of arguments provided to the command.  Then, make sure to call
the superclass's method of the same name to set the KickstartData.
Finally, do your specialized behavior.

It is even possible to force your handlers to accept more arguments,
though it is slightly more complicated.  This requires making a subclass
of KickstartParser in addition to KickstartHandlers.

        from pykickstart.data import *
        from pykickstart.parser import *

        class KSHandlers(KickstartHandlers):
                def doBootloader (self, userarg, args):
                        KickstartHandlers.doBootloader(self, args)
                        print "%s bootloader location = %s" % (userarg, self.ksdata.bootloader["location"])

                ...

        class KSParser(KickstartParser):
                def __init__ (self, ksdata, kshandlers, userarg):
                        self.userarg = userarg
                        KickstartParser.__init__(self, ksdata, kshandlers)

                def handleCommand (self, cmd, args):
                        if not self.handler:
                                return

                        if not self.handler.handlers.has_key(cmd):
                                raise KickstartParseError, (cmd + " " + string.join(args))
                        else:
                                if self.handler.handlers[cmd] != None:
                                        self.handler.currentCmd = cmd
                                        self.handler.handlers[cmd](self.userarg,
args)

        ksdata = KickstartData()
        kshandlers = VNCHandlers(ksdata)
        ksparser = KSParser(ksdata, kshandlers, "note: ")
        ksparser.readKickstart("ks.cfg")

Let's examine this example a little more closely.  First, you need to
create a subclass of KickstartParser whose __init__ method stores your
argument.  Then you also need to override handleCommand.  In the
overridden method, the only difference is that on the last line you'll
need to pass your argument.

In your subclassed KickstartHandlers, you will need to make sure every
handler accepts your new argument.  You could possibly get around this
requirement by further modifying handleCommand to strip off your argument
for commands that do not accept it.  However, I'm not demonstrating that
here.  Then, make sure to call the superclass's handler method without any
additional arguments.

Adding a new command
--------------------
Adding support for a new command is fairly straightforward.  You'll need
to create a KickstartHandlers and KickstartWriter subclass and add your
methods to the handler lists.

        from pykickstart.data import *
        from pykickstart.parser import *
        from pykickstart.writer import *

        class SpecialHandlers(KickstartHandlers):
                def __init__ (self, ksdata):
                        KickstartHandlers.__init__(self, ksdata)
                        self.handlers["log"] = self.doLog

                def doLog (self, args):
                        op = KSOptionParser()
                        op.add_option("--level", dest="level")
                        op.add_option("--host", dest="host")

                        (opts, extra) = op.parse_args(args=args)

                        self.ksdata.log["level"] = getattr(opts, "level")
                        self.ksdata.log["host"] = getattr(opts, "host")

        class SpecialWriter(KickstartWriter):
                def __init__ (self, ksdata):
                        KickstartWriter.__init__(self, ksdata)
                        self.handlers.insert(1, self.doLog)

                def doLog(self):
                        argstr = ""

                        if self.ksdata.log["level"]:
                                argstr = "--level=%s" % self.ksdata.log["level"]
                        if self.ksdata.log["host"]:
                                argstr = argstr + "--host=%s" % self.ksdata.log["host"]

                        if argstr != "":
                                return "log %s" % argstr
                        else
                                return

        ksdata = KickstartData()
        kshandlers = SpecialHandlers(ksdata)
        ksparser = KickstartParser(ksdata, kshandlers)
        ksparser.readKickstart("ks.cfg")

This is all fairly straightforward, with the possible exception of the
OptionParser stuff.  Without getting into it too much, you'll need to
create a new instance of KSOptionParser, and then use the Python
documentation on how to use it.  It's a very complex, powerful class.
Make sure to set the KickstartData afterwards.

The KickstartWriter object is also pretty simple.  Make sure your method
returns an empty string if there's nothing set on this option, and the
appropriate string otherwise.

Adding a new section
--------------------
Currently, there is no simple way to add a new section.  This requires
adding more code to readKickstart as well as additional states.
readKickstart is not set up to do this sort of thing easily.


Class Reference
===============
class DeprecatedOption:
    def __init__ ():
        Creates a new Option type that supports a "deprecated" option
        attribute.  Command options with deprecated=1 will still be
        processed, but will raise a DeprecationWarning that may be logged.

class KickstartData:
    def __init__ ():
        Creates a new KickstartData object and sets default values on
        attributes.

class KickstartError:
    def __init__ (val):
        Creates a new KickstartError exception object with the given val
        as the message.  This is a generic exception.

class KickstartHandlers:
    def __init__ (ksdata):
        Creates a new KickstartHandlers object and initializes the handler
        dictionary to point at the default handlers.

    def resetHandlers ():
        Clears each handler to None.  This can be used to quickly clear
        out the handlers before setting only the handlers a subclass is
        interested in.

    def deprecatedCommand (cmd):
        Uses the Python warnings framework to issue a DeprecationWarning
        for the given command keyword.

    handlers:
        A dictionary mapping commands to handling methods.  Individual
        handlers may be set to None, in which case the command will be
        ignored.  A subclass may override any or all of the handlers,
        though each one should call the superclass's method to ensure the
        ksdata is correct.  The default handlers only set ksdata.

class KickstartLogVolData:
    def __init__ ():
        Creates a new KickstartLogVolData object and sets default values
        on attributes.  Instances of this class should be appended to
        KickstartData.logVolList.

class KickstartNetworkData:
    def __init__ ():
        Creates a new KickstartNetworkData object and sets default values
        on attributes.  Instances of this class should be appended to
        KickstartData.networkList.

class KickstartParser:
    def __init__ (ksdata, kshandlers):
        Creates a new KickstartParser object and initializes the parser
        state.  kshandlers may be None, in which case no function will be
        run when a kickstart command is seen in the input file.

    def addScript ():
        Called when the parser determines that it has found the end of
        a script section and should add the script to the ksdata's list.
        This method may be overriden if specialized behavior is required,
        though the superclass's method should still be called to make sure
        the script ends up in the ksdata.

    def addPackages (line):
        Called while in the %packages section.  Takes a line from the
        file, determines whether it is a group, package, or excluded
        package, and adds it to the correct list in the ksdata.  This
        method may be overriden if specialized behavior is required,
        though the superclass's method should still be called to make sure
        the package lists in the ksdata are correct.

    def handleCommand (cmd, lineno, args):
        Called while in the commands section.  Takes a line from the
        file, gets the commands from the head of the line, and dispatches
        the correct method from the KickstartHandlers.  If the
        KickstartParser object is created with kshandlers=None, no method
        will be called for the command but no error will be raised.  This
        method may be overridden if specialized behavior is needed.

    def handlePackageHdr (lineno, args):
        Called when the %packages section header is first seen.  Handles
        the options that may be given to %packages and sets these values
        in the ksdata.  This method may be overridden if specialized
        behavior is needed, though the superclass's method should still be
        called.

    def handleScriptHdr (lineno, args):
        Called when one of the script headers - %pre, %post, or %traceback -
        is first seen.  Handles the optiosn that may be given to those
        sections headers and sets those falues in the internal script
        representation.  This method may be overridden if specialized
        behavior is needed, though the superclass's method should still be
        called.

    def readKickstart (file):
        Parse the input file.  This method reads the input file, switches
        between states, and calls the handlers listed above.  This
        method should not need to be overridden by any subclass as it
        defines the kickstart file format.

    followIncludes:
        Defaults to True.  Set to False if you don't want %include
        files to be parsed as well.

class KickstartParseError:
    def __init__ (msg):
        Creates a new KickstartParseError exception object.  The message
        will include msg, which should be a properly formatted error
        message produced by formatErrorMsg.

class KickstartPartData:
    def __init__ ():
        Creates a new KickstartPartData object and sets default values
        on attributes.  Instances of this class should be appended to
        KickstartData.partList.

class KickstartRaidData:
    def __init__ ():
        Creates a new KickstartRaidData object and sets default values
        on attributes.  Instances of this class should be appended to
        KickstartData.raidList.

class KickstartValueError:
    def __init (val):
        Creates a new KickstartValueError exception object corresponding
        to a problem with the provided val, which should be a properly
        formatted error message produced by formatErrorMsg.

class KickstartVolGroupData:
    def __init__ ():
        Creates a new KickstartVolGroupData object and sets default values
        on attributes.  Instances of this class should be appended to
        KickstartData.volGroupList.

class KickstartWriter:
    def __init__ (ksdata):
        Creates a new KickstartWriter object and initializes the order
        KickstartData options should be printed out in.

    def write ():
        Returns a valid kickstart file generated from ksdata as a string
        that is suitable for writing out to a file.

class KSOptionParser:
    def __init__ (map=[], lineno=None):
        Creates a specialized subclass of python's OptionParser making use
        of the MappableOption, RequiredOption, and DeprecatedOption
        classes.  This class is used within command handlers and should
        not need to be subclassed unless extremely complex behavior is
        required.  The map option is needed if this object will have any
        MappableOption instances.  lineno is used in error reporting.

class MappableOption:
    def __init__ ():
        Creates a new Option type that supports a "map" action.  A map
        action on a given option stores into the destination the value
        given by:

                KSOptionParser.map[key]

        where key is the option with leading dashes stripped off.

        This also adds a "map_extend" action.  A map_extend action on
        a given set of options appends to the destination the value
        given by:

                KSOptionParser.map[key]

        where key is the option with leading dashes stripped off.

class RequiredOption:
    def __init__ ():
        Creates a new Option type that supports a "required" option
        attribute.  Command options with required=1 must be provided or
        an OptionError exception is raised.

class Script:
    def __init__ (script, interp="/bin/sh", inChroot=False, logfile=None,
                  errorOnFail=False, type=KS_SCRIPT_PRE):
        Creates a new Script object and sets the defaults for a %pre
        script.  inChroot only makes sense for %post scripts.

    def __repr__ ():
        Returns a string representation of the script useful for debugging
        output.

    def write ():
        Returns a valid kickstart file script generated from ksdata as a
        string that is suitable for writing out to a file.  This string
        includes the script header.