docs/api_extension.html.in

   1 <?xml version="1.0" encoding="UTF-8"?>
   2 <!DOCTYPE html>
   3 <html xmlns="http://www.w3.org/1999/xhtml">
   4   <body>
   5     <h1>Implementing a new API in Libvirt</h1>
   6
   7     <ul id="toc"></ul>
   8
   9     <p>
  10       This document walks you through the process of implementing a new
  11       API in libvirt.  Remember that new API consists of any new public
  12       functions, as well as the addition of flags or extensions of XML used by
  13       existing functions.
  14     </p>
  15
  16     <p>
  17       Before you begin coding, it is critical that you propose your
  18       changes on the libvirt mailing list and get feedback on your ideas to
  19       make sure what you're proposing fits with the general direction of the
  20       project.  Even before doing a proof of concept implementation, send an
  21       email giving an overview of the functionality you think should be
  22       added to libvirt.  Someone may already be working on the feature you
  23       want.  Also, recognize that everything you write is likely to undergo
  24       significant rework as you discuss it with the other developers, so
  25       don't wait too long before getting feedback.
  26     </p>
  27
  28     <p>
  29       Adding a new API to libvirt is not difficult, but there are quite a
  30       few steps.  This document assumes that you are familiar with C
  31       programming and have checked out the libvirt code from the source code
  32       repository and successfully built the existing tree.  Instructions on
  33       how to check out and build the code can be found at:
  34     </p>
  35
  36     <p>
  37       <a href="https://libvirt.org/downloads.html">https://libvirt.org/downloads.html</a>
  38     </p>
  39
  40     <p>
  41       Once you have a working development environment, the steps to create a
  42       new API are:
  43     </p>
  44     <ol>
  45       <li>define the public API</li>
  46       <li>define the internal driver API</li>
  47       <li>implement the public API</li>
  48       <li>implement the remote protocol:
  49         <ol>
  50           <li>define the wire protocol format</li>
  51           <li>implement the RPC client</li>
  52           <li>implement the server side dispatcher</li>
  53         </ol>
  54       </li>
  55       <li>use new API where appropriate in drivers</li>
  56       <li>add virsh support</li>
  57       <li>add common handling for new API</li>
  58       <li>for each driver that can support the new API:
  59         <ol>
  60           <li>add prerequisite support</li>
  61           <li>fully implement new API</li>
  62         </ol>
  63       </li>
  64     </ol>
  65
  66     <p>
  67       It is, of course, possible to implement the pieces in any order, but
  68       if the development tasks are completed in the order listed, the code
  69       will compile after each step.  Given the number of changes required,
  70       verification after each step is highly recommended.
  71     </p>
  72
  73     <p>
  74       Submit new code in the form of one patch per step.  That's not to say
  75       submit patches before you have working functionality--get the whole thing
  76       working and make sure you're happy with it.  Then use git to break the
  77       changes into pieces so you don't drop a big blob of code on the
  78       mailing list in one go.  Also, you should follow the upstream tree, and
  79       rebase your series to adapt your patches to work with any other changes
  80       that were accepted upstream during your development.
  81     </p>
  82
  83     <p>
  84       Don't mix anything else into the patches you submit.  The patches
  85       should be the minimal changes required to implement the functionality
  86       you're adding.  If you notice a bug in unrelated code (i.e., code you
  87       don't have to touch to implement your API change) during development,
  88       create a patch that just addresses that bug and submit it
  89       separately.
  90     </p>
  91
  92     <h2><a name='publicapi'>Defining the public API</a></h2>
  93
  94     <p>The first task is to define the public API.  If the new API
  95       involves an XML extension, you have to enhance the RelaxNG
  96       schema and document the new elements or attributes:</p>
  97
  98     <p><code>
  99         docs/schemas/domaincommon.rng<br/>
 100         docs/formatdomain.html.in
 101     </code></p>
 102
 103     <p>If the API extension involves a new function, you have to add a
 104       declaration in the public header, and arrange to export the
 105       function name (symbol) so other programs can link against the
 106       libvirt library and call the new function:</p>
 107
 108     <p><code>
 109         include/libvirt/libvirt-$MODULE.h.in
 110         src/libvirt_public.syms
 111     </code></p>
 112
 113     <p>
 114       This task is in many ways the most important to get right, since once
 115       the API has been committed to the repository, it's libvirt's policy
 116       never to change it.  Mistakes in the implementation are bugs that you
 117       can fix.  Make a mistake in the API definition and you're stuck with
 118       it, so think carefully about the interface and don't be afraid to
 119       rework it as you go through the process of implementing it.
 120     </p>
 121
 122     <h2><a name='internalapi'>Defining the internal API</a></h2>
 123
 124     <p>
 125       Each public API call is associated with a driver, such as a host
 126       virtualization driver, a network virtualization driver, a storage
 127       virtualization driver, a state driver, or a device monitor.  Adding
 128       the internal API is ordinarily a matter of adding a new member to the
 129       struct representing one of these drivers.
 130     </p>
 131
 132     <p>
 133       Of course, it's possible that the new API will involve the creation of
 134       an entirely new driver type, in which case the changes will include the
 135       creation of a new struct type to represent the new driver type.
 136     </p>
 137
 138     <p>The driver structs are defined in:</p>
 139
 140     <p><code>src/driver-$MODULE.h</code></p>
 141
 142     <p>
 143       To define the internal API, first typedef the driver function
 144       prototype and then add a new field for it to the relevant driver
 145       struct.  Then, update all existing instances of the driver to
 146       provide a <code>NULL</code> stub for the new function.
 147     </p>
 148
 149     <h2><a name='implpublic'>Implementing the public API</a></h2>
 150
 151     <p>
 152       Implementing the public API is largely a formality in which we wire up
 153       public API to the internal driver API.  The public API implementation
 154       takes care of some basic validity checks before passing control to the
 155       driver implementation.  In RFC 2119 vocabulary, this function:
 156     </p>
 157
 158     <ol class="ordinarylist">
 159       <li>SHOULD log a message with VIR_DEBUG() indicating that it is
 160         being called and its parameters;</li>
 161       <li>MUST call virResetLastError();</li>
 162       <li>SHOULD confirm that the connection is valid with
 163         virCheckConnectReturn() or virCheckConnectGoto();</li>
 164       <li><strong>SECURITY: If the API requires a connection with write
 165           privileges, MUST confirm that the connection flags do not
 166           indicate that the connection is read-only with
 167           virCheckReadOnlyGoto();</strong></li>
 168       <li>SHOULD do basic validation of the parameters that are being
 169         passed in, using helpers like virCheckNonNullArgGoto();</li>
 170       <li>MUST confirm that the driver for this connection exists and that
 171         it implements this function;</li>
 172       <li>MUST call the internal API;</li>
 173       <li>SHOULD log a message with VIR_DEBUG() indicating that it is
 174         returning, its return value, and status.</li>
 175       <li>MUST return status to the caller.</li>
 176     </ol>
 177
 178     <p>The public API calls are implemented in:</p>
 179
 180     <p><code>src/libvirt-$MODULE.c</code></p>
 181
 182     <h2><a name='remoteproto'>Implementing the remote protocol</a></h2>
 183
 184     <p>
 185       Implementing the remote protocol is essentially a
 186       straightforward exercise which is probably most easily
 187       understood by referring to the existing code.
 188     </p>
 189
 190     <h3><a name='wireproto'>Defining the wire protocol format</a></h3>
 191
 192     <p>
 193       Defining the wire protocol involves making additions to:
 194     </p>
 195
 196     <p><code>src/remote/remote_protocol.x</code></p>
 197
 198     <p>
 199       First, create two new structs for each new function that you're adding
 200       to the API.  One struct describes the parameters to be passed to the
 201       remote function, and a second struct describes the value returned by
 202       the remote function.  The one exception to this rule is that functions
 203       that return only 0 or -1 for status do not require a struct for returned
 204       data.
 205     </p>
 206
 207     <p>
 208       Second, add values to the remote_procedure enum for each new function
 209       added to the API.
 210     </p>
 211
 212     <p>
 213       Once these changes are in place, it's necessary to run 'make rpcgen'
 214       in the src directory to create the .c and .h files required by the
 215       remote protocol code. This must be done on a Linux host using the
 216       GLibC rpcgen program. Other rpcgen versions may generate code which
 217       results in bogus compile time warnings.  This regenerates the
 218       following files:
 219     </p>
 220
 221     <p><code>
 222         src/remote/remote_daemon_dispatch_stubs.h
 223         src/remote/remote_daemon_dispatch.h
 224         src/remote/remote_daemon_dispatch.c
 225         src/remote/remote_protocol.c
 226         src/remote/remote_protocol.h
 227     </code></p>
 228
 229     <h3><a name='rpcclient'>Implement the RPC client</a></h3>
 230
 231     <p>
 232       Implementing the uses the rpcgen generated .h files.  The remote
 233       method calls go in:
 234     </p>
 235
 236     <p><code>src/remote/remote_driver.c</code></p>
 237
 238     <p>Each remote method invocation does the following:</p>
 239
 240     <ol class="ordinarylist">
 241       <li>locks the remote driver;</li>
 242       <li>sets up the method arguments;</li>
 243       <li>invokes the remote function;</li>
 244       <li>checks the return value, if necessary;</li>
 245       <li>extracts any returned data;</li>
 246       <li>frees any returned data;</li>
 247       <li>unlocks the remote driver.</li>
 248     </ol>
 249
 250     <h3><a id="serverdispatch">Implement the server side dispatcher</a></h3>
 251
 252     <p>
 253       Implementing the server side of the remote function call is simply a
 254       matter of deserializing the parameters passed in from the remote
 255       caller and passing them to the corresponding internal API function.
 256       The server side dispatchers are implemented in:
 257     </p>
 258
 259     <p><code>src/remote/daemon_dispatch.c</code></p>
 260
 261     <p>Again, this step uses the .h files generated by make rpcgen.</p>
 262
 263     <p>
 264       After all three pieces of the remote protocol are complete, and
 265       the generated files have been updated, it will be necessary to
 266       update the file:</p>
 267
 268     <p><code>src/remote_protocol-structs</code></p>
 269
 270     <p>
 271       This file should only have new lines added; modifications to
 272       existing lines probably imply a backwards-incompatible API change.
 273     </p>
 274
 275     <h2><a id="internaluseapi">Use the new API internally</a></h2>
 276
 277     <p>
 278       Sometimes, a new API serves as a superset of existing API, by
 279       adding more granularity in what can be managed.  When this is
 280       the case, it makes sense to share a common implementation by
 281       making the older API become a trivial wrapper around the new
 282       API, rather than duplicating the common code.  This step should
 283       not introduce any semantic differences for the old API, and is
 284       not necessary if the new API has no relation to existing API.
 285     </p>
 286
 287     <h2><a id="virshuseapi">Expose the new API in virsh</a></h2>
 288
 289     <p>
 290       All new API should be manageable from the virsh command line
 291       shell.  This proves that the API is sufficient for the intended
 292       purpose, and helps to identify whether the proposed API needs
 293       slight changes for easier usage.  However, remember that virsh
 294       is used to connect to hosts running older versions of libvirtd,
 295       so new commands should have fallbacks to an older API if
 296       possible; implementing the virsh hooks at this point makes it
 297       very easy to test these fallbacks.  Also remember to document
 298       virsh additions.
 299     </p>
 300
 301     <p>
 302       A virsh command is composed of a few pieces of code.  You need to
 303       define an array of vshCmdInfo structs for each new command that
 304       contain the help text and the command description text.  You also need
 305       an array of vshCmdOptDef structs to describe the command options.
 306       Once you have those pieces in place you can write the function
 307       implementing the virsh command.  Finally, you need to add the new
 308       command to the commands[] array.  The following files need changes:
 309     </p>
 310
 311     <p><code>
 312         tools/virsh-$MODULE.c<br/>
 313         tools/virsh.pod
 314     </code></p>
 315
 316     <h2><a id="driverimpl">Implement the driver methods</a></h2>
 317
 318     <p>
 319       So, after all that, we get to the fun part.  All functionality in
 320       libvirt is implemented inside a driver.  Thus, here is where you
 321       implement whatever functionality you're adding to libvirt.  You'll
 322       either need to add additional files to the src directory or extend
 323       files that are already there, depending on what functionality you're
 324       adding.
 325     </p>
 326
 327     <h3><a id="commonimpl">Implement common handling</a></h3>
 328
 329     <p>
 330       If the new API is applicable to more than one driver, it may
 331       make sense to provide some utility routines, or to factor some
 332       of the work into the dispatcher, to avoid reimplementing the
 333       same code in every driver.  In the example code, this involved
 334       adding a member to the virDomainDefPtr struct for mapping
 335       between the XML API addition and the in-memory representation of
 336       a domain, along with updating all clients to use the new member.
 337       Up to this point, there have been no changes to existing
 338       semantics, and the new APIs will fail unless they are used in
 339       the same way as the older API wrappers.
 340     </p>
 341
 342     <h3><a id="drivercode">Implement driver handling</a></h3>
 343
 344     <p>
 345       The remaining patches should only touch one driver at a time.
 346       It is possible to implement all changes for a driver in one
 347       patch, but for review purposes it may still make sense to break
 348       things into simpler steps.  Here is where the new APIs finally
 349       start working.
 350     </p>
 351
 352     <p>
 353       It is always a good idea to patch the test driver in addition to the
 354       target driver, to prove that the API can be used for more than one
 355       driver.
 356     </p>
 357
 358     <p>
 359       Any cleanups resulting from the changes should be added as separate
 360       patches at the end of the series.
 361     </p>
 362
 363     <p>
 364       Once you have working functionality, run make check and make
 365       syntax-check on each patch of the series before submitting
 366       patches.  It may also be worth writing tests for the libvirt-TCK
 367       testsuite to exercise your new API, although those patches are
 368       not kept in the libvirt repository.
 369     </p>
 370   </body>
 371 </html>