site/index.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
   2 <html xmlns="http://www.w3.org/1999/xhtml">
   3 <head>
   4 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
   5 <title>god - process and task monitoring done right</title>
   6 <link href="styles.css" rel="stylesheet" type="text/css" />
   7 <style type="text/css" media="screen">
   8 * {
   9         margin: 0;
  10         font-size: 100%;
  11 }
  12
  13 body {
  14         font: normal .8em/1.5em "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
  15         color: #484848;
  16         background: #E6EAE9 url(images/bg_grey.gif);
  17 }
  18
  19 a {
  20         color: #c75f3e;
  21         text-decoration: none;
  22 }
  23
  24 a:hover,
  25 a:active {
  26         text-decoration: underline;
  27 }
  28
  29 #mothership {
  30         width: 307px;
  31         height: 117px;
  32         margin: 0 auto;
  33         background: url(images/god_logo1.gif);
  34 }
  35
  36 #content {
  37         width: 700px;
  38         margin: 3px auto;
  39         background: white;
  40         border: 1px solid #444;
  41         padding: 0 24px;
  42         background: #f8f8ff;
  43         overflow: hidden;
  44 }
  45
  46 .banner {
  47         margin-top: 24px;
  48         border: 1px solid #ddd;
  49         width: 698px;
  50         height: 150px;
  51         background: url(images/banner.jpg);
  52 }
  53
  54 #menu {
  55         margin-top: 5px;
  56 }
  57
  58 #menu div.dots {
  59         background: url(images/red_dot.gif) repeat;
  60         height: 5px;
  61         width: 700px;
  62         font-size: 0;
  63 }
  64
  65 #menu ul {
  66         font-family: "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
  67         font-weight: bold;
  68         text-transform: uppercase;
  69         color: #4D4D4D;
  70         font-size: 12px;
  71         padding: 0;
  72         margin: 0;
  73         margin-top: 0 !important;
  74         margin-top: -2px;
  75 }
  76
  77 #menu li {
  78         display: inline;
  79         margin: 0 30px 0 0;
  80 }
  81
  82 #menu a:link,
  83 #menu a:visited {
  84         color: #4D4D4D;
  85         text-decoration: none;
  86 }
  87
  88 #menu a:hover,
  89 #menu a:active {
  90         color: black;
  91         text-decoration: none;
  92 }
  93
  94 #page_home #menu li.menu_home a {
  95         color: #A70000;
  96 }
  97
  98 .columnleft {
  99         float: left;
 100         width: 325px;
 101         margin-bottom: 20px;
 102 }
 103
 104 .columnleft p {
 105                 text-align: justify;
 106 }
 107
 108 .columnright {
 109         float: right;
 110         width: 325px;
 111         margin-bottom: 20px;
 112 }
 113
 114 h1 {
 115         font: bold 1.5em "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
 116         color: #f36e21;
 117         text-transform: uppercase;
 118         margin: 1.5em 0 .5em 0;
 119         clear: both;
 120 }
 121
 122 p {
 123         margin-bottom: 1em;
 124 }
 125
 126 ul {
 127   margin-bottom: 1em;
 128 }
 129
 130 ul.features {
 131         padding: 0;
 132         margin-left: 1.5em !important;
 133         margin-left: 1.3em;
 134 }
 135
 136 ul.features li {
 137         list-style-position: outside;
 138         list-style-type: circle;
 139         list-style-image: url(images/bullet.jpg);
 140         line-height: 1.4em;
 141 }
 142
 143 #footer {
 144         text-align: center;
 145         color: white;
 146         margin-bottom: 50px;
 147 }
 148
 149
 150
 151 pre {
 152   line-height: 1.3;
 153   border: 1px solid #ccc;
 154   padding: 1em;
 155   background-color: #efefef;
 156   margin: 1em 0;
 157 }
 158
 159 code {
 160   font-size: 1.2em;
 161 }
 162
 163 .ruby .keywords {
 164   color: blue;
 165 }
 166
 167 .ruby .comment {
 168         color : green;
 169 }
 170
 171 .ruby .string {
 172         color : teal;
 173 }
 174
 175 .ruby .keywords {
 176         color : navy;
 177 }
 178
 179 .ruby .brackets {
 180         color : navy;
 181 }
 182 </style>
 183
 184 <script type="text/javascript" src="javascripts/code_highlighter.js"></script>
 185 <script type="text/javascript" src="javascripts/ruby.js"></script>
 186
 187 </head>
 188
 189 <body id="page_home">
 190
 191 <div id="mothership">
 192
 193 </div>
 194 <div id="content">
 195         <div class="banner">
 196
 197         </div>
 198
 199   <!-- <div id="menu">
 200     <div class="dots"></div>
 201     <ul>
 202       <li class="menu_home"><a href="/">Home</a></li>
 203       <li class="menu_contact"><a href="mailto:tom@projectmothership.com">Contact</a></li>
 204     </ul>
 205     <div class="dots"></div>
 206   </div> -->
 207
 208         <div class="columnleft">
 209                 <h1>A Better Way to Monitor</h1>
 210                 <p>God is an easy to configure, easy to extend monitoring framework written in Ruby.</p>
 211                 <p>Keeping your server processes and tasks running should be a simple part of your deployment process. God aims to be the simplest, most powerful monitoring application available.</p>
 212                 <p style="text-align: right">Tom Preston-Werner<br />tom at rubyisawesome dot com</p>
 213                 <p style="text-align: right">Google Group: <a href="http://groups.google.com/group/god-rb">http://groups.google.com/group/god-rb</a></p>
 214         </div>
 215
 216         <div class="columnright">
 217         <h1>Features</h1>
 218         <ul class="features">
 219                 <li>Config file is written in Ruby</li>
 220                 <li>Easily write your own custom conditions in Ruby</li>
 221                 <li>Supports both poll and event based conditions</li>
 222                 <li>Different poll conditions can have different intervals</li>
 223                 <li>Integrated notification system (write your own too!)</li>
 224                 <li>Easily control non-daemonizing scripts</li>
 225         </ul>
 226         </div>
 227
 228         <h1>Installation (v 0.6)</h1>
 229         <p>The best way to get god is via rubygems:</p>
 230         <pre>$ sudo gem install god</pre>
 231         <p>You can also peruse or clone the code from <a href="http://repo.or.cz/w/god.git">http://repo.or.cz/w/god.git</a></p>
 232
 233         <h1>Requirements</h1>
 234
 235         <p>God currently only works on <b>Linux (kernel 2.6.15+), BSD,</b> and <b>Darwin</b> systems. No support for Windows is planned. Event based conditions on Linux systems require the <code>cn</code> (connector) kernel module loaded or compiled in to the kernel and god must be run as root.</p>
 236
 237         <p>The following systems have been tested. Help us test it on others!</p>
 238
 239         <ul>
 240           <li>Darwin 10.4.10</li>
 241           <li>RedHat Fedora Core 6</li>
 242           <li>Ubuntu Dapper (no events)</li>
 243           <li>Ubuntu Feisty</li>
 244           <li>CentOS 4.5 (no events)</li>
 245         </ul>
 246
 247         <h1>Finally, a Config File that Makes Sense</h1>
 248         <p>The easiest way to understand how god will make your life better is by looking at a sample config file. The following configuration file is what I use at <a href="http://site.gravatar.com/">gravatar.com</a> to keep the mongrels running:</p>
 249
 250 <pre><code class="ruby"># run with:  god -c /path/to/gravatar.god
 251 #
 252 # This is the actual config file used to keep the mongrels of
 253 # gravatar.com running.
 254
 255 RAILS_ROOT = "/Users/tom/dev/gravatar2"
 256
 257 %w{8200 8201 8202}.each do |port|
 258   God.watch do |w|
 259     w.name = "gravatar2-mongrel-#{port}"
 260     w.interval = 30.seconds # default
 261     w.start = "mongrel_rails start -c #{RAILS_ROOT} -p #{port} \
 262       -P #{RAILS_ROOT}/log/mongrel.#{port}.pid  -d"
 263     w.stop = "mongrel_rails stop -P #{RAILS_ROOT}/log/mongrel.#{port}.pid"
 264     w.restart = "mongrel_rails restart -P #{RAILS_ROOT}/log/mongrel.#{port}.pid"
 265     w.start_grace = 10.seconds
 266     w.restart_grace = 10.seconds
 267     w.pid_file = File.join(RAILS_ROOT, "log/mongrel.#{port}.pid")
 268
 269     w.behavior(:clean_pid_file)
 270
 271     w.start_if do |start|
 272       start.condition(:process_running) do |c|
 273         c.interval = 5.seconds
 274         c.running = false
 275       end
 276     end
 277
 278     w.restart_if do |restart|
 279       restart.condition(:memory_usage) do |c|
 280         c.above = 150.megabytes
 281         c.times = [3, 5] # 3 out of 5 intervals
 282       end
 283
 284       restart.condition(:cpu_usage) do |c|
 285         c.above = 50.percent
 286         c.times = 5
 287       end
 288     end
 289
 290     # lifecycle
 291     w.lifecycle do |on|
 292       on.condition(:flapping) do |c|
 293         c.to_state = [:start, :restart]
 294         c.times = 5
 295         c.within = 5.minute
 296         c.transition = :unmonitored
 297         c.retry_in = 10.minutes
 298         c.retry_times = 5
 299         c.retry_within = 2.hours
 300       end
 301     end
 302   end
 303 end</code></pre>
 304
 305 <p>That's a lot to take in at once, so I'll break it down by section and explain what's going on in each.</p>
 306
 307 <pre><code class="ruby">RAILS_ROOT = "/var/www/gravatar2/current"</code></pre>
 308
 309 <p>Here I've set a constant that is used throughout the file. Keeping the <code>RAILS_ROOT</code> value in a constant makes it easy to adapt this script to other applications. Because the config file is Ruby code, I can set whatever variables or constants I want that make the configuration more concise and easier to work with.</p>
 310
 311 <pre><code class="ruby">%w{8200 8201 8202}.each do |port|
 312   ...
 313 end</code></pre>
 314
 315 <p>Because the config file is written in actual Ruby code, we can construct loops and do other intelligent things that are impossible in your every day, run of the mill config file. I need to watch three mongrels, so I simply loop over their port numbers, eliminating duplication and making my life a whole lot easier.</p>
 316
 317 <pre><code class="ruby">  God.watch do |w|
 318     w.name = "gravatar2-mongrel-#{port}"
 319     w.interval = 30.seconds # default
 320     w.start = "mongrel_rails start -c #{RAILS_ROOT} -p #{port} \
 321       -P #{RAILS_ROOT}/log/mongrel.#{port}.pid  -d"
 322     w.stop = "mongrel_rails stop -P #{RAILS_ROOT}/log/mongrel.#{port}.pid"
 323     w.restart = "mongrel_rails restart -P #{RAILS_ROOT}/log/mongrel.#{port}.pid"
 324     w.start_grace = 10.seconds
 325     w.restart_grace = 10.seconds
 326     w.pid_file = File.join(RAILS_ROOT, "log/mongrel.#{port}.pid")
 327
 328     ...
 329   end</code></pre>
 330
 331 <p>A <code>watch</code> represents a single process that has concrete start, stop, and/or restart operations. You can define as many watches as you like. In the example above, I've got some Rails instances running in Mongrels that I need to keep alive. Every watch must have a unique <code>name</code> so that it can be identified later on. The <code>interval</code> option sets the default poll interval (this can be overridden in each condition). The <code>start</code> and <code>stop</code> attributes specify the commands to start and stop the process. If no <code>restart</code> attribute is set, restart will be represented by a call to stop followed by a call to start. The optional <code>grace</code> attribute sets the amount of time following a start/stop/restart command to wait before resuming normal monitoring operations. To be more specific, I can set just <code>start_grace</code>, <code>stop_grace</code>, and/or <code>restart_grace</code>. If the process you're watching runs as a daemon (as mine does), you'll need to set the <code>pid_file</code> attribute.</p>
 332
 333 <pre><code class="ruby">    w.behavior(:clean_pid_file)</code></pre>
 334
 335 <p>Behaviors allow you to execute additional commands around start/stop/restart commands. In our case, if the process dies it will leave a PID file behind. The next time a start command is issued, it will fail, complaining about the leftover PID file. We'd like the PID file cleaned up before a start command is issued. The built-in behavior <code>clean_pid_file</code> will do just that.</p>
 336
 337 <pre><code class="ruby">    w.start_if do |start|
 338       start.condition(:process_running) do |c|
 339         c.interval = 5.seconds
 340         c.running = false
 341       end
 342     end</code></pre>
 343
 344 <p>Watches contain conditions grouped by the action to execute should they return <code>true</code>. I start with a <code>start_if</code> block that contains a single condition. Conditions are specified by calling <code>condition</code> with an identifier, in this case
 345 <code>:process_running</code>. Each condition can specify a poll interval that will override the default watch interval. In this case, I want to check that the process is still running every 5 seconds instead of the 30 second interval that other conditions will inherit. The ability to set condition specific poll intervals makes it possible to run critical tests (such as :process_running) more often than less critical tests (such as :memory_usage and :cpu_usage).</p>
 346
 347 <pre><code class="ruby">    w.restart_if do |restart|
 348       restart.condition(:memory_usage) do |c|
 349         c.above = 150.megabytes
 350         c.times = [3, 5] # 3 out of 5 intervals
 351       end
 352
 353       ...
 354     end</code></pre>
 355
 356 <p>Similar to <code>start_if</code> there is a <code>restart_if</code> command that groups conditions that should trigger a restart. The <code>memory_usage</code> condition will fail if the specified process is using too much memory. The maximum allowable amount of memory is specified with the <code>above</code> attribute (you can use the kilobytes, megabytes, or gigabytes helpers). The number of times the test needs to fail in order to trigger a restart is set with <code>times</code>. This can be either an integer or an array. An integer means it must fail that many times in a row while an array [x, y] means it must fail x times out of the last y tests.</p>
 357
 358 <pre><code class="ruby">    w.restart_if do |restart|
 359       ...
 360
 361       restart.condition(:cpu_usage) do |c|
 362         c.above = 50.percent
 363         c.times = 5
 364       end
 365     end</code></pre>
 366
 367 <p>To keep an eye on CPU usage, I've employed the <code>cpu_usage</code> condition. When CPU usage for a Mongrel process is over 50% for 5 consecutive intervals, it will be restarted.</p>
 368
 369 <pre><code class="ruby">    w.lifecycle do |on|
 370       on.condition(:flapping) do |c|
 371         c.to_state = [:start, :restart]
 372         c.times = 5
 373         c.within = 5.minute
 374         c.transition = :unmonitored
 375         c.retry_in = 10.minutes
 376         c.retry_times = 5
 377         c.retry_within = 2.hours
 378       end
 379     end</code></pre>
 380
 381 <p>Conditions inside a <code>lifecycle</code> section are active as long as the process is being monitored (they live across state changes).
 382
 383 <p>The <code>:flapping</code> condition guards against the edge case wherein god rapidly starts or restarts your application. Things like server configuration changes or the unavailability of external services could make it impossible for my process to start. In that case, god will try to start my process over and over to no avail. The <code>:flapping</code> condition provides two levels of giving up on flapping processes. If I were to translate the options of the code above, it would be something like: If this watch is started or restarted five times withing 5 minutes, then unmonitor it...then after ten minutes, monitor it again to see if it was just a temporary problem; if the process is seen to be flapping five times within two hours, then give up completely.</p>
 384
 385 <p>That's it! Simple, huh?</p>
 386
 387 <!-- ------------------------------------------------------------------------- -->
 388
 389 <h1>Changing UID/GID for processes</h1>
 390
 391 <p>It is possible to have god run your start/stop/restart commands as a specific user/group. This can be done by setting the <code>uid</code> and/or <code>gid</code> attributes of a watch.</p>
 392
 393 <pre><code class="ruby">  God.watch do |w|
 394     ...
 395
 396     w.uid = 'tom'
 397     w.gid = 'devs'
 398
 399     ...
 400   end</code></pre>
 401
 402 </p>This only works for commands specified as a string. Lambda commands are unaffected.</p>
 403
 404
 405 <!-- ------------------------------------------------------------------------- -->
 406
 407 <h1>Lambda commands</h1>
 408
 409 <p>In addition to specifying start/stop/restart commands as strings (to be executed via the shell), you can specify a lambda that will be called.</p>
 410
 411 <pre><code class="ruby">  God.watch do |w|
 412     ...
 413
 414     w.start = lambda { ENV['APACHE'] ? `apachectl -k graceful` : `lighttpd restart` }
 415
 416     ...
 417   end</code></pre>
 418
 419 <!-- ------------------------------------------------------------------------- -->
 420
 421 <h1>Starting and Controlling God</h1>
 422
 423 <p>To start the god monitoring process as a daemon simply run the <code>god</code> executable passing in the path to the config file (you need to sudo if you're using events on Linux or want to use the setuid/setgid functionality):</p>
 424
 425 <pre>$ sudo god -c /path/to/config.god</pre>
 426
 427 <p>While you're writing your config file, it can be helpful to run god in the foreground so you can see the log messages. You can do that with:</p>
 428
 429 <pre>$ sudo god -c /path/to/config.god -D</pre>
 430
 431 <p>You can start/restart/stop/monitor/unmonitor your Watches with the same utility like so:</p>
 432
 433 <pre>$ sudo god stop gravatar2-mongrel-8200</pre>
 434
 435 <!-- ------------------------------------------------------------------------- -->
 436
 437 <h1>Grouping Watches</h1>
 438
 439 <p>Watches can be assigned to groups. These groups can then be controlled together from the command line.</p>
 440
 441 <pre><code class="ruby">  God.watch do |w|
 442     ...
 443
 444     w.group = 'mongrels'
 445
 446     ...
 447   end</code></pre>
 448
 449 <p>The above configuration now allows you to control the watch (and any others that are in the group) with a single command:</p>
 450
 451 <pre>$ sudo god stop mongrels</pre>
 452
 453 <!-- ------------------------------------------------------------------------- -->
 454
 455 <h1>Advanced Configuration with Transitions and Events</h1>
 456
 457 <p>So far you've been introduced to a simple poll-based config file and seen how to run it. Poll-based monitoring works great for simple things, but falls short for highly critical tasks. God has native support for kqueue/netlink events on BSD/Darwin/Linux systems. For instance, instead of using the <code>process_running</code> condition to poll for the status of your process, you can use the <code>process_exits</code> condition that will be notified <b>immediately</b> upon the exit of your process. This means less load on your system and shorter downtime after a crash.</p>
 458
 459 <p>While the configuration syntax you saw in the previous example is very simple, it lacks the power that we need to deal with event based monitoring. In fact, the <code>start_if</code> and <code>restart_if</code> methods are really just calling out to a lower-level API. If we use the low-level API directly, we can harness the full power of god's event based lifecycle system. Let's look at another example config file.</p>
 460
 461 <pre><code class="ruby">RAILS_ROOT = "/Users/tom/dev/gravatar2"
 462
 463 God.watch do |w|
 464   w.name = "local-3000"
 465   w.interval = 5.seconds # default
 466   w.start = "mongrel_rails start -c #{RAILS_ROOT} -P #{RAILS_ROOT}/log/mongrel.pid -p 3000 -d"
 467   w.stop = "mongrel_rails stop -P #{RAILS_ROOT}/log/mongrel.pid"
 468   w.restart = "mongrel_rails restart -P #{RAILS_ROOT}/log/mongrel.pid"
 469   w.pid_file = File.join(RAILS_ROOT, "log/mongrel.pid")
 470
 471   # clean pid files before start if necessary
 472   w.behavior(:clean_pid_file)
 473
 474   # determine the state on startup
 475   w.transition(:init, { true => :up, false => :start }) do |on|
 476     on.condition(:process_running) do |c|
 477       c.running = true
 478     end
 479   end
 480
 481   # determine when process has finished starting
 482   w.transition([:start, :restart], :up) do |on|
 483     on.condition(:process_running) do |c|
 484       c.running = true
 485     end
 486
 487     # failsafe
 488     on.condition(:tries) do |c|
 489       c.times = 5
 490       c.transition = :start
 491     end
 492   end
 493
 494   # start if process is not running
 495   w.transition(:up, :start) do |on|
 496     on.condition(:process_exits)
 497   end
 498
 499   # restart if memory or cpu is too high
 500   w.transition(:up, :restart) do |on|
 501     on.condition(:memory_usage) do |c|
 502       c.interval = 20
 503       c.above = 50.megabytes
 504       c.times = [3, 5]
 505     end
 506
 507     on.condition(:cpu_usage) do |c|
 508       c.interval = 10
 509       c.above = 10.percent
 510       c.times = [3, 5]
 511     end
 512   end
 513
 514   # lifecycle
 515   w.lifecycle do |on|
 516     on.condition(:flapping) do |c|
 517       c.to_state = [:start, :restart]
 518       c.times = 5
 519       c.within = 5.minute
 520       c.transition = :unmonitored
 521       c.retry_in = 10.minutes
 522       c.retry_times = 5
 523       c.retry_within = 2.hours
 524     end
 525   end
 526 end
 527 </code></pre>
 528
 529 <p>A bit longer, I know, but very straighforward once you understand how the <code>transition</code> calls work. The <code>name</code>, <code>interval</code>, <code>start</code>, <code>stop</code>, and <code>pid_file</code> attributes should be familiar. We also specify the <code>clean_pid_file</code> behavior.</p>
 530
 531 <p>Before jumping into the code, it's important to understand the different states that a Watch can have, and how that state changes over time. At any given time, a Watch will be in one of the <code>init</code>, <code>up</code>, <code>start</code>, or <code>restart</code> states. As different conditions are satisfied, the Watch will progress from state to state, enabling and disabling conditions along the way.</p>
 532
 533 <p>When god first starts, each Watch is placed in the <code>init</code> state.</p>
 534
 535 <p>You'll use the <code>transition</code> method to tell god how to transition between states. It takes two arguments. The first argument may be either a symbol or an array of symbols representing the state or states during which the specified conditions should be enabled. The second argument may be either a symbol or a hash. If it is a symbol, then that is the state that will be transitioned to if any of the conditions return <code>true</code>. If it is a hash, then that hash must have both <code>true</code> and <code>false</code> keys, each of which point to a symbol that represents the state to transition to given the corresponding return from the single condition that must be specified.</p>
 536
 537 <pre><code class="ruby">  # determine the state on startup
 538   w.transition(:init, { true => :up, false => :start }) do |on|
 539     on.condition(:process_running) do |c|
 540       c.running = true
 541     end
 542   end</code></pre>
 543
 544 <p>The first transition block tells god what to do when the Watch is in the <code>init</code> state (first argument). This is where I tell god how to determine if my task is already running. Since I'm monitoring a process, I can use the <code>process_running</code> condition to determine whether the process is running. If the process is running, it will return true, otherwise it will return false. Since I sent a hash as the second argument to <code>transition</code>, the return from <code>process_running</code> will determine which of the two states will be transitioned to. If the process is running, the return is true and god will put the Watch into the <code>up</code> state. If the process is not running, the return is false and god will put the Watch into the <code>start</code> state.</p>
 545
 546 <pre><code class="ruby">  # determine when process has finished starting
 547   w.transition([:start, :restart], :up) do |on|
 548     on.condition(:process_running) do |c|
 549       c.running = true
 550     end
 551
 552     ...
 553   end</code></pre>
 554
 555 <p>If god has determined that my process isn't running, the Watch will be put into the <code>start</code> state. Upon entering this state, the <code>start</code> command that I specified on the Watch will be called. In addition, the above transition specifies a condition that should be enabled when in either the <code>start</code> or <code>restart</code> states. The condition is another <code>process_running</code>, however this time I'm only interested in moving to another state once it returns <code>true</code>. A <code>true</code> return from this condition means that the process is running and it's ok to transition to the <code>up</code> state (second argument to <code>transition</code>).</p>
 556
 557 <pre><code class="ruby">  # determine when process has finished starting
 558   w.transition([:start, :restart], :up) do |on|
 559     ...
 560
 561     # failsafe
 562     on.condition(:tries) do |c|
 563       c.times = 5
 564       c.transition = :start
 565     end
 566   end</code></pre>
 567
 568 <p>The other half of this transition uses the <code>tries</code> condition to ensure that god doesn't get stuck in this state. It's possible that the process could go down while the transition is being made, in which case god would end up polling forever to see if the process is up. Here I've specified that if this condition is called five times, god should override the normal transition destination and move to the <code>start</code> state instead. If you specify a <code>transition</code> attribute on any condition, that state will be transferred to instead of the normal transfer destination.</p>
 569
 570 <pre><code class="ruby">  # start if process is not running
 571   w.transition(:up, :start) do |on|
 572     on.condition(:process_exits)
 573   end</code></pre>
 574
 575 <p>This is where the event based system comes into play. Once in the <code>up</code> state, I want to be notified when my process exits. The <code>process_exits</code> condition registers a callback that will trigger a transition change when it is fired off. Event conditions (like this one) cannot be used in transitions that have a hash for the second argument (as they do not return true or false).</p>
 576
 577 <pre><code class="ruby">  # restart if memory or cpu is too high
 578   w.transition(:up, :restart) do |on|
 579     on.condition(:memory_usage) do |c|
 580       c.interval = 20
 581       c.above = 50.megabytes
 582       c.times = [3, 5]
 583     end
 584
 585     on.condition(:cpu_usage) do |c|
 586       c.interval = 10
 587       c.above = 10.percent
 588       c.times = [3, 5]
 589     end
 590   end</code></pre>
 591
 592 <p>Notice that I can have multiple transitions with the same start state. In this case, I want to have the <code>memory_usage</code> and <code>cpu_usage</code> poll conditions going at the same time that I listen for the process exit event. In the case of runaway CPU or memory usage, however, I want to transition to the <code>restart</code> state. When a Watch enters the <code>restart</code> state it will either call the <code>restart</code> command that you specified, or if none has been set, call the <code>stop</code> and then <code>start</code> commands.</p>
 593
 594 <!-- ------------------------------------------------------------------------- -->
 595
 596 <h1>Watching Non-Daemon Processes</h1>
 597
 598 <p>Need to watch a script that doesn't have built in daemonization? No problem! God will daemonize and keep track of your process for you. If you don't specify a <code>pid_file</code> attribute for a watch, it will be auto-daemonized and a PID file will be stored for it in <code>/var/run/god</code>. If you'd rather have the PID file stored in a different location, you can set it at the top of your config:</p>
 599
 600 <pre><code class="ruby">God.pid_file_directory = '/home/tom/pids'
 601
 602 God.watch do |w|
 603   # watch with no pid_file attribute set
 604 end</code></pre>
 605
 606 <p>The directory you specify must be writable by god.</p>
 607
 608 <!-- ------------------------------------------------------------------------- -->
 609
 610 <h1>Loading Other Config Files</h1>
 611
 612 <p>You should feel free to separate your god configs into separate files for easier organization. You can load in other configs using Ruby's normal <code>load</code> method, or use the convenience method <code>God.load</code> which allows for glob-style paths:</p>
 613
 614 <pre><code class="ruby"># load in all god configs
 615 God.load "/usr/local/conf/*.god"</code></pre>
 616
 617 <p>God won't start its monitoring operations until all configurations have been loaded.</p>
 618
 619 <!-- ------------------------------------------------------------------------- -->
 620
 621 <h1>Dynamically Loading Config Files Into an Already Running God</h1>
 622
 623 <p>God allows you to load or reload configurations into an already running instance. There are a few things to consider when doing this:</p>
 624
 625 <ul>
 626   <li>Existng Watches with the same <code>name</code> as the incoming Watches will be overidden by the new config.</li>
 627   <li>All paths must be either absolute or relative to the path from which god was started.</li>
 628 </ul>
 629
 630 <p>To load a config into a running god, issue the following command:</p>
 631
 632 <pre>$ sudo god load path/to/config.god</pre>
 633
 634 <p>Config files that are loaded dynamically can contain anything that a normal config file contains, however, global options such as <code>God.pid_file_directory</code> blocks will be ignored (and produce a warning in the logs).</p>
 635
 636 <!-- ------------------------------------------------------------------------- -->
 637
 638 <h1>Getting Logs for a Single Watch</h1>
 639
 640 <p>Sifting through the god logs for statements specific to a specific Watch can be frustrating when you have many of them. You can get the realtime logs for a single Watch via the command line:</p>
 641
 642 <pre>$ sudo god log local-3000</pre>
 643
 644 <p>This will display the last 1000 lines of log for the 'local-3000' Watch and update every second with new log messages.</p>
 645
 646 <p>You can also supply a shorthand to the log command that will match one of your watches. If it happens to match several, the first match will be used:</p>
 647
 648 <pre>$ sudo god log l3</pre>
 649
 650 <!-- ------------------------------------------------------------------------- -->
 651
 652 <h1>Notifications</h1>
 653
 654 <p>God has an extensible notification framework built in that makes it easy to have notifications sent when conditions are triggered. There are three steps to enabling notifications.</p>
 655
 656 <p>Step 1: Set the options for the notification subsystem(s) that you'll be using. Let's look at how to setup email notifications.</p>
 657
 658 <pre><code class="ruby">God::Contacts::Email.message_settings = {
 659   :from => 'god@example.com'
 660 }
 661
 662 God::Contacts::Email.server_settings = {
 663   :address => "smtp.example.com",
 664   :port => 25,
 665   :domain => "example.com",
 666   :authentication => :plain,
 667   :user_name => "john",
 668   :password => "s3kr3ts"
 669 }</code></pre>
 670
 671 <p>Step 2: Configure some contacts.</p>
 672
 673 <pre><code class="ruby">God.contact(:email) do |c|
 674   c.name = 'tom'
 675   c.email = 'tom@example.com'
 676 end
 677
 678 God.contact(:email) do |c|
 679   c.name = 'vanpelt'
 680   c.email = 'vanpelt@example.com'
 681   c.group = 'developers'
 682 end
 683
 684 God.contact(:email) do |c|
 685   c.name = 'kevin'
 686   c.email = 'kevin@example.com'
 687   c.group = 'developers'
 688 end</code></pre>
 689
 690 <p>Step 3: Attach to a condition:</p>
 691
 692 <pre><code class="ruby">  w.transition(:up, :start) do |on|
 693     on.condition(:process_exits) do |c|
 694       c.notify = 'tom'
 695     end
 696   end</code></pre>
 697
 698 <p>There are two ways to specify that a notification should be sent. The first, easier way is shown above. Every condition can take an optional <code>notify</code> attribute that specifies which contacts should be notified when the condition is triggered. The value can be a contact name or contact group *or* an array of contact names and/or contact groups.</p>
 699
 700 <pre><code class="ruby">  w.transition(:up, :start) do |on|
 701     on.condition(:process_exits) do |c|
 702       c.notify = {:contacts => ['tom', 'developers'], :priority => 1, :category => 'product'}
 703     end
 704   end</code></pre>
 705
 706 </p>The second way allows you to specify the <code>priority</code> and <code>category</code> in addition to the contacts. The extra attributes can be arbitrary integers or strings and will be passed as-is to the notification subsystem.</p>
 707
 708 <p>The above notification will arrive as an email similar to the following.</p>
 709
 710 <pre><code>From: god &lt;god@example.com&gt;
 711 To: tom &lt;tom@example.com&gt;
 712 Subject: [god] mongrel-8600 [trigger] process exited (ProcessExits)
 713
 714 Message: mongre-8600 [trigger] process exited (ProcessExits)
 715 Host: candymountain.example.com
 716 Priority: 1
 717 Category: product</code></pre>
 718
 719 <!-- ------------------------------------------------------------------------- -->
 720
 721 <h1>Extend God with your own Conditions</h1>
 722
 723 <p>God was designed from the start to allow you to easily write your own custom conditions, making it simple to add tests that are application specific.</p>
 724
 725 </div>
 726 <div id="footer">
 727         <p>Brought to you by <a href="http://rubyisawesome.com/">Ruby is Awesome</a></p>
 728 </div>
 729
 730 <script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
 731 </script>
 732 <script type="text/javascript">
 733 _uacct = "UA-2196727-1";
 734 urchinTracker();
 735 </script>
 736
 737 </body>
 738 </html>