change status interface to return hash instead of string
[god.git] / site / index.html
blobaa5ccc0575eb66053db4959988475424da5f6997
1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2 <html xmlns="http://www.w3.org/1999/xhtml">
3 <head>
4 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
5 <title>god - process and task monitoring done right</title>
6 <link href="styles.css" rel="stylesheet" type="text/css" />
7 <style type="text/css" media="screen">
8 * {
9 margin: 0;
10 font-size: 100%;
13 body {
14 font: normal .8em/1.5em "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
15 color: #484848;
16 background: #E6EAE9 url(images/bg_grey.gif);
19 a {
20 color: #c75f3e;
21 text-decoration: none;
24 a:hover,
25 a:active {
26 text-decoration: underline;
29 #mothership {
30 width: 307px;
31 height: 117px;
32 margin: 0 auto;
33 background: url(images/god_logo1.gif);
36 #content {
37 width: 700px;
38 margin: 3px auto;
39 background: white;
40 border: 1px solid #444;
41 padding: 0 24px;
42 background: #f8f8ff;
43 overflow: hidden;
46 .banner {
47 margin-top: 24px;
48 border: 1px solid #ddd;
49 width: 698px;
50 height: 150px;
51 background: url(images/banner.jpg);
54 #menu {
55 margin-top: 5px;
58 #menu div.dots {
59 background: url(images/red_dot.gif) repeat;
60 height: 5px;
61 width: 700px;
62 font-size: 0;
65 #menu ul {
66 font-family: "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
67 font-weight: bold;
68 text-transform: uppercase;
69 color: #4D4D4D;
70 font-size: 12px;
71 padding: 0;
72 margin: 0;
73 margin-top: 0 !important;
74 margin-top: -2px;
77 #menu li {
78 display: inline;
79 margin: 0 30px 0 0;
82 #menu a:link,
83 #menu a:visited {
84 color: #4D4D4D;
85 text-decoration: none;
88 #menu a:hover,
89 #menu a:active {
90 color: black;
91 text-decoration: none;
94 #page_home #menu li.menu_home a {
95 color: #A70000;
98 .columnleft {
99 float: left;
100 width: 325px;
101 margin-bottom: 20px;
104 .columnleft p {
105 text-align: justify;
108 .columnright {
109 float: right;
110 width: 325px;
111 margin-bottom: 20px;
114 h1 {
115 font: bold 1.5em "Trebuchet MS", Verdana, Arial, Helvetica, sans-serif;
116 color: #f36e21;
117 text-transform: uppercase;
118 margin: 1.5em 0 .5em 0;
119 clear: both;
123 margin-bottom: 1em;
126 ul {
127 margin-bottom: 1em;
130 ul.features {
131 padding: 0;
132 margin-left: 1.5em !important;
133 margin-left: 1.3em;
136 ul.features li {
137 list-style-position: outside;
138 list-style-type: circle;
139 list-style-image: url(images/bullet.jpg);
140 line-height: 1.4em;
143 #footer {
144 text-align: center;
145 color: white;
146 margin-bottom: 50px;
151 pre {
152 line-height: 1.3;
153 border: 1px solid #ccc;
154 padding: 1em;
155 background-color: #efefef;
156 margin: 1em 0;
159 code {
160 font-size: 1.2em;
163 .ruby .keywords {
164 color: blue;
167 .ruby .comment {
168 color : green;
171 .ruby .string {
172 color : teal;
175 .ruby .keywords {
176 color : navy;
179 .ruby .brackets {
180 color : navy;
182 </style>
184 <script type="text/javascript" src="javascripts/code_highlighter.js"></script>
185 <script type="text/javascript" src="javascripts/ruby.js"></script>
187 </head>
189 <body id="page_home">
191 <div id="mothership">
193 </div>
194 <div id="content">
195 <div class="banner">
197 </div>
199 <!-- <div id="menu">
200 <div class="dots"></div>
201 <ul>
202 <li class="menu_home"><a href="/">Home</a></li>
203 <li class="menu_contact"><a href="mailto:tom@projectmothership.com">Contact</a></li>
204 </ul>
205 <div class="dots"></div>
206 </div> -->
208 <div class="columnleft">
209 <h1>A Better Way to Monitor</h1>
210 <p>God is an easy to configure, easy to extend monitoring framework written in Ruby.</p>
211 <p>Keeping your server processes and tasks running should be a simple part of your deployment process. God aims to be the simplest, most powerful monitoring application available.</p>
212 <p style="text-align: right">Tom Preston-Werner<br />tom at rubyisawesome dot com</p>
213 </div>
215 <div class="columnright">
216 <h1>Features</h1>
217 <ul class="features">
218 <li>Config file is written in Ruby</li>
219 <li>Easily write your own custom conditions in Ruby</li>
220 <li>Supports both poll and event based conditions</li>
221 <li>Different poll conditions can have different intervals</li>
222 <li>Easily control non-daemonized processes</li>
223 </ul>
224 </div>
226 <h1>Installation (v 0.4.0)</h1>
227 <p>The best way to get god is via rubygems:</p>
228 <pre>$ sudo gem install god</pre>
229 <p>You can also peruse or clone the code from <a href="http://repo.or.cz/w/god.git">http://repo.or.cz/w/god.git</a></p>
231 <h1>Requirements</h1>
233 <p>God currently only works on <b>Linux (kernel 2.6.15+), BSD,</b> and <b>Darwin</b> systems. No support for Windows is planned. Event based conditions on Linux systems require the <code>cn</code> (connector) kernel module loaded or compiled in to the kernel and god must be run as root.</p>
235 <p>The following systems have been tested. Help us test it on others!</p>
237 <ul>
238 <li>Darwin 10.4.10</li>
239 <li>RedHat Fedora Core 6</li>
240 <li>Ubuntu Dapper (no events)</li>
241 <li>Ubuntu Feisty</li>
242 <li>CentOS 4.5 (no events)</li>
243 </ul>
245 <h1>Finally, a Config File that Makes Sense</h1>
246 <p>The easiest way to understand how god will make your life better is by looking at a sample config file. The following configuration file is what I use at <a href="http://site.gravatar.com/">gravatar.com</a> to keep the mongrels running:</p>
248 <pre><code class="ruby"># file: gravatar.god
249 # run with: god -c /path/to/gravatar.god
251 # This is the actual config file used to keep the mongrels of
252 # gravatar.com running.
254 RAILS_ROOT = "/var/www/gravatar2/current"
256 %w{8200 8201 8202}.each do |port|
257 God.watch do |w|
258 w.name = "gravatar2-mongrel-#{port}"
259 w.interval = 30.seconds # default
260 w.start = "mongrel_rails cluster::start --only #{port} \
261 -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
262 w.stop = "mongrel_rails cluster::stop --only #{port} \
263 -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
264 w.grace = 10.seconds
265 w.pid_file = File.join(RAILS_ROOT, "log/mongrel.#{port}.pid")
267 w.behavior(:clean_pid_file)
269 w.start_if do |start|
270 start.condition(:process_running) do |c|
271 c.interval = 5.seconds
272 c.running = false
276 w.restart_if do |restart|
277 restart.condition(:memory_usage) do |c|
278 c.above = 150.megabytes
279 c.times = [3, 5] # 3 out of 5 intervals
282 restart.condition(:cpu_usage) do |c|
283 c.above = 50.percent
284 c.times = 5
288 end</code></pre>
290 <p>That's a lot to take in at once, so I'll break it down by section and explain what's going on in each.</p>
292 <pre><code class="ruby">RAILS_ROOT = "/var/www/gravatar2/current"</code></pre>
294 <p>Here I've set a constant that is used throughout the file. Keeping the <code>RAILS_ROOT</code> value in a constant makes it easy to adapt this script to other applications. Because the config file is Ruby code, I can set whatever variables or constants I want that make the configuration more concise and easier to work with.</p>
296 <pre><code class="ruby">%w{8200 8201 8202}.each do |port|
298 end</code></pre>
300 <p>Because the config file is written in actual Ruby code, we can construct loops and do other intelligent things that are impossible in your every day, run of the mill config file. I need to watch three mongrels, so I simply loop over their port numbers, eliminating duplication and making my life a whole lot easier.</p>
302 <pre><code class="ruby"> God.watch do |w|
303 w.name = "gravatar2-mongrel-#{port}"
304 w.interval = 30.seconds # default
305 w.start = "mongrel_rails cluster::start --only #{port} \
306 -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
307 w.stop = "mongrel_rails cluster::stop --only #{port} \
308 -C #{RAILS_ROOT}/config/mongrel_cluster.yml"
309 w.grace = 10.seconds
310 pid_file = File.join(RAILS_ROOT, "log/mongrel.#{port}.pid")
313 end</code></pre>
315 <p>A <code>watch</code> represents a single process that has concrete start, stop, and/or restart operations. You can define as many watches as you like. In the example above, I've got a Rails instance running in a Mongrel that I need to keep alive. Every watch must have a unique <code>name</code> so that it can be identified later on. The <code>interval</code> option sets the default poll interval (this can be overridden in each condition). The <code>start</code> and <code>stop</code> attributes specify the commands to start and stop the process. If no <code>restart</code> attribute is set, restart will be represented by a call to stop followed by a call to start. The optional <code>grace</code> attribute sets the amount of time following a start/stop/restart command to wait before resuming normal monitoring operations. If the process you're watches runs as a daemon, you'll need to set the <code>pid_file</code> attribute in order for the behaviors and conditions to function.</p>
317 <pre><code class="ruby"> w.behavior(:clean_pid_file)</code></pre>
319 <p>Behaviors allow you to execute additional commands around start/stop/restart commands. In our case, if the process dies it will leave a PID file behind. The next time a start command is issued, it will fail, complaining about the leftover PID file. We'd like the PID file cleaned up before a start command is issued. The built-in behavior <code>clean_pid_file</code> will do just that.</p>
321 <pre><code class="ruby"> w.start_if do |start|
322 start.condition(:process_running) do |c|
323 c.interval = 5.seconds
324 c.running = false
326 end</code></pre>
328 <p>Watches contain conditions grouped by the action to execute should they return <code>true</code>. I start with a <code>start_if</code> block that contains a single condition. Conditions are specified by calling <code>condition</code> with an identifier, in this case
329 <code>:process_running</code>. Each condition can specify a poll interval that will override the default watch interval. In this case, I want to check that the process is still running every 5 seconds instead of the 30 second interval that other conditions will inherit. The ability to set condition specific poll intervals makes it possible to run costly tests less often then cheap tests.</p>
331 <pre><code class="ruby"> w.restart_if do |restart|
332 restart.condition(:memory_usage) do |c|
333 c.above = 150.megabytes
334 c.times = [3, 5] # 3 out of 5 intervals
338 end</code></pre>
340 <p>Similar to <code>start_if</code> there is a <code>restart_if</code> command that groups conditions that should trigger a restart. The <code>memory_usage</code> condition will fail if the specified process is using too much memory. The maximum allowable amount of memory is specified with the <code>above</code> attribute (you can use the kilobytes, megabytes, or gigabytes helpers). The number of times the test needs to fail in order to trigger a restart is set with <code>times</code>. This can be either an integer or an array. An integer means it must fail that many times in a row while an array [x, y] means it must fail x times out of the last y tests.</p>
342 <pre><code class="ruby"> w.restart_if do |restart|
345 restart.condition(:cpu_usage) do |c|
346 c.above = 50.percent
347 c.times = 5
349 end</code></pre>
351 <p>To keep an eye on CPU usage, I've employed the <code>cpu_usage</code> condition. When CPU usage for a Mongrel process is over 50% for 5 consecutive intervals, it will be restarted.</p>
353 <p>That's it! Simple, huh?</p>
355 <!-- ------------------------------------------------------------------------- -->
357 <h1>Changing UID/GID for processes</h1>
359 <p>It is possible to have god run your start/stop/restart commands as a specific user/group. This can be done by setting the <code>uid</code> and/or <code>gid</code> attributes of a watch.</p>
361 <pre><code class="ruby"> God.watch do |w|
364 w.uid = 'tom'
365 w.gid = 'devs'
368 end</code></pre>
370 </p>This only works for commands specified as a string. Lambda commands are unaffected.</p>
373 <!-- ------------------------------------------------------------------------- -->
375 <h1>Lambda commands</h1>
377 <p>In addition to specifying start/stop/restart commands as strings (to be executed via the shell), you can specify a lambda that will be called.</p>
379 <pre><code class="ruby"> God.watch do |w|
382 w.start = lambda { ENV['APACHE'] ? `apachectl -k graceful` : `lighttpd restart` }
385 end</code></pre>
387 <!-- ------------------------------------------------------------------------- -->
389 <h1>Starting and Controlling God</h1>
391 <p>To start the god monitoring process as a daemon simply run the <code>god</code> executable passing in the path to the config file (you need to sudo if you're using events on Linux or want to use the setuid/setgid functionality):</p>
393 <pre>$ sudo god -c /path/to/config.god</pre>
395 <p>While you're writing your config file, it can be helpful to run god in the foreground so you can see the log messages. You can do that with:</p>
397 <pre>$ sudo god -c /path/to/config.god -D</pre>
399 <p>You can start/restart/stop/monitor/unmonitor your Watches with the same utility like so:</p>
401 <pre>$ sudo god stop 'gravatar2-mongrel-8200'</pre>
403 <!-- ------------------------------------------------------------------------- -->
405 <h1>Grouping Watches</h1>
407 <p>Watches can be assigned to groups. These groups can then be controlled together from the command line.</p>
409 <pre><code class="ruby"> God.watch do |w|
412 w.group = 'mongrels'
415 end</code></pre>
417 <p>The above configuration now allows you to control the watch (and any others that are in the group) with a single command:</p>
419 <pre>$ sudo god stop 'mongrels'</pre>
421 <!-- ------------------------------------------------------------------------- -->
423 <h1>Advanced Configuration with Transitions and Events</h1>
425 <p>So far you've been introduced to a simple poll-based config file and seen how to run it. Poll-based monitoring works great for simple things, but falls short for highly critical tasks. God has native support for kqueue/netlink events on BSD/Darwin/Linux systems. For instance, instead of using the <code>process_running</code> condition to poll for the status of your process, you can use the <code>process_exits</code> condition that will be notified <b>immediately</b> upon the exit of your process. This means less load on your system and shorter downtime after a crash.</p>
427 <p>While the configuration syntax you saw in the previous example is very simple, it lacks the power that we need to deal with event based monitoring. In fact, the <code>start_if</code> and <code>restart_if</code> methods are really just calling out to a lower-level API. If we use the low-level API directly, we can harness the full power of god's event based lifecycle system. Let's look at another example config file.</p>
429 <pre><code class="ruby">RAILS_ROOT = "/Users/tom/dev/git/helloworld"
431 God.watch do |w|
432 w.name = "local-3000"
433 w.interval = 5.seconds # default
434 w.start = "mongrel_rails start -P ./log/mongrel.pid -c #{RAILS_ROOT}"
435 w.stop = "mongrel_rails stop -P ./log/mongrel.pid -c #{RAILS_ROOT}"
436 w.pid_file = File.join(RAILS_ROOT, "log/mongrel.pid")
438 # clean pid files before start if necessary
439 w.behavior(:clean_pid_file)
441 # determine the state on startup
442 w.transition(:init, { true => :up, false => :start }) do |on|
443 on.condition(:process_running) do |c|
444 c.running = true
448 # determine when process has finished starting
449 w.transition([:start, :restart], :up) do |on|
450 on.condition(:process_running) do |c|
451 c.running = true
454 # failsafe
455 on.condition(:tries) do |c|
456 c.times = 3
457 c.transition = :start
461 # start if process is not running
462 w.transition(:up, :start) do |on|
463 on.condition(:process_exits)
466 # restart if memory or cpu is too high
467 w.transition(:up, :restart) do |on|
468 on.condition(:memory_usage) do |c|
469 c.interval = 20
470 c.above = 50.megabytes
471 c.times = [3, 5]
474 on.condition(:cpu_usage) do |c|
475 c.interval = 10
476 c.above = 10.percent
477 c.times = [3, 5]
481 </code></pre>
483 <p>A bit longer, I know, but very straighforward once you understand how the <code>transition</code> calls work. The <code>name</code>, <code>interval</code>, <code>start</code>, <code>stop</code>, and <code>pid_file</code> attributes should be familiar. We also specify the <code>clean_pid_file</code> behavior.</p>
485 <p>Before jumping into the code, it's important to understand the different states that a Watch can have, and how that state changes over time. At any given time, a Watch will be in one of the <code>init</code>, <code>up</code>, <code>start</code>, or <code>restart</code> states. As different conditions are satisfied, the Watch will progress from state to state, enabling and disabling conditions along the way.</p>
487 <p>When god first starts, each Watch is placed in the <code>init</code> state.</p>
489 <p>You'll use the <code>transition</code> method to tell god how to transition between states. It takes two arguments. The first argument may be either a symbol or an array of symbols representing the state or states during which the specified conditions should be enabled. The second argument may be either a symbol or a hash. If it is a symbol, then that is the state that will be transitioned to if any of the conditions return <code>true</code>. If it is a hash, then that hash must have both <code>true</code> and <code>false</code> keys, each of which point to a symbol that represents the state to transition to given the corresponding return from the single condition that must be specified.</p>
491 <pre><code class="ruby"> # determine the state on startup
492 w.transition(:init, { true => :up, false => :start }) do |on|
493 on.condition(:process_running) do |c|
494 c.running = true
496 end</code></pre>
498 <p>The first transition block tells god what to do when the Watch is in the <code>init</code> state (first argument). This is where I tell god how to determine if my task is already running. Since I'm monitoring a process, I can use the <code>process_running</code> condition to determine whether the process is running. If the process is running, it will return true, otherwise it will return false. Since I sent a hash as the second argument to <code>transition</code>, the return from <code>process_running</code> will determine which of the two states will be transitioned to. If the process is running, the return is true and god will put the Watch into the <code>up</code> state. If the process is not running, the return is false and god will put the Watch into the <code>start</code> state.</p>
500 <pre><code class="ruby"> # determine when process has finished starting
501 w.transition([:start, :restart], :up) do |on|
502 on.condition(:process_running) do |c|
503 c.running = true
507 end</code></pre>
509 <p>If god has determined that my process isn't running, the Watch will be put into the <code>start</code> state. Upon entering this state, the <code>start</code> command that I specified on the Watch will be called. In addition, the above transition specifies a condition that should be enabled when in either the <code>start</code> or <code>restart</code> states. The condition is another <code>process_running</code>, however this time I'm only interested in moving to another state once it returns <code>true</code>. A <code>true</code> return from this condition means that the process is running and it's ok to transition to the <code>up</code> state (second argument to <code>transition</code>).</p>
511 <pre><code class="ruby"> # determine when process has finished starting
512 w.transition([:start, :restart], :up) do |on|
515 # failsafe
516 on.condition(:tries) do |c|
517 c.times = 3
518 c.transition = :start
520 end</code></pre>
522 <p>The other half of this transition uses the <code>tries</code> condition to ensure that god doesn't get stuck in this state. It's possible that the process could go down while the transition is being made, in which case god would end up polling forever to see if the process is up. Here I've specified that if this condition is called three times, god should override the normal transition destination and move to the <code>start</code> state instead. If you specify a <code>transition</code> attribute on any condition, that state will be transferred to instead of the normal transfer destination.</p>
524 <pre><code class="ruby"> # start if process is not running
525 w.transition(:up, :start) do |on|
526 on.condition(:process_exits)
527 end</code></pre>
529 <p>This is where the event based system comes into play. Once in the <code>up</code> state, I want to be notified when my process exits. The <code>process_exits</code> condition registers a callback that will trigger a transition change when it is fired off. Event conditions (like this one) cannot be used in transitions that have a hash for the second argument (as they do not return true or false).</p>
531 <pre><code class="ruby"> # restart if memory or cpu is too high
532 w.transition(:up, :restart) do |on|
533 on.condition(:memory_usage) do |c|
534 c.interval = 20
535 c.above = 50.megabytes
536 c.times = [3, 5]
539 on.condition(:cpu_usage) do |c|
540 c.interval = 10
541 c.above = 10.percent
542 c.times = [3, 5]
544 end</code></pre>
546 <p>Notice that I can have multiple transitions with the same start state. In this case, I want to have the <code>memory_usage</code> and <code>cpu_usage</code> poll conditions going at the same time that I listen for the process exit event. In the case of runaway CPU or memory usage, however, I want to transition to the <code>restart</code> state. When a Watch enters the <code>restart</code> state it will either call the <code>restart</code> command that you specified, or if none has been set, call the <code>stop</code> and then <code>start</code> commands.</p>
548 <!-- ------------------------------------------------------------------------- -->
550 <h1>Watching Non-Daemon Processes</h1>
552 <p>Need to watch a script that doesn't have built in daemonization? No problem! God will daemonize and keep track of your process for you. If you don't specify a <code>pid_file</code> attribute for a watch, it will be auto-daemonized and a PID file will be stored for it in <code>/var/run/god</code>. If you'd rather have the PID file stored in a different location, you can set it in a <code>God.init</code> block at the top of your config:</p>
554 <pre><code class="ruby">God.init do |god|
555 god.pid_file_directory = '/home/tom/pids'
558 God.watch do |w|
559 # watch with no pid_file attribute set
560 end</code></pre>
562 <p>The directory you specify must be writable by god.</p>
564 <!-- ------------------------------------------------------------------------- -->
566 <h1>Loading Other Config Files</h1>
568 <p>You should feel free to separate your god configs into separate files for easier organization. You can load in other configs using Ruby's normal <code>load</code> method, or use the convenience method <code>God.load</code> which allows for glob-style paths:</p>
570 <pre><code class="ruby"># load in all god configs
571 God.load "/usr/local/conf/*.god"</code></pre>
573 <p>God won't start it's monitoring operations until all configurations have been loaded.</p>
575 <!-- ------------------------------------------------------------------------- -->
577 <h1>Dynamically Loading Config Files Into an Already Running God</h1>
579 <p>God allows you to load or reload configurations into an already running instance. There are a few things to consider when doing this:</p>
581 <ul>
582 <li>Existng Watches with the same <code>name</code> as the incoming Watches will be overidden by the new config.</li>
583 <li>All paths must be either absolute or relative to the path from which god was started.</li>
584 </ul>
586 <p>To load a config into a running god, issue the following command:</p>
588 <pre>$ sudo god load path/to/config.god</pre>
590 <p>Config files that are loaded dynamically can contain anything that a normal config file contains, however, <code>God.init</code> blocks will be ignored.</p>
592 <!-- ------------------------------------------------------------------------- -->
594 <h1>Getting Logs for a Single Watch</h1>
596 <p>Sifting through the god logs for statements specific to a specific Watch can be frustrating when you have many of them. You can get the realtime logs for a single Watch via the command line:</p>
598 <pre>$ sudo god log 'local-3000'</pre>
600 <p>This will display the last 100 lines of log for the 'local-3000' Watch and update every second with new log messages.</p>
602 <!-- ------------------------------------------------------------------------- -->
604 <h1>Extend God with your own Conditions</h1>
606 <p>God was designed from the start to allow you to easily write your own custom conditions, making it simple to add tests that are application specific.</p>
608 <pre><code class="ruby">module God
609 module Conditions
611 class ProcessRunning < PollCondition
612 attr_accessor :running
614 def valid?
615 valid = true
616 valid &= complain("You must specify the 'pid_file' attribute \
617 on the Watch for :process_running") if self.watch.pid_file.nil?
618 valid &= complain("You must specify the 'running' attribute \
619 for :process_running") if self.running.nil?
620 valid
623 def test
624 return !self.running unless File.exist?(self.watch.pid_file)
626 pid = File.read(self.watch.pid_file).strip
627 active = System::Process.new(pid).exists?
629 (self.running && active) || (!self.running && !active)
634 end</code></pre>
636 </div>
637 <div id="footer">
638 <p>Brought to you by <a href="http://rubyisawesome.com/">Ruby is Awesome</a></p>
639 </div>
641 <script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
642 </script>
643 <script type="text/javascript">
644 _uacct = "UA-2196727-1";
645 urchinTracker();
646 </script>
648 </body>
649 </html>