index.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   2
   3 <html xmlns="http://www.w3.org/1999/xhtml">
   4 <head>
   5 <meta http-equiv="content-type" content="text/html; charset=utf-8" />
   6 <title>Light Weight Event System</title>
   7 <meta name="keywords" content="" />
   8 <meta name="description" content="" />
   9 <link href="/css/style.css" rel="stylesheet" type="text/css" />
  10 </head>
  11 <body>
  12 <div id="wrapper">
  13 <!-- start header -->
  14 <div id="logo">
  15         <h1><a href="http://www.lwes.org">Light Weight Event System </a></h1>
  16 </div>
  17 <div id="header">
  18         <div id="menu">
  19                 <ul>
  20                         <li class="current_page_item"><a href="/">Introduction</a></li>
  21                         <li><a href="/docs">Documentation</a></li>
  22                         <li><a href="/download">Download</a></li>
  23                         <li class="last"><a href="/contact">Contact</a></li>
  24                 </ul>
  25         </div>
  26 </div>
  27 <!-- end header -->
  28 </div>
  29 <!-- start page -->
  30 <div id="page">
  31         <!-- start content -->
  32         <div id="content">
  33                 <div class="post">
  34                         <h1 class="title">Introduction</h1>
  35                         <div class="entry">
  36                                 <p>
  37                                         The Light Weight Event System (LWES) is an open source toolkit allowing the exchange of information between machines in a platform agnostic, language neutral, decoupled way.  The exchange of information is done in a connectionless fashion using multicast or unicast UDP, using self-describing data so that any platform or language can translate it into its local dialect.
  38                                 </p>
  39                                 <p>This leads to a system with the following features:
  40                                 <ul>
  41                                         <li>Fire and forget messaging</li>
  42                                         <li>Decoupling of senders (emitter) from receivers (listeners)</li>
  43                                         <li>Centerless, no single point of failure</li>
  44                                         <li>Support for many to many communications where nodes can enter and leave the system at any time</li>
  45                                         <li>Computer language and hardware platform independent</li>
  46                                 </ul>
  47                                 </p>
  48                                 <p>The system consists of two major components:
  49                                 <ul>
  50                                         <li>Emitter: a source of events, this is usually a server or process that wants to announce something</li>
  51                                         <li>Listener: a sink of events which deserializes and processes events on the fly as they are received.</li>
  52                                 </ul>
  53                                 </p>
  54                                 <p>Optionally, one can run a "journaller", which is a listener that writes raw events to a compressed log instead of deserializing the data, effectively deferring deserialization to a later time.</p>
  55                                 <p>LWES uses key-value pair data types and supports strings as keys and a set of primitive types as values.  Note that LWES does not support complex types or nested objects.  This was a deliberate design decision in order to constrain the scope of the system and limit data exchanges to just key-value pairs.  If you need to exchange complex types or do RPC, tools such as <a href="http://incubator.apache.org/thrift/">Thrift</a> or <a href="http://code.google.com/p/protobuf/">Protocol Buffers</a> are probably more appropriate for you.
  56                                 </p>
  57                                 <p>The Light Weight Event System was originally developed at <a href="http://web.archive.org/web/20010830140743/http://www.goto.com/">GoTo.com</a>, which became <a href="http://web.archive.org/web/20011118191954/www.overture.com/">Overture</a>, which was acquired by <a href="http://www.yahoo.com">Yahoo!</a>, where it was open sourced in 2007.  Currently, it is maintained by developers at <a href="http://www.openx.org">OpenX</a> as well as some of the original contributors.
  58                         </div>
  59                 </div>
  60                 <div class="post">
  61                         <h2 class="title">LWES Design Features</h2>
  62                         <div class="entry">
  63                                 <p>
  64                                         LWES is specifically designed for non-blocking, best-effort delivery of data from a sender (emitter) to a set of receivers (listeners).  The LWES library provides UDP unicast and multicast implementations of emitters and listeners which provide these functions in various programming languages.  However, you could easily write your own transport and only use the LWES API and serialization methods to generate network-ordered byte arrays.  But the original intention was to use UDP to provide connectionless, best effort delivery of data.
  65                                 </p>
  66                                 <h4>But Isn't UDP Unreliable?</h4>
  67                                 <p>
  68                                         One of the most common questions regarding the Light Weight Event System is the design choice to use UDP multicast by default.  Packets can get dropped, machines can have hardware issues, and networks can get congested, in which case data can be lost forever.  This should be taken account before using LWES -- it is a best-effort system designed to provide minimal impact to applications that are senders (emitters).  Some key points:
  69                                 <ul>
  70                                         <li>Establishing TCP connections is expensive.  If you don't need reliable, guaranteed delivery, it can be overkill.</li>
  71                                         <li>You can run multiple journallers or listeners when using multicast, to protect against listener failure.</li>
  72                                         <li>Implementing non-blocking clients over TCP (for example, using libevent) is non-trivial.</li>
  73                                         <li>When getting data from point A to point B, you have to do flow control <i>somewhere</i>.  LWES puts the burden of flow control on the receiver so that applications sending data are minimally impacted from a performance standpoint.</li>
  74                                         <li>You can always implement your own transport if you don't like UDP.</li>
  75                                 </ul>
  76                                 </p>
  77                         </div>
  78                 </div>
  79                 <div class="post">
  80                         <h2 class="title">LWES Use Cases</h2>
  81                         <div class="entry">
  82                                 <p>
  83                                         Taking these considerations into account, LWES is useful in a number of specialized scenarios:
  84                                         <ul>
  85                                                 <li>Monitoring and Instrumentation: machines can report about their status using LWES, and applications are minimally blocked by its presence.  Other machines can receive these events and process them.  <a href="http://www.sourceforge.net/projects/mondemand">MonDemand</a> is an implementation of an monitoring and instrumentation system using LWES.</li>
  86                                                 <li>Logging: machines can log events remotely over LWES to journaller machines to be processed in batch at a later date.  The LWES deserialization and access methods perform very well on batch systems such as <a href="http://hadoop.apache.org">Hadoop</a>.</li>
  87                                                 <li>Feedback Loops: services can report their status, and a separate listener can make feedback decisions and pass the data back to the running service.</li>
  88                                                 <li>Ad-Hoc Debugging: if a system is sending diagnostic events over UDP multicast, you could easily start up a listener and start to see what's happening in real-time.  When you're finished, you simply stop the listener and the system is unaffected.</li>
  89                                         </ul>
  90                                 </p>
  91                         </div>
  92                 </div>
  93
  94         </div>
  95         <!-- end content -->
  96         <!-- start sidebar -->
  97         <div id="sidebar">
  98                 <ul>
  99
 100                         <li>
 101                                 <h2>Useful Links</h2>
 102                                 <ul>
 103                                         <li><a href="http://sourceforge.net/projects/lwes/">SourceForge Project Page</a></li>
 104                                         <li><a href="http://sourceforge.net/project/showfiles.php?group_id=234456">Downloads</a></li>
 105                                         <li><a href="http://sourceforge.net/scm/?type=svn&group_id=234456">Source Code</a></li>
 106                                         <li><a href="http://sourceforge.net/tracker/?group_id=234456">Bug Reports</a></li>
 107                                 </ul>
 108                         </li>
 109                         <li>
 110                                 <h2>Contributors</h2>
 111                                 <a href="http://www.openx.org"><img src="/images/openx.jpg"/></a>
 112                                 <p/>
 113                                 <a href="http://www.att.com"><img src="/images/att.png"/></a>
 114                         </li>
 115                 </ul>
 116         </div>
 117         <!-- end sidebar -->
 118         <div style="clear: both;">&nbsp;</div>
 119 </div>
 120 <!-- end page -->
 121 <!-- start footer -->
 122 <div id="footer">
 123         <p id="legal">&copy; 2009 Light Weight Event System. All Rights Reserved.</p>
 124 </div>
 125 <!-- end footer -->
 126 </body>
 127 </html>