doc/ext_ffi_tutorial.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
   2 <html>
   3 <head>
   4 <title>FFI Tutorial</title>
   5 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   6 <meta name="Author" content="Mike Pall">
   7 <meta name="Copyright" content="Copyright (C) 2005-2011, Mike Pall">
   8 <meta name="Language" content="en">
   9 <link rel="stylesheet" type="text/css" href="bluequad.css" media="screen">
  10 <link rel="stylesheet" type="text/css" href="bluequad-print.css" media="print">
  11 <style type="text/css">
  12 span.codemark { position:absolute; left: 16em; color: #4040c0; }
  13 span.mark { color: #4040c0; font-family: Courier New, Courier, monospace;
  14   line-height: 1.1; }
  15 pre.mark { padding-left: 2em; }
  16 table.idiomtable { line-height: 1.2; }
  17 table.idiomtable tt { font-size: 100%; }
  18 tr.idiomhead td { font-weight: bold; }
  19 td.idiomc { width: 12em; }
  20 td.idiomlua { width: 14em; }
  21 td.idiomlua b { font-weight: normal; color: #2142bf; }
  22 </style>
  23 </head>
  24 <body>
  25 <div id="site">
  26 <a href="http://luajit.org"><span>Lua<span id="logo">JIT</span></span></a>
  27 </div>
  28 <div id="head">
  29 <h1>FFI Tutorial</h1>
  30 </div>
  31 <div id="nav">
  32 <ul><li>
  33 <a href="luajit.html">LuaJIT</a>
  34 <ul><li>
  35 <a href="install.html">Installation</a>
  36 </li><li>
  37 <a href="running.html">Running</a>
  38 </li></ul>
  39 </li><li>
  40 <a href="extensions.html">Extensions</a>
  41 <ul><li>
  42 <a href="ext_ffi.html">FFI Library</a>
  43 <ul><li>
  44 <a class="current" href="ext_ffi_tutorial.html">FFI Tutorial</a>
  45 </li><li>
  46 <a href="ext_ffi_api.html">ffi.* API</a>
  47 </li><li>
  48 <a href="ext_ffi_semantics.html">FFI Semantics</a>
  49 </li></ul>
  50 </li><li>
  51 <a href="ext_jit.html">jit.* Library</a>
  52 </li><li>
  53 <a href="ext_c_api.html">Lua/C API</a>
  54 </li></ul>
  55 </li><li>
  56 <a href="status.html">Status</a>
  57 <ul><li>
  58 <a href="changes.html">Changes</a>
  59 </li></ul>
  60 </li><li>
  61 <a href="faq.html">FAQ</a>
  62 </li><li>
  63 <a href="http://luajit.org/performance.html">Performance <span class="ext">&raquo;</span></a>
  64 </li><li>
  65 <a href="http://luajit.org/download.html">Download <span class="ext">&raquo;</span></a>
  66 </li></ul>
  67 </div>
  68 <div id="main">
  69 <p>
  70 This page is intended to give you an overview of the features of the FFI
  71 library by presenting a few use cases and guidelines.
  72 </p>
  73 <p>
  74 This page makes no attempt to explain all of the FFI library, though.
  75 You'll want to have a look at the <a href="ext_ffi_api.html">ffi.* API
  76 function reference</a> and the <a href="ext_ffi_semantics.html">FFI
  77 semantics</a> to learn more.
  78 </p>
  79
  80 <h2 id="load">Loading the FFI Library</h2>
  81 <p>
  82 The FFI library is built into LuaJIT by default, but it's not loaded
  83 and initialized by default. The suggested way to use the FFI library
  84 is to add the following to the start of every Lua file that needs one
  85 of its functions:
  86 </p>
  87 <pre class="code">
  88 local ffi = require("ffi")
  89 </pre>
  90 <p>
  91 Please note this doesn't define an <tt>ffi</tt> variable in the table
  92 of globals &mdash; you really need to use the local variable. The
  93 <tt>require</tt> function ensures the library is only loaded once.
  94 </p>
  95
  96 <h2 id="sleep">Accessing Standard System Functions</h2>
  97 <p>
  98 The following code explains how to access standard system functions.
  99 We slowly print two lines of dots by sleeping for 10&nbsp;milliseconds
 100 after each dot:
 101 </p>
 102 <pre class="code mark">
 103 <span class="codemark">&nbsp;
 104 &#9312;
 105
 106
 107
 108
 109
 110 &#9313;
 111 &#9314;
 112 &#9315;
 113
 114
 115
 116 &#9316;
 117
 118
 119
 120
 121
 122 &#9317;</span>local ffi = require("ffi")
 123 ffi.cdef[[
 124 <span style="color:#00a000;">void Sleep(int ms);
 125 int poll(struct pollfd *fds, unsigned long nfds, int timeout);</span>
 126 ]]
 127
 128 local sleep
 129 if ffi.os == "Windows" then
 130   function sleep(s)
 131     ffi.C.Sleep(s*1000)
 132   end
 133 else
 134   function sleep(s)
 135     ffi.C.poll(nil, 0, s*1000)
 136   end
 137 end
 138
 139 for i=1,160 do
 140   io.write("."); io.flush()
 141   sleep(0.01)
 142 end
 143 io.write("\n")
 144 </pre>
 145 <p>
 146 Here's the step-by-step explanation:
 147 </p>
 148 <p>
 149 <span class="mark">&#9312;</span> This defines the
 150 C&nbsp;library functions we're going to use. The part inside the
 151 double-brackets (in green) is just standard C&nbsp;syntax. You can
 152 usually get this info from the C&nbsp;header files or the
 153 documentation provided by each C&nbsp;library or C&nbsp;compiler.
 154 </p>
 155 <p>
 156 <span class="mark">&#9313;</span> The difficulty we're
 157 facing here, is that there are different standards to choose from.
 158 Windows has a simple <tt>Sleep()</tt> function. On other systems there
 159 are a variety of functions available to achieve sub-second sleeps, but
 160 with no clear consensus. Thankfully <tt>poll()</tt> can be used for
 161 this task, too, and it's present on most non-Windows systems. The
 162 check for <tt>ffi.os</tt> makes sure we use the Windows-specific
 163 function only on Windows systems.
 164 </p>
 165 <p>
 166 <span class="mark">&#9314;</span> Here we're wrapping the
 167 call to the C&nbsp;function in a Lua function. This isn't strictly
 168 necessary, but it's helpful to deal with system-specific issues only
 169 in one part of the code. The way we're wrapping it ensures the check
 170 for the OS is only done during initialization and not for every call.
 171 </p>
 172 <p>
 173 <span class="mark">&#9315;</span> A more subtle point is
 174 that we defined our <tt>sleep()</tt> function (for the sake of this
 175 example) as taking the number of seconds, but accepting fractional
 176 seconds. Multiplying this by 1000 gets us milliseconds, but that still
 177 leaves it a Lua number, which is a floating-point value. Alas, the
 178 <tt>Sleep()</tt> function only accepts an integer value. Luckily for
 179 us, the FFI library automatically performs the conversion when calling
 180 the function (truncating the FP value towards zero, like in C).
 181 </p>
 182 <p style="font-size: 8pt;">
 183 Some readers will notice that <tt>Sleep()</tt> is part of
 184 <tt>KERNEL32.DLL</tt> and is also a <tt>stdcall</tt> function. So how
 185 can this possibly work? The FFI library provides the <tt>ffi.C</tt>
 186 default C&nbsp;library namespace, which allows calling functions from
 187 the default set of libraries, like a C&nbsp;compiler would. Also, the
 188 FFI library automatically detects <tt>stdcall</tt> functions, so you
 189 don't need to declare them as such.
 190 </p>
 191 <p>
 192 <span class="mark">&#9316;</span> The <tt>poll()</tt>
 193 function takes a couple more arguments we're not going to use. You can
 194 simply use <tt>nil</tt> to pass a <tt>NULL</tt> pointer and <tt>0</tt>
 195 for the <tt>nfds</tt> parameter. Please note that the
 196 number&nbsp;<tt>0</tt> <em>does not convert to a pointer value</em>,
 197 unlike in C++. You really have to pass pointers to pointer arguments
 198 and numbers to number arguments.
 199 </p>
 200 <p style="font-size: 8pt;">
 201 The page on <a href="ext_ffi_semantics.html">FFI semantics</a> has all
 202 of the gory details about
 203 <a href="ext_ffi_semantics.html#convert">conversions between Lua
 204 objects and C&nbsp;types</a>. For the most part you don't have to deal
 205 with this, as it's performed automatically and it's carefully designed
 206 to bridge the semantic differences between Lua and C.
 207 </p>
 208 <p>
 209 <span class="mark">&#9317;</span> Now that we have defined
 210 our own <tt>sleep()</tt> function, we can just call it from plain Lua
 211 code. That wasn't so bad, huh? Turning these boring animated dots into
 212 a fascinating best-selling game is left as an exercise for the reader.
 213 :-)
 214 </p>
 215
 216 <h2 id="zlib">Accessing the zlib Compression Library</h2>
 217 <p>
 218 The following code shows how to access the <a
 219 href="http://zlib.net/">zlib</a> compression library from Lua code.
 220 We'll define two convenience wrapper functions that take a string and
 221 compress or uncompress it to another string:
 222 </p>
 223 <pre class="code mark">
 224 <span class="codemark">&nbsp;
 225 &#9312;
 226
 227
 228
 229
 230
 231
 232 &#9313;
 233
 234
 235 &#9314;
 236
 237 &#9315;
 238
 239
 240 &#9316;
 241
 242
 243 &#9317;
 244
 245
 246
 247
 248
 249
 250
 251 &#9318;</span>local ffi = require("ffi")
 252 ffi.cdef[[
 253 <span style="color:#00a000;">unsigned long compressBound(unsigned long sourceLen);
 254 int compress2(uint8_t *dest, unsigned long *destLen,
 255               const uint8_t *source, unsigned long sourceLen, int level);
 256 int uncompress(uint8_t *dest, unsigned long *destLen,
 257                const uint8_t *source, unsigned long sourceLen);</span>
 258 ]]
 259 local zlib = ffi.load(ffi.os == "Windows" and "zlib1" or "z")
 260
 261 local function compress(txt)
 262   local n = zlib.compressBound(#txt)
 263   local buf = ffi.new("uint8_t[?]", n)
 264   local buflen = ffi.new("unsigned long[1]", n)
 265   local res = zlib.compress2(buf, buflen, txt, #txt, 9)
 266   assert(res == 0)
 267   return ffi.string(buf, buflen[0])
 268 end
 269
 270 local function uncompress(comp, n)
 271   local buf = ffi.new("uint8_t[?]", n)
 272   local buflen = ffi.new("unsigned long[1]", n)
 273   local res = zlib.uncompress(buf, buflen, comp, #comp)
 274   assert(res == 0)
 275   return ffi.string(buf, buflen[0])
 276 end
 277
 278 -- Simple test code.
 279 local txt = string.rep("abcd", 1000)
 280 print("Uncompressed size: ", #txt)
 281 local c = compress(txt)
 282 print("Compressed size: ", #c)
 283 local txt2 = uncompress(c, #txt)
 284 assert(txt2 == txt)
 285 </pre>
 286 <p>
 287 Here's the step-by-step explanation:
 288 </p>
 289 <p>
 290 <span class="mark">&#9312;</span> This defines some of the
 291 C&nbsp;functions provided by zlib. For the sake of this example, some
 292 type indirections have been reduced and it uses the pre-defined
 293 fixed-size integer types, while still adhering to the zlib API/ABI.
 294 </p>
 295 <p>
 296 <span class="mark">&#9313;</span> This loads the zlib shared
 297 library. On POSIX systems it's named <tt>libz.so</tt> and usually
 298 comes pre-installed. Since <tt>ffi.load()</tt> automatically adds any
 299 missing standard prefixes/suffixes, we can simply load the
 300 <tt>"z"</tt> library. On Windows it's named <tt>zlib1.dll</tt> and
 301 you'll have to download it first from the
 302 <a href="http://zlib.net/"><span class="ext">&raquo;</span>&nbsp;zlib site</a>. The check for
 303 <tt>ffi.os</tt> makes sure we pass the right name to
 304 <tt>ffi.load()</tt>.
 305 </p>
 306 <p>
 307 <span class="mark">&#9314;</span> First, the maximum size of
 308 the compression buffer is obtained by calling the
 309 <tt>zlib.compressBound</tt> function with the length of the
 310 uncompressed string. The next line allocates a byte buffer of this
 311 size. The <tt>[?]</tt> in the type specification indicates a
 312 variable-length array (VLA). The actual number of elements of this
 313 array is given as the 2nd argument to <tt>ffi.new()</tt>.
 314 </p>
 315 <p>
 316 <span class="mark">&#9315;</span> This may look strange at
 317 first, but have a look at the declaration of the <tt>compress2</tt>
 318 function from zlib: the destination length is defined as a pointer!
 319 This is because you pass in the maximum buffer size and get back the
 320 actual length that was used.
 321 </p>
 322 <p>
 323 In C you'd pass in the address of a local variable
 324 (<tt>&amp;buflen</tt>). But since there's no address-of operator in
 325 Lua, we'll just pass in a one-element array. Conveniently it can be
 326 initialized with the maximum buffer size in one step. Calling the
 327 actual <tt>zlib.compress2</tt> function is then straightforward.
 328 </p>
 329 <p>
 330 <span class="mark">&#9316;</span> We want to return the
 331 compressed data as a Lua string, so we'll use <tt>ffi.string()</tt>.
 332 It needs a pointer to the start of the data and the actual length. The
 333 length has been returned in the <tt>buflen</tt> array, so we'll just
 334 get it from there.
 335 </p>
 336 <p style="font-size: 8pt;">
 337 Note that since the function returns now, the <tt>buf</tt> and
 338 <tt>buflen</tt> variables will eventually be garbage collected. This
 339 is fine, because <tt>ffi.string()</tt> has copied the contents to a
 340 newly created (interned) Lua string. If you plan to call this function
 341 lots of times, consider reusing the buffers and/or handing back the
 342 results in buffers instead of strings. This will reduce the overhead
 343 for garbage collection and string interning.
 344 </p>
 345 <p>
 346 <span class="mark">&#9317;</span> The <tt>uncompress</tt>
 347 functions does the exact opposite of the <tt>compress</tt> function.
 348 The compressed data doesn't include the size of the original string,
 349 so this needs to be passed in. Otherwise no surprises here.
 350 </p>
 351 <p>
 352 <span class="mark">&#9318;</span> The code, that makes use
 353 of the functions we just defined, is just plain Lua code. It doesn't
 354 need to know anything about the LuaJIT FFI &mdash; the convenience
 355 wrapper functions completely hide it.
 356 </p>
 357 <p>
 358 One major advantage of the LuaJIT FFI is that you are now able to
 359 write those wrappers <em>in Lua</em>. And at a fraction of the time it
 360 would cost you to create an extra C&nbsp;module using the Lua/C API.
 361 Many of the simpler C&nbsp;functions can probably be used directly
 362 from your Lua code, without any wrappers.
 363 </p>
 364 <p style="font-size: 8pt;">
 365 Side note: the zlib API uses the <tt>long</tt> type for passing
 366 lengths and sizes around. But all those zlib functions actually only
 367 deal with 32&nbsp;bit values. This is an unfortunate choice for a
 368 public API, but may be explained by zlib's history &mdash; we'll just
 369 have to deal with it.
 370 </p>
 371 <p style="font-size: 8pt;">
 372 First, you should know that a <tt>long</tt> is a 64&nbsp;bit type e.g.
 373 on POSIX/x64 systems, but a 32&nbsp;bit type on Windows/x64 and on
 374 32&nbsp;bit systems. Thus a <tt>long</tt> result can be either a plain
 375 Lua number or a boxed 64&nbsp;bit integer cdata object, depending on
 376 the target system.
 377 </p>
 378 <p style="font-size: 8pt;">
 379 Ok, so the <tt>ffi.*</tt> functions generally accept cdata objects
 380 wherever you'd want to use a number. That's why we get a away with
 381 passing <tt>n</tt> to <tt>ffi.string()</tt> above. But other Lua
 382 library functions or modules don't know how to deal with this. So for
 383 maximum portability one needs to use <tt>tonumber()</tt> on returned
 384 <tt>long</tt> results before passing them on. Otherwise the
 385 application might work on some systems, but would fail in a POSIX/x64
 386 environment.
 387 </p>
 388
 389 <h2 id="idioms">Translating C&nbsp;Idioms</h2>
 390 <p>
 391 Here's a list of common C&nbsp;idioms and their translation to the
 392 LuaJIT FFI:
 393 </p>
 394 <table class="idiomtable">
 395 <tr class="idiomhead">
 396 <td class="idiomdesc">Idiom</td>
 397 <td class="idiomc">C&nbsp;code</td>
 398 <td class="idiomlua">Lua code</td>
 399 </tr>
 400 <tr class="odd separate">
 401 <td class="idiomdesc">Pointer dereference<br><tt>int *p;</tt></td><td class="idiomc"><tt>x = *p;<br>*p = y;</tt></td><td class="idiomlua"><tt>x = <b>p[0]</b><br><b>p[0]</b> = y</tt></td></tr>
 402 <tr class="even">
 403 <td class="idiomdesc">Pointer indexing<br><tt>int i, *p;</tt></td><td class="idiomc"><tt>x = p[i];<br>p[i+1] = y;</tt></td><td class="idiomlua"><tt>x = p[i]<br>p[i+1] = y</tt></td></tr>
 404 <tr class="odd">
 405 <td class="idiomdesc">Array indexing<br><tt>int i, a[];</tt></td><td class="idiomc"><tt>x = a[i];<br>a[i+1] = y;</tt></td><td class="idiomlua"><tt>x = a[i]<br>a[i+1] = y</tt></td></tr>
 406 <tr class="even separate">
 407 <td class="idiomdesc"><tt>struct</tt>/<tt>union</tt> dereference<br><tt>struct foo s;</tt></td><td class="idiomc"><tt>x = s.field;<br>s.field = y;</tt></td><td class="idiomlua"><tt>x = s.field<br>s.field = y</tt></td></tr>
 408 <tr class="odd">
 409 <td class="idiomdesc"><tt>struct</tt>/<tt>union</tt> pointer deref.<br><tt>struct foo *sp;</tt></td><td class="idiomc"><tt>x = sp->field;<br>sp->field = y;</tt></td><td class="idiomlua"><tt>x = <b>s.field</b><br><b>s.field</b> = y</tt></td></tr>
 410 <tr class="even separate">
 411 <td class="idiomdesc">Pointer arithmetic<br><tt>int i, *p;</tt></td><td class="idiomc"><tt>x = p + i;<br>y = p - i;</tt></td><td class="idiomlua"><tt>x = p + i<br>y = p - i</tt></td></tr>
 412 <tr class="odd">
 413 <td class="idiomdesc">Pointer difference<br><tt>int *p1, *p2;</tt></td><td class="idiomc"><tt>x = p1 - p2;</tt></td><td class="idiomlua"><tt>x = p1 - p2</tt></td></tr>
 414 <tr class="even">
 415 <td class="idiomdesc">Array element pointer<br><tt>int i, a[];</tt></td><td class="idiomc"><tt>x = &amp;a[i];</tt></td><td class="idiomlua"><tt>x = <b>a+i</b></tt></td></tr>
 416 <tr class="odd">
 417 <td class="idiomdesc">Cast pointer to address<br><tt>int *p;</tt></td><td class="idiomc"><tt>x = (intptr_t)p;</tt></td><td class="idiomlua"><tt>x = <b>tonumber(<br>&nbsp;ffi.cast("intptr_t",<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;p))</b></tt></td></tr>
 418 <tr class="even separate">
 419 <td class="idiomdesc">Functions with outargs<br><tt>void foo(int *inoutlen);</tt></td><td class="idiomc"><tt>int len = x;<br>foo(&amp;len);<br>y = len;</tt></td><td class="idiomlua"><tt><b>local len =<br>&nbsp;&nbsp;ffi.new("int[1]", x)<br>foo(len)<br>y = len[0]</b></tt></td></tr>
 420 <tr class="odd">
 421 <td class="idiomdesc"><a href="ext_ffi_semantics.html#convert_vararg">Vararg conversions</a><br><tt>int printf(char *fmt, ...);</tt></td><td class="idiomc"><tt>printf("%g", 1.0);<br>printf("%d", 1);<br>&nbsp;</tt></td><td class="idiomlua"><tt>printf("%g", 1);<br>printf("%d",<br>&nbsp;&nbsp;<b>ffi.new("int", 1)</b>)</tt></td></tr>
 422 </table>
 423
 424 <h2 id="cache">To Cache or Not to Cache</h2>
 425 <p>
 426 It's a common Lua idiom to cache library functions in local variables
 427 or upvalues, e.g.:
 428 </p>
 429 <pre class="code">
 430 local byte, char = string.byte, string.char
 431 local function foo(x)
 432   return char(byte(x)+1)
 433 end
 434 </pre>
 435 <p>
 436 This replaces several hash-table lookups with a (faster) direct use of
 437 a local or an upvalue. This is less important with LuaJIT, since the
 438 JIT compiler optimizes hash-table lookups a lot and is even able to
 439 hoist most of them out of the inner loops. It can't eliminate
 440 <em>all</em> of them, though, and it saves some typing for often-used
 441 functions. So there's still a place for this, even with LuaJIT.
 442 </p>
 443 <p>
 444 The situation is a bit different with C&nbsp;function calls via the
 445 FFI library. The JIT compiler has special logic to eliminate <em>all
 446 of the lookup overhead</em> for functions resolved from a
 447 <a href="ext_ffi_semantics.html#clib">C&nbsp;library namespace</a>!
 448 Thus it's not helpful and actually counter-productive to cache
 449 individual C&nbsp;functions like this:
 450 </p>
 451 <pre class="code">
 452 local <b>funca</b>, <b>funcb</b> = ffi.C.funcb, ffi.C.funcb -- <span style="color:#c00000;">Not helpful!</span>
 453 local function foo(x, n)
 454   for i=1,n do <b>funcb</b>(<b>funca</b>(x, i), 1) end
 455 end
 456 </pre>
 457 <p>
 458 This turns them into indirect calls and generates bigger and slower
 459 machine code. Instead you'll want to cache the namespace itself and
 460 rely on the JIT compiler to eliminate the lookups:
 461 </p>
 462 <pre class="code">
 463 local <b>C</b> = ffi.C          -- <span style="color:#00a000;">Instead use this!</span>
 464 local function foo(x, n)
 465   for i=1,n do <b>C.funcb</b>(<b>C.funca</b>(x, i), 1) end
 466 end
 467 </pre>
 468 <p>
 469 This generates both shorter and faster code. So <b>don't cache
 470 C&nbsp;functions</b>, but <b>do</b> cache namespaces! Most often the
 471 namespace is already in a local variable at an outer scope, e.g. from
 472 <tt>local&nbsp;lib&nbsp;=&nbsp;ffi.load(...)</tt>. Note that copying
 473 it to a local variable in the function scope is unnecessary.
 474 </p>
 475 <br class="flush">
 476 </div>
 477 <div id="foot">
 478 <hr class="hide">
 479 Copyright &copy; 2005-2011 Mike Pall
 480 <span class="noprint">
 481 &middot;
 482 <a href="contact.html">Contact</a>
 483 </span>
 484 </div>
 485 </body>
 486 </html>