1 #+OPTIONS: H:3 num:nil toc:2 \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:(HIDE) tags:not-in-toc
2 #+STARTUP: align fold nodlcheck hidestars oddeven lognotestate hideblocks
3 #+SEQ_TODO: TODO(t) INPROGRESS(i) WAITING(w@) | DONE(d) CANCELED(c@)
4 #+TAGS: Write(w) Update(u) Fix(f) Check(c)
6 #+AUTHOR: Dan Davison, Eric Schulte
7 #+EMAIL: davison at stats dot ox dot ac dot uk
13 <p>executable source code blocks in org-mode</p>
17 <img src="images/tower-of-babel.png" alt="images/tower-of-babel.png"
18 title="And the Lord said, Behold, the people is one, and they have all one language; and this they begin to do; and now nothing will be restrained from them, which they have imagined to do. Genesis 11:1-9"/>
21 <a href="http://www.flickr.com/photos/23379658@N05/" title=""><b>Martijn Streefkerk</b></a>
34 :CUSTOM_ID: introduction
36 Org-babel is an extension to the very excellent [[http://orgmode.org/][Org-mode]]; an [[http://www.gnu.org/software/emacs/][Emacs]]
37 major mode for doing almost anything with plain text. If you are
38 not familiar with Org-mode please take a moment to read [[http://orgmode.org/][the Org-mode
39 homepage]] before continuing.
41 Org-babel provides the following modifications to [[http://orgmode.org/manual/Literal-examples.html][the existing
42 support]] for blocks of source code examples in the org-mode core.
44 1. Interactive source code execution
45 2. Arguments to source code blocks
46 3. Exportation of source code blocks to files (literate programming)
48 Much of this document consists of interactive examples, and it might
49 therefore make most sense to grab the plain text version of this file
50 #+HTML: <a href="org-babel-worg.org">org-babel-worg.org</a>
51 and work through it in Emacs. Alternatively you can the htmlized
52 version of the plain text of this file at
53 #+HTML: <a href="org-babel-worg.org.html">org-babel-worg.html</a>
54 allows the plain text version to be viewed (non-interactively) in a web browser.
58 :CUSTOM_ID: getting-started
61 1) Grab the latest code from the git repo at [[http://github.com/eschulte/org-babel/tree/master][github/org-babel]]
63 git clone git://github.com/eschulte/org-babel.git
66 2) Add the following lines to your .emacs, replacing the path as
67 appropriate. A good place to check that things are up and running
68 would the examples in [[* Basic org-babel functionality][Basic org-babel functionality]].
69 #+begin_src emacs-lisp
70 (add-to-list 'load-path "/path/to/org-babel/lisp")
71 (require 'org-babel-init)
74 3) Then activate the subset of supported Org-babel languages which
75 you will want to be able to execute on your system. The
76 following can be added to your .emacs and used to activate
77 languages. It includes a brief list of the requirements for each
78 language. *Note*: this also serves as the list of languages
79 currently supported by Org-babel.
80 #+begin_src emacs-lisp
81 ;; Uncomment each of the following require lines if you want org-babel
82 ;; to support that language. Each language has a comment explaining
83 ;; it's dependencies. See the related files in lisp/langs for more
84 ;; detailed explanations of requirements.
85 ;; (require 'org-babel-R) ;; R and ess-mode
86 ;; (require 'org-babel-asymptote) ;; asymptote
87 ;; (require 'org-babel-css) ;; none
88 ;; (require 'org-babel-ditaa) ;; ditaa
89 ;; (require 'org-babel-dot) ;; dot
90 ;; (require 'org-babel-gnuplot) ;; gnuplot, and gnuplot-mode
91 ;; (require 'org-babel-haskell) ;; haskell, haskell-mode, inf-haskell
92 ;; (require 'org-babel-ocaml) ;; ocaml, and tuareg-mode
93 ;; (require 'org-babel-python) ;; python, and python-mode
94 ;; (require 'org-babel-ruby) ;; ruby, irb, ruby-mode, and inf-ruby
95 ;; (require 'org-babel-sass) ;; sass, sass-mode
96 ;; (require 'org-babel-sql) ;; none
98 ;; Once you've activated languages, load the library of babel for
99 ;; pre-built helpers in the languages you will be using.
100 (org-babel-load-library-of-babel)
103 * Basic org-babel functionality
105 :CUSTOM_ID: basic-functionality
107 *** Source code execution
109 :CUSTOM_ID: source-code-execution
111 For interpreted languages such as shell, python, R, etc, org-babel
112 allows source blocks to be executed: the code is passed to the
113 interpreter and you have control over what is done with the
114 results of excecution. E.g. place point anywhere in the following
115 block and use =C-c C-c= to run the code:
117 *Note:* calling =C-c C-o= on a source-code block will open the
120 [[http://www.ruby-lang.org/][Ruby]] source code
122 "This file was last evaluated on #{Date.today}"
125 Results of Ruby evaluation
127 : This file was last evaluated on 2009-08-09
129 [[http://www.r-project.org/][R]] source code
130 #+begin_src R :results value
136 Results of R evaluation
141 [[http://ditaa.sourceforge.net/][ditaa]] source code
142 #+begin_src ditaa :file images/blue.png :cmdline -r
153 [[file:images/blue.png]]
155 *** What happens to the results?
159 Org-babel provides two fundamentally different modes for capturing
160 the results of code evaluation, specified by the :results header
163 This means that the 'result' of code evaluation is defined to be
164 the *value* of the last statement in the block. Thus with this
165 setting, one can view the code block as a function with a return
166 value. And not only can one view it that way, but you can
167 actually use the return value of one source block as input for
168 another (see later). This setting is the default.
170 With this setting, org-babel captures all the text output of the
171 code block and places it in the org buffer. One can think of this
172 as a 'scripting' mode: the code block contains a series of
173 commands, and you get the output of all the commands. Unlike in
174 the 'functional' mode specified by =:results value=, the code
175 block has no return value. (This mode will be familiar to Sweave
177 **** Additional :results settings
179 *** Arguments to source code blocks
181 :CUSTOM_ID: arguments-to-source-code-blocks
183 In addition to evaluation of code blocks, org-babel allows them to
184 be parameterised (i.e. have arguments). Thus source code blocks
185 now have the status of *functions*.
187 Inputs for fibonacci-seq
189 #+tblname: fibonacci-inputs
190 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
191 | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 |
193 in the Org-mode buffer this looks like
194 : #+tblname: fibonacci-inputs
195 : | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
196 : | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 |
198 [[http://www.gnu.org/software/emacs/manual/elisp.html][Emacs Lisp]] source code
199 #+srcname: fibonacci-seq
200 #+begin_src emacs-lisp :var fib-inputs=fibonacci-inputs
202 (if (or (= n 0) (= n 1))
204 (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))
206 (mapcar (lambda (row)
207 (mapcar #'fibonacci row)) fib-inputs)
210 in the Org-mode buffer this looks like
211 : #+srcname: fibonacci-seq
212 : #+begin_src emacs-lisp :var fib-inputs=fibonacci-inputs
213 : (defun fibonacci (n)
214 : (if (or (= n 0) (= n 1))
216 : (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))
218 : (mapcar (lambda (row)
219 : (mapcar #'fibonacci row)) fib-inputs)
222 Results of Emacs Lisp code evaluation
224 | 1 | 1 | 2 | 3 | 5 | 8 | 13 | 21 | 34 | 55 |
225 | 1 | 3 | 8 | 21 | 55 | 144 | 377 | 987 | 2584 | 6765 |
227 * A meta-programming language for org-mode
229 :CUSTOM_ID: meta-programming-language
232 Since information can pass freely between source-code blocks and
233 org-mode tables you can mix and match languages using each language
234 for those tasks to which it is suited. This makes Org-mode files with
235 Org-babel into a kind of meta-functional programming language in which
236 functions from many languages can work together.
238 As an example, lets take some system diagnostics in the shell, and
239 then graph them with R.
242 #+srcname: directories
243 #+begin_src bash :results replace
244 cd ~ && du -sc * |grep -v total
246 2. Results of the shell source code (on my system, grab this org-mode
247 files and try running it on your own)
248 #+resname: directories
250 | 12156104 | "Documents" |
251 | 3482440 | "Downloads" |
252 | 2901720 | "Library" |
254 | 16548024 | "Music" |
256 | 7649472 | "Pictures" |
264 3. R source code (which calls the previous shell source code)
265 #+srcname: directory-pie
266 #+begin_src R :var dirs = directories :session R-pie-example
267 pie(dirs[,1], labels = dirs[,2])
269 4. Results of R code [[file:images/dirs.png]]
271 * Spreadsheet plugins for org-mode in any language
273 :CUSTOM_ID: spreadsheet
276 *NOTE*: Maybe in-addition-to/in-stead-of this example we should do a
277 more traditional "spreadsheet" example with R [Eric]
279 Not only can Org-babel pass entire tables of data to source code
280 blocks (see [[arguments-to-source-code-blocks]]), Org-babel can also be
281 used to call source code blocks from *within* tables using the
282 Org-mode's [[http://orgmode.org/manual/The-spreadsheet.html#The-spreadsheet][existing spreadsheet functionality]].
284 In fact the functional test suite for Org-babel is implemented as a
285 large Org-mode table. To run the entire test suite you simple
286 evaluate the table =C-u C-c C-c=, and all of the tests are run
287 updating the table with pass/fail statistics.
289 Here's a sample of our test suite.
291 #+TBLNAME: org-babel-tests
292 | functionality | block | arg | expected | results | pass |
293 |------------------+--------------+-----+-------------+-------------+------|
294 | basic evaluation | | | | | pass |
295 |------------------+--------------+-----+-------------+-------------+------|
296 | emacs lisp | basic-elisp | 2 | 4 | 4 | pass |
297 | shell | basic-shell | | 6 | 6 | pass |
298 | ruby | basic-ruby | | org-babel | org-babel | pass |
299 | python | basic-python | | hello world | hello world | pass |
300 | R | basic-R | | 13 | 13 | pass |
301 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
302 #+TBLFM: $5=""::$6=""
304 *** code blocks for tests
306 #+srcname: basic-elisp
307 #+begin_src emacs-lisp :var n=7
311 #+srcname: basic-shell
312 #+begin_src sh :results silent
316 #+srcname: date-simple
317 #+begin_src sh :results silent
321 #+srcname: basic-ruby
322 #+begin_src ruby :results silent
326 #+srcname: basic-python
327 #+begin_src python :results silent
332 #+begin_src R :results silent
339 :CUSTOM_ID: library-of-babel
341 What about those source code blocks which are so useful you want to
342 have them available in every org-mode buffer?
344 The [[file:library-of-babel.org][Library of Babel]] is an extensible collection of ready-made and
345 easily-shortcut-callable source-code blocks for handling common
346 tasks. Org-babel comes pre-populated with the source-code blocks
347 located in the [[file:library-of-babel.org][library-of-babel.org]] file. It is possible to add
348 source-code blocks from any org-mode file to the library by calling
350 #+srcname: add-file-to-lob
351 #+begin_src emacs-lisp
352 (org-babel-lob-ingest "path/to/file.org")
355 * Reproducible Research
357 :CUSTOM_ID: reproducable-research
360 An article about computational science in a scientific publication is
361 not the scholarship itself, it is merely advertising of the
362 scholarship. The actual scholarship is the complete software
363 development environment and the complete set of instructions which
364 generated the figures.
369 [[http://reproducibleresearch.net/index.php/Main_Page][Reproducible Research]] (RR) is the practice of distributing along with
370 an article of research all data, code, and tools required to reproduce
371 the results discussed in the paper. As such the paper becomes not
372 only a document describing the research but a complete laboratory
373 reproducing the research.
375 Org-mode already has exceptional support for [[http://orgmode.org/manual/Exporting.html#Exporting][exporting to html and
376 LaTeX]]. Org-babel makes Org-mode a tool for RR by *activating* the
377 data and source code embedded into Org-mode documents making the
378 entire document executable. This makes it not only possible, but
379 natural to distribute research in a format that encourages readers to
380 recreate your results, and perform their own analysis.
382 Existing RR tools like [[http://en.wikipedia.org/wiki/Sweave][Sweave]] provide for the embedding of [[http://www.r-project.org/][R]] code into
383 LaTeX documents. While this is very useful, such documents often
384 still require a large degree of "glue code" in the form of external
385 shell scripts, python scripts, and Makefiles. To our knowledge
386 Org-babel is the only RR tool which allows multiple languages and data
387 to coexist and cooperate inside of a single document.
389 * Literate programming
391 :CUSTOM_ID: literate-programming
395 Let us change our traditional attitude to the construction of
396 programs: Instead of imagining that our main task is to instruct a
397 /computer/ what to do, let us concentrate rather on explaining to
398 /human beings/ what we want a computer to do.
400 The practitioner of literate programming can be regarded as an
401 essayist, whose main concern is with exposition and excellence of
402 style. Such an author, with thesaurus in hand, chooses the names of
403 variables carefully and explains what each variable means. He or she
404 strives for a program that is comprehensible because its concepts have
405 been introduced in an order that is best for human understanding,
406 using a mixture of formal and informal methods that reinforce each
412 Org-babel supports [[http://en.wikipedia.org/wiki/Literate_programming][Literate Programming]] (LP) by allowing the act of
413 programming to take place inside of Org-mode documents. The Org-mode
414 file can then be exported (*woven* in LP speak) to html or LaTeX for
415 consumption by a human, and the embedded source code can be extracted
416 (*tangled* in LP speak) into structured source code files for
417 consumption by a computer.
419 To support these operations Org-babel relies on Org-mode's [[http://orgmode.org/manual/Exporting.html#Exporting][existing
420 exporting functionality]] for *weaving* of documentation, and on the
421 =org-babel-tangle= function which makes use of [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]] [[reference-expansion][reference syntax]]
422 for *tangling* of code files.
424 The [[literate-programming-example][following example]] demonstrates the process of *tangling* in
427 *** Simple Literate Programming Example (Noweb syntax)
429 :CUSTOM_ID: literate-programming-example
432 Tangling functionality is controlled by the =tangle= family of
433 [[header-arguments]]. These arguments can be used to turn tangling on or
434 off (the default) on the source code block, or the outline heading
437 The following demonstrates the combination of three source code blocks
438 into a single source code file using =org-babel-tangle=.
440 The following two blocks will not be tangled by default since they
441 have no =tangle= header arguments.
443 #+srcname: hello-world-prefix
444 #+begin_src sh :exports none
445 echo "/-----------------------------------------------------------\\"
448 : #+srcname: hello-world-prefix
449 : #+begin_src sh :exports none
450 : echo "/-----------------------------------------------------------\\"
453 #+srcname: hello-world-postfix
454 #+begin_src sh :exports none
455 echo "\-----------------------------------------------------------/"
458 : #+srcname: hello-world-postfix
459 : #+begin_src sh :exports none
460 : echo "\-----------------------------------------------------------/"
464 The third block does have a =tangle= header argument indicating the
465 name of the file to which it should be written. It also has [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]]
466 style references to the two previous source code blocks which will be
467 expanded during tangling to include them in the output file as well.
469 #+srcname: hello-world
470 #+begin_src sh :tangle hello :exports none
471 # <<hello-world-prefix>>
472 echo "| hello world |"
473 # <<hello-world-postfix>>
476 : #+srcname: hello-world
477 : #+begin_src sh :tangle hello :exports none
478 : # <<hello-world-prefix>>
479 : echo "| hello world |"
480 : # <<hello-world-postfix>>
483 Calling =org-babel-tangle= will result in the following being written
484 to the =hello.sh= file.
486 #+srcname: hello-world-output
489 # generated by org-babel-tangle
491 # [[file:~/src/org-babel/org-babel-worg.org::#literate-programming-example][block-16]]
492 # <<hello-world-prefix>>
493 echo "/-----------------------------------------------------------\\"
495 echo "| hello world |"
496 # <<hello-world-postfix>>
497 echo "\-----------------------------------------------------------/"
501 *** Emacs Initialization with Org-babel
502 Org-babel has special support for embedding your emacs initialization
503 into Org-mode files. The =org-babel-load-file= function can be used
504 to load the emacs lisp embedded in a literate Org-mode file in the
505 same way that you might load a regular elisp file.
507 This allows you to have all the niceness of Org-mode (folding, tags,
508 notes, html export, etc...) available in your emacs initialization.
510 To try this out either see the simple [[literate-emacs-init][Literate Emacs Initialization]]
511 example directly below, or check out the Org-babel Literate
512 Programming version of Phil Hagelberg's excellent [[http://github.com/technomancy/emacs-starter-kit/tree/master][emacs-starter-kit]]
513 available at [[http://github.com/eschulte/emacs-starter-kit/tree/master][Org-babel-emacs-starter-kit]].
515 ***** Literate Emacs Initialization
517 :CUSTOM_ID: literate-emacs-init
520 For a simple example of usage follow these 4 steps.
522 1) create a directory named =.emacs.d= in the base of your home
527 2) checkout the latest versions of Org-mode and Org-babel into the src
528 subdirectory of this new directory
533 git clone git://repo.or.cz/org-mode.git
534 git clone git://github.com/eschulte/org-babel.git
536 3) place the following in a file called =init.el= in your emacs
537 initialization directory (=~/.emacs.d=).
538 #+srcname: emacs-init
539 #+begin_src emacs-lisp
540 ;;; init.el --- Where all the magic begins
542 ;; This file loads both
543 ;; - Org-mode : http://orgmode.org/ and
544 ;; - Org-babel: http://eschulte.github.com/org-babel/
546 ;; It then loads the rest of our Emacs initialization from Emacs lisp
547 ;; embedded in literate Org-mode files.
549 ;; Load up Org Mode and Org Babel for elisp embedded in Org Mode files
550 (setq dotfiles-dir (file-name-directory (or (buffer-file-name) load-file-name)))
551 (add-to-list 'load-path (expand-file-name
552 "lisp" (expand-file-name
553 "org" (expand-file-name
554 "src" dotfiles-dir))))
555 (add-to-list 'load-path (expand-file-name
556 "lisp" (expand-file-name
557 "org-babel" (expand-file-name
558 "src" dotfiles-dir))))
559 (require 'org-babel-init)
561 ;; load up all literate org-mode files in this directory
562 (mapc #'org-babel-load-file (directory-files dotfiles-dir t "\\.org$"))
564 ;;; init.el ends here
566 4) Implement all of your emacs customizations inside of elisp
567 source-code blocks located in Org-mode files in this directory.
568 They will be loaded by emacs on startup.
570 * Reference / Documentation
572 :CUSTOM_ID: reference-and-documentation
575 *** Source Code block syntax
577 The basic syntax of source-code blocks is as follows:
580 : #+begin_src language header-arguments
584 - name :: This name is associated with the source-code block. This is
585 similar to the =#+TBLNAME= lines which can be used to name tables
586 in org-mode files. By referencing the srcname of a source-code
587 block it is possible to evaluate the block for other places,
588 files, or from inside tables.
589 - language :: The language of the code in the source-code block, valid
590 values must be members of `org-babel-interpreters'.
591 - header-arguments :: Header arguments control many facets of the
592 input to, evaluation of, and output of source-code blocks. See
593 the [[* Header Arguments][Header Arguments]] section for a complete review of available
595 - body :: The actual source code which will be evaluated. This can be
596 edited with `org-edit-special'.
600 :CUSTOM_ID: header-arguments
603 - results :: results arguments specify what should be done with the
604 output of source-code blocks
605 - The following options are mutually exclusive, and specify how the
606 results should be collected from the source-code block
609 - The following options are mutually exclusive and specify what type
610 of results the code block will return
611 - vector :: specifies that the results should be interpreted as a
612 multidimensional vector (even if the vector is
613 trivial), and will be inserted into the org-mode file
615 - scalar :: specifies that the results should be interpreted as a
616 scalar value, and will be inserted into the org-mode
618 - file :: specifies that the results should be interpreted as the
619 path to a file, and will be inserted into the org-mode
621 - The following options specify how the results should be inserted
622 into the org-mode file
623 - replace :: the current results replace any previously inserted
624 results from the code block
625 - silent :: rather than being inserted into the org-mode file the
626 results are echoed into the message bar
627 - exports :: exports arguments specify what should be included in html
628 or latex exports of the org-mode file
629 - code :: the body of code is included into the exported file
630 - results :: the results of evaluating the code is included in the
632 - both :: both the code and results are included in the exported
634 - none :: nothing is included in the exported file
635 - tangle :: tangle arguments specify whether or not the source-code
636 block should be included in tangled extraction of
638 - yes :: the source-code block is exported to a source-code file
639 named after the basename (name w/o extension) of the
641 - no :: (default) the source-code block is not exported to a
643 - other :: any other string passed to the =tangle= header argument
644 is interpreted as a file basename to which the block will
647 *** Noweb reference syntax
648 The [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]] Literate Programming system allows named blocks of code to
649 be referenced by using a =<<code-block-name>>= syntax. When a
650 document is tangled these references are replaced with the named code.
651 An example is provided in the [[literate-programming-example]] in this