1 #+OPTIONS: H:3 num:nil toc:2 \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:(HIDE) tags:not-in-toc
2 #+STARTUP: align fold nodlcheck hidestars oddeven lognotestate hideblocks
3 #+SEQ_TODO: TODO(t) INPROGRESS(i) WAITING(w@) | DONE(d) CANCELED(c@)
4 #+TAGS: Write(w) Update(u) Fix(f) Check(c)
6 #+AUTHOR: Dan Davison, Eric Schulte
7 #+EMAIL: davison at stats dot ox dot ac dot uk
13 <p>executable source code blocks in org-mode</p>
17 <img src="images/tower-of-babel.png" alt="images/tower-of-babel.png"
18 title="And the Lord said, Behold, the people is one, and they have all one language; and this they begin to do; and now nothing will be restrained from them, which they have imagined to do. Genesis 11:1-9"/>
21 <a href="http://www.flickr.com/photos/23379658@N05/" title=""><b>Martijn Streefkerk</b></a>
34 :CUSTOM_ID: introduction
36 Org-babel provides the following modifications to [[http://orgmode.org/manual/Literal-examples.html][the existing
37 support]] for blocks of source code examples in the org-mode core.
38 1. interactive source code execution
39 2. arguments to source code blocks
40 3. exportation of source code blocks to files (literate programming)
42 Much of this document includes interactive examples. The reader is
43 encouraged to grab either the plain text version of this file
44 #+HTML: <a href="org-babel-worg.org">org-babel-worg.org</a>
45 which can be read and run in Emacs or the htmlized version of the
46 plain text of this file
47 #+HTML: <a href="org-babel-worg.org.html">org-babel-worg.html</a>
48 which is viewable through the web browser.
52 :CUSTOM_ID: getting-started
55 1) Grab the latest code from the git repo at [[http://github.com/eschulte/org-babel/tree/master][github/org-babel]]
57 git clone git://github.com/eschulte/org-babel.git
60 2) Add the following lines to your .emacs, replacing the path as
61 appropriate. A good place to check that things are up and running
62 would the examples in [[* Basic org-babel functionality][Basic org-babel functionality]].
63 #+begin_src emacs-lisp
64 (add-to-list 'load-path "/path/to/org-babel/lisp")
65 (require 'org-babel-init)
68 3) Then activate the subset of supported Org-babel languages which
69 you will want to be able to execute on your system. The
70 following can be added to your .emacs and used to activate
71 languages. It includes a brief list of the requirements for each
73 #+begin_src emacs-lisp
74 ;; Uncomment each of the following require lines if you want org-babel
75 ;; to support that language. Each language has a comment explaining
76 ;; it's dependencies. See the related files in lisp/langs for more
77 ;; detailed explanations of requirements.
78 ;; (require 'org-babel-R) ;; R and ess-mode
79 ;; (require 'org-babel-asymptote) ;; asymptote
80 ;; (require 'org-babel-css) ;; none
81 ;; (require 'org-babel-ditaa) ;; ditaa
82 ;; (require 'org-babel-dot) ;; dot
83 ;; (require 'org-babel-gnuplot) ;; gnuplot, and gnuplot-mode
84 ;; (require 'org-babel-ocaml) ;; ocaml, and tuareg-mode
85 ;; (require 'org-babel-python) ;; python, and python-mode
86 ;; (require 'org-babel-ruby) ;; ruby, irb, ruby-mode, and inf-ruby mode
87 ;; (require 'org-babel-sass) ;; sass, sass-mode
88 ;; (require 'org-babel-sql) ;; none
90 ;; Once you've activated languages, load the library of babel for
91 ;; pre-built helpers in the languages you will be using.
92 (org-babel-load-library-of-babel)
95 * Basic org-babel functionality
97 :CUSTOM_ID: basic-functionality
99 *** Source code execution
101 :CUSTOM_ID: source-code-execution
103 For interpreted languages such as shell, python, R, etc, org-babel
104 allows source blocks to be executed: the code is passed to the
105 interpreter and you have control over what is done with the
106 results of excecution. E.g. place point anywhere in the following
107 block and use C-c C-c to run the code:
109 [[http://www.ruby-lang.org/][Ruby]] source code
111 "This file was last evaluated on #{Date.today}"
114 Results of Ruby evaluation
116 : This file was last evaluated on 2009-08-09
118 [[http://www.r-project.org/][R]] source code
119 #+begin_src R :results value
125 Results of R evaluation
130 [[http://ditaa.sourceforge.net/][ditaa]] source code
131 #+begin_src ditaa :file images/blue.png :cmdline -r
142 [[file:images/blue.png]]
144 *** What happens to the results?
148 Org-babel provides two fundamentally different modes for capturing
149 the results of code evaluation, specified by the :results header
152 This means that the 'result' of code evaluation is defined to be
153 the *value* of the last statement in the block. Thus with this
154 setting, one can view the code block as a function with a return
155 value. And not only can one view it that way, but you can
156 actually use the return value of one source block as input for
157 another (see later). This setting is the default.
159 With this setting, org-babel captures all the text output of the
160 code block and places it in the org buffer. One can think of this
161 as a 'scripting' mode: the code block contains a series of
162 commands, and you get the output of all the commands. Unlike in
163 the 'functional' mode specified by =:results value=, the code
164 block has no return value. (This mode will be familiar to Sweave
166 **** Additional :results settings
168 *** Arguments to source code blocks
170 :CUSTOM_ID: arguments-to-source-code-blocks
172 In addition to evaluation of code blocks, org-babel allows them to
173 be parameterised (i.e. have arguments). Thus source code blocks
174 now have the status of *functions*.
176 Inputs for fibonacci-seq
178 #+tblname: fibonacci-inputs
179 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
180 | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 |
182 in the Org-mode buffer this looks like
183 : #+tblname: fibonacci-inputs
184 : | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
185 : | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 |
187 [[http://www.gnu.org/software/emacs/manual/elisp.html][Emacs Lisp]] source code
188 #+srcname: fibonacci-seq
189 #+begin_src emacs-lisp :var fib-inputs=fibonacci-inputs
191 (if (or (= n 0) (= n 1))
193 (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))
195 (mapcar (lambda (row)
196 (mapcar #'fibonacci row)) fib-inputs)
199 in the Org-mode buffer this looks like
200 : #+srcname: fibonacci-seq
201 : #+begin_src emacs-lisp :var fib-inputs=fibonacci-inputs
202 : (defun fibonacci (n)
203 : (if (or (= n 0) (= n 1))
205 : (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))
207 : (mapcar (lambda (row)
208 : (mapcar #'fibonacci row)) fib-inputs)
211 Results of Emacs Lisp code evaluation
213 | 1 | 1 | 2 | 3 | 5 | 8 | 13 | 21 | 34 | 55 |
214 | 1 | 3 | 8 | 21 | 55 | 144 | 377 | 987 | 2584 | 6765 |
216 * A meta-programming language for org-mode
218 :CUSTOM_ID: meta-programming-language
221 Since information can pass freely between source-code blocks and
222 org-mode tables you can mix and match languages using each language
223 for those tasks to which it is suited. This makes Org-mode files with
224 Org-babel into a kind of meta-functional programming language in which
225 functions from many languages can work together.
227 As an example, lets take some system diagnostics in the shell, and
228 then graph them with R.
231 #+srcname: directories
232 #+begin_src bash :results replace
233 cd ~ && du -sc * |grep -v total
235 2. Results of the shell source code (on my system, grab this org-mode
236 files and try running it on your own)
237 #+resname: directories
239 | 12156104 | "Documents" |
240 | 3482440 | "Downloads" |
241 | 2901720 | "Library" |
243 | 16548024 | "Music" |
245 | 7649472 | "Pictures" |
253 3. R source code (which calls the previous shell source code)
254 #+srcname: directory-pie
255 #+begin_src R :var dirs = directories :session R-pie-example
256 pie(dirs[,1], labels = dirs[,2])
258 4. Results of R code [[file:images/dirs.png]]
260 * Spreadsheet plugins for org-mode in any language
262 :CUSTOM_ID: spreadsheet
265 *NOTE*: Maybe in-addition-to/in-stead-of this example we should do a
266 more traditional "spreadsheet" example with R [Eric]
268 Not only can Org-babel pass entire tables of data to source code
269 blocks (see [[arguments-to-source-code-blocks]]), Org-babel can also be
270 used to call source code blocks from *within* tables using the
271 Org-mode's [[http://orgmode.org/manual/The-spreadsheet.html#The-spreadsheet][existing spreadsheet functionality]].
273 In fact the functional test suite for Org-babel is implemented as a
274 large Org-mode table. To run the entire test suite you simple
275 evaluate the table =C-u C-c C-c=, and all of the tests are run
276 updating the table with pass/fail statistics.
278 Here's a sample of our test suite.
280 #+TBLNAME: org-babel-tests
281 | functionality | block | arg | expected | results | pass |
282 |------------------+--------------+-----+-------------+-------------+------|
283 | basic evaluation | | | | | pass |
284 |------------------+--------------+-----+-------------+-------------+------|
285 | emacs lisp | basic-elisp | 2 | 4 | 4 | pass |
286 | shell | basic-shell | | 6 | 6 | pass |
287 | ruby | basic-ruby | | org-babel | org-babel | pass |
288 | python | basic-python | | hello world | hello world | pass |
289 | R | basic-R | | 13 | 13 | pass |
290 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
291 #+TBLFM: $5=""::$6=""
293 *** code blocks for tests
295 #+srcname: basic-elisp
296 #+begin_src emacs-lisp :var n=7
300 #+srcname: basic-shell
301 #+begin_src sh :results silent
305 #+srcname: date-simple
306 #+begin_src sh :results silent
310 #+srcname: basic-ruby
311 #+begin_src ruby :results silent
315 #+srcname: basic-python
316 #+begin_src python :results silent
321 #+begin_src R :results silent
328 :CUSTOM_ID: library-of-babel
330 What about those source code blocks which are so useful you want to
331 have them available in every org-mode buffer?
333 The [[file:library-of-babel.org][Library of Babel]] is an extensible collection of ready-made and
334 easily-shortcut-callable source-code blocks for handling common
335 tasks. Org-babel comes pre-populated with the source-code blocks
336 located in the [[file:library-of-babel.org][library-of-babel.org]] file. It is possible to add
337 source-code blocks from any org-mode file to the library by calling
339 #+srcname: add-file-to-lob
340 #+begin_src emacs-lisp
341 (org-babel-lob-ingest "path/to/file.org")
344 * Reproducible Research
346 :CUSTOM_ID: reproducable-research
349 An article about computational science in a scientific publication is
350 not the scholarship itself, it is merely advertising of the
351 scholarship. The actual scholarship is the complete software
352 development environment and the complete set of instructions which
353 generated the figures.
358 [[http://reproducibleresearch.net/index.php/Main_Page][Reproducible Research]] (RR) is the practice of distributing along with
359 an article of research all data, code, and tools required to reproduce
360 the results discussed in the paper. As such the paper becomes not
361 only a document describing the research but a complete laboratory
362 reproducing the research.
364 Org-mode already has exceptional support for [[http://orgmode.org/manual/Exporting.html#Exporting][exporting to html and
365 LaTeX]]. Org-babel makes Org-mode a tool for RR by *activating* the
366 data and source code embedded into Org-mode documents making the
367 entire document executable. This makes it not only possible, but
368 natural to distribute research in a format that encourages readers to
369 recreate your results, and perform their own analysis.
371 Existing RR tools like [[http://en.wikipedia.org/wiki/Sweave][Sweave]] provide for the embedding of [[http://www.r-project.org/][R]] code into
372 LaTeX documents. While this is very useful, such documents often
373 still require a large degree of "glue code" in the form of external
374 shell scripts, python scripts, and Makefiles. To our knowledge
375 Org-babel is the only RR tool which allows multiple languages and data
376 to coexist and cooperate inside of a single document.
378 * Literate programming
380 :CUSTOM_ID: literate-programming
384 Let us change our traditional attitude to the construction of
385 programs: Instead of imagining that our main task is to instruct a
386 /computer/ what to do, let us concentrate rather on explaining to
387 /human beings/ what we want a computer to do.
389 The practitioner of literate programming can be regarded as an
390 essayist, whose main concern is with exposition and excellence of
391 style. Such an author, with thesaurus in hand, chooses the names of
392 variables carefully and explains what each variable means. He or she
393 strives for a program that is comprehensible because its concepts have
394 been introduced in an order that is best for human understanding,
395 using a mixture of formal and informal methods that reinforce each
401 Org-babel supports [[http://en.wikipedia.org/wiki/Literate_programming][Literate Programming]] (LP) by allowing the act of
402 programming to take place inside of Org-mode documents. The Org-mode
403 file can then be exported (*woven* in LP speak) to html or LaTeX for
404 consumption by a human, and the embedded source code can be extracted
405 (*tangled* in LP speak) into structured source code files for
406 consumption by a computer.
408 To support these operations Org-babel relies on Org-mode's [[http://orgmode.org/manual/Exporting.html#Exporting][existing
409 exporting functionality]] for *weaving* of documentation, and on the
410 =org-babel-tangle= function which makes use of [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]] [[reference-expansion][reference syntax]]
411 for *tangling* of code files.
413 The [[literate-programming-example][following example]] demonstrates the process of *tangling* in
416 *** Simple Literate Programming Example (Noweb syntax)
418 :CUSTOM_ID: literate-programming-example
421 Tangling functionality is controlled by the =tangle= family of
422 [[header-arguments]]. These arguments can be used to turn tangling on or
423 off (the default) on the source code block, or the outline heading
426 The following demonstrates the combination of three source code blocks
427 into a single source code file using =org-babel-tangle=.
429 The following two blocks will not be tangled by default since they
430 have no =tangle= header arguments.
432 #+srcname: hello-world-prefix
433 #+begin_src sh :exports none
434 echo "/-----------------------------------------------------------\\"
437 : #+srcname: hello-world-prefix
438 : #+begin_src sh :exports none
439 : echo "/-----------------------------------------------------------\\"
442 #+srcname: hello-world-postfix
443 #+begin_src sh :exports none
444 echo "\-----------------------------------------------------------/"
447 : #+srcname: hello-world-postfix
448 : #+begin_src sh :exports none
449 : echo "\-----------------------------------------------------------/"
453 The third block does have a =tangle= header argument indicating the
454 name of the file to which it should be written. It also has [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]]
455 style references to the two previous source code blocks which will be
456 expanded during tangling to include them in the output file as well.
458 #+srcname: hello-world
459 #+begin_src sh :tangle hello :exports none
460 # <<hello-world-prefix>>
461 echo "| hello world |"
462 # <<hello-world-postfix>>
465 : #+srcname: hello-world
466 : #+begin_src sh :tangle hello :exports none
467 : # <<hello-world-prefix>>
468 : echo "| hello world |"
469 : # <<hello-world-postfix>>
472 Calling =org-babel-tangle= will result in the following being written
473 to the =hello.sh= file.
475 #+srcname: hello-world-output
478 # generated by org-babel-tangle
480 # [[file:~/src/org-babel/org-babel-worg.org::#literate-programming-example][block-16]]
481 # <<hello-world-prefix>>
482 echo "/-----------------------------------------------------------\\"
484 echo "| hello world |"
485 # <<hello-world-postfix>>
486 echo "\-----------------------------------------------------------/"
490 *** Emacs Initialization with Org-babel
491 Org-babel has special support for embedding your emacs initialization
492 into Org-mode files. The =org-babel-load-file= function can be used
493 to load the emacs lisp embedded in a literate Org-mode file in the
494 same way that you might load a regular elisp file.
496 This allows you to have all the niceness of Org-mode (folding, tags,
497 notes, html export, etc...) available in your emacs initialization.
499 To try this out either see the simple [[literate-emacs-init][Literate Emacs Initialization]]
500 example directly below, or check out the Org-babel Literate
501 Programming version of Phil Hagelberg's excellent [[http://github.com/technomancy/emacs-starter-kit/tree/master][emacs-starter-kit]]
502 available at [[http://github.com/eschulte/emacs-starter-kit/tree/master][Org-babel-emacs-starter-kit]].
504 ***** Literate Emacs Initialization
506 :CUSTOM_ID: literate-emacs-init
509 For a simple example of usage follow these 4 steps.
511 1) create a directory named =.emacs.d= in the base of your home
516 2) checkout the latest versions of Org-mode and Org-babel into the src
517 subdirectory of this new directory
522 git clone git://repo.or.cz/org-mode.git
523 git clone git://github.com/eschulte/org-babel.git
525 3) place the following in a file called =init.el= in your emacs
526 initialization directory (=~/.emacs.d=).
527 #+srcname: emacs-init
528 #+begin_src emacs-lisp
529 ;;; init.el --- Where all the magic begins
531 ;; This file loads both
532 ;; - Org-mode : http://orgmode.org/ and
533 ;; - Org-babel: http://eschulte.github.com/org-babel/
535 ;; It then loads the rest of our Emacs initialization from Emacs lisp
536 ;; embedded in literate Org-mode files.
538 ;; Load up Org Mode and Org Babel for elisp embedded in Org Mode files
539 (setq dotfiles-dir (file-name-directory (or (buffer-file-name) load-file-name)))
540 (add-to-list 'load-path (expand-file-name
541 "lisp" (expand-file-name
542 "org" (expand-file-name
543 "src" dotfiles-dir))))
544 (add-to-list 'load-path (expand-file-name
545 "lisp" (expand-file-name
546 "org-babel" (expand-file-name
547 "src" dotfiles-dir))))
548 (require 'org-babel-init)
550 ;; load up all literate org-mode files in this directory
551 (mapc #'org-babel-load-file (directory-files ditfiles-dir t "\\.org$"))
553 ;;; init.el ends here
555 4) Implement all of your emacs customizations inside of elisp
556 source-code blocks located in Org-mode files in this directory.
557 They will be loaded by emacs on startup.
559 * Reference / Documentation
561 :CUSTOM_ID: reference-and-documentation
564 *** Source Code block syntax
566 The basic syntax of source-code blocks is as follows:
569 : #+begin_src language header-arguments
573 - name :: This name is associated with the source-code block. This is
574 similar to the =#+TBLNAME= lines which can be used to name tables
575 in org-mode files. By referencing the srcname of a source-code
576 block it is possible to evaluate the block for other places,
577 files, or from inside tables.
578 - language :: The language of the code in the source-code block, valid
579 values must be members of `org-babel-interpreters'.
580 - header-arguments :: Header arguments control many facets of the
581 input to, evaluation of, and output of source-code blocks. See
582 the [[* Header Arguments][Header Arguments]] section for a complete review of available
584 - body :: The actual source code which will be evaluated. This can be
585 edited with `org-edit-special'.
589 :CUSTOM_ID: header-arguments
592 - results :: results arguments specify what should be done with the
593 output of source-code blocks
594 - The following options are mutually exclusive, and specify how the
595 results should be collected from the source-code block
598 - The following options are mutually exclusive and specify what type
599 of results the code block will return
600 - vector :: specifies that the results should be interpreted as a
601 multidimensional vector (even if the vector is
602 trivial), and will be inserted into the org-mode file
604 - scalar :: specifies that the results should be interpreted as a
605 scalar value, and will be inserted into the org-mode
607 - file :: specifies that the results should be interpreted as the
608 path to a file, and will be inserted into the org-mode
610 - The following options specify how the results should be inserted
611 into the org-mode file
612 - replace :: the current results replace any previously inserted
613 results from the code block
614 - silent :: rather than being inserted into the org-mode file the
615 results are echoed into the message bar
616 - exports :: exports arguments specify what should be included in html
617 or latex exports of the org-mode file
618 - code :: the body of code is included into the exported file
619 - results :: the results of evaluating the code is included in the
621 - both :: both the code and results are included in the exported
623 - none :: nothing is included in the exported file
624 - tangle :: tangle arguments specify whether or not the source-code
625 block should be included in tangled extraction of
627 - yes :: the source-code block is exported to a source-code file
628 named after the basename (name w/o extension) of the
630 - no :: (default) the source-code block is not exported to a
632 - other :: any other string passed to the =tangle= header argument
633 is interpreted as a file basename to which the block will
636 *** Noweb reference syntax
637 The [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]] Literate Programming system allows named blocks of code to
638 be referenced by using a =<<code-block-name>>= syntax. When a
639 document is tangled these references are replaced with the named code.
640 An example is provided in the [[literate-programming-example]] in this