1 #+OPTIONS: H:3 num:nil toc:2 \n:nil @:t ::t |:t ^:{} -:t f:t *:t TeX:t LaTeX:t skip:nil d:(HIDE) tags:not-in-toc
2 #+STARTUP: align fold nodlcheck hidestars oddeven lognotestate hideblocks
3 #+SEQ_TODO: TODO(t) INPROGRESS(i) WAITING(w@) | DONE(d) CANCELED(c@)
4 #+TAGS: Write(w) Update(u) Fix(f) Check(c)
6 #+AUTHOR: Dan Davison, Eric Schulte
7 #+EMAIL: davison at stats dot ox dot ac dot uk
13 <p>executable source code blocks in org-mode</p>
17 <img src="images/tower-of-babel.png" alt="images/tower-of-babel.png"
18 title="And the Lord said, Behold, the people is one, and they have all one language; and this they begin to do; and now nothing will be restrained from them, which they have imagined to do. Genesis 11:1-9"/>
21 <a href="http://www.flickr.com/photos/23379658@N05/" title=""><b>Martijn Streefkerk</b></a>
34 :CUSTOM_ID: introduction
36 Org-babel is an extension to the very excellent [[http://orgmode.org/][Org-mode]]; an [[http://www.gnu.org/software/emacs/][Emacs]]
37 major mode for doing almost anything with plain text. If you are
38 not familiar with Org-mode please take a moment to read [[http://orgmode.org/][the Org-mode
39 homepage]] before continuing.
41 Org-babel provides the following modifications to [[http://orgmode.org/manual/Literal-examples.html][the existing
42 support]] for blocks of source code examples in the org-mode core.
44 1. Interactive source code execution
45 2. Arguments to source code blocks
46 3. Exportation of source code blocks to files (literate programming)
50 :CUSTOM_ID: getting-started
53 1) Grab the latest code from the git repo at [[http://github.com/eschulte/org-babel/tree/master][github/org-babel]]
55 git clone git://github.com/eschulte/org-babel.git
58 2) Add the following lines to your .emacs, replacing the path as
59 appropriate. A good place to check that things are up and running
60 would the examples in [[* Basic org-babel functionality][Basic org-babel functionality]].
61 #+begin_src emacs-lisp
62 (add-to-list 'load-path "/path/to/org-babel/lisp")
63 (require 'org-babel-init)
66 3) Then activate the subset of supported Org-babel languages which
67 you will want to be able to execute on your system. The
68 following can be added to your .emacs and used to activate
69 languages. It includes a brief list of the requirements for each
70 language. *Note*: this also serves as the list of languages
71 currently supported by Org-babel.
72 #+begin_src emacs-lisp
73 ;; Uncomment each of the following require lines if you want org-babel
74 ;; to support that language. Each language has a comment explaining
75 ;; it's dependencies. See the related files in lisp/langs for more
76 ;; detailed explanations of requirements.
77 ;; (require 'org-babel-R) ;; R and ess-mode
78 ;; (require 'org-babel-asymptote) ;; asymptote
79 ;; (require 'org-babel-css) ;; none
80 ;; (require 'org-babel-ditaa) ;; ditaa
81 ;; (require 'org-babel-dot) ;; dot
82 ;; (require 'org-babel-gnuplot) ;; gnuplot, and gnuplot-mode
83 ;; (require 'org-babel-haskell) ;; haskell, haskell-mode, inf-haskell
84 ;; (require 'org-babel-ocaml) ;; ocaml, and tuareg-mode
85 ;; (require 'org-babel-python) ;; python, and python-mode
86 ;; (require 'org-babel-ruby) ;; ruby, irb, ruby-mode, and inf-ruby
87 ;; (require 'org-babel-sass) ;; sass, sass-mode
88 ;; (require 'org-babel-sql) ;; none
90 ;; Once you've activated languages, load the library of babel for
91 ;; pre-built helpers in the languages you will be using.
92 (org-babel-load-library-of-babel)
95 * Basic org-babel functionality
97 :CUSTOM_ID: basic-functionality
99 *** Source code blocks
101 :CUSTOM_ID: source-code-blocks
104 Org-babel is all about *source code blocks* in org mode. These are
105 blocks of code (in whatever language), surrounded by special
106 starting and ending lines. For example, the following is a source
107 block containing [[http://www.ruby-lang.org/][ruby]] code:
110 : "This file was last evaluated on #{Date.today}"
113 If you are unfamiliar with the notion of source code blocks in
114 org-mode, please have a look at the [[http://orgmode.org/manual/Literal-examples.html][relevant manual section]] before
117 Note that above is what the source block looks like in the org-mode
118 file. We had to take [[http://orgmode.org/manual/Literal-examples.html#Literal-examples][special steps]] to make it look that way in the
119 HTML output. Normally, when exported to HTML, source blocks are
120 fontified according to their language, and the begin_src...end_src
121 mark-up is omitted, like this:
124 "This file was last evaluated on #{Date.today}"
127 From now on, if you are viewing the HTML version, you will see the
128 HTML output only. However, much of this document consists of
129 interactive examples, and therefore in order to get a feeling for the
130 mechanics of Org-babel it might make most sense to grab the plain text
132 #+HTML: <a href="org-babel-worg.org">org-babel-worg.org</a>
133 and work through it in Emacs. Alternatively the htmlized
134 version of the plain text of this file at
135 #+HTML: <a href="org-babel-worg.org.html">org-babel-worg.html</a>
136 allows the plain text version to be viewed (non-interactively) in a web browser.
137 *** Source code execution
139 :CUSTOM_ID: source-code-execution
141 For interpreted languages such as shell, python, R, etc, org-babel
142 allows source blocks to be executed: the code is passed to the
143 interpreter and you have control over what is done with the results of
144 excecution. E.g. place point anywhere in the following blocks and use
145 =C-c C-c= to run the code[fn:1]. In each case the code comes first,
146 followed by the results of evlauting the block.
150 "This file was last evaluated on #{Date.today}"
154 : This file was last evaluated on 2009-08-09
156 **** [[http://www.r-project.org/][R]]
157 #+begin_src R :results value
158 matrix(rnorm(6), nrow=2)
162 | 0.496600061063252 | "-1.44355317891110" | 0.106411785870013 |
163 | "-1.81619611674921" | "-1.25542979009380" | 0.00969467528507845 |
165 **** [[http://ditaa.sourceforge.net/][ditaa]]
166 #+begin_src ditaa :file images/blue.png :cmdline -r
177 [[file:images/blue.png]]
179 *** New source block syntax
180 Org-babel adds some new syntactical elements to source blocks in
181 org-mode, illustrated here:
183 : #+srcname: source-block-name(arg1=value1, arg2=value2)
184 : #+begin_src language :var1 var1-value :var2 var2-value ...
190 Note the following two new features
191 **** =srcname= with optional arguments
192 The optional =srcname= line allows a name to be given to the
193 source block, together with any arguments to the source block
194 (see [[#arguments-to-source-code-blocks][below]])
195 **** Header arguments
196 An arbitrary series of :variable value pairs can be specified
197 after the language, controlling various aspects of org-babel
198 operation. The available header arguments are documented [[#header-arguments][below.]]
199 *** What happens to the results?
203 Org-babel provides two fundamentally different modes for capturing
204 the results of code evaluation, specified by the =:results= header
206 **** =:results value= (functional mode)
207 This means that the 'result' of code evaluation is defined to be
208 the *value* of the last statement in the block. Thus with this
209 setting, one can view the code block as a function with a return
210 value. And not only can you view it that way, but you can
211 actually use the return value of one source block as input for
212 another (see later). This setting is the default.
214 As an example, consider the following block of python code and its
217 #+begin_src python :results value
219 print("Hello, today's date is %s" % time.ctime())
220 print('Two plus two is')
227 Notice that in functional mode, the output consists of the value of
228 the last statement, and nothing else.
230 **** =:results output= (scripting mode)
231 With this setting, org-babel captures all the text output of the
232 code block and places it in the org buffer. One can think of this
233 as a 'scripting' mode: the code block contains a series of
234 commands, and you get the output of all the commands. Unlike in
235 the 'functional' mode, the code block has no return value. (This
236 mode will be familiar to Sweave users).
238 Now consider the result of evaluating the same source block as
239 before, but under scripting mode.
242 #+begin_src python :results output
244 print("Hello, today's date is %s" % time.ctime())
245 print('Two plus two is')
250 : Hello, today's date is Fri Sep 4 19:49:06 2009
253 So, we got what we asked for: all the text output by python
254 (stdout). Since we didn't print the last value (2 + 2), we didn't get
257 *** Arguments to source code blocks
259 :CUSTOM_ID: arguments-to-source-code-blocks
261 In addition to evaluation of code blocks, org-babel allows them to
262 be parameterised (i.e. have arguments). Thus source code blocks
263 now have the status of *functions*.
265 Inputs for fibonacci-seq
267 #+tblname: fibonacci-inputs
268 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
269 | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 |
271 in the Org-mode buffer this looks like
272 : #+tblname: fibonacci-inputs
273 : | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
274 : | 2 | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 | 20 |
276 [[http://www.gnu.org/software/emacs/manual/elisp.html][Emacs Lisp]] source code
277 #+srcname: fibonacci-seq
278 #+begin_src emacs-lisp :var fib-inputs=fibonacci-inputs
280 (if (or (= n 0) (= n 1))
282 (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))
284 (mapcar (lambda (row)
285 (mapcar #'fibonacci row)) fib-inputs)
288 in the Org-mode buffer this looks like
289 : #+srcname: fibonacci-seq
290 : #+begin_src emacs-lisp :var fib-inputs=fibonacci-inputs
291 : (defun fibonacci (n)
292 : (if (or (= n 0) (= n 1))
294 : (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))
296 : (mapcar (lambda (row)
297 : (mapcar #'fibonacci row)) fib-inputs)
300 Results of Emacs Lisp code evaluation
302 | 1 | 1 | 2 | 3 | 5 | 8 | 13 | 21 | 34 | 55 |
303 | 1 | 3 | 8 | 21 | 55 | 144 | 377 | 987 | 2584 | 6765 |
305 * A meta-programming language for org-mode
307 :CUSTOM_ID: meta-programming-language
310 Since information can pass freely between source-code blocks and
311 org-mode tables you can mix and match languages using each language
312 for those tasks to which it is suited. This makes Org-mode files with
313 Org-babel into a kind of meta-functional programming language in which
314 functions from many languages can work together.
316 As an example, lets take some system diagnostics in the shell, and
317 then graph them with R.
320 #+srcname: directories
321 #+begin_src bash :results replace
322 cd ~ && du -sc * |grep -v total
324 2. Results of the shell source code (on my system, grab this org-mode
325 files and try running it on your own)
326 #+resname: directories
328 | 12156104 | "Documents" |
329 | 3482440 | "Downloads" |
330 | 2901720 | "Library" |
332 | 16548024 | "Music" |
334 | 7649472 | "Pictures" |
342 3. R source code (which calls the previous shell source code)
343 #+srcname: directory-pie
344 #+begin_src R :var dirs = directories :session R-pie-example
345 pie(dirs[,1], labels = dirs[,2])
347 4. Results of R code [[file:images/dirs.png]]
349 * Spreadsheet plugins for org-mode in any language
351 :CUSTOM_ID: spreadsheet
354 *NOTE*: Maybe in-addition-to/in-stead-of this example we should do a
355 more traditional "spreadsheet" example with R [Eric]
357 Not only can Org-babel pass entire tables of data to source code
358 blocks (see [[arguments-to-source-code-blocks]]), Org-babel can also be
359 used to call source code blocks from *within* tables using the
360 Org-mode's [[http://orgmode.org/manual/The-spreadsheet.html#The-spreadsheet][existing spreadsheet functionality]].
362 In fact the functional test suite for Org-babel is implemented as a
363 large Org-mode table. To run the entire test suite you simple
364 evaluate the table =C-u C-c C-c=, and all of the tests are run
365 updating the table with pass/fail statistics.
367 Here's a sample of our test suite.
369 #+TBLNAME: org-babel-tests
370 | functionality | block | arg | expected | results | pass |
371 |------------------+--------------+-----+-------------+-------------+------|
372 | basic evaluation | | | | | pass |
373 |------------------+--------------+-----+-------------+-------------+------|
374 | emacs lisp | basic-elisp | 2 | 4 | 4 | pass |
375 | shell | basic-shell | | 6 | 6 | pass |
376 | ruby | basic-ruby | | org-babel | org-babel | pass |
377 | python | basic-python | | hello world | hello world | pass |
378 | R | basic-R | | 13 | 13 | pass |
379 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
380 #+TBLFM: $5=""::$6=""
382 *** code blocks for tests
384 #+srcname: basic-elisp
385 #+begin_src emacs-lisp :var n=7
389 #+srcname: basic-shell
390 #+begin_src sh :results silent
394 #+srcname: date-simple
395 #+begin_src sh :results silent
399 #+srcname: basic-ruby
400 #+begin_src ruby :results silent
404 #+srcname: basic-python
405 #+begin_src python :results silent
410 #+begin_src R :results silent
417 :CUSTOM_ID: library-of-babel
419 What about those source code blocks which are so useful you want to
420 have them available in every org-mode buffer?
422 The [[file:library-of-babel.org][Library of Babel]] is an extensible collection of ready-made and
423 easily-shortcut-callable source-code blocks for handling common
424 tasks. Org-babel comes pre-populated with the source-code blocks
425 located in the [[file:library-of-babel.org][library-of-babel.org]] file. It is possible to add
426 source-code blocks from any org-mode file to the library by calling
428 #+srcname: add-file-to-lob
429 #+begin_src emacs-lisp
430 (org-babel-lob-ingest "path/to/file.org")
433 * Reproducible Research
435 :CUSTOM_ID: reproducable-research
438 An article about computational science in a scientific publication is
439 not the scholarship itself, it is merely advertising of the
440 scholarship. The actual scholarship is the complete software
441 development environment and the complete set of instructions which
442 generated the figures.
447 [[http://reproducibleresearch.net/index.php/Main_Page][Reproducible Research]] (RR) is the practice of distributing along with
448 an article of research all data, code, and tools required to reproduce
449 the results discussed in the paper. As such the paper becomes not
450 only a document describing the research but a complete laboratory
451 reproducing the research.
453 Org-mode already has exceptional support for [[http://orgmode.org/manual/Exporting.html#Exporting][exporting to html and
454 LaTeX]]. Org-babel makes Org-mode a tool for RR by *activating* the
455 data and source code embedded into Org-mode documents making the
456 entire document executable. This makes it not only possible, but
457 natural to distribute research in a format that encourages readers to
458 recreate your results, and perform their own analysis.
460 Existing RR tools like [[http://en.wikipedia.org/wiki/Sweave][Sweave]] provide for the embedding of [[http://www.r-project.org/][R]] code into
461 LaTeX documents. While this is very useful, such documents often
462 still require a large degree of "glue code" in the form of external
463 shell scripts, python scripts, and Makefiles. To our knowledge
464 Org-babel is the only RR tool which allows multiple languages and data
465 to coexist and cooperate inside of a single document.
467 * Literate programming
469 :CUSTOM_ID: literate-programming
473 Let us change our traditional attitude to the construction of
474 programs: Instead of imagining that our main task is to instruct a
475 /computer/ what to do, let us concentrate rather on explaining to
476 /human beings/ what we want a computer to do.
478 The practitioner of literate programming can be regarded as an
479 essayist, whose main concern is with exposition and excellence of
480 style. Such an author, with thesaurus in hand, chooses the names of
481 variables carefully and explains what each variable means. He or she
482 strives for a program that is comprehensible because its concepts have
483 been introduced in an order that is best for human understanding,
484 using a mixture of formal and informal methods that reinforce each
490 Org-babel supports [[http://en.wikipedia.org/wiki/Literate_programming][Literate Programming]] (LP) by allowing the act of
491 programming to take place inside of Org-mode documents. The Org-mode
492 file can then be exported (*woven* in LP speak) to html or LaTeX for
493 consumption by a human, and the embedded source code can be extracted
494 (*tangled* in LP speak) into structured source code files for
495 consumption by a computer.
497 To support these operations Org-babel relies on Org-mode's [[http://orgmode.org/manual/Exporting.html#Exporting][existing
498 exporting functionality]] for *weaving* of documentation, and on the
499 =org-babel-tangle= function which makes use of [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]] [[reference-expansion][reference syntax]]
500 for *tangling* of code files.
502 The [[literate-programming-example][following example]] demonstrates the process of *tangling* in
505 *** Simple Literate Programming Example (Noweb syntax)
507 :CUSTOM_ID: literate-programming-example
510 Tangling functionality is controlled by the =tangle= family of
511 [[header-arguments]]. These arguments can be used to turn tangling on or
512 off (the default) on the source code block, or the outline heading
515 The following demonstrates the combination of three source code blocks
516 into a single source code file using =org-babel-tangle=.
518 The following two blocks will not be tangled by default since they
519 have no =tangle= header arguments.
521 #+srcname: hello-world-prefix
522 #+begin_src sh :exports none
523 echo "/-----------------------------------------------------------\\"
526 : #+srcname: hello-world-prefix
527 : #+begin_src sh :exports none
528 : echo "/-----------------------------------------------------------\\"
531 #+srcname: hello-world-postfix
532 #+begin_src sh :exports none
533 echo "\-----------------------------------------------------------/"
536 : #+srcname: hello-world-postfix
537 : #+begin_src sh :exports none
538 : echo "\-----------------------------------------------------------/"
542 The third block does have a =tangle= header argument indicating the
543 name of the file to which it should be written. It also has [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]]
544 style references to the two previous source code blocks which will be
545 expanded during tangling to include them in the output file as well.
547 #+srcname: hello-world
548 #+begin_src sh :tangle hello :exports none
549 # <<hello-world-prefix>>
550 echo "| hello world |"
551 # <<hello-world-postfix>>
554 : #+srcname: hello-world
555 : #+begin_src sh :tangle hello :exports none
556 : # <<hello-world-prefix>>
557 : echo "| hello world |"
558 : # <<hello-world-postfix>>
561 Calling =org-babel-tangle= will result in the following being written
562 to the =hello.sh= file.
564 #+srcname: hello-world-output
567 # generated by org-babel-tangle
569 # [[file:~/src/org-babel/org-babel-worg.org::#literate-programming-example][block-16]]
570 # <<hello-world-prefix>>
571 echo "/-----------------------------------------------------------\\"
573 echo "| hello world |"
574 # <<hello-world-postfix>>
575 echo "\-----------------------------------------------------------/"
579 *** Emacs Initialization with Org-babel
580 Org-babel has special support for embedding your emacs initialization
581 into Org-mode files. The =org-babel-load-file= function can be used
582 to load the emacs lisp embedded in a literate Org-mode file in the
583 same way that you might load a regular elisp file.
585 This allows you to have all the niceness of Org-mode (folding, tags,
586 notes, html export, etc...) available in your emacs initialization.
588 To try this out either see the simple [[literate-emacs-init][Literate Emacs Initialization]]
589 example directly below, or check out the Org-babel Literate
590 Programming version of Phil Hagelberg's excellent [[http://github.com/technomancy/emacs-starter-kit/tree/master][emacs-starter-kit]]
591 available at [[http://github.com/eschulte/emacs-starter-kit/tree/master][Org-babel-emacs-starter-kit]].
593 ***** Literate Emacs Initialization
595 :CUSTOM_ID: literate-emacs-init
598 For a simple example of usage follow these 4 steps.
600 1) create a directory named =.emacs.d= in the base of your home
605 2) checkout the latest versions of Org-mode and Org-babel into the src
606 subdirectory of this new directory
611 git clone git://repo.or.cz/org-mode.git
612 git clone git://github.com/eschulte/org-babel.git
614 3) place the following in a file called =init.el= in your emacs
615 initialization directory (=~/.emacs.d=).
616 #+srcname: emacs-init
617 #+begin_src emacs-lisp
618 ;;; init.el --- Where all the magic begins
620 ;; This file loads both
621 ;; - Org-mode : http://orgmode.org/ and
622 ;; - Org-babel: http://eschulte.github.com/org-babel/
624 ;; It then loads the rest of our Emacs initialization from Emacs lisp
625 ;; embedded in literate Org-mode files.
627 ;; Load up Org Mode and Org Babel for elisp embedded in Org Mode files
628 (setq dotfiles-dir (file-name-directory (or (buffer-file-name) load-file-name)))
629 (add-to-list 'load-path (expand-file-name
630 "lisp" (expand-file-name
631 "org" (expand-file-name
632 "src" dotfiles-dir))))
633 (add-to-list 'load-path (expand-file-name
634 "lisp" (expand-file-name
635 "org-babel" (expand-file-name
636 "src" dotfiles-dir))))
637 (require 'org-babel-init)
639 ;; load up all literate org-mode files in this directory
640 (mapc #'org-babel-load-file (directory-files dotfiles-dir t "\\.org$"))
642 ;;; init.el ends here
644 4) Implement all of your emacs customizations inside of elisp
645 source-code blocks located in Org-mode files in this directory.
646 They will be loaded by emacs on startup.
648 * Reference / Documentation
650 :CUSTOM_ID: reference-and-documentation
653 *** Source Code block syntax
655 The basic syntax of source-code blocks is as follows:
658 : #+begin_src language header-arguments
662 - name :: This name is associated with the source-code block. This is
663 similar to the =#+TBLNAME= lines which can be used to name tables
664 in org-mode files. By referencing the srcname of a source-code
665 block it is possible to evaluate the block for other places,
666 files, or from inside tables.
667 - language :: The language of the code in the source-code block, valid
668 values must be members of `org-babel-interpreters'.
669 - header-arguments :: Header arguments control many facets of the
670 input to, evaluation of, and output of source-code blocks. See
671 the [[* Header Arguments][Header Arguments]] section for a complete review of available
673 - body :: The actual source code which will be evaluated. This can be
674 edited with `org-edit-special'.
678 :CUSTOM_ID: header-arguments
681 - results :: results arguments specify what should be done with the
682 output of source-code blocks
683 - The following options are mutually exclusive, and specify how the
684 results should be collected from the source-code block
687 - The following options are mutually exclusive and specify what type
688 of results the code block will return
689 - vector :: specifies that the results should be interpreted as a
690 multidimensional vector (even if the vector is
691 trivial), and will be inserted into the org-mode file
693 - scalar :: specifies that the results should be interpreted as a
694 scalar value, and will be inserted into the org-mode
696 - file :: specifies that the results should be interpreted as the
697 path to a file, and will be inserted into the org-mode
699 - The following options specify how the results should be inserted
700 into the org-mode file
701 - replace :: the current results replace any previously inserted
702 results from the code block
703 - silent :: rather than being inserted into the org-mode file the
704 results are echoed into the message bar
705 - exports :: exports arguments specify what should be included in html
706 or latex exports of the org-mode file
707 - code :: the body of code is included into the exported file
708 - results :: the results of evaluating the code is included in the
710 - both :: both the code and results are included in the exported
712 - none :: nothing is included in the exported file
713 - tangle :: tangle arguments specify whether or not the source-code
714 block should be included in tangled extraction of
716 - yes :: the source-code block is exported to a source-code file
717 named after the basename (name w/o extension) of the
719 - no :: (default) the source-code block is not exported to a
721 - other :: any other string passed to the =tangle= header argument
722 is interpreted as a file basename to which the block will
725 *** Noweb reference syntax
726 The [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]] Literate Programming system allows named blocks of code to
727 be referenced by using a =<<code-block-name>>= syntax. When a
728 document is tangled these references are replaced with the named code.
729 An example is provided in the [[literate-programming-example]] in this