1 #+OPTIONS: H:3 num:nil toc:t
2 #+TITLE: org-babel --- facilitating communication between programming languages and people
3 #+SEQ_TODO: TODO PROPOSED | DONE DEFERRED REJECTED
8 Org-Babel enables *communication* between programming languages and
12 - communication between programs :: Data passes seamlessly between
13 different programming languages, text, and tables.
14 - communication between people :: Data and calculations are embedded
15 in the same document as notes explanations and reports.
17 ** communication between programs
19 Org-Mode supports embedded blocks of source code (in any language)
20 inside of Org documents. Org-Babel allows these blocks of code to be
21 executed from within Org-Mode with natural handling of their inputs
25 with both scalar, file, and table output
28 *** reading information from tables
31 *** reading information from other source blocks (disk usage in your home directory)
33 This will work for Linux and Mac users, not so sure about shell
34 commands for windows users.
36 To run place the cursor on the =#+begin_src= line of the source block
37 labeled directory-pie and press =\C-c\C-c=.
39 #+srcname: directories
40 #+begin_src bash :results replace
41 cd ~ && du -sc * |grep -v total
44 #+resname: directories
46 | 11882808 | "Documents" |
47 | 8210024 | "Downloads" |
48 | 879800 | "Library" |
51 | 5307664 | "Pictures" |
60 #+srcname: directory-pie
61 #+begin_src R :var dirs = directories
62 pie(dirs[,1], labels = dirs[,2])
66 *** operations in/on tables
68 #+tblname: grades-table
69 | student | grade | letter |
70 |---------+-------+--------|
77 #+TBLFM: $2='(sbe random-score-generator)::$3='(sbe assign-grade (score $2))
79 #+srcname: assign-grade
80 #+begin_src ruby :var score=99
91 #+srcname: random-score-generator
96 #+srcname: show-distribution
97 #+begin_src R :var grades=grades-table
102 ** communication between people
103 Quick overview of Org-Mode's exportation abilities, with links to the
104 online Org-Mode documentation, a focus on source-code blocks, and the
105 exportation options provided by Org-Babel.
107 *** Interactive tutorial
108 This would demonstrate applicability to Reproducible Research, and
109 Literate Programming.
111 *** Tests embedded in documentation
112 org-babels own functional tests are contained in a large org-mode
113 table, allowing the test suite to be run be evaluation of the table
114 and the results to be collected in the same table.
118 ** TODO Create objects in top level (global) environment in R?
119 *** initial requirement statement [DED]
120 At the moment, objects created by computations performed in the
121 code block are evaluated in the scope of the
122 code-block-function-body and therefore disappear when the code
123 block is evaluated {unless you employ some extra trickery like
124 assign('name', object, env=globalenv()) }. I think it will be
125 desirable to also allow for a style wherein objects that are
126 created in one code block persist in the R global environment and
127 can be re-used in a separate block.
129 This is what Sweave does, and while I'm not saying we have to be
130 the same as Sweave, it wouldn't be hard for us to provide the same
131 behaviour in this case; if we don't, we risk undeservedly being
132 written off as an oddity by some.
134 IOW one aspect of org-babel is that of a sort of functional
135 meta-programming language. This is crazy, in a very good
136 way. Nevertheless, wrt R I think there's going to be a lot of value
137 in providing for a working style in which the objects are stored in
138 the R session, rather than elisp/org buffer. This will be a very
139 familiar working style to lots of people.
141 There are no doubt a number of different ways of accomplishing
142 this, the simplest being a hack like adding
146 assign(objname, get(objname), envir=globalenv())
149 to the source code block function body. (Maybe wrap it in an on.exit() call).
151 However this may deserve to be thought about more carefully, perhaps
152 with a view to having a uniform approach across languages. E.g. shell
153 code blocks have the same semantics at the moment (no persistence of
154 variables across code blocks), because the body is evaluated in a new
155 bash shell process rather than a running shell. And I guess the same
156 is true for python. However, in both these cases, you could imagine
157 implementing the alternative in which the body is evaluated in a
158 persistent interactive session. It's just that it's particularly
159 natural for R, seeing as both ESS and org-babel evaluate commands in a
160 single persistent R session.
164 Thanks for bringing this up. I think you are absolutely correct that we
165 should provide support for a persistent environment (maybe called a
166 *session*) in which to evaluate code blocks. I think the current setup
167 demonstrates my personal bias for a functional style of programming
168 which is certainly not ideal in all contexts.
170 While the R function you mention does look like an elegant solution, I
171 think we should choose an implementation that would be the same across
172 all source code types. Specifically I think we should allow the user to
173 specify an optional *session* as a header variable (when not present we
174 assume a default session for each language). The session name could be
175 used to name a comint buffer (like the *R* buffer) in which all
176 evaluation would take place (within which variables would retain their
177 values --at least once I remove some of the functional method wrappings
178 currently in place-- ).
180 This would allow multiple environments to be used in the same buffer,
181 and once this setup was implemented we should be able to fairly easily
182 implement commands for jumping between source code blocks and the
183 related session buffers, as well as for dumping the last N commands from
184 a session into a new or existing source code block.
186 Please let me know if you foresee any problems with this proposed setup,
187 or if you think any parts might be confusing for people coming from
188 Sweave. I'll hopefully find some time to work on this later in the
190 *** further requirement: make it play nicely with ESS [DED]
191 I think something like this would be great. You've probably already
192 thought of this, but just to note it down: it would be really nice if
193 org-babel's notion of a buffer's 'session/process' played nicely with
194 ESS's notion of the buffer's session/process. ESS keeps the current
195 process name for a buffer in a buffer-local variable
196 ess-local-process-name.
199 in [[file:lisp/org-babel-comint.el][org-babel-comint.el]]
201 Currently I've coppied and begun generalizing the functions for
202 interacting with R buffers.
203 ** TODO fully purge org-babel-R of direct comint interaction
204 try to remove all code under the [[file:lisp/org-babel-R.el::functions%20for%20evaluation%20of%20R%20code][;; functions for evaluation of R code]] line
206 ** TODO improve the source-block snippet
208 [[file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name%20Chapter%20title][file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name Chapter title]]
210 ,#name : Chapter title
213 ${1:$(make-string (string-width text) ?\=)}
218 [[file:snippets/org-mode/sb][sb -- snippet]]
220 waiting for guidance from those more familiar with yasnippets
222 ** TODO resolve references to other buffers
223 This would allow source blocks to call upon tables, source-blocks,
224 and results in other buffers.
227 - [[file:lisp/org-babel-ref.el::TODO%20allow%20searching%20for%20names%20in%20other%20buffers][org-babel-ref.el:searching-in-other-buffers]]
228 - [[file:lisp/org-babel.el::defun%20org-babel%20find%20named%20result%20name][org-babel.el#org-babel-find-named-result]]
230 ** TODO figure out how to handle graphic output
231 This is listed under [[* graphical output][graphical output]] in out objectives.
233 This should take advantage of the =:results file= option, and
234 languages which almost always produce graphical output should set
235 =:results file= to true by default. That would handle placing these
236 results in the buffer. Then if there is a combination of =silent= and
237 =file= =:results= headers we could drop the results to a temp buffer
238 and pop open that buffer...
240 ** TODO share org-babel
241 how should we share org-babel?
243 - post to org-mode and ess mailing lists
244 - create a org-babel page on worg
245 - create a short screencast demonstrating org-babel in action
248 we need to think up some good examples
250 **** interactive tutorials
251 This could be a place to use [[* org-babel assertions][org-babel assertions]].
253 for example the first step of a tutorial could assert that the version
254 of the software-package (or whatever) is equal to some value, then
255 source-code blocks could be used with confidence (and executed
256 directly from) the rest of the tutorial.
258 **** answering a text-book question w/code example
259 org-babel is an ideal environment enabling both the development and
260 demonstrationg of the code snippets required as answers to many
263 **** something using tables
264 maybe something along the lines of calculations from collected grades
267 Maybe something like the following which outputs sizes of directories
268 under the home directory, and then instead of the trivial =emacs-lisp=
269 block we could use an R block to create a nice pie chart of the
273 #+begin_src bash :results replace
277 #+begin_src emacs-lisp :var sizes=sizes :results replace
281 ** TODO command line execution
282 Allow source code blocks to be called form the command line. This
283 will be easy using the =sbe= function in [[file:lisp/org-babel-table.el][org-babel-table.el]].
285 This will rely upon [[* resolve references to other buffers][resolve references to other buffers]].
287 ** TODO inline source code blocks [3/5]
288 Like the =\R{ code }= blocks
290 not sure what the format should be, maybe just something simple
291 like =src_lang[]{}= where lang is the name of the source code
292 language to be evaluated, =[]= is optional and contains any header
293 arguments and ={}= contains the code.
295 (see [[* (sandbox) inline source blocks][the-sandbox]])
297 *** DONE evaluation with \C-c\C-c
298 Putting aside the header argument issue for now we can just run these
299 with the following default header arguments
300 - =:results= :: silent
301 - =:exports= :: results
303 *** DONE inline exportation
304 Need to add an interblock hook (or some such) through org-exp-blocks
305 *** DONE header arguments
306 We should make it possible to use header arguments.
308 *** TODO fontification
309 we should color these blocks differently
311 *** TODO refine html exportation
312 should use a span class, and should show original source in tool-tip
314 ** TODO allow tables with hline to be passed as args into R
315 This doesn't seem to work at the moment (example below). It would
316 also be nice to have a natural way for the column names of the org
317 table to become the column names of the R data frame, and to have
318 the option to specify that the first column is to be used as row
319 names in R (these must be unique). But this might require a bit of
324 | col1 | col2 | col3 |
325 |------+---------+------|
333 #+begin_src R var tabel=egtable
338 | "col1" | "col2" | "col3" |
339 |--------+-----------+--------|
341 | 4 | "schulte" | 6 |
344 Another example is in the [[*operations%20in%20on%20tables][grades example]].
346 ** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format
347 I haven't thought about this properly. Just noting it down. What
348 Sweave uses is called "R noweb" (.Rnw).
350 I found a good description of noweb in the following article (see
351 the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]).
353 I think there are two parts to noweb, the construction of
354 documentation and the extraction of source-code (with notangle).
356 *documentation*: org-mode handles all of our documentation needs in
357 a manner that I believe is superior to noweb.
359 *source extraction* At this point I don't see anyone writing large
360 applications with 100% of the source code contained in org-babel
361 files, rather I see org-babel files containing things like
362 - notes with active code chunks
363 - interactive tutorials
364 - requirements documents with code running test suites
365 - and of course experimental reports with the code to run the
366 experiment, and perform analysis
368 Basically I think the scope of the programs written in org-babel
369 (at least initially) will be small enough that it wont require the
370 addition of a tangle type program to extract all of the source code
371 into a running application.
373 On the other hand, since we already have named blocks of source
374 code which reference other blocks on which they rely, this
375 shouldn't be too hard to implement either on our own, or possibly
376 relying on something like noweb/notangle.
378 ** PROPOSED support for passing paths to files between source blocks
379 Maybe this should be it's own result type (in addition to scalars and
380 vectors). The reason being that some source-code blocks (for example
381 ditaa or anything that results in the creation of a file) may want to
382 pass a file path back to org-mode which could then be inserted into
383 the org-mode buffer as a link to the file...
385 This would allow for display of images upon export providing
386 functionality similar to =org-exp-blocks= only in a more general
389 ** PROPOSED re-implement helper functions from org-R
390 Much of the power of org-R seems to be in it's helper functions for
391 the quick graphing of tables. Should we try to re-implement these
392 functions on top of org-babel?
394 I'm thinking this may be useful both to add features to org-babel-R and
395 also to potentially suggest extensions of the framework. For example
396 one that comes to mind is the ability to treat a source-code block
397 like a function which accepts arguments and returns results. Actually
398 this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]).
400 ** DONE pass multiple reference arguments into R
401 Can we do this? I wasn't sure how to supply multiple 'var' header
402 args. Just delete this if I'm being dense.
404 This should be working, see the following example...
406 #+srcname: two-arg-example
407 #+begin_src R :var n=2 :var m=8
411 #+resname: two-arg-example
414 ** DEFERRED use textConnection to pass tsv to R?
415 When passing args from the org buffer to R, the following route is
416 used: arg in buffer -> elisp -> tsv on file -> data frame in R. I
417 think it would be possible to avoid having to write to file by
418 constructing an R expression in org-babel-R-assign-elisp, something
421 #+begin_src emacs-lisp
422 (org-babel-R-input-command
423 (format "%s <- read.table(textConnection(\"%s\"), sep=\"\\t\", as.is=TRUE)"
424 name (orgtbl-to-tsv value '(:sep "\t" :fmt org-babel-R-quote-tsv-field))))
427 I haven't tried to implement this yet as it's basically just
428 fiddling with something that works. The only reason for it I can
429 think of would be efficiency and I haven't tested that.
431 This Didn't work after an initial test. I still think this is a
432 good idea (I also think we should try to do something similar when
433 writing out results frmo R to elisp) however as it wouldn't result
434 in any functional changes I'm bumping it down to deferred for
439 #+tblname: quick-test
442 #+srcname: quick-test-src-blk
443 #+begin_src R :var vec=quick-test
449 ** DEFERRED re-implement R evaluation using ess-command or ess-execute
450 I don't have any complaints with the current R evaluation code or
451 behaviour, but I think it would be good to use the ESS functions
452 from a political point of view. Plus of course it has the normal
453 benefits of an API (insulates us from any underlying changes etc). [DED]
455 I'll look into this. I believe that I looked at and rejected these
456 functions initially but now I can't remember why. I agree with
457 your overall point about using API's where available. I will take
458 a look back at these and either switch to using the ess commands,
459 or at least articulate under this TODO the reasons for using our
460 custom R-interaction commands. [Eric]
464 Lets just replace =org-babel-R-input-command= with =ess-execute=.
466 I tried this, and although it works in some situations, I find that
467 =ess-command= will often just hang indefinitely without returning
468 results. Also =ess-execute= will occasionally hang, and pops up
469 the buffer containing the results of the command's execution, which
470 is undesirable. For now these functions can not be used. Maybe
471 someone more familiar with the ESS code can recommend proper usage
472 of =ess-command= or some other lower-level function which could be
473 used in place of [[file:lisp/org-babel-R.el::defun%20org-babel%20R%20input%20command%20command][org-babel-R-input-command]].
477 #+begin_quote ess-command
478 (ess-command COM &optional BUF SLEEP NO-PROMPT-CHECK)
480 Send the ESS process command COM and delete the output
481 from the ESS process buffer. If an optional second argument BUF exists
482 save the output in that buffer. BUF is erased before use.
483 COM should have a terminating newline.
484 Guarantees that the value of .Last.value will be preserved.
485 When optional third arg SLEEP is non-nil, `(sleep-for (* a SLEEP))'
486 will be used in a few places where `a' is proportional to `ess-cmd-delay'.
489 #+begin_quote ess-execute
490 (ess-execute COMMAND &optional INVERT BUFF MESSAGE)
492 Send a command to the ESS process.
493 A newline is automatically added to COMMAND. Prefix arg (or second arg
494 INVERT) means invert the meaning of
495 `ess-execute-in-process-buffer'. If INVERT is 'buffer, output is
496 forced to go to the process buffer. If the output is going to a
497 buffer, name it *BUFF*. This buffer is erased before use. Optional
498 fourth arg MESSAGE is text to print at the top of the buffer (defaults
499 to the command if BUFF is not given.)
502 *** out current setup
504 1) The body of the R source code block is wrapped in a function
505 2) The function is called inside of a =write.table= function call
506 writing the results to a table
507 3) The table is read using =org-table-import=
509 ** DEFERRED Rework Interaction with Running Processes [0/3]
510 *** TODO ability to select which of multiple sessions is being used
511 Increasingly it is looking like we're going to want to run all
512 source code blocks in comint buffer (sessions). Which will have
514 1) allowing background execution
515 2) maintaining state between source-blocks
516 - allowing inline blocks w/o header arguments
519 (like ess-switch-process in .R buffers)
521 Maybe this could be packaged into a header argument, something
522 like =:R_session= which could accept either the name of the
523 session to use, or the string =prompt=, in which case we could use
524 the =ess-switch-process= command to select a new process.
526 *** TODO evaluation of shell code as background process?
527 After C-c C-c on an R code block, the process may appear to
528 block, but C-g can be used to reclaim control of the .org buffer,
529 without interrupting the R evalution. However I believe this is not
530 true of bash/sh evaluation. [Haven't tried other languages] Perhaps
531 a solution is just to background the individual shell commands.
533 The other languages (aside from emacs lisp) are run through the
534 shell, so if we find a shell solution it should work for them as
537 Adding an ampersand seems to be a supported way to run commands in
538 the background (see [[http://www.emacswiki.org/emacs/ExecuteExternalCommand#toc4][external-commands]]). Although a more extensible
539 solution may involve the use of the [[elisp:(progn (describe-function 'call-process-region) nil)][call-process-region]] function.
541 Going to try this out in a new file [[file:lisp/org-babel-proc.el][org-babel-proc.el]]. This should
542 contain functions for asynchronously running generic shell commands
543 in the background, and then returning their input.
545 **** partial update of org-mode buffer
546 The sleekest solution to this may be using a comint buffer, and
547 then defining a filter function which would incrementally interpret
548 the results as they are returned, including insertion into the
549 org-mode buffer. This may actually cause more problems than it is
550 worth, what with the complexities of identifying the types of
551 incrementally returned results, and the need for maintenance of a
552 process marker in the org buffer.
554 **** 'working' spinner
555 It may be nice and not too difficult to place a spinner on/near the
556 evaluating source code block
558 *** TODO conversion of output from interactive shell, R (and python) sessions to org-babel buffers
559 [DED] This would be a nice feature I think. Although an org-babel
560 purist would say that it's working the wrong way round... After
561 some interactive work in a *R* buffer, you save the buffer, maybe
562 edit out some lines, and then convert it to org-babel format for
563 posterity. Same for a shell session either in a *shell* buffer, or
564 pasted from another terminal emulator. And python of course.
566 ** DONE ensure that table ranges work
567 when a table range is passed to org-babel as an argument, it should be
568 interpreted as a vector.
572 | 3 | 4 | Array:123456 |
576 #+TBLFM: @1$3='(sbe simple-sbe-example (n 4))::@2$3='(sbe task-table-range (n @1$1..@6$1))::@3$3='(sbe task-table-range (n (@1$1..@6$1)))
578 #+srcname: simple-sbe-example
579 #+begin_src emacs-lisp
583 #+srcname: task-table-range
584 #+begin_src ruby :var n=simple-sbe-example
588 #+srcname: simple-results
589 #+begin_src emacs-lisp :var n=task-table-range(n=(1 2 3))
593 #+resname: simple-results
596 #+srcname: task-arr-referent
597 #+begin_src ruby :var ar=(1 2 3)
601 #+resname: task-arr-referent
604 ** DONE global variable indicating default to vector output
605 how about an alist... =org-babel-default-header-args= this may already
606 exist... just execute the following and all source blocks will default
609 #+begin_src emacs-lisp
610 (setq org-babel-default-header-args '((:results . "vector")))
613 ** DONE name named results if source block is named
614 currently this isn't happening although it should be
616 #+srcname: test-naming-named-source-blocks
617 #+begin_src emacs-lisp
621 #+resname: test-naming-named-source-blocks
623 ** DONE (simple caching) check for named results before source blocks
624 see the TODO comment in [[file:lisp/org-babel-ref.el::TODO%20This%20should%20explicitly%20look%20for%20resname%20lines%20before][org-babel-ref.el#org-babel-ref-resolve-reference]]
625 ** DONE set =:results silent= when eval with prefix argument
627 #+begin_src emacs-lisp
630 ** DONE results-type header (vector/file) [3/3]
631 In response to a point in Dan's email. We should allow the user to
632 force scalar or vector results. This could be done with a header
633 argument, and the default behavior could be controlled through a
634 configuration variable.
636 #+srcname: task-trivial-vector
637 #+begin_src ruby :results replace vector
644 since it doesn't make sense to turn a vector into a scalar, lets
645 just add a two values...
647 - vector :: forces the results to be a vector (potentially 1 dimensional)
648 - file :: this throws an error if the result isn't a string, and
649 tries to treat it as a path to a file.
651 I'm just going to cram all of these into the =:results= header
652 argument. Then if we allow multiple header arguments it should
653 work out, for example one possible header argument string could be
654 =:results replace vector file=, which would *replace* any existing
655 results forcing the results into an org-mode table, and
656 interpreting any strings as file paths.
658 *** DONE multiple =:results= headers
660 #+srcname: multiple-result-headers
661 #+begin_src ruby :results replace silent
667 *** DONE file result types
668 When inserting into an org-mode buffer create a link with the path
669 being the value, and optionally the display being the
670 =file-name-nondirectory= if it exists.
672 #+srcname: task-file-result
673 #+begin_src python :results replace file
678 [[something][something]]
681 This will be useful because blocks like =ditaa= and =dot= can return
682 the string path of their files, and can add =file= to their results
685 *** DONE vector result types
687 #+srcname: task-force-results
688 #+begin_src emacs-lisp :results vector
696 In order to do this we will need to start naming our results.
697 Since the source blocks are named with =#+srcname:= lines we can
698 name results with =#+resname:= lines (if the source block has no
699 name then no name is given to the =#+resname:= line on creation,
700 otherwise the name of the source block is used).
702 This will have the additional benefit of allowing results and
703 source blocks to be located in different places in a buffer (and
704 eventually in different buffers entirely).
706 #+srcname: developing-resnames
707 #+begin_src emacs-lisp :results silent
711 Once source blocks are able to find their own =#+resname:= lines
714 #+srcname: sbe-w-new-results
715 #+begin_src emacs-lisp :results replace
716 (sbe "developing-resnames")
722 *** TODO change the results insertion functions to use these lines
724 *** TODO teach references to resolve =#+resname= lines.
726 ** DONE org-babel tests org-babel [1/1]
727 since we are accumulating this nice collection of source-code blocks
728 in the sandbox section we should make use of them as unit tests.
729 What's more, we should be able to actually use org-babel to run these
732 We would just need to cycle over every source code block under the
733 sandbox, run it, and assert that the return value is equal to what we
736 I have the feeling that this should be possible using only org-babel
737 functions with minimal or no additional elisp. It would be very cool
738 for org-babel to be able to test itself.
740 This is now done, see [[* Tests]].
742 *** DEFERRED org-babel assertions (may not be necessary)
743 These could be used to make assertions about the results of a
744 source-code block. If the assertion fails then the point could be
745 moved to the block, and error messages and highlighting etc... could
748 ** DONE make C-c C-c work anywhere within source code block?
749 This seems like it would be nice to me, but perhaps it would be
750 inefficient or ugly in implementation? I suppose you could search
751 forward, and if you find #+end_src before you find #+begin_src,
752 then you're inside one. [DED]
754 Agreed, I think inside of the =#+srcname: line= would be useful as
757 #+srcname: testing-out-cc
758 #+begin_src emacs-lisp
762 ** DONE integration with org tables
763 We should make it easy to call org-babel source blocks from org-mode
764 table formulas. This is practical now that it is possible to pass
765 arguments to org-babel source blocks.
767 See the related [[* (sandbox) integration w/org tables][sandbox]] header for tests/examples.
769 *** digging in org-table.el
770 In the past [[file:~/src/org/lisp/org-table.el::org%20table%20el%20The%20table%20editor%20for%20Org%20mode][org-table.el]] has proven difficult to work with.
772 Should be a hook in [[file:~/src/org/lisp/org-table.el::defun%20org%20table%20eval%20formula%20optional%20arg%20equation][org-table-eval-formula]].
774 Looks like I need to change this [[file:~/src/org/lisp/org-table.el::if%20lispp][if statement]] (line 2239) into a cond
777 ** DONE source blocks as functions
779 Allow source code blocks to be called like functions, with arguments
780 specified. We are already able to call a source-code block and assign
781 it's return result to a variable. This would just add the ability to
782 specify the values of the arguments to the source code block assuming
783 any exist. For an example see
785 When a variable appears in a header argument, how do we differentiate
786 between it's value being a reference or a literal value? I guess this
787 could work just like a programming language. If it's escaped or in
788 quotes, then we count it as a literal, otherwise we try to look it up
791 ** DONE folding of code blocks? [2/2]
792 [DED] In similar way to using outline-minor-mode for folding function
793 bodies, can we fold code blocks? #+begin whatever statements are
794 pretty ugly, and in any case when you're thinking about the overall
795 game plan you don't necessarily want to see the code for each Step.
797 *** DONE folding of source code block
798 Sounds good, and wasn't too hard to implement. Code blocks should
799 now be fold-able in the same manner as headlines (by pressing TAB
802 *** REJECTED folding of results
803 So, lets do a three-stage tab cycle... First fold the src block,
804 then fold the results, then unfold.
806 There's no way to tell if the results are a table or not w/o
807 actually executing the block which would be too expensive of an
810 ** DONE selective export of text, code, figures
811 [DED] The org-babel buffer contains everything (code, headings and
812 notes/prose describing what you're up to, textual/numeric/graphical
813 code output, etc). However on export to html / LaTeX one might want
814 to include only a subset of that content. For example you might
815 want to create a presentation of what you've done which omits the
818 [EMS] So I think this should be implemented as a property which can
819 be set globally or on the outline header level (I need to review
820 the mechanics of org-mode properties). And then as a source block
821 header argument which will apply only to a specific source code
822 block. A header argument of =:export= with values of
824 - =code= :: just show the code in the source code block
825 - =none= :: don't show the code or the results of the evaluation
826 - =results= :: just show the results of the code evaluation (don't
827 show the actual code)
828 - =both= :: show both the source code, and the results
830 this will be done in [[* (sandbox) selective export][(sandbox) selective export]].
832 ** DONE a header argument specifying silent evaluation (no output)
833 This would be useful across all types of source block. Currently
834 there is a =:replace t= option to control output, this could be
835 generalized to an =:output= option which could take the following
838 - =t= :: this would be the default, and would simply insert the
839 results after the source block
840 - =replace= :: to replace any results which may already be there
841 - =silent= :: this would inhibit any insertion of the results
843 This is now implemented see the example in the [[* silent evaluation][sandbox]]
845 ** DONE assign variables from tables in R
846 This is now working (see [[* (sandbox table) R][(sandbox-table)-R]]). Although it's not that
847 impressive until we are able to print table results from R.
849 ** DONE insert 2-D R results as tables
850 everything is working but R and shell
856 This has already been tackled by Dan in [[file:existing_tools/org-R.el::defconst%20org%20R%20write%20org%20table%20def][org-R:check-dimensions]]. The
857 functions there should be useful in combination with [[http://cran.r-project.org/doc/manuals/R-data.html#Export-to-text-files][R-export-to-csv]]
858 as a means of converting multidimensional R objects to emacs lisp.
860 It may be as simple as first checking if the data is multidimensional,
861 and then, if so using =write= to write the data out to a temporary
862 file from which emacs can read the data in using =org-table-import=.
864 Looking into this further, is seems that there is no such thing as a
865 scalar in R [[http://tolstoy.newcastle.edu.au/R/help/03a/3733.html][R-scalar-vs-vector]] In that light I am not sure how to
866 deal with trivial vectors (scalars) in R. I'm tempted to just treat
867 them as vectors, but then that would lead to a proliferation of
868 trivial 1-cell tables...
870 ** DONE allow variable initialization from source blocks
871 Currently it is possible to initialize a variable from an org-mode
872 table with a block argument like =table=sandbox= (note that the
873 variable doesn't have to named =table=) as in the following example
879 #+begin_src emacs-lisp :var table=sandbox :results replace
880 (message (format "table = %S" table))
883 : "table = ((1 2 3) (4 \"schulte\" 6))"
885 It would be good to allow initialization of variables from the results
886 of other source blocks in the same manner. This would probably
887 require the addition of =#+SRCNAME: example= lines for the naming of
888 source blocks, also the =table=sandbox= syntax may have to be expanded
889 to specify whether the target is a source code block or a table
890 (alternately we could just match the first one with the given name
891 whether it's a table or a source code block).
893 At least initially I'll try to implement this so that there is no need
894 to specify whether the reference is to a table or a source-code block.
895 That seems to be simpler both in terms of use and implementation.
897 This is now working for emacs-lisp, ruby and python (and mixtures of
898 the three) source blocks. See the examples in the [[* (sandbox) referencing other source blocks][sandbox]].
900 This is currently working only with emacs lisp as in the following
901 example in the [[* emacs lisp source reference][emacs lisp source reference]].
904 ** TODO Add languages [0/5]
905 I'm sure there are many more that aren't listed here. Please add
906 them, and bubble any that you particularly care about up to the top.
908 Any new language should be implemented in a org-babel-lang.el file.
909 Follow the pattern set by [[file:lisp/org-babel-script.el][org-babel-script.el]], [[file:lisp/org-babel-shell.el][org-babel-shell.el]] and
910 [[file:lisp/org-babel-R.el][org-babel-R.el]].
913 This could probably be added to [[file:lisp/org-babel-script.el][org-babel-script.el]]
918 (see [[* file result types][file result types]])
921 (see [[* file result types][file result types]])
924 (see [[* file result types][file result types]])
929 ** TODO non-orgtbl formatted lists
932 #+srcname: this-doesn't-match-orgtbl
933 #+begin_src emacs-lisp :results replace
934 '((:results . "replace"))
937 #+resname: this-doesn't-match-orgtbl
939 ** TODO collapsing consecutive newlines in string output
941 #+srcname: multi-line-string-output
942 #+begin_src ruby :results replace
943 "the first line ends here
946 and this is the second one
952 : the first line ends here
953 : and this is the second one
954 : return even a third
956 ** TODO cursor movement when evaluating source blocks
957 E.g. the pie chart example. Despite the save-window-excursion in
958 org-babel-execute:R. (I never learned how to do this properly: org-R
959 jumps all over the place...)
961 ** DONE R-code broke on "org-babel" rename
963 #+srcname: bug-R-babels
968 ** DONE error on trivial R results
970 So I know it's generally not a good idea to squash error without
971 handling them, but in this case the error almost always means that
972 there was no file contents to be read by =org-table-import=, so I
975 #+srcname: bug-trivial-r1
976 #+begin_src R :results replace
977 pie(c(1, 2, 3), labels = c(1, 2, 3))
980 #+srcname: bug-trivial-r2
981 #+begin_src R :results replace
985 #+resname: bug-trivial-r2
988 #+srcname: bug-trivial-r3
989 #+begin_src R :results replace
993 #+resname: bug-trivial-r3
998 ** DONE ruby new variable creation (multi-line ruby blocks)
999 Actually it looks like we were dropping all but the last line.
1001 #+srcname: multi-line-ruby-test
1002 #+begin_src ruby :var table=bug-numerical-table :results replace
1004 table.each{|n| total += n}
1011 ** DONE R code execution seems to choke on certain inputs
1012 Currently the R code seems to work on vertical (but not landscape)
1015 #+srcname: little-fake
1016 #+begin_src emacs-lisp
1020 #+begin_src R :var num=little-fake
1034 #+srcname: set-debug-on-error
1035 #+begin_src emacs-lisp :results silent
1036 (setq debug-on-error t)
1039 #+srcname: bug-numerical-table
1040 #+begin_src emacs-lisp :results silent
1044 #+srcname: bug-R-number-evaluation
1045 #+begin_src R :var table=bug-numerical-table :results replace
1052 #+tblname: bug-vert-table
1057 #+srcname: bug-R-vertical-table
1058 #+begin_src R :var table=bug-vert-table :results silent
1062 ** DEFERRED org bug/request: prevent certain org behaviour within code blocks
1063 E.g. [[]] gets recognised as a link (when there's text inside the
1064 brackets). This is bad for R code at least, and more generally
1065 could be argued to be inappropriate. Is it difficult to get org to
1066 ignore text in code blocks? [DED]
1068 I believe Carsten addressed this recently on the mailing list with
1069 the comment that it was indeed a difficult issue. I believe this
1070 may be one area where we could wait for an upstream (org-mode) fix.
1071 ** DONE with :results replace, non-table output doesn't replace table output
1072 And vice versa. E.g. Try this first with table and then with len(table) [DED]
1073 #+begin_src python :var table=sandbox :results replace
1078 | 4 | "schulte" | 6 |
1081 Yes, this is certainly a problem. I fear that if we begin replacing
1082 anything immediately following a source block (regardless of whether
1083 it matches the type of our current results) we may accidentally delete
1084 hand written portions of the user's org-mode buffer.
1086 I think that the best solution here would be to actually start
1087 labeling results with a line that looks something like...
1091 This would have a couple of benefits...
1092 1) we wouldn't have to worry about possibly deleting non-results
1093 (which is currently an issue)
1094 2) we could reliably replace results even if there are different types
1095 3) we could reference the results of a source-code block in variable
1096 definitions, which would be useful if for example we don't wish to
1097 re-run a source-block every time because it is long-running.
1099 Thoughts? If no-one objects, I believe I will implement the labeling
1102 ** DONE extra quotes for nested string
1103 Well R appears to be reading the tables without issue...
1105 these *should* be quoted
1107 #+begin_src sh :results replace
1112 | "README.markdown" |
1115 | "existing_tools" |
1119 | "test-export.html" |
1120 | "test-export.org" |
1122 #+srcname: test-quotes
1123 #+begin_src ruby :var tab=ls
1129 #+srcname: test-quotes
1130 #+begin_src R :var tab=ls
1136 ** DONE simple ruby arrays not working
1138 As an example eval the following. Adding a line to test
1140 #+srcname: simple-ruby-array
1145 #+srcname: ruby-array-test
1146 #+begin_src ruby :var ar = simple-ruby-array
1150 ** DONE space trailing language name
1151 fix regexp so it works when there's a space trailing the language name
1153 #+srcname: test-trailing-space
1158 ** DONE Args out of range error
1160 The following block resulted in the error below [DED]. It ran without
1161 error directly in the shell.
1164 for platf in ill aff ; do
1165 for pop in CEU YRI ASI ; do
1166 rm -f $platf/hapmap-genos-$pop-all $platf/hapmap-rs-all
1167 cat $platf/hapmap-genos-$pop-* > $platf/hapmap-genos-$pop-all
1168 cat $platf/hapmap-rs-* > $platf/hapmap-rs-all
1173 executing source block with sh...
1174 finished executing source block
1175 string-equal: Args out of range: "", -1, 0
1177 the error =string-equal: Args out of range: "", -1, 0= looks like what
1178 used to be output when the block returned an empty results string.
1179 This should be fixed in the current version, you should now see the
1180 following message =no result returned by source block=.
1182 ** DONE ruby arrays not recognized as such
1184 Something is wrong in [[file:lisp/org-babel-script.el]] related to the
1185 recognition of ruby arrays as such.
1187 #+begin_src ruby :results replace
1193 #+begin_src python :results replace
1202 Evaluate all the cells in this table for a comprehensive test of the
1203 org-babel functionality.
1205 *Note*: if you have customized =org-babel-default-header-args= then some
1206 of these tests may fail.
1208 #+TBLNAME: org-babel-tests
1209 | functionality | block | arg | expected | results | pass |
1210 |-------------------------+----------------------------+-----+-------------+-------------+------|
1211 | basic evaluation | | | | | pass |
1212 |-------------------------+----------------------------+-----+-------------+-------------+------|
1213 | emacs lisp | basic-elisp | | 5 | 5 | pass |
1214 | shell | basic-shell | | 6 | 6 | pass |
1215 | ruby | basic-ruby | | org-babel | org-babel | pass |
1216 | python | basic-python | | hello world | hello world | pass |
1217 | R | basic-R | | 13 | 13 | pass |
1218 |-------------------------+----------------------------+-----+-------------+-------------+------|
1219 | tables | | | | | pass |
1220 |-------------------------+----------------------------+-----+-------------+-------------+------|
1221 | emacs lisp | table-elisp | | 3 | 3 | pass |
1222 | ruby | table-ruby | | 1-2-3 | 1-2-3 | pass |
1223 | python | table-python | | 5 | 5 | pass |
1224 | R | table-R | | 3.5 | 3.5 | pass |
1225 |-------------------------+----------------------------+-----+-------------+-------------+------|
1226 | source block references | | | | | pass |
1227 |-------------------------+----------------------------+-----+-------------+-------------+------|
1228 | all languages | chained-ref-last | | Array | Array | pass |
1229 |-------------------------+----------------------------+-----+-------------+-------------+------|
1230 | source block functions | | | | | pass |
1231 |-------------------------+----------------------------+-----+-------------+-------------+------|
1232 | emacs lisp | defun-fibb | | fibbd | fibbd | pass |
1233 | run over | Fibonacci | 0 | 1 | 1 | pass |
1234 | a | Fibonacci | 1 | 1 | 1 | pass |
1235 | variety | Fibonacci | 2 | 2 | 2 | pass |
1236 | of | Fibonacci | 3 | 3 | 3 | pass |
1237 | different | Fibonacci | 4 | 5 | 5 | pass |
1238 | arguments | Fibonacci | 5 | 8 | 8 | pass |
1239 |-------------------------+----------------------------+-----+-------------+-------------+------|
1240 | bugs and tasks | | | | | pass |
1241 |-------------------------+----------------------------+-----+-------------+-------------+------|
1242 | simple ruby arrays | ruby-array-test | | 3 | 3 | pass |
1243 | R number evaluation | bug-R-number-evaluation | | 2 | 2 | pass |
1244 | multi-line ruby blocks | multi-line-ruby-test | | 2 | 2 | pass |
1245 | forcing vector results | test-forced-vector-results | | Array | Array | pass |
1246 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
1250 #+srcname: basic-elisp
1251 #+begin_src emacs-lisp :results silent
1255 #+srcname: basic-shell
1256 #+begin_src sh :results silent
1261 #+srcname: basic-ruby
1262 #+begin_src ruby :results silent
1266 #+srcname: basic-python
1267 #+begin_src python :results silent
1272 #+begin_src R :results silent
1279 #+tblname: test-table
1283 #+srcname: table-elisp
1284 #+begin_src emacs-lisp :results silent :var table=test-table
1285 (length (car table))
1288 #+srcname: table-ruby
1289 #+begin_src ruby :results silent :var table=test-table
1290 table.first.join("-")
1293 #+srcname: table-python
1294 #+begin_src python :var table=test-table
1299 #+begin_src R :var table=test-table
1305 Lets pass a references through all of our languages...
1307 Lets start by reversing the table from the previous examples
1309 #+srcname: chained-ref-first
1310 #+begin_src python :var table = test-table
1315 #+resname: chained-ref-first
1319 Take the first part of the list
1321 #+srcname: chained-ref-second
1322 #+begin_src R :var table = chained-ref-first
1326 #+resname: chained-ref-second
1330 Turn the numbers into string
1332 #+srcname: chained-ref-third
1333 #+begin_src emacs-lisp :var table = chained-ref-second
1334 (mapcar (lambda (el) (format "%S" el)) table)
1337 #+resname: chained-ref-third
1340 and Check that it is still a list
1342 #+srcname: chained-ref-last
1343 #+begin_src ruby :var table=chained-ref-third
1348 ** source blocks as functions
1350 #+srcname: defun-fibb
1351 #+begin_src emacs-lisp :results silent
1352 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
1355 #+srcname: fibonacci
1356 #+begin_src emacs-lisp :results silent :var n=7
1361 Testing the insertion of results into org-mode tables.
1363 #+srcname: multi-line-output
1364 #+begin_src ruby :results replace
1365 "the first line ends here
1368 and this is the second one
1374 : the first line ends here
1375 : and this is the second one
1376 : return even a third
1378 #+srcname: multi-line-error
1379 #+begin_src ruby :results replace
1380 raise "oh nooooooooooo"
1384 : -:5: warning: parenthesize argument(s) for future version
1385 : -:5:in `main': oh nooooooooooo (RuntimeError)
1388 | the first line ends here... | -:5: warning: parenthesize argument(s) for future version... |
1389 #+TBLFM: $1='(sbe "multi-line-output")::$2='(sbe "multi-line-error")
1391 ** forcing results types tests
1393 #+srcname: test-trivial-vector
1394 #+begin_src emacs-lisp :results vector silent
1398 #+srcname: test-forced-vector-results
1399 #+begin_src ruby :var triv=test-trivial-vector :results silent
1408 To run these examples evaluate [[file:lisp/org-babel-init.el][org-babel-init.el]]
1410 ** org-babel.el beginning functionality
1412 #+begin_src sh :results replace
1416 : Thu May 14 18:52:25 EDT 2009
1422 : Thu May 14 18:59:09 -0400 2009
1433 #+begin_src R :results replace
1442 hist(rgamma(20,3,3))
1446 ** org-babel plays with tables
1447 Alright, this should demonstrate both the ability of org-babel to read
1448 tables into a lisp source code block, and to then convert the results
1449 of the source code block into an org table. It's using the classic
1450 "lisp is elegant" demonstration transpose function. To try this
1453 1. evaluate [[file:lisp/org-babel-init.el]] to load org-babel and friends
1454 2. evaluate the transpose definition =\C-c\C-c= on the beginning of
1456 3. evaluate the next source code block, this should read in the table
1457 because of the =:var table=previous=, then transpose the table, and
1458 finally it should insert the transposed table into the buffer
1459 immediately following the block
1463 #+begin_src emacs-lisp :results silent
1464 (defun transpose (table)
1465 (apply #'mapcar* #'list table))
1473 #+begin_src emacs-lisp :var table=sandbox :results replace
1478 #+begin_src emacs-lisp
1482 | 1 | 2 | 3 | 4 | 5 |
1486 #+begin_src ruby :var table=sandbox :results replace
1487 table.first.join(" - ")
1492 #+begin_src python :var table=sandbox :results replace
1498 #+begin_src ruby :var table=sandbox :results replace
1503 | 4 | "schulte" | 6 |
1505 #+begin_src python :var table=sandbox :results replace
1511 | "__add__" | "__class__" | "__contains__" | "__delattr__" | "__delitem__" | "__delslice__" | "__doc__" | "__eq__" | "__format__" | "__ge__" | "__getattribute__" | "__getitem__" | "__getslice__" | "__gt__" | "__hash__" | "__iadd__" | "__imul__" | "__init__" | "__iter__" | "__le__" | "__len__" | "__lt__" | "__mul__" | "__ne__" | "__new__" | "__reduce__" | "__reduce_ex__" | "__repr__" | "__reversed__" | "__rmul__" | "__setattr__" | "__setitem__" | "__setslice__" | "__sizeof__" | "__str__" | "__subclasshook__" | "append" | "count" | "extend" | "index" | "insert" | "pop" | "remove" | "reverse" | "sort" |
1513 *** (sandbox table) R
1515 #+TBLNAME: sandbox_r
1519 #+begin_src R :results replace
1520 x <- c(rnorm(10, mean=-3, sd=1), rnorm(10, mean=3, sd=1))
1524 | -3.35473133869346 |
1526 | -3.32819924928633 |
1527 | -2.97310212756194 |
1528 | -2.09640758369576 |
1529 | -5.06054014378736 |
1530 | -2.20713700711221 |
1531 | -1.37618039712037 |
1532 | -1.95839385821742 |
1533 | -3.90407396475502 |
1534 | 2.51168071590226 |
1535 | 3.96753011570494 |
1536 | 3.31793212627865 |
1537 | 1.99829753972341 |
1538 | 4.00403686419829 |
1539 | 4.63723764452927 |
1540 | 3.94636744261313 |
1541 | 3.58355906547775 |
1542 | 3.01563442274226 |
1545 #+begin_src R var tabel=sandbox_r :results replace
1550 | 4 | "schulte" | 6 |
1553 Now shell commands are converted to tables using =org-table-import=
1554 and if these tables are non-trivial (i.e. have multiple elements) then
1555 they are imported as org-mode tables...
1557 #+begin_src sh :results replace
1561 | "total" | 208 | "" | "" | "" | "" | "" | "" |
1562 | "-rw-r--r--" | 1 | "dan" | "dan" | 57 | 2009 | 15 | "block" |
1563 | "-rw-r--r--" | 1 | "dan" | "dan" | 35147 | 2009 | 15 | "COPYING" |
1564 | "-rw-r--r--" | 1 | "dan" | "dan" | 722 | 2009 | 18 | "examples.org" |
1565 | "drwxr-xr-x" | 4 | "dan" | "dan" | 4096 | 2009 | 19 | "existing_tools" |
1566 | "-rw-r--r--" | 1 | "dan" | "dan" | 2207 | 2009 | 14 | "intro.org" |
1567 | "drwxr-xr-x" | 2 | "dan" | "dan" | 4096 | 2009 | 18 | "org-babel" |
1568 | "-rw-r--r--" | 1 | "dan" | "dan" | 277 | 2009 | 20 | "README.markdown" |
1569 | "-rw-r--r--" | 1 | "dan" | "dan" | 11837 | 2009 | 18 | "rorg.html" |
1570 | "-rw-r--r--" | 1 | "dan" | "dan" | 61829 | 2009 | 19 | "#rorg.org#" |
1571 | "-rw-r--r--" | 1 | "dan" | "dan" | 60190 | 2009 | 19 | "rorg.org" |
1572 | "-rw-r--r--" | 1 | "dan" | "dan" | 972 | 2009 | 11 | "test-export.org" |
1575 ** silent evaluation
1583 #+begin_src ruby :results silent
1587 #+begin_src ruby :results replace
1594 ** (sandbox) referencing other source blocks
1595 Doing this in emacs-lisp first because it's trivial to convert
1596 emacs-lisp results to and from emacs-lisp.
1598 *** emacs lisp source reference
1599 This first example performs a calculation in the first source block
1600 named =top=, the results of this calculation are then saved into the
1601 variable =first= by the header argument =:var first=top=, and it is
1602 used in the calculations of the second source block.
1605 #+begin_src emacs-lisp
1609 #+begin_src emacs-lisp :var first=top :results replace
1615 This example is the same as the previous only the variable being
1616 passed through is a table rather than a number.
1618 #+begin_src emacs-lisp :results silent
1619 (defun transpose (table)
1620 (apply #'mapcar* #'list table))
1623 #+TBLNAME: top_table
1627 #+SRCNAME: second_src_example
1628 #+begin_src emacs-lisp :var table=top_table
1632 #+begin_src emacs-lisp :var table=second_src_example :results replace
1637 | 4 | "schulte" | 6 |
1639 Now working for ruby
1646 #+begin_src ruby :var other=start :results replace
1652 #+SRCNAME: start_two
1657 #+begin_src python :var another=start_two :results replace
1662 Since all variables are converted into Emacs Lisp it is no problem to
1663 reference variables specified in another language.
1665 #+SRCNAME: ruby-block
1670 #+SRCNAME: lisp_block
1671 #+begin_src emacs-lisp :var ruby-variable=ruby-block
1675 #+begin_src python :var lisp_var=lisp_block
1684 #+begin_src R :results replace
1691 #+begin_src R :var other=first_r :results replace
1698 ** (sandbox) selective export
1700 For exportation tests and examples see (including exportation of
1701 inline source code blocks) [[file:test-export.org]]
1704 ** (sandbox) source blocks as functions
1707 #+begin_src emacs-lisp :results silent
1712 #+begin_src emacs-lisp :var n=default :results replace
1718 #+begin_src emacs-lisp :var result=triple(n=3, m=98) :results replace
1724 The following just demonstrates the ability to assign variables to
1725 literal values, which was not implemented until recently.
1727 #+begin_src ruby :var num="eric" :results replace
1734 ** (sandbox) inline source blocks
1736 This is an inline source code block src_ruby{1 + 6}. And another
1737 source block with text output src_emacs-lisp{"eric"}.
1739 This is an inline source code block with header
1740 arguments. src_ruby[:var n=fibbd( n = 0 )]{n}
1743 ** (sandbox) integration w/org tables
1745 #+begin_src emacs-lisp :results silent
1746 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
1750 #+begin_src emacs-lisp :var n=4 :results silent
1754 #+begin_src emacs-lisp :results silent
1755 (mapcar #'fibbd '(0 1 2 3 4 5 6 7 8))
1758 Something is not working here. The function `sbe ' works fine when
1759 called from outside of the table (see the source block below), but
1760 produces an error when called from inside the table. I think there
1761 must be some narrowing going on during intra-table emacs-lisp
1764 | original | fibbd |
1765 |----------+-------|
1776 #+TBLFM: $2='(sbe "fibbd" (n $1))
1780 #+begin_src emacs-lisp :results silent
1781 (sbe 'fibbd (n "8"))
1786 LocalWords: DBlocks dblocks org-babel el eric fontification