1 #+TITLE: org-babel --- facilitating communication between programming languages and people
2 #+SEQ_TODO: TODO PROPOSED | DONE DEFERRED REJECTED
3 #+OPTIONS: H:3 num:nil toc:t
4 #+STARTUP: oddeven hideblocks
8 Org-Babel enables *communication* between programming languages and
12 - communication between programs :: Data passes seamlessly between
13 different programming languages, text, and tables.
14 - communication between people :: Data and calculations are embedded
15 in the same document as notes explanations and reports.
17 ** communication between programs
19 Org-Mode supports embedded blocks of source code (in any language)
20 inside of Org documents. Org-Babel allows these blocks of code to be
21 executed from within Org-Mode with natural handling of their inputs
25 with both scalar, file, and table output
28 *** reading information from tables
31 *** reading information from other source blocks (disk usage in your home directory)
33 This will work for Linux and Mac users, not so sure about shell
34 commands for windows users.
36 To run place the cursor on the =#+begin_src= line of the source block
37 labeled directory-pie and press =\C-c\C-c=.
39 #+srcname: directories
40 #+begin_src bash :results replace
41 cd ~ && du -sc * |grep -v total
44 #+resname: directories
46 | 11882808 | "Documents" |
47 | 8210024 | "Downloads" |
48 | 879800 | "Library" |
51 | 5307664 | "Pictures" |
60 #+srcname: directory-pie
61 #+begin_src R :var dirs = directories
62 pie(dirs[,1], labels = dirs[,2])
66 *** operations in/on tables
68 #+tblname: grades-table
69 | student | grade | letter |
70 |---------+-------+--------|
77 #+TBLFM: $2='(sbe random-score-generator)::$3='(sbe assign-grade (score $2))
79 #+srcname: assign-grade
80 #+begin_src ruby :var score=99
91 #+srcname: random-score-generator
96 #+srcname: show-distribution
97 #+begin_src R :var grades=grades-table
102 ** communication between people
103 Quick overview of Org-Mode's exportation abilities, with links to the
104 online Org-Mode documentation, a focus on source-code blocks, and the
105 exportation options provided by Org-Babel.
107 *** Interactive tutorial
108 This would demonstrate applicability to Reproducible Research, and
109 Literate Programming.
111 *** Tests embedded in documentation
112 org-babels own functional tests are contained in a large org-mode
113 table, allowing the test suite to be run be evaluation of the table
114 and the results to be collected in the same table.
118 ** TODO add a function to jump to a source-block by name
119 ** TODO Create objects in top level (global) environment [3/5]
122 *** initial requirement statement [DED]
123 At the moment, objects created by computations performed in the
124 code block are evaluated in the scope of the
125 code-block-function-body and therefore disappear when the code
126 block is evaluated {unless you employ some extra trickery like
127 assign('name', object, env=globalenv()) }. I think it will be
128 desirable to also allow for a style wherein objects that are
129 created in one code block persist in the R global environment and
130 can be re-used in a separate block.
132 This is what Sweave does, and while I'm not saying we have to be
133 the same as Sweave, it wouldn't be hard for us to provide the same
134 behaviour in this case; if we don't, we risk undeservedly being
135 written off as an oddity by some.
137 IOW one aspect of org-babel is that of a sort of functional
138 meta-programming language. This is crazy, in a very good
139 way. Nevertheless, wrt R I think there's going to be a lot of value
140 in providing for a working style in which the objects are stored in
141 the R session, rather than elisp/org buffer. This will be a very
142 familiar working style to lots of people.
144 There are no doubt a number of different ways of accomplishing
145 this, the simplest being a hack like adding
149 assign(objname, get(objname), envir=globalenv())
152 to the source code block function body. (Maybe wrap it in an on.exit() call).
154 However this may deserve to be thought about more carefully, perhaps
155 with a view to having a uniform approach across languages. E.g. shell
156 code blocks have the same semantics at the moment (no persistence of
157 variables across code blocks), because the body is evaluated in a new
158 bash shell process rather than a running shell. And I guess the same
159 is true for python. However, in both these cases, you could imagine
160 implementing the alternative in which the body is evaluated in a
161 persistent interactive session. It's just that it's particularly
162 natural for R, seeing as both ESS and org-babel evaluate commands in a
163 single persistent R session.
167 Thanks for bringing this up. I think you are absolutely correct that we
168 should provide support for a persistent environment (maybe called a
169 *session*) in which to evaluate code blocks. I think the current setup
170 demonstrates my personal bias for a functional style of programming
171 which is certainly not ideal in all contexts.
173 While the R function you mention does look like an elegant solution, I
174 think we should choose an implementation that would be the same across
175 all source code types. Specifically I think we should allow the user to
176 specify an optional *session* as a header variable (when not present we
177 assume a default session for each language). The session name could be
178 used to name a comint buffer (like the *R* buffer) in which all
179 evaluation would take place (within which variables would retain their
180 values --at least once I remove some of the functional method wrappings
181 currently in place-- ).
183 This would allow multiple environments to be used in the same buffer,
184 and once this setup was implemented we should be able to fairly easily
185 implement commands for jumping between source code blocks and the
186 related session buffers, as well as for dumping the last N commands from
187 a session into a new or existing source code block.
189 Please let me know if you foresee any problems with this proposed setup,
190 or if you think any parts might be confusing for people coming from
191 Sweave. I'll hopefully find some time to work on this later in the
194 *** can functional and interpreted/interactive models coexist?
196 Even though both of these use the same =*R*= buffer the value of =a=
197 is not preserved because it is assigned inside of a functional
200 #+srcname: task-R-sessions
207 #+srcname: task-R-same-session
212 This functional wrapper was implemented in order to efficiently return
213 the results of the execution of the entire source code block. However
214 it inhibits the evaluation of source code blocks in the top level,
215 which would allow for persistence of variable assignment across
216 evaluations. How can we allow *both* evaluation in the top level, and
217 efficient capture of the return value of an entire source code block
218 in a language independent manner?
220 Possible solutions...
221 1) we can't so we will have to implement two types of evaluation
222 depending on which is appropriate (functional or imperative)
223 2) we remove the functional wrapper and parse the source code block
224 into it's top level statements (most often but not always on line
225 breaks) so that we can isolate the final segment which is our
227 3) we add some sort of "#+return" line to the code block
228 4) we take advantage of each languages support for meta-programming
229 through =eval= type functions, and use said to evaluate the entire
230 blocks in such a way that their environment can be combined with the
231 global environment, and their results are still captured.
232 5) I believe that most modern languages which support interactive
233 sessions have support for a =last_result= type function, which
234 returns the result of the last input without re-calculation. If
235 widely enough present this would be the ideal solution to a
236 combination of functional and imperative styles.
238 None of these solutions seem very desirable, but for now I don't see
239 what else would be possible.
241 Of these options I was leaning towards (1) and (4) but now believe
242 that if it is possible option (5) will be ideal.
244 **** (1) both functional and imperative evaluation
246 - can take advantage of built in functions for sending regions to the
248 - retains the proven tested and working functional wrappers
251 - introduces the complication of keeping track of which type of
252 evaluation is best suited to a particular context
253 - the current functional wrappers may require some changes in order to
254 include the existing global context
256 **** (4) exploit language meta-programming constructs to explicitly evaluate code
258 - only one type of evaluation
261 - some languages may not have sufficient meta-programming constructs
263 **** (5) exploit some =last_value= functionality if present
265 Need to ensure that most languages have such a function, those without
266 will simply have to implement their own similar solution...
268 | language | =last_value= function |
269 |------------+-----------------------------|
273 | shell | see [[* last command for shells][last command for shells]] |
274 | emacs-lisp | see [[* emacs-lisp will be a special case][special-case]] |
276 #+srcname: task-last-value
281 ***** last command for shells
282 Do this using the =tee= shell command, and continually pipe the output
285 Got this idea from the following [[http://linux.derkeiler.com/Mailing-Lists/Fedora/2004-01/0898.html][email-thread]].
287 suggested from mailing list
289 #+srcname: bash-save-last-output-to-file
293 bash -c "$line" | tee /tmp/last.out1
294 mv /tmp/last.out1 /tmp/last.out
298 another proposed solution from the above thread
300 #+srcname: bash-save-in-variable
303 # so - Save Output. Saves output of command in OUT shell variable.
313 "^M": " | tee /tmp/h_lastcmd.out ^[k"
316 export __=/tmp/h_lastcmd.out
318 If you try it, Alt-k will stand for the old Enter; use "command $__" to
319 access the last output.
325 Herculano de Lima Einloft Neto
328 ***** emacs-lisp will be a special case
329 While it is possible for emacs-lisp to be run in a console type
330 environment (see the =elim= function) it is *not* possible to run
331 emacs-lisp in a different *session*. Meaning any variable set top
332 level of the console environment will be set *everywhere* inside
333 emacs. For this reason I think that it doesn't make any sense to
334 worry about session support for emacs-lisp.
336 *** Further thoughts on 'scripting' vs. functional approaches
338 These are just thoughts, I don't know how sure I am about this.
339 And again, perhaps I'm not saying anything very radical, just that
340 it would be nice to have some options supporting things like
341 receiving text output in the org buffer.
343 I can see that you've already gone some way down the road towards
344 the 'last value' approach, so sorry if my comments come rather
345 late. I am concerned that we are not giving sufficient attention
346 to stdout / the text that is returned by the interpreters. In
347 contrast, many of our potential users will be accustomed to a
348 'scripting' approach, where they are outputting text at various
349 points in the code block, not just at the end. I am leaning
350 towards thinking that we should have 2 modes of evaluation:
351 'script' mode, and 'functional' mode.
353 In script mode, evaluation of a code block would result in *all*
354 text output from that code block appearing as output in the org
355 buffer, presumably as an #+begin_example...#+end_example. There
356 could be an :echo option controlling whether the input commands
357 also appear in the output. [This is like Sweave].
359 In functional mode, the *result* of the code block is available as
360 an elisp object, and may appear in the org buffer as an org
361 table/string, via the mechanisms you have developed already.
363 One thing I'm wondering about is whether, in script mode, there
364 simply should not be a return value. Perhaps this is not so
365 different from what exists: script mode would be new, and what
366 exists currently would be functional mode.
368 I think it's likely that, while code evaluation will be exciting
369 to people, a large majority of our users in a large majority of
370 their usage will not attempt to actually use the return value from
371 a source code block in any meaningful way. In that case, it seems
372 rather restrictive to only allow them to see output from the end
375 Instead I think the most accessible way to introduce org-babel to
376 people, at least while they are learning it, is as an immensely
377 powerful environment in which to embed their 'scripts', which now
378 also allows them to 'run' their 'scripts'. Especially as such
379 people are likely to be the least capable of the user-base, a
380 possible design-rule would be to make the scripting style of usage
381 easy (default?), perhaps requiring a special option to enable a
382 functional style. Those who will use the functional style won't
383 have a problem understanding what's going on, whereas the 'skript
384 kiddies' might not even know the syntax for defining a function in
385 their language of choice. And of course we can allow the user to
386 set a variable in their .emacs controlling the preference, so that
387 functional users are not inconveniennced by having to provide
388 header args the whole time.
390 Please don't get the impression that I am down-valuing the
391 functional style of org-babel. I am constantly horrified at the
392 messy 'scripts' that my colleagues produce in perl or R or
393 whatever! Nevertheless that seems to be how a lot of people work.
395 I think you were leaning towards the last-value approach because
396 it offered the possibility of unified code supporting both the
397 single evaluation environment and the functional style. If you
398 agree with any of the above then perhaps it will impact upon this
399 and mean that the code in the two branches has to differ a bit. In
400 that case, functional mode could perhaps after all evaluate each
401 code block in its own environment, thus (re)approaching 'true'
402 functional programming (side-effects are hard to achieve).
406 echo "There are `wc -l files` files in this directory"
410 *** even more thoughts on evaluation, results, models and options
412 Thanks Dan, These comments are invaluable.
414 What do you think about this as a new list of priorities/requirements
415 for the execution of source-code blocks.
418 1) we want the evaluation of the source code block to take place in a
419 session which can persist state (variables, current directory,
421 2) source code blocks can specify their session with a header argument
422 3) each session should correspond to an Emacs comint buffer so that the
423 user can drop into the session and experiment with live code
426 1) each source-code block generates some form of results which (as
427 we have already implemented) is transfered into emacs-lisp
428 after which it can be inserted into the org-mode buffer, or
429 used by other source-code blocks
430 2) when the results are translated into emacs-lisp, forced to be
431 interpreted as a scalar (dumping their raw values into the
432 org-mode buffer), as a vector (which is often desirable with R
433 code blocks), or interpreted on the fly (the default option).
434 Note that this is very nearly currently implemented through the
435 [[* DONE results-type header (vector/file)][results-type-header]].
436 3) there should be *two* means of collecting results from the
437 execution of a source code block. *Either* the value of the
438 last statement of the source code block, or the collection of
439 all that has been passed to STDOUT during the evaluation.
441 **** header argument or return line (*header argument*)
443 Rather than using a header argument to specify how the return value
444 should be passed back, I'm leaning towards the use of a =#+RETURN=
445 line inside the block. If such a line *is not present* then we
446 default to using STDOUT to collect results, but if such a line *is
447 present* then we use it's value as the results of the block. I
448 think this will allow for the most elegant specification between
449 functional and script execution. This also cleans up some issues
450 of implementation and finding which statement is the last
453 Having given this more thought, I think a header argument is
454 preferable. The =#+return:= line adds new complicating syntax for
455 something that does little more than we would accomplish through
456 the addition of a header argument. The only benefit being that we
457 know where the final statement starts, which is not an issue in
458 those languages which contain 'last value' operators.
460 new header =:results= arguments
461 - script :: explicitly states that we want to use STDOUT to
462 initialize our results
463 - return_last :: stdout is ignored instead the *value* of the final
464 statement in the block is returned
465 - echo :: means echo the contents of the source-code block along
466 with the results (this implies the *script* =:results=
469 *** DONE rework evaluation lang-by-lang [4/4]
471 This should include...
472 - functional results working with the comint buffer
474 - script :: return the output of STDOUT
475 - write a macro which runs the first redirection, executes the
476 body, then runs the second redirection
477 - last :: return the value of the last statement
480 - sessions in comint buffers
483 - [X] functional results working with comint
485 - [X] ensure scalar/vector results args are taken into consideration
486 - [X] ensure callable by other source block
488 #+srcname: ruby-use-last-output
489 #+begin_src ruby :results replace
496 #+resname: ruby-use-last-output
499 #+srcname: task-call-use-last-output
500 #+begin_src ruby :var last=ruby-use-last-output :results replace
501 last.flatten.size + 1
504 #+resname: task-call-use-last-output
509 #+srcname: first-ruby-session-task
510 #+begin_src ruby :session schulte :results silent
514 #+srcname: second-ruby-session-task
515 #+begin_src ruby :session schulte :results silent
519 #+srcname: without-the-right-session
520 #+begin_src ruby :results silent
526 - [X] functional results working with comint
528 - [X] ensure scalar/vector results args are taken into consideration
529 - [X] ensure callable by other source block
531 To redirect output to a file, you can use the =sink()= command.
534 #+begin_src R :results value vector silent
541 #+srcname: task-R-use-other-output
542 #+begin_src R :var twoentyseven=task_R_B() :results replace value
547 #+resname: task-R-use-other-output
550 **** DONE Python [4/4]
551 - [X] functional results working with comint
553 - [X] ensure scalar/vector results args are taken into consideration
554 - [X] ensure callable by other source block
556 #+srcname: task-new-eval-for-python
557 #+begin_src python :results silent output scalar
563 #+srcname: task-use-new-eval
564 #+begin_src python :var tasking=task-new-eval-for-python() :results replace
568 #+resname: task-use-new-eval
571 **** DONE Shells [4/4]
572 - [X] functional results working with comint
574 - [X] ensure scalar/vector results args are taken into consideration
575 - [X] ensure callable by other source block
577 #+srcname: task-shell-new-evaluation
578 #+begin_src sh :results silent value scalar
583 #+srcname: task-call-other-shell
584 #+begin_src sh :var other=task-shell-new-evaluation() :results replace scalar
585 echo $other ' is the old date'
588 #+resname: task-call-other-shell
589 : $ Fri Jun 12 13:08:37 PDT 2009 is the old date
591 *** DONE implement a *session* header argument [4/4]
592 =:session= header argument to override the default *session* buffer
596 #+srcname: task-ruby-named-session
597 #+begin_src ruby :session schulte :results replace
598 schulte = :in_schulte
601 #+resname: task-ruby-named-session
604 #+srcname: another-in-schulte
605 #+begin_src ruby :session schulte
609 #+resname: another-in-schulte
616 #+srcname: python-session-task
617 #+begin_src python :session what :results silent
621 #+srcname: python-get-from-session
622 #+begin_src python :session what :results replace
626 #+resname: python-get-from-session
631 #+srcname: task-shell-sessions
632 #+begin_src sh :session what
636 #+srcname: task-shell-sessions-what
637 #+begin_src sh :session what :results replace
641 #+resname: task-shell-sessions-what
646 #+srcname: task-R-session
647 #+begin_src R :session what :results replace
653 #+resname: task-R-session
656 #+srcname: another-task-R-session
657 #+begin_src R :session what :results replace
661 *** TODO function to bring up inferior-process buffer
663 This should be callable from inside of a source-code block in an
664 org-mode buffer. It should evaluate the header arguments, then bring
665 up the inf-proc buffer using =pop-to-buffer=.
667 *** TODO function to dump last N lines from inf-proc buffer into the current source block
669 Callable with a prefix argument to specify how many lines should be
670 dumped into the source-code buffer.
672 *** REJECTED comint notes
674 Implementing comint integration in [[file:lisp/org-babel-comint.el][org-babel-comint.el]].
677 - handling of outputs
678 - split raw output from process by prompts
679 - a ring of the outputs, buffer-local, `org-babel-comint-output-ring'
680 - a switch for dumping all outputs to a buffer
683 Lets drop all this language specific stuff, and just use
684 org-babel-comint to split up our outputs, and return either the last
685 value of an execution or the combination of values from the
688 **** comint filter functions
689 : ;; comint-input-filter-functions hook process-in-a-buffer
690 : ;; comint-output-filter-functions hook function modes.
691 : ;; comint-preoutput-filter-functions hook
692 : ;; comint-input-filter function ...
694 #+srcname: obc-filter-ruby
695 #+begin_src ruby :results last
703 ** TODO support for working with =*Org Edit Src Example*= buffers [1/4]
704 *** TODO set buffer-local-process variables appropriately [DED]
705 I think something like this would be great. You've probably
706 already thought of this, but just to note it down: it would be really
707 nice if org-babel's notion of a buffer's 'session/process' played
708 nicely with ESS's notion of the buffer's session/process. ESS keeps
709 the current process name for a buffer in a buffer-local variable
710 ess-local-process-name. So one thing we will probably want to do is
711 make sure that the *Org Edit Src Example* buffer sets that variable
714 I had not thought of that, but I agree whole heartedly. [Eric]
716 Once this is done every variable should be able to dump regions into
717 their inferior-process buffer using major-mode functions.
719 *** TODO some possible requests/proposed changes for Carsten [2/3]
720 While I remember, some possible requests/proposed changes for Carsten
721 come to mind in that regard:
723 **** DONE Remap C-x C-s to save the source to the org buffer?
724 I've done this personally and I find it essential. I'm using
725 #+begin_src emacs-lisp
726 (defun org-edit-src-save ()
727 "Update the parent org buffer with the edited source code, save
728 the parent org-buffer, and return to the source code edit
737 (define-key org-exit-edit-mode-map "\C-x\C-s" 'org-edit-src-save)
741 I think this is great, but I think it should be implemented in the
743 **** TODO Rename buffer and minor mode?
744 Something shorter than *Org Edit Src Example* for the buffer
745 name. org-babel is bringing org's source code interaction to a
746 level of maturity where the 'example' is no longer
747 appropriate. And if further keybindings are going to be added to
748 the minor mode then maybe org-edit-src-mode is a better name than
751 Maybe we should name the buffer with a combination of the source
752 code and the session. I think that makes sense.
754 [ES] Are you also suggesting a new org-edit-src minor mode?
755 [DED] org-exit-edit-mode is a minor mode that already exists:
757 Minor mode installing a single key binding, "C-c '" to exit special edit.
759 org-edit-src-save now has a binding in that mode, so I guess all
760 I'm saying at this stage is that it's a bit of a misnomer. But
761 perhaps we will also have more functionality to add to that minor
762 mode, making it even more of a misnomer. Perhaps something like
763 org-src-mode would be better.
765 **** DEFERRED a hook called when the src edit buffer is created
766 This should be implemented in the org-mode core
769 *** DEFERRED send code to inferior process
770 Another thought on this topic: I think we will want users to send
771 chunks of code to the interpreter from within the *Org Edit Src*
772 buffer, and I think that's what you have in mind already. In ESS that
773 is done using the ess-eval-* functions. [DED]
775 I think we can leave this up to the major-mode in the source code
776 buffer, as almost every source-code major mode will have functions for
777 doing things like sending regions to the inferior process. If
778 anything we might need to set the value of the buffer local inferior
779 process variable. [Eric]
781 *** TODO optionally evaluate header references when we switch to =*Org Edit Src*= buffer
782 That seems to imply that the header references need to be evaluated
783 and transformed into the target language object when we hit C-c ' to
784 enter the *Org Edit Src* buffer [DED]
786 Good point, I heartily agree that this should be supported [Eric]
788 (or at least before the first time we attempt to evaluate code in that
789 buffer -- I suppose there might be an argument for lazy evaluation, in
790 case someone hits C-c ' but is "just looking" and not actually
791 evaluating anything.) Of course if evaluating the reference is
792 computationally intensive then the user might have to wait before they
793 get the *Org Edit Src* buffer. [DED]
795 I fear that it may be hard to anticipate when the references will be
796 needed, some major-modes do on-the-fly evaluation while the buffer is
797 being edited. I think that we should either do this before the buffer
798 is opened or not at all, specifically I think we should resolve
799 references if the user calls C-c ' with a prefix argument. Does that
800 sound reasonable? [Eric]
804 ** TODO fully purge org-babel-R of direct comint interaction
805 try to remove all code under the [[file:lisp/org-babel-R.el::functions%20for%20evaluation%20of%20R%20code][;; functions for evaluation of R code]] line
807 ** TODO improve the source-block snippet
809 [[file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name%20Chapter%20title][file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name Chapter title]]
811 ,#name : Chapter title
814 ${1:$(make-string (string-width text) ?\=)}
819 [[file:snippets/org-mode/sb][sb -- snippet]]
821 waiting for guidance from those more familiar with yasnippets
823 ** TODO resolve references to other org buffers/files
824 This would allow source blocks to call upon tables, source-blocks,
825 and results in other org buffers/files.
828 - [[file:lisp/org-babel-ref.el::TODO%20allow%20searching%20for%20names%20in%20other%20buffers][org-babel-ref.el:searching-in-other-buffers]]
829 - [[file:lisp/org-babel.el::defun%20org-babel%20find%20named%20result%20name][org-babel.el#org-babel-find-named-result]]
830 ** TODO resolve references to other non-org files
831 - tabular data in .csv, .tsv etc format
832 - files of interpreted code: anything stopping us giving such files
833 similar status to a source code block?
834 - Would be nice to allow org and non-org files to be remote
835 ** TODO figure out how to handle errors during evaluation
836 R has a try function, with error handling, along the lines of
837 python. I bet ruby does too. Maybe more of an issue for functional
838 style; in my proposed scripting style the error just gets dumped to
839 the org buffer and the user is thus alerted.
840 ** TODO figure out how to handle graphic output
841 This is listed under [[* graphical output][graphical output]] in out objectives.
843 This should take advantage of the =:results file= option, and
844 languages which almost always produce graphical output should set
845 =:results file= to true by default. That would handle placing these
846 results in the buffer. Then if there is a combination of =silent= and
847 =file= =:results= headers we could drop the results to a temp buffer
848 and pop open that buffer...
850 ** TODO Finalise behaviour regarding vector/scalar output
851 *** DONE Stop spaces causing vector output
852 This simple example of multilingual chaining produces vector output if
853 there are spaces in the message and scalar otherwise.
857 #+begin_src R :var msg=msg-from-python
858 paste(msg, "und R", sep=" ")
862 : org-babel speaks elisp y python und R
864 #+srcname: msg-from-python
865 #+begin_src python :var msg=msg-from-elisp
869 #+srcname: msg-from-elisp
870 #+begin_src emacs-lisp :var msg="org-babel speaks"
871 (concat msg " elisp")
874 ** TODO re-implement helper functions from org-R
875 *** Initial statement [Eric]
876 Much of the power of org-R seems to be in it's helper functions for
877 the quick graphing of tables. Should we try to re-implement these
878 functions on top of org-babel?
880 I'm thinking this may be useful both to add features to org-babel-R and
881 also to potentially suggest extensions of the framework. For example
882 one that comes to mind is the ability to treat a source-code block
883 like a function which accepts arguments and returns results. Actually
884 this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]).
886 - We want to provide convenient off-the-shelf actions
887 (e.g. plotting data) that make use of our new code evaluation
888 environment but do not require any actual coding.
890 - *Input data* will be specified using the same mechanism as :var
891 references, thus the input data may come from a table, or
892 another source block, and it is initially available as an elisp
894 - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that
895 line will apply an *action* to the referenced data.
896 - *Actions correspond to source blocks*: our library of available
897 actions will be a library of org-babel source blocks. Thus the
898 code for executing an action, and the code for dealing with the
899 output of the action will be the same code as for executing
900 source blocks in general
901 - Optionally, the user can have the relevant source block inserted
902 into the org buffer after the (say) #+BABELDO line. This will
903 allow the user to fine tune the action by modifying the code
904 (especially useful for plots).
905 - So maybe a #+BABELDO line will have header args
906 - :data (a reference to a table or source code block)
907 - :action (or should that be :srcname?) which will be something
908 like :action pie-chart, referring to a source block which will
909 be executed with the :data referent passed in using a :var arg.
910 - :showcode or something controlling whether to show the code
912 ** TODO share org-babel
913 how should we share org-babel?
915 - post to org-mode and ess mailing lists
916 - create a org-babel page on worg
917 - create a short screencast demonstrating org-babel in action
920 we need to think up some good examples
922 **** interactive tutorials
923 This could be a place to use [[* org-babel assertions][org-babel assertions]].
925 for example the first step of a tutorial could assert that the version
926 of the software-package (or whatever) is equal to some value, then
927 source-code blocks could be used with confidence (and executed
928 directly from) the rest of the tutorial.
930 **** answering a text-book question w/code example
931 org-babel is an ideal environment enabling both the development and
932 demonstrationg of the code snippets required as answers to many
935 **** something using tables
936 maybe something along the lines of calculations from collected grades
939 Maybe something like the following which outputs sizes of directories
940 under the home directory, and then instead of the trivial =emacs-lisp=
941 block we could use an R block to create a nice pie chart of the
945 #+begin_src bash :results replace
949 #+begin_src emacs-lisp :var sizes=sizes :results replace
953 ** TODO command line execution
954 Allow source code blocks to be called form the command line. This
955 will be easy using the =sbe= function in [[file:lisp/org-babel-table.el][org-babel-table.el]].
957 This will rely upon [[* resolve references to other buffers][resolve references to other buffers]].
959 ** TODO inline source code blocks [3/5]
960 Like the =\R{ code }= blocks
962 not sure what the format should be, maybe just something simple
963 like =src_lang[]{}= where lang is the name of the source code
964 language to be evaluated, =[]= is optional and contains any header
965 arguments and ={}= contains the code.
967 (see [[* (sandbox) inline source blocks][the-sandbox]])
969 *** DONE evaluation with \C-c\C-c
970 Putting aside the header argument issue for now we can just run these
971 with the following default header arguments
972 - =:results= :: silent
973 - =:exports= :: results
975 *** DONE inline exportation
976 Need to add an interblock hook (or some such) through org-exp-blocks
977 *** DONE header arguments
978 We should make it possible to use header arguments.
980 *** TODO fontification
981 we should color these blocks differently
983 *** TODO refine html exportation
984 should use a span class, and should show original source in tool-tip
986 ** TODO formulate general rules for handling vectors and tables / matrices with names
987 This is non-trivial, but may be worth doing, in particular to
988 develop a nice framework for sending data to/from R.
990 In R, indexing vector elements, and rows and columns, using
991 strings rather than integers is an important part of the
993 - elements of a vector may have names
994 - matrices and data.frames may have "column names" and "row names"
995 which can be used for indexing
996 - In a data frame, row names *must* be unique
1000 > vec <- c(a=1, b=2)
1004 > mat <- matrix(1:4, nrow=2, ncol=2, dimnames=list(c("r1","r2"), c("c1","c2")))
1009 > # The names are separate from the data: they do not interfere with operations on the data
1016 > df <- data.frame(var1=1:26, var2=26:1, row.names=letters)
1018 [1] 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
1024 So it's tempting to try to provide support for this in org-babel. For example
1025 - allow R to refer to columns of a :var reference by their names
1026 - When appropriate, results from R appear in the org buffer with "named
1029 However none (?) of the other languages we are currently supporting
1030 really have a native matrix type, let alone "column names" or "row
1031 names". Names are used in e.g. python and perl to refer to entries
1034 It currently seems to me that support for this in org-babel would
1035 require setting rules about when org tables are considered to have
1036 named columns/fields, and ensuring that (a) languages with a notion
1037 of named columns/fields use them appropriately and (b) languages
1038 with no such notion do not treat then as data.
1040 - Org allows something that *looks* like column names to be separated
1042 - Org also allows a row to *function* as column names when special
1043 markers are placed in the first column. An hline is unnecessary
1044 (indeed hlines are purely cosmetic in org [correct?]
1045 - Org does not have a notion of "row names" [correct?]
1047 The full org table functionality exeplified [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]] has features that
1048 we would not support in e.g. R (like names for the row below).
1050 *** Initial statement: allow tables with hline to be passed as args into R
1051 This doesn't seem to work at the moment (example below). It would
1052 also be nice to have a natural way for the column names of the org
1053 table to become the column names of the R data frame, and to have
1054 the option to specify that the first column is to be used as row
1055 names in R (these must be unique). But this might require a bit of
1060 | col1 | col2 | col3 |
1061 |------+---------+------|
1069 #+begin_src R var tabel=egtable
1074 | "col1" | "col2" | "col3" |
1075 |--------+-----------+--------|
1077 | 4 | "schulte" | 6 |
1080 Another example is in the [[*operations%20in%20on%20tables][grades example]].
1082 ** PROPOSED Are we happy with current behaviour regarding vector/scalar output?
1083 This simple example of multilingual chaining produces vector output if
1084 there are spaces in the message and scalar otherwise.
1086 #+begin_src R :var msg=msg-from-python
1087 paste(msg, "und_R", sep="_")
1090 #+srcname: msg-from-python
1091 #+begin_src python :var msg=msg-from-elisp
1095 #+srcname: msg-from-elisp
1096 #+begin_src emacs-lisp :var msg="org-babel_speaks"
1097 (concat msg "_elisp")
1100 ** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format
1101 I haven't thought about this properly. Just noting it down. What
1102 Sweave uses is called "R noweb" (.Rnw).
1104 I found a good description of noweb in the following article (see
1105 the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]).
1107 I think there are two parts to noweb, the construction of
1108 documentation and the extraction of source-code (with notangle).
1110 *documentation*: org-mode handles all of our documentation needs in
1111 a manner that I believe is superior to noweb.
1113 *source extraction* At this point I don't see anyone writing large
1114 applications with 100% of the source code contained in org-babel
1115 files, rather I see org-babel files containing things like
1116 - notes with active code chunks
1117 - interactive tutorials
1118 - requirements documents with code running test suites
1119 - and of course experimental reports with the code to run the
1120 experiment, and perform analysis
1122 Basically I think the scope of the programs written in org-babel
1123 (at least initially) will be small enough that it wont require the
1124 addition of a tangle type program to extract all of the source code
1125 into a running application.
1127 On the other hand, since we already have named blocks of source
1128 code which reference other blocks on which they rely, this
1129 shouldn't be too hard to implement either on our own, or possibly
1130 relying on something like noweb/notangle.
1132 ** PROPOSED support for passing paths to files between source blocks
1133 Maybe this should be it's own result type (in addition to scalars and
1134 vectors). The reason being that some source-code blocks (for example
1135 ditaa or anything that results in the creation of a file) may want to
1136 pass a file path back to org-mode which could then be inserted into
1137 the org-mode buffer as a link to the file...
1139 This would allow for display of images upon export providing
1140 functionality similar to =org-exp-blocks= only in a more general
1143 ** DEFERRED use textConnection to pass tsv to R?
1144 When passing args from the org buffer to R, the following route is
1145 used: arg in buffer -> elisp -> tsv on file -> data frame in R. I
1146 think it would be possible to avoid having to write to file by
1147 constructing an R expression in org-babel-R-assign-elisp, something
1150 #+begin_src emacs-lisp
1151 (org-babel-R-input-command
1152 (format "%s <- read.table(textConnection(\"%s\"), sep=\"\\t\", as.is=TRUE)"
1153 name (orgtbl-to-tsv value '(:sep "\t" :fmt org-babel-R-quote-tsv-field))))
1156 I haven't tried to implement this yet as it's basically just
1157 fiddling with something that works. The only reason for it I can
1158 think of would be efficiency and I haven't tested that.
1160 This Didn't work after an initial test. I still think this is a
1161 good idea (I also think we should try to do something similar when
1162 writing out results frmo R to elisp) however as it wouldn't result
1163 in any functional changes I'm bumping it down to deferred for
1168 #+tblname: quick-test
1171 #+srcname: quick-test-src-blk
1172 #+begin_src R :var vec=quick-test
1178 ** DEFERRED re-implement R evaluation using ess-command or ess-execute
1179 I don't have any complaints with the current R evaluation code or
1180 behaviour, but I think it would be good to use the ESS functions
1181 from a political point of view. Plus of course it has the normal
1182 benefits of an API (insulates us from any underlying changes etc). [DED]
1184 I'll look into this. I believe that I looked at and rejected these
1185 functions initially but now I can't remember why. I agree with
1186 your overall point about using API's where available. I will take
1187 a look back at these and either switch to using the ess commands,
1188 or at least articulate under this TODO the reasons for using our
1189 custom R-interaction commands. [Eric]
1193 Lets just replace =org-babel-R-input-command= with =ess-execute=.
1195 I tried this, and although it works in some situations, I find that
1196 =ess-command= will often just hang indefinitely without returning
1197 results. Also =ess-execute= will occasionally hang, and pops up
1198 the buffer containing the results of the command's execution, which
1199 is undesirable. For now these functions can not be used. Maybe
1200 someone more familiar with the ESS code can recommend proper usage
1201 of =ess-command= or some other lower-level function which could be
1202 used in place of [[file:lisp/org-babel-R.el::defun%20org-babel%20R%20input%20command%20command][org-babel-R-input-command]].
1206 #+begin_quote ess-command
1207 (ess-command COM &optional BUF SLEEP NO-PROMPT-CHECK)
1209 Send the ESS process command COM and delete the output
1210 from the ESS process buffer. If an optional second argument BUF exists
1211 save the output in that buffer. BUF is erased before use.
1212 COM should have a terminating newline.
1213 Guarantees that the value of .Last.value will be preserved.
1214 When optional third arg SLEEP is non-nil, `(sleep-for (* a SLEEP))'
1215 will be used in a few places where `a' is proportional to `ess-cmd-delay'.
1218 #+begin_quote ess-execute
1219 (ess-execute COMMAND &optional INVERT BUFF MESSAGE)
1221 Send a command to the ESS process.
1222 A newline is automatically added to COMMAND. Prefix arg (or second arg
1223 INVERT) means invert the meaning of
1224 `ess-execute-in-process-buffer'. If INVERT is 'buffer, output is
1225 forced to go to the process buffer. If the output is going to a
1226 buffer, name it *BUFF*. This buffer is erased before use. Optional
1227 fourth arg MESSAGE is text to print at the top of the buffer (defaults
1228 to the command if BUFF is not given.)
1231 *** out current setup
1233 1) The body of the R source code block is wrapped in a function
1234 2) The function is called inside of a =write.table= function call
1235 writing the results to a table
1236 3) The table is read using =org-table-import=
1238 ** DEFERRED Rework Interaction with Running Processes [0/3]
1239 *** TODO ability to select which of multiple sessions is being used
1240 Increasingly it is looking like we're going to want to run all
1241 source code blocks in comint buffer (sessions). Which will have
1243 1) allowing background execution
1244 2) maintaining state between source-blocks
1245 - allowing inline blocks w/o header arguments
1248 (like ess-switch-process in .R buffers)
1250 Maybe this could be packaged into a header argument, something
1251 like =:R_session= which could accept either the name of the
1252 session to use, or the string =prompt=, in which case we could use
1253 the =ess-switch-process= command to select a new process.
1255 *** TODO evaluation of shell code as background process?
1256 After C-c C-c on an R code block, the process may appear to
1257 block, but C-g can be used to reclaim control of the .org buffer,
1258 without interrupting the R evalution. However I believe this is not
1259 true of bash/sh evaluation. [Haven't tried other languages] Perhaps
1260 a solution is just to background the individual shell commands.
1262 The other languages (aside from emacs lisp) are run through the
1263 shell, so if we find a shell solution it should work for them as
1266 Adding an ampersand seems to be a supported way to run commands in
1267 the background (see [[http://www.emacswiki.org/emacs/ExecuteExternalCommand#toc4][external-commands]]). Although a more extensible
1268 solution may involve the use of the [[elisp:(progn (describe-function 'call-process-region) nil)][call-process-region]] function.
1270 Going to try this out in a new file [[file:lisp/org-babel-proc.el][org-babel-proc.el]]. This should
1271 contain functions for asynchronously running generic shell commands
1272 in the background, and then returning their input.
1274 **** partial update of org-mode buffer
1275 The sleekest solution to this may be using a comint buffer, and
1276 then defining a filter function which would incrementally interpret
1277 the results as they are returned, including insertion into the
1278 org-mode buffer. This may actually cause more problems than it is
1279 worth, what with the complexities of identifying the types of
1280 incrementally returned results, and the need for maintenance of a
1281 process marker in the org buffer.
1283 **** 'working' spinner
1284 It may be nice and not too difficult to place a spinner on/near the
1285 evaluating source code block
1287 *** TODO conversion of output from interactive shell, R (and python) sessions to org-babel buffers
1288 [DED] This would be a nice feature I think. Although an org-babel
1289 purist would say that it's working the wrong way round... After
1290 some interactive work in a *R* buffer, you save the buffer, maybe
1291 edit out some lines, and then convert it to org-babel format for
1292 posterity. Same for a shell session either in a *shell* buffer, or
1293 pasted from another terminal emulator. And python of course.
1295 ** DONE Remove protective commas from # comments before evaluating
1296 org inserts protective commas in front of ## comments in language
1297 modes that use them. We need to remove them prior to sending code
1300 #+srcname: testing-removal-of-protective-comas
1302 ,# this one might break it??
1306 ** DONE pass multiple reference arguments into R
1307 Can we do this? I wasn't sure how to supply multiple 'var' header
1308 args. Just delete this if I'm being dense.
1310 This should be working, see the following example...
1312 #+srcname: two-arg-example
1313 #+begin_src R :var n=2 :var m=8
1317 #+resname: two-arg-example
1320 ** DONE ensure that table ranges work
1321 when a table range is passed to org-babel as an argument, it should be
1322 interpreted as a vector.
1325 | 2 | 3 | Fixnum:1 |
1326 | 3 | 4 | Array:123456 |
1330 #+TBLFM: @1$3='(sbe simple-sbe-example (n 4))::@2$3='(sbe task-table-range (n @1$1..@6$1))::@3$3='(sbe task-table-range (n (@1$1..@6$1)))
1332 #+srcname: simple-sbe-example
1333 #+begin_src emacs-lisp
1337 #+srcname: task-table-range
1338 #+begin_src ruby :var n=simple-sbe-example
1342 #+srcname: simple-results
1343 #+begin_src emacs-lisp :var n=task-table-range(n=(1 2 3))
1347 #+resname: simple-results
1350 #+srcname: task-arr-referent
1351 #+begin_src ruby :var ar=(1 2 3)
1355 #+resname: task-arr-referent
1358 ** DONE global variable indicating default to vector output
1359 how about an alist... =org-babel-default-header-args= this may already
1360 exist... just execute the following and all source blocks will default
1363 #+begin_src emacs-lisp
1364 (setq org-babel-default-header-args '((:results . "vector")))
1367 ** DONE name named results if source block is named
1368 currently this isn't happening although it should be
1370 #+srcname: test-naming-named-source-blocks
1371 #+begin_src emacs-lisp
1375 #+resname: test-naming-named-source-blocks
1377 ** DONE (simple caching) check for named results before source blocks
1378 see the TODO comment in [[file:lisp/org-babel-ref.el::TODO%20This%20should%20explicitly%20look%20for%20resname%20lines%20before][org-babel-ref.el#org-babel-ref-resolve-reference]]
1379 ** DONE set =:results silent= when eval with prefix argument
1381 #+begin_src emacs-lisp
1384 ** DONE results-type header (vector/file) [3/3]
1385 In response to a point in Dan's email. We should allow the user to
1386 force scalar or vector results. This could be done with a header
1387 argument, and the default behavior could be controlled through a
1388 configuration variable.
1390 #+srcname: task-trivial-vector
1391 #+begin_src ruby :results replace vector
1398 since it doesn't make sense to turn a vector into a scalar, lets
1399 just add a two values...
1401 - vector :: forces the results to be a vector (potentially 1 dimensional)
1402 - file :: this throws an error if the result isn't a string, and
1403 tries to treat it as a path to a file.
1405 I'm just going to cram all of these into the =:results= header
1406 argument. Then if we allow multiple header arguments it should
1407 work out, for example one possible header argument string could be
1408 =:results replace vector file=, which would *replace* any existing
1409 results forcing the results into an org-mode table, and
1410 interpreting any strings as file paths.
1412 *** DONE multiple =:results= headers
1414 #+srcname: multiple-result-headers
1415 #+begin_src ruby :results replace silent
1421 *** DONE file result types
1422 When inserting into an org-mode buffer create a link with the path
1423 being the value, and optionally the display being the
1424 =file-name-nondirectory= if it exists.
1426 #+srcname: task-file-result
1427 #+begin_src python :results replace file
1432 [[something][something]]
1435 This will be useful because blocks like =ditaa= and =dot= can return
1436 the string path of their files, and can add =file= to their results
1439 *** DONE vector result types
1441 #+srcname: task-force-results
1442 #+begin_src emacs-lisp :results vector
1449 ** DONE results name
1450 In order to do this we will need to start naming our results.
1451 Since the source blocks are named with =#+srcname:= lines we can
1452 name results with =#+resname:= lines (if the source block has no
1453 name then no name is given to the =#+resname:= line on creation,
1454 otherwise the name of the source block is used).
1456 This will have the additional benefit of allowing results and
1457 source blocks to be located in different places in a buffer (and
1458 eventually in different buffers entirely).
1460 #+srcname: developing-resnames
1461 #+begin_src emacs-lisp :results silent
1465 Once source blocks are able to find their own =#+resname:= lines
1468 #+srcname: sbe-w-new-results
1469 #+begin_src emacs-lisp :results replace
1470 (sbe "developing-resnames")
1476 *** TODO change the results insertion functions to use these lines
1478 *** TODO teach references to resolve =#+resname= lines.
1480 ** DONE org-babel tests org-babel [1/1]
1481 since we are accumulating this nice collection of source-code blocks
1482 in the sandbox section we should make use of them as unit tests.
1483 What's more, we should be able to actually use org-babel to run these
1486 We would just need to cycle over every source code block under the
1487 sandbox, run it, and assert that the return value is equal to what we
1490 I have the feeling that this should be possible using only org-babel
1491 functions with minimal or no additional elisp. It would be very cool
1492 for org-babel to be able to test itself.
1494 This is now done, see [[* Tests]].
1496 *** DEFERRED org-babel assertions (may not be necessary)
1497 These could be used to make assertions about the results of a
1498 source-code block. If the assertion fails then the point could be
1499 moved to the block, and error messages and highlighting etc... could
1502 ** DONE make C-c C-c work anywhere within source code block?
1503 This seems like it would be nice to me, but perhaps it would be
1504 inefficient or ugly in implementation? I suppose you could search
1505 forward, and if you find #+end_src before you find #+begin_src,
1506 then you're inside one. [DED]
1508 Agreed, I think inside of the =#+srcname: line= would be useful as
1511 #+srcname: testing-out-cc
1512 #+begin_src emacs-lisp
1516 ** DONE integration with org tables
1517 We should make it easy to call org-babel source blocks from org-mode
1518 table formulas. This is practical now that it is possible to pass
1519 arguments to org-babel source blocks.
1521 See the related [[* (sandbox) integration w/org tables][sandbox]] header for tests/examples.
1523 *** digging in org-table.el
1524 In the past [[file:~/src/org/lisp/org-table.el::org%20table%20el%20The%20table%20editor%20for%20Org%20mode][org-table.el]] has proven difficult to work with.
1526 Should be a hook in [[file:~/src/org/lisp/org-table.el::defun%20org%20table%20eval%20formula%20optional%20arg%20equation][org-table-eval-formula]].
1528 Looks like I need to change this [[file:~/src/org/lisp/org-table.el::if%20lispp][if statement]] (line 2239) into a cond
1531 ** DONE source blocks as functions
1533 Allow source code blocks to be called like functions, with arguments
1534 specified. We are already able to call a source-code block and assign
1535 it's return result to a variable. This would just add the ability to
1536 specify the values of the arguments to the source code block assuming
1537 any exist. For an example see
1539 When a variable appears in a header argument, how do we differentiate
1540 between it's value being a reference or a literal value? I guess this
1541 could work just like a programming language. If it's escaped or in
1542 quotes, then we count it as a literal, otherwise we try to look it up
1545 ** DONE folding of code blocks? [2/2]
1546 [DED] In similar way to using outline-minor-mode for folding function
1547 bodies, can we fold code blocks? #+begin whatever statements are
1548 pretty ugly, and in any case when you're thinking about the overall
1549 game plan you don't necessarily want to see the code for each Step.
1551 *** DONE folding of source code block
1552 Sounds good, and wasn't too hard to implement. Code blocks should
1553 now be fold-able in the same manner as headlines (by pressing TAB
1556 *** REJECTED folding of results
1557 So, lets do a three-stage tab cycle... First fold the src block,
1558 then fold the results, then unfold.
1560 There's no way to tell if the results are a table or not w/o
1561 actually executing the block which would be too expensive of an
1564 ** DONE selective export of text, code, figures
1565 [DED] The org-babel buffer contains everything (code, headings and
1566 notes/prose describing what you're up to, textual/numeric/graphical
1567 code output, etc). However on export to html / LaTeX one might want
1568 to include only a subset of that content. For example you might
1569 want to create a presentation of what you've done which omits the
1572 [EMS] So I think this should be implemented as a property which can
1573 be set globally or on the outline header level (I need to review
1574 the mechanics of org-mode properties). And then as a source block
1575 header argument which will apply only to a specific source code
1576 block. A header argument of =:export= with values of
1578 - =code= :: just show the code in the source code block
1579 - =none= :: don't show the code or the results of the evaluation
1580 - =results= :: just show the results of the code evaluation (don't
1581 show the actual code)
1582 - =both= :: show both the source code, and the results
1584 this will be done in [[* (sandbox) selective export][(sandbox) selective export]].
1586 ** DONE a header argument specifying silent evaluation (no output)
1587 This would be useful across all types of source block. Currently
1588 there is a =:replace t= option to control output, this could be
1589 generalized to an =:output= option which could take the following
1590 options (maybe more)
1592 - =t= :: this would be the default, and would simply insert the
1593 results after the source block
1594 - =replace= :: to replace any results which may already be there
1595 - =silent= :: this would inhibit any insertion of the results
1597 This is now implemented see the example in the [[* silent evaluation][sandbox]]
1599 ** DONE assign variables from tables in R
1600 This is now working (see [[* (sandbox table) R][(sandbox-table)-R]]). Although it's not that
1601 impressive until we are able to print table results from R.
1603 ** DONE insert 2-D R results as tables
1604 everything is working but R and shell
1610 This has already been tackled by Dan in [[file:existing_tools/org-R.el::defconst%20org%20R%20write%20org%20table%20def][org-R:check-dimensions]]. The
1611 functions there should be useful in combination with [[http://cran.r-project.org/doc/manuals/R-data.html#Export-to-text-files][R-export-to-csv]]
1612 as a means of converting multidimensional R objects to emacs lisp.
1614 It may be as simple as first checking if the data is multidimensional,
1615 and then, if so using =write= to write the data out to a temporary
1616 file from which emacs can read the data in using =org-table-import=.
1618 Looking into this further, is seems that there is no such thing as a
1619 scalar in R [[http://tolstoy.newcastle.edu.au/R/help/03a/3733.html][R-scalar-vs-vector]] In that light I am not sure how to
1620 deal with trivial vectors (scalars) in R. I'm tempted to just treat
1621 them as vectors, but then that would lead to a proliferation of
1622 trivial 1-cell tables...
1624 ** DONE allow variable initialization from source blocks
1625 Currently it is possible to initialize a variable from an org-mode
1626 table with a block argument like =table=sandbox= (note that the
1627 variable doesn't have to named =table=) as in the following example
1633 #+begin_src emacs-lisp :var table=sandbox :results replace
1634 (message (format "table = %S" table))
1637 : "table = ((1 2 3) (4 \"schulte\" 6))"
1639 It would be good to allow initialization of variables from the results
1640 of other source blocks in the same manner. This would probably
1641 require the addition of =#+SRCNAME: example= lines for the naming of
1642 source blocks, also the =table=sandbox= syntax may have to be expanded
1643 to specify whether the target is a source code block or a table
1644 (alternately we could just match the first one with the given name
1645 whether it's a table or a source code block).
1647 At least initially I'll try to implement this so that there is no need
1648 to specify whether the reference is to a table or a source-code block.
1649 That seems to be simpler both in terms of use and implementation.
1651 This is now working for emacs-lisp, ruby and python (and mixtures of
1652 the three) source blocks. See the examples in the [[* (sandbox) referencing other source blocks][sandbox]].
1654 This is currently working only with emacs lisp as in the following
1655 example in the [[* emacs lisp source reference][emacs lisp source reference]].
1658 ** TODO Add languages [0/5]
1659 I'm sure there are many more that aren't listed here. Please add
1660 them, and bubble any that you particularly care about up to the top.
1662 Any new language should be implemented in a org-babel-lang.el file.
1663 Follow the pattern set by [[file:lisp/org-babel-script.el][org-babel-script.el]], [[file:lisp/org-babel-shell.el][org-babel-shell.el]] and
1664 [[file:lisp/org-babel-R.el][org-babel-R.el]].
1667 This could probably be added to [[file:lisp/org-babel-script.el][org-babel-script.el]]
1672 (see [[* file result types][file result types]])
1675 (see [[* file result types][file result types]])
1678 (see [[* file result types][file result types]])
1683 ** TODO when reading results from =#+resname= line
1685 Errors when trying to read from resname lines.
1687 #+resname: bug-in-resname
1690 #+srcname: bug-in-resname-reader
1691 #+begin_src emacs-lisp :var buggy=bug-in-resname() :results silent
1695 ** TODO non-orgtbl formatted lists
1698 #+srcname: this-doesn't-match-orgtbl
1699 #+begin_src emacs-lisp :results replace
1700 '((:results . "replace"))
1703 #+resname: this-doesn't-match-orgtbl
1705 ** TODO collapsing consecutive newlines in string output
1707 #+srcname: multi-line-string-output
1708 #+begin_src ruby :results replace
1709 "the first line ends here
1712 and this is the second one
1718 : the first line ends here
1719 : and this is the second one
1720 : return even a third
1722 ** TODO cursor movement when evaluating source blocks
1723 E.g. the pie chart example. Despite the save-window-excursion in
1724 org-babel-execute:R. (I never learned how to do this properly: org-R
1725 jumps all over the place...)
1727 ** DONE R-code broke on "org-babel" rename
1729 #+srcname: bug-R-babels
1734 ** DONE error on trivial R results
1736 So I know it's generally not a good idea to squash error without
1737 handling them, but in this case the error almost always means that
1738 there was no file contents to be read by =org-table-import=, so I
1741 #+srcname: bug-trivial-r1
1742 #+begin_src R :results replace
1743 pie(c(1, 2, 3), labels = c(1, 2, 3))
1746 #+srcname: bug-trivial-r2
1747 #+begin_src R :results replace
1751 #+resname: bug-trivial-r2
1754 #+srcname: bug-trivial-r3
1755 #+begin_src R :results replace
1759 #+resname: bug-trivial-r3
1764 ** DONE ruby new variable creation (multi-line ruby blocks)
1765 Actually it looks like we were dropping all but the last line.
1767 #+srcname: multi-line-ruby-test
1768 #+begin_src ruby :var table=bug-numerical-table :results replace
1770 table.each{|n| total += n}
1777 ** DONE R code execution seems to choke on certain inputs
1778 Currently the R code seems to work on vertical (but not landscape)
1781 #+srcname: little-fake
1782 #+begin_src emacs-lisp
1786 #+begin_src R :var num=little-fake
1800 #+srcname: set-debug-on-error
1801 #+begin_src emacs-lisp :results silent
1802 (setq debug-on-error t)
1805 #+srcname: bug-numerical-table
1806 #+begin_src emacs-lisp :results silent
1810 #+srcname: bug-R-number-evaluation
1811 #+begin_src R :var table=bug-numerical-table :results replace
1818 #+tblname: bug-vert-table
1823 #+srcname: bug-R-vertical-table
1824 #+begin_src R :var table=bug-vert-table :results silent
1828 ** DEFERRED org bug/request: prevent certain org behaviour within code blocks
1829 E.g. [[]] gets recognised as a link (when there's text inside the
1830 brackets). This is bad for R code at least, and more generally
1831 could be argued to be inappropriate. Is it difficult to get org to
1832 ignore text in code blocks? [DED]
1834 I believe Carsten addressed this recently on the mailing list with
1835 the comment that it was indeed a difficult issue. I believe this
1836 may be one area where we could wait for an upstream (org-mode) fix.
1837 ** DONE with :results replace, non-table output doesn't replace table output
1838 And vice versa. E.g. Try this first with table and then with len(table) [DED]
1839 #+begin_src python :var table=sandbox :results replace
1844 | 4 | "schulte" | 6 |
1847 Yes, this is certainly a problem. I fear that if we begin replacing
1848 anything immediately following a source block (regardless of whether
1849 it matches the type of our current results) we may accidentally delete
1850 hand written portions of the user's org-mode buffer.
1852 I think that the best solution here would be to actually start
1853 labeling results with a line that looks something like...
1857 This would have a couple of benefits...
1858 1) we wouldn't have to worry about possibly deleting non-results
1859 (which is currently an issue)
1860 2) we could reliably replace results even if there are different types
1861 3) we could reference the results of a source-code block in variable
1862 definitions, which would be useful if for example we don't wish to
1863 re-run a source-block every time because it is long-running.
1865 Thoughts? If no-one objects, I believe I will implement the labeling
1868 ** DONE extra quotes for nested string
1869 Well R appears to be reading the tables without issue...
1871 these *should* be quoted
1873 #+begin_src sh :results replace
1878 | "README.markdown" |
1881 | "existing_tools" |
1885 | "test-export.html" |
1886 | "test-export.org" |
1888 #+srcname: test-quotes
1889 #+begin_src ruby :var tab=ls
1895 #+srcname: test-quotes
1896 #+begin_src R :var tab=ls
1902 ** DONE simple ruby arrays not working
1904 As an example eval the following. Adding a line to test
1906 #+srcname: simple-ruby-array
1911 #+srcname: ruby-array-test
1912 #+begin_src ruby :var ar = simple-ruby-array :results silent
1916 ** DONE space trailing language name
1917 fix regexp so it works when there's a space trailing the language name
1919 #+srcname: test-trailing-space
1924 ** DONE Args out of range error
1926 The following block resulted in the error below [DED]. It ran without
1927 error directly in the shell.
1930 for platf in ill aff ; do
1931 for pop in CEU YRI ASI ; do
1932 rm -f $platf/hapmap-genos-$pop-all $platf/hapmap-rs-all
1933 cat $platf/hapmap-genos-$pop-* > $platf/hapmap-genos-$pop-all
1934 cat $platf/hapmap-rs-* > $platf/hapmap-rs-all
1939 executing source block with sh...
1940 finished executing source block
1941 string-equal: Args out of range: "", -1, 0
1943 the error =string-equal: Args out of range: "", -1, 0= looks like what
1944 used to be output when the block returned an empty results string.
1945 This should be fixed in the current version, you should now see the
1946 following message =no result returned by source block=.
1948 ** DONE ruby arrays not recognized as such
1950 Something is wrong in [[file:lisp/org-babel-script.el]] related to the
1951 recognition of ruby arrays as such.
1953 #+begin_src ruby :results replace
1959 #+begin_src python :results replace
1967 Evaluate all the cells in this table for a comprehensive test of the
1968 org-babel functionality.
1970 *Note*: if you have customized =org-babel-default-header-args= then some
1971 of these tests may fail.
1973 #+TBLNAME: org-babel-tests
1974 | functionality | block | arg | expected | results | pass |
1975 |-------------------------+----------------------------+-----+-------------+-------------+------|
1976 | basic evaluation | | | | | pass |
1977 |-------------------------+----------------------------+-----+-------------+-------------+------|
1978 | emacs lisp | basic-elisp | | 5 | 5 | pass |
1979 | shell | basic-shell | | 6 | 6 | pass |
1980 | ruby | basic-ruby | | org-babel | org-babel | pass |
1981 | python | basic-python | | hello world | hello world | pass |
1982 | R | basic-R | | 13 | 13 | pass |
1983 |-------------------------+----------------------------+-----+-------------+-------------+------|
1984 | tables | | | | | pass |
1985 |-------------------------+----------------------------+-----+-------------+-------------+------|
1986 | emacs lisp | table-elisp | | 3 | 3 | pass |
1987 | ruby | table-ruby | | 1-2-3 | 1-2-3 | pass |
1988 | python | table-python | | 5 | 5 | pass |
1989 | R | table-R | | 3.5 | 3.5 | pass |
1990 |-------------------------+----------------------------+-----+-------------+-------------+------|
1991 | source block references | | | | | pass |
1992 |-------------------------+----------------------------+-----+-------------+-------------+------|
1993 | all languages | chained-ref-last | | Array | Array | pass |
1994 |-------------------------+----------------------------+-----+-------------+-------------+------|
1995 | source block functions | | | | | pass |
1996 |-------------------------+----------------------------+-----+-------------+-------------+------|
1997 | emacs lisp | defun-fibb | | fibbd | fibbd | pass |
1998 | run over | Fibonacci | 0 | 1 | 1 | pass |
1999 | a | Fibonacci | 1 | 1 | 1 | pass |
2000 | variety | Fibonacci | 2 | 2 | 2 | pass |
2001 | of | Fibonacci | 3 | 3 | 3 | pass |
2002 | different | Fibonacci | 4 | 5 | 5 | pass |
2003 | arguments | Fibonacci | 5 | 8 | 8 | pass |
2004 |-------------------------+----------------------------+-----+-------------+-------------+------|
2005 | bugs and tasks | | | | | pass |
2006 |-------------------------+----------------------------+-----+-------------+-------------+------|
2007 | simple ruby arrays | ruby-array-test | | 3 | 3 | pass |
2008 | R number evaluation | bug-R-number-evaluation | | 2 | 2 | pass |
2009 | multi-line ruby blocks | multi-line-ruby-test | | 2 | 2 | pass |
2010 | forcing vector results | test-forced-vector-results | | Array | Array | pass |
2011 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
2015 #+srcname: basic-elisp
2016 #+begin_src emacs-lisp :results silent
2020 #+srcname: basic-shell
2021 #+begin_src sh :results silent
2026 #+srcname: basic-ruby
2027 #+begin_src ruby :results silent
2031 #+srcname: basic-python
2032 #+begin_src python :results silent
2037 #+begin_src R :results silent
2044 #+tblname: test-table
2048 #+srcname: table-elisp
2049 #+begin_src emacs-lisp :results silent :var table=test-table
2050 (length (car table))
2053 #+srcname: table-ruby
2054 #+begin_src ruby :results silent :var table=test-table
2055 table.first.join("-")
2058 #+srcname: table-python
2059 #+begin_src python :var table=test-table
2064 #+begin_src R :var table=test-table
2070 Lets pass a references through all of our languages...
2072 Lets start by reversing the table from the previous examples
2074 #+srcname: chained-ref-first
2075 #+begin_src python :var table = test-table
2080 #+resname: chained-ref-first
2084 Take the first part of the list
2086 #+srcname: chained-ref-second
2087 #+begin_src R :var table = chained-ref-first
2091 #+resname: chained-ref-second
2095 Turn the numbers into string
2097 #+srcname: chained-ref-third
2098 #+begin_src emacs-lisp :var table = chained-ref-second
2099 (mapcar (lambda (el) (format "%S" el)) table)
2102 #+resname: chained-ref-third
2105 and Check that it is still a list
2107 #+srcname: chained-ref-last
2108 #+begin_src ruby :var table=chained-ref-third
2112 ** source blocks as functions
2114 #+srcname: defun-fibb
2115 #+begin_src emacs-lisp :results silent
2116 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
2119 #+srcname: fibonacci
2120 #+begin_src emacs-lisp :results silent :var n=7
2125 Testing the insertion of results into org-mode tables.
2127 #+srcname: multi-line-output
2128 #+begin_src ruby :results replace
2129 "the first line ends here
2132 and this is the second one
2138 : the first line ends here
2139 : and this is the second one
2140 : return even a third
2142 #+srcname: multi-line-error
2143 #+begin_src ruby :results replace
2144 raise "oh nooooooooooo"
2148 : -:5: warning: parenthesize argument(s) for future version
2149 : -:5:in `main': oh nooooooooooo (RuntimeError)
2152 | the first line ends here... | -:5: warning: parenthesize argument(s) for future version... |
2153 #+TBLFM: $1='(sbe "multi-line-output")::$2='(sbe "multi-line-error")
2155 ** forcing results types tests
2157 #+srcname: test-trivial-vector
2158 #+begin_src emacs-lisp :results vector silent
2162 #+srcname: test-forced-vector-results
2163 #+begin_src ruby :var triv=test-trivial-vector :results silent
2172 To run these examples evaluate [[file:lisp/org-babel-init.el][org-babel-init.el]]
2174 ** org-babel.el beginning functionality
2176 #+begin_src sh :results replace
2180 : Thu May 14 18:52:25 EDT 2009
2186 : Thu May 14 18:59:09 -0400 2009
2197 #+begin_src R :results replace
2206 hist(rgamma(20,3,3))
2210 ** org-babel plays with tables
2211 Alright, this should demonstrate both the ability of org-babel to read
2212 tables into a lisp source code block, and to then convert the results
2213 of the source code block into an org table. It's using the classic
2214 "lisp is elegant" demonstration transpose function. To try this
2217 1. evaluate [[file:lisp/org-babel-init.el]] to load org-babel and friends
2218 2. evaluate the transpose definition =\C-c\C-c= on the beginning of
2220 3. evaluate the next source code block, this should read in the table
2221 because of the =:var table=previous=, then transpose the table, and
2222 finally it should insert the transposed table into the buffer
2223 immediately following the block
2227 #+begin_src emacs-lisp :results silent
2228 (defun transpose (table)
2229 (apply #'mapcar* #'list table))
2237 #+begin_src emacs-lisp :var table=sandbox :results replace
2242 #+begin_src emacs-lisp
2246 | 1 | 2 | 3 | 4 | 5 |
2250 #+begin_src ruby :var table=sandbox :results replace
2251 table.first.join(" - ")
2256 #+begin_src python :var table=sandbox :results replace
2262 #+begin_src ruby :var table=sandbox :results replace
2267 | 4 | "schulte" | 6 |
2269 #+begin_src python :var table=sandbox :results replace
2275 | "__add__" | "__class__" | "__contains__" | "__delattr__" | "__delitem__" | "__delslice__" | "__doc__" | "__eq__" | "__format__" | "__ge__" | "__getattribute__" | "__getitem__" | "__getslice__" | "__gt__" | "__hash__" | "__iadd__" | "__imul__" | "__init__" | "__iter__" | "__le__" | "__len__" | "__lt__" | "__mul__" | "__ne__" | "__new__" | "__reduce__" | "__reduce_ex__" | "__repr__" | "__reversed__" | "__rmul__" | "__setattr__" | "__setitem__" | "__setslice__" | "__sizeof__" | "__str__" | "__subclasshook__" | "append" | "count" | "extend" | "index" | "insert" | "pop" | "remove" | "reverse" | "sort" |
2277 *** (sandbox table) R
2279 #+TBLNAME: sandbox_r
2283 #+begin_src R :results replace
2284 x <- c(rnorm(10, mean=-3, sd=1), rnorm(10, mean=3, sd=1))
2288 | -3.35473133869346 |
2290 | -3.32819924928633 |
2291 | -2.97310212756194 |
2292 | -2.09640758369576 |
2293 | -5.06054014378736 |
2294 | -2.20713700711221 |
2295 | -1.37618039712037 |
2296 | -1.95839385821742 |
2297 | -3.90407396475502 |
2298 | 2.51168071590226 |
2299 | 3.96753011570494 |
2300 | 3.31793212627865 |
2301 | 1.99829753972341 |
2302 | 4.00403686419829 |
2303 | 4.63723764452927 |
2304 | 3.94636744261313 |
2305 | 3.58355906547775 |
2306 | 3.01563442274226 |
2309 #+begin_src R var tabel=sandbox_r :results replace
2314 | 4 | "schulte" | 6 |
2317 Now shell commands are converted to tables using =org-table-import=
2318 and if these tables are non-trivial (i.e. have multiple elements) then
2319 they are imported as org-mode tables...
2321 #+begin_src sh :results replace
2325 | "total" | 208 | "" | "" | "" | "" | "" | "" |
2326 | "-rw-r--r--" | 1 | "dan" | "dan" | 57 | 2009 | 15 | "block" |
2327 | "-rw-r--r--" | 1 | "dan" | "dan" | 35147 | 2009 | 15 | "COPYING" |
2328 | "-rw-r--r--" | 1 | "dan" | "dan" | 722 | 2009 | 18 | "examples.org" |
2329 | "drwxr-xr-x" | 4 | "dan" | "dan" | 4096 | 2009 | 19 | "existing_tools" |
2330 | "-rw-r--r--" | 1 | "dan" | "dan" | 2207 | 2009 | 14 | "intro.org" |
2331 | "drwxr-xr-x" | 2 | "dan" | "dan" | 4096 | 2009 | 18 | "org-babel" |
2332 | "-rw-r--r--" | 1 | "dan" | "dan" | 277 | 2009 | 20 | "README.markdown" |
2333 | "-rw-r--r--" | 1 | "dan" | "dan" | 11837 | 2009 | 18 | "rorg.html" |
2334 | "-rw-r--r--" | 1 | "dan" | "dan" | 61829 | 2009 | 19 | "#rorg.org#" |
2335 | "-rw-r--r--" | 1 | "dan" | "dan" | 60190 | 2009 | 19 | "rorg.org" |
2336 | "-rw-r--r--" | 1 | "dan" | "dan" | 972 | 2009 | 11 | "test-export.org" |
2339 ** silent evaluation
2347 #+begin_src ruby :results silent
2351 #+begin_src ruby :results replace
2358 ** (sandbox) referencing other source blocks
2359 Doing this in emacs-lisp first because it's trivial to convert
2360 emacs-lisp results to and from emacs-lisp.
2362 *** emacs lisp source reference
2363 This first example performs a calculation in the first source block
2364 named =top=, the results of this calculation are then saved into the
2365 variable =first= by the header argument =:var first=top=, and it is
2366 used in the calculations of the second source block.
2369 #+begin_src emacs-lisp
2373 #+begin_src emacs-lisp :var first=top :results replace
2379 This example is the same as the previous only the variable being
2380 passed through is a table rather than a number.
2382 #+begin_src emacs-lisp :results silent
2383 (defun transpose (table)
2384 (apply #'mapcar* #'list table))
2387 #+TBLNAME: top_table
2391 #+SRCNAME: second_src_example
2392 #+begin_src emacs-lisp :var table=top_table
2396 #+begin_src emacs-lisp :var table=second_src_example :results replace
2401 | 4 | "schulte" | 6 |
2403 Now working for ruby
2410 #+begin_src ruby :var other=start :results replace
2416 #+SRCNAME: start_two
2421 #+begin_src python :var another=start_two :results replace
2426 Since all variables are converted into Emacs Lisp it is no problem to
2427 reference variables specified in another language.
2429 #+SRCNAME: ruby-block
2434 #+SRCNAME: lisp_block
2435 #+begin_src emacs-lisp :var ruby-variable=ruby-block
2439 #+begin_src python :var lisp_var=lisp_block
2448 #+begin_src R :results replace
2455 #+begin_src R :var other=first_r :results replace
2462 ** (sandbox) selective export
2464 For exportation tests and examples see (including exportation of
2465 inline source code blocks) [[file:test-export.org]]
2468 ** (sandbox) source blocks as functions
2471 #+begin_src emacs-lisp :results silent
2476 #+begin_src emacs-lisp :var n=default :results replace
2482 #+begin_src emacs-lisp :var result=triple(n=3, m=98) :results replace
2488 The following just demonstrates the ability to assign variables to
2489 literal values, which was not implemented until recently.
2491 #+begin_src ruby :var num="eric" :results replace
2498 ** (sandbox) inline source blocks
2500 This is an inline source code block src_ruby{1 + 6}. And another
2501 source block with text output src_emacs-lisp{"eric"}.
2503 This is an inline source code block with header
2504 arguments. src_ruby[:var n=fibbd( n = 0 )]{n}
2507 ** (sandbox) integration w/org tables
2509 #+begin_src emacs-lisp :results silent
2510 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
2514 #+begin_src emacs-lisp :var n=4 :results silent
2518 #+begin_src emacs-lisp :results silent
2519 (mapcar #'fibbd '(0 1 2 3 4 5 6 7 8))
2522 Something is not working here. The function `sbe ' works fine when
2523 called from outside of the table (see the source block below), but
2524 produces an error when called from inside the table. I think there
2525 must be some narrowing going on during intra-table emacs-lisp
2528 | original | fibbd |
2529 |----------+-------|
2540 #+TBLFM: $2='(sbe "fibbd" (n $1))
2544 #+begin_src emacs-lisp :results silent
2545 (sbe 'fibbd (n "8"))
2550 LocalWords: DBlocks dblocks org-babel el eric fontification