1 #+TITLE: org-babel --- facilitating communication between programming languages and people
2 #+SEQ_TODO: PROPOSED TODO STARTED | DONE DEFERRED REJECTED
3 #+OPTIONS: H:3 num:nil toc:t
4 #+STARTUP: oddeven hideblocks
6 Through Org-Babel Org-Mode can communicate with programming languages.
7 Code contained in source-code blocks can be evaluated and data can
8 pass seamlessly between different programming languages, Org-Mode
9 constructs (tables, file links, example text) and interactive comint
13 - The [[* Introduction][Introduction]] :: provides a brief overview of the design and use
14 of Org-Babel including tutorials and examples.
15 - In [[* Getting started][Getting Started]] :: find instructions for installing org-babel
16 into your emacs configuration.
17 - The [[* Tasks][Tasks]] :: section contains current and past tasks roughly ordered
18 by TODO state, then importance or date-completed. This would be
19 a good place to suggest ideas for development.
20 - The [[* Bugs][Bugs]] :: section contains bug reports.
21 - The [[* Tests][Tests]] :: section consists of a large table which can be
22 evaluated to run Org-Babel's functional test suite. This
23 provides a good overview of the current functionality with
24 pointers to example source blocks.
25 - The [[* Sandbox][Sandbox]] :: demonstrates much of the early/basic functionality
26 through commented source-code blocks.
28 Also see the [[file:library-of-babel.org][Library of Babel]], an extensible collection of ready-made
29 and easily-shortcut-callable source-code blocks for handling common
34 Org-Babel enables *communication* between programming languages and
38 - communication between programs :: Data passes seamlessly between
39 different programming languages, Org-Mode constructs (tables,
40 file links, example text) and interactive comint buffers.
41 - communication between people :: Data and calculations are embedded
42 in the same document as notes explanations and reports.
44 ** communication between programs
46 Org-Mode supports embedded blocks of source code (in any language)
47 inside of Org documents. Org-Babel allows these blocks of code to be
48 executed from within Org-Mode with natural handling of their inputs
52 with both scalar, file, and table output
54 *** reading information from tables
56 *** reading information from other source blocks (disk usage in your home directory)
58 This will work for Linux and Mac users, not so sure about shell
59 commands for windows users.
61 To run place the cursor on the =#+begin_src= line of the source block
62 labeled directory-pie and press =\C-c\C-c=.
64 #+srcname: directories
65 #+begin_src bash :results replace
66 cd ~ && du -sc * |grep -v total
69 #+resname: directories
71 | 11882808 | "Documents" |
72 | 8210024 | "Downloads" |
73 | 879800 | "Library" |
76 | 5307664 | "Pictures" |
85 #+srcname: directory-pie
86 #+begin_src R :var dirs = directories
87 pie(dirs[,1], labels = dirs[,2])
90 *** operations in/on tables
92 #+tblname: grades-table
93 | student | grade | letter |
94 |---------+-------+--------|
101 #+TBLFM: $2='(sbe random-score-generator)::$3='(sbe assign-grade (score $2))
103 #+srcname: assign-grade
104 #+begin_src ruby :var score=99
115 #+srcname: random-score-generator
120 #+srcname: show-distribution
121 #+begin_src R :var grades=grades-table
126 ** communication between people
127 Quick overview of Org-Mode's exportation abilities, with links to the
128 online Org-Mode documentation, a focus on source-code blocks, and the
129 exportation options provided by Org-Babel.
131 *** Interactive tutorial
132 This would demonstrate applicability to Reproducible Research, and
133 Literate Programming.
135 *** Tests embedded in documentation
136 org-babels own functional tests are contained in a large org-mode
137 table, allowing the test suite to be run be evaluation of the table
138 and the results to be collected in the same table.
140 *** Emacs initialization files stored in Org-Mode buffers
141 Using `org-babel-tangle' it is possible to embed your Emacs
142 initialization into org-mode files. This allows for folding,
143 note-taking, todo's etc... embedded with the source-code of your Emacs
144 initialization, and through org-mode's publishing features aids in
145 sharing your customizations with others.
147 It may be worthwhile to create a fork of Phil Hagelberg's
148 [[http://github.com/technomancy/emacs-starter-kit/tree/master][emacs-starter-kit]] which uses literate org-mode files for all of the
149 actual elisp customization. These org-mode files could then be
150 exported to html and used to populate the repositories wiki on [[http://github.com/][github]].
155 *** code evaluation (comint buffer sessions and external processes)
156 There are two main ways to evaluate source blocks with org-babel.
158 - external :: By default (if the =:session= header argument is not
159 present) all source code blocks are evaluated in
160 external processes. In these cases an external process
161 is used to evaluate the source-code blocks.
162 - session :: Session based evaluation uses persistent sessions in
163 comint buffers. Sessions can be used across multiple
164 source blocks setting and accessing variables in the
167 Evaluating source blocks in sessions also allows for
168 interaction with the code. To jump to the session of a
169 source block use the `org-babel-pop-to-session' command
170 or press =M-[down]= while inside of a source code block.
171 When called with a prefix argument
172 `org-babel-pop-to-session' will evaluate all header
173 arguments before jumping to the source-code block.
175 *** results (values and outputs)
176 Either the *value* or the *output* of source code blocks can be
177 collected after evaluation.
179 - value :: The default way to collect results from a source-code block
180 is to return the value of the last statement in the block.
181 This can be thought of as the return value of the block.
182 In this case any printed output of the block is ignored.
183 This can be though of a similar to a "functional" value of
185 - output :: Another way of generating results from a source-code block
186 is to collect the output generated by the execution of the
187 block. In this case all printed output is collected
188 throughout the execution of the block. This can be
189 thought of as similar to a "script" style of evaluation.
193 Add the following lines to your .emacs, replacing the path as
194 appropriate. A good place to check that things are up and running
195 would then be [[#sandbox][the sandbox]].
196 #+begin_src emacs-lisp
197 (add-to-list 'load-path "/path/to/org-babel/lisp")
198 (require 'org-babel-init)
203 ** PROPOSED use example block for large amounts of stdout output?
204 We're currently `examplizing' with : at the beginning of the line,
205 but should larger amounts of output be in a
206 \#+begin_example...\#+end_example block? What's the cutoff? > 1 line?
207 ** TODO make tangle files read-only?
208 With a file-local variable setting, yea that makes sense. Maybe
209 the header should reference the related org-mode file.
210 ** TODO take default values for header args from properties
211 Use file-wide and subtree wide properties to set default values for
213 ** TODO support for working with =*Org Edit Src Example*= buffers [2/4]
214 *** TODO optionally evaluate header references when we switch to =*Org Edit Src*= buffer
215 That seems to imply that the header references need to be evaluated
216 and transformed into the target language object when we hit C-c ' to
217 enter the *Org Edit Src* buffer [DED]
219 Good point, I heartily agree that this should be supported [Eric]
221 (or at least before the first time we attempt to evaluate code in that
222 buffer -- I suppose there might be an argument for lazy evaluation, in
223 case someone hits C-c ' but is "just looking" and not actually
224 evaluating anything.) Of course if evaluating the reference is
225 computationally intensive then the user might have to wait before they
226 get the *Org Edit Src* buffer. [DED]
228 I fear that it may be hard to anticipate when the references will be
229 needed, some major-modes do on-the-fly evaluation while the buffer is
230 being edited. I think that we should either do this before the buffer
231 is opened or not at all, specifically I think we should resolve
232 references if the user calls C-c ' with a prefix argument. Does that
233 sound reasonable? [Eric]
237 [Dan] So now that we have org-src-mode and org-src-mode-hook, I guess
238 org-babel should do this by using the hook to make sure that, when C-c
239 C-' is issued on a source block, any references are resolved and
240 assignments are made in the appropriate session.
241 *** TODO set buffer-local-process variables appropriately [DED]
242 I think something like this would be great. You've probably
243 already thought of this, but just to note it down: it would be really
244 nice if org-babel's notion of a buffer's 'session/process' played
245 nicely with ESS's notion of the buffer's session/process. ESS keeps
246 the current process name for a buffer in a buffer-local variable
247 ess-local-process-name. So one thing we will probably want to do is
248 make sure that the *Org Edit Src Example* buffer sets that variable
251 I had not thought of that, but I agree whole heartedly. [Eric]
253 Once this is done every variable should be able to dump regions into
254 their inferior-process buffer using major-mode functions.
255 *** DEFERRED send code to inferior process
256 Another thought on this topic: I think we will want users to send
257 chunks of code to the interpreter from within the *Org Edit Src*
258 buffer, and I think that's what you have in mind already. In ESS that
259 is done using the ess-eval-* functions. [DED]
261 I think we can leave this up to the major-mode in the source code
262 buffer, as almost every source-code major mode will have functions for
263 doing things like sending regions to the inferior process. If
264 anything we might need to set the value of the buffer local inferior
265 process variable. [Eric]
267 *** DONE some possible requests/proposed changes for Carsten [4/4]
268 While I remember, some possible requests/proposed changes for Carsten
269 come to mind in that regard:
271 **** DONE Remap C-x C-s to save the source to the org buffer?
272 I've done this personally and I find it essential. I'm using
273 #+begin_src emacs-lisp
274 (defun org-edit-src-save ()
275 "Update the parent org buffer with the edited source code, save
276 the parent org-buffer, and return to the source code edit
285 (define-key org-exit-edit-mode-map "\C-x\C-s" 'org-edit-src-save)
289 I think this is great, but I think it should be implemented in the
292 **** DEFERRED Rename buffer and minor mode?
293 Something shorter than *Org Edit Src Example* for the buffer
294 name. org-babel is bringing org's source code interaction to a
295 level of maturity where the 'example' is no longer
296 appropriate. And if further keybindings are going to be added to
297 the minor mode then maybe org-edit-src-mode is a better name than
300 Maybe we should name the buffer with a combination of the source
301 code and the session. I think that makes sense.
303 [ES] Are you also suggesting a new org-edit-src minor mode?
304 [DED] org-exit-edit-mode is a minor mode that already exists:
306 Minor mode installing a single key binding, "C-c '" to exit special edit.
308 org-edit-src-save now has a binding in that mode, so I guess all
309 I'm saying at this stage is that it's a bit of a misnomer. But
310 perhaps we will also have more functionality to add to that minor
311 mode, making it even more of a misnomer. Perhaps something like
312 org-src-mode would be better.
313 **** DONE Changed minor mode name and added hooks
315 **** DEFERRED a hook called when the src edit buffer is created
316 This should be implemented in the org-mode core
318 ** TODO resolve references to other org buffers/files
319 This would allow source blocks to call upon tables, source-blocks,
320 and results in other org buffers/files.
323 - [[file:lisp/org-babel-ref.el::TODO%20allow%20searching%20for%20names%20in%20other%20buffers][org-babel-ref.el:searching-in-other-buffers]]
324 - [[file:lisp/org-babel.el::defun%20org-babel%20find%20named%20result%20name][org-babel.el#org-babel-find-named-result]]
325 ** TODO resolve references to other non-org files
326 - tabular data in .csv, .tsv etc format
327 - files of interpreted code: anything stopping us giving such files
328 similar status to a source code block?
329 - Would be nice to allow org and non-org files to be remote
330 ** TODO figure out how to handle errors during evaluation
331 R has a try function, with error handling, along the lines of
332 python. I bet ruby does too. Maybe more of an issue for functional
333 style; in my proposed scripting style the error just gets dumped to
334 the org buffer and the user is thus alerted.
335 ** TODO figure out how to handle graphic output
336 This is listed under [[* graphical output][graphical output]] in out objectives.
338 This should take advantage of the =:results file= option, and
339 languages which almost always produce graphical output should set
340 =:results file= to true by default. That would handle placing these
341 results in the buffer. Then if there is a combination of =silent= and
342 =file= =:results= headers we could drop the results to a temp buffer
343 and pop open that buffer...
344 ** TODO =\C-c \C-o= to open results of source block
345 by adding a =defadvice= to =org-open-at-point= we can use the common
346 =\C-c \C-o= keybinding to open the results of a source-code block.
347 This would be especially useful for source-code blocks which generate
348 graphical results and insert a file link as the results in the
349 org-mode buffer. (see [[* figure out how to handle graphic output][TODO figure out how to handle graphic output]]).
350 This could also act reasonably with other results types...
352 - file :: use org-open-at-point to open the file
353 - scalar :: open results unquoted in a new buffer
354 - tabular :: export the table to a new buffer and open that buffer
356 ** TODO Finalise behaviour regarding vector/scalar output
357 *** DONE Stop spaces causing vector output
358 This simple example of multilingual chaining produces vector output if
359 there are spaces in the message and scalar otherwise.
363 #+begin_src R :var msg=msg-from-python
364 paste(msg, "und R", sep=" ")
368 : org-babel speaks elisp y python und R
370 #+srcname: msg-from-python
371 #+begin_src python :var msg=msg-from-elisp
375 #+srcname: msg-from-elisp
376 #+begin_src emacs-lisp :var msg="org-babel speaks"
377 (concat msg " elisp")
379 ** STARTED share org-babel [1/4]
380 how should we share org-babel?
382 *** DONE post to org-mode
383 *** TODO post to ess mailing list
384 *** TODO create a org-babel page on worg
385 *** TODO create a short screencast demonstrating org-babel in action
388 we need to think up some good examples
390 **** interactive tutorials
391 This could be a place to use [[* org-babel assertions][org-babel assertions]].
393 for example the first step of a tutorial could assert that the version
394 of the software-package (or whatever) is equal to some value, then
395 source-code blocks could be used with confidence (and executed
396 directly from) the rest of the tutorial.
398 **** answering a text-book question w/code example
399 org-babel is an ideal environment enabling both the development and
400 demonstrationg of the code snippets required as answers to many
403 **** something using tables
404 maybe something along the lines of calculations from collected grades
407 Maybe something like the following which outputs sizes of directories
408 under the home directory, and then instead of the trivial =emacs-lisp=
409 block we could use an R block to create a nice pie chart of the
413 #+begin_src bash :results replace
417 #+begin_src emacs-lisp :var sizes=sizes :results replace
421 ** TODO command line execution
422 Allow source code blocks to be called form the command line. This
423 will be easy using the =sbe= function in [[file:lisp/org-babel-table.el][org-babel-table.el]].
425 This will rely upon [[* resolve references to other buffers][resolve references to other buffers]].
427 ** TODO inline source code blocks [3/5]
428 Like the =\R{ code }= blocks
430 not sure what the format should be, maybe just something simple
431 like =src_lang[]{}= where lang is the name of the source code
432 language to be evaluated, =[]= is optional and contains any header
433 arguments and ={}= contains the code.
435 (see [[* (sandbox) inline source blocks][the-sandbox]])
437 *** DONE evaluation with \C-c\C-c
438 Putting aside the header argument issue for now we can just run these
439 with the following default header arguments
440 - =:results= :: silent
441 - =:exports= :: results
443 *** DONE inline exportation
444 Need to add an interblock hook (or some such) through org-exp-blocks
445 *** DONE header arguments
446 We should make it possible to use header arguments.
448 *** TODO fontification
449 we should color these blocks differently
451 *** TODO refine html exportation
452 should use a span class, and should show original source in tool-tip
454 ** TODO formulate general rules for handling vectors and tables / matrices with names
455 This is non-trivial, but may be worth doing, in particular to
456 develop a nice framework for sending data to/from R.
458 In R, indexing vector elements, and rows and columns, using
459 strings rather than integers is an important part of the
461 - elements of a vector may have names
462 - matrices and data.frames may have "column names" and "row names"
463 which can be used for indexing
464 - In a data frame, row names *must* be unique
472 > mat <- matrix(1:4, nrow=2, ncol=2, dimnames=list(c("r1","r2"), c("c1","c2")))
477 > # The names are separate from the data: they do not interfere with operations on the data
484 > df <- data.frame(var1=1:26, var2=26:1, row.names=letters)
486 [1] 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
492 So it's tempting to try to provide support for this in org-babel. For example
493 - allow R to refer to columns of a :var reference by their names
494 - When appropriate, results from R appear in the org buffer with "named
497 However none (?) of the other languages we are currently supporting
498 really have a native matrix type, let alone "column names" or "row
499 names". Names are used in e.g. python and perl to refer to entries
502 It currently seems to me that support for this in org-babel would
503 require setting rules about when org tables are considered to have
504 named columns/fields, and ensuring that (a) languages with a notion
505 of named columns/fields use them appropriately and (b) languages
506 with no such notion do not treat then as data.
508 - Org allows something that *looks* like column names to be separated
510 - Org also allows a row to *function* as column names when special
511 markers are placed in the first column. An hline is unnecessary
512 (indeed hlines are purely cosmetic in org [correct?]
513 - Org does not have a notion of "row names" [correct?]
515 The full org table functionality exeplified [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]] has features that
516 we would not support in e.g. R (like names for the row below).
518 *** Initial statement: allow tables with hline to be passed as args into R
519 This doesn't seem to work at the moment (example below). It would
520 also be nice to have a natural way for the column names of the org
521 table to become the column names of the R data frame, and to have
522 the option to specify that the first column is to be used as row
523 names in R (these must be unique). But this might require a bit of
528 | col1 | col2 | col3 |
529 |------+---------+------|
537 #+begin_src R var tabel=egtable
542 | "col1" | "col2" | "col3" |
543 |--------+-----------+--------|
545 | 4 | "schulte" | 6 |
548 Another example is in the [[*operations%20in%20on%20tables][grades example]].
550 ** TODO re-implement helper functions from org-R
551 *** Initial statement [Eric]
552 Much of the power of org-R seems to be in it's helper functions for
553 the quick graphing of tables. Should we try to re-implement these
554 functions on top of org-babel?
556 I'm thinking this may be useful both to add features to org-babel-R and
557 also to potentially suggest extensions of the framework. For example
558 one that comes to mind is the ability to treat a source-code block
559 like a function which accepts arguments and returns results. Actually
560 this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]).
562 - We want to provide convenient off-the-shelf actions
563 (e.g. plotting data) that make use of our new code evaluation
564 environment but do not require any actual coding.
565 *** Initial Design proposal [Dan]
566 - *Input data* will be specified using the same mechanism as :var
567 references, thus the input data may come from a table, or
568 another source block, and it is initially available as an elisp
570 - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that
571 line will apply an *action* to the referenced data.
572 - *Actions correspond to source blocks*: our library of available
573 actions will be a library of org-babel source blocks. Thus the
574 code for executing an action, and the code for dealing with the
575 output of the action will be the same code as for executing
576 source blocks in general
577 - Optionally, the user can have the relevant source block inserted
578 into the org buffer after the (say) #+BABELDO line. This will
579 allow the user to fine tune the action by modifying the code
580 (especially useful for plots).
581 - So maybe a #+BABELDO line will have header args
582 - :data (a reference to a table or source code block)
583 - :action (or should that be :srcname?) which will be something
584 like :action pie-chart, referring to a source block which will
585 be executed with the :data referent passed in using a :var arg.
586 - :showcode or something controlling whether to show the code
588 *** Modification to design
589 I'm implementing this, at least initially, as a new interpreter
590 named 'babel', which has an empty body. 'babel' blocks take
591 a :srcname header arg, and look for the source-code block with
592 that name. They then execute the referenced block, after first
593 appending their own header args on to the target block's header
596 If the target block is in the library of babel (a.o.t. e.g. the
597 current buffer), then the code in the block will refer to the
598 input data with a name dictated by convention (e.g. __data__
599 (something which is syntactically legal in all languages...). Thus
600 the babel block will use a :var __data__ = whatever header arg to
601 reference the data to be plotted.
604 This is covered by the [[file:library-of-babel.org][Library of Babel]], which will contain
605 ready-made source blocks designed to carry out useful common tasks.
606 ** PROPOSED Are we happy with current behaviour regarding vector/scalar output?
607 This simple example of multilingual chaining produces vector output if
608 there are spaces in the message and scalar otherwise.
610 #+begin_src R :var msg=msg-from-python
611 paste(msg, "und_R", sep="_")
614 #+srcname: msg-from-python
615 #+begin_src python :var msg=msg-from-elisp
619 #+srcname: msg-from-elisp
620 #+begin_src emacs-lisp :var msg="org-babel_speaks"
621 (concat msg "_elisp")
624 ** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format
625 I haven't thought about this properly. Just noting it down. What
626 Sweave uses is called "R noweb" (.Rnw).
628 I found a good description of noweb in the following article (see
629 the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]).
631 I think there are two parts to noweb, the construction of
632 documentation and the extraction of source-code (with notangle).
634 *documentation*: org-mode handles all of our documentation needs in
635 a manner that I believe is superior to noweb.
637 *source extraction* At this point I don't see anyone writing large
638 applications with 100% of the source code contained in org-babel
639 files, rather I see org-babel files containing things like
640 - notes with active code chunks
641 - interactive tutorials
642 - requirements documents with code running test suites
643 - and of course experimental reports with the code to run the
644 experiment, and perform analysis
646 Basically I think the scope of the programs written in org-babel
647 (at least initially) will be small enough that it wont require the
648 addition of a tangle type program to extract all of the source code
649 into a running application.
651 On the other hand, since we already have named blocks of source
652 code which reference other blocks on which they rely, this
653 shouldn't be too hard to implement either on our own, or possibly
654 relying on something like noweb/notangle.
656 ** PROPOSED support for passing paths to files between source blocks
657 Maybe this should be it's own result type (in addition to scalars and
658 vectors). The reason being that some source-code blocks (for example
659 ditaa or anything that results in the creation of a file) may want to
660 pass a file path back to org-mode which could then be inserted into
661 the org-mode buffer as a link to the file...
663 This would allow for display of images upon export providing
664 functionality similar to =org-exp-blocks= only in a more general
667 ** TODO Add languages [1/6]
668 I'm sure there are many more that aren't listed here. Please add
669 them, and bubble any that you particularly care about up to the top.
671 Any new language should be implemented in a org-babel-lang.el file.
672 Follow the pattern set by [[file:lisp/org-babel-script.el][org-babel-script.el]], [[file:lisp/org-babel-shell.el][org-babel-shell.el]] and
673 [[file:lisp/org-babel-R.el][org-babel-R.el]].
676 This could probably be added to [[file:lisp/org-babel-script.el][org-babel-script.el]]
681 (see [[* file result types][file result types]])
683 #+srcname: implementing-ditaa
684 #+begin_src ditaa :results replace :file blue.png :cmdline -r
694 #+resname: implementing-ditaa
695 [[file:blue.png][blue.png]]
698 (see [[* file result types][file result types]])
700 - a =file= header argument
701 - a =cmdline= header argument
703 - scalar variables should be replaced in the body of the gnuplot code
704 - vector variables should be exported to tab-separated files, and
705 the variable names should be replaced with the path to the files
707 #+PLOT: title:"Citas" ind:1 deps:(3) type:2d with:histograms set:"yrange [0:]"
708 #+TBLNAME: gnuplot-data
709 | independent var | first dependent var | second dependent var |
710 |-----------------+---------------------+----------------------|
711 | 0.1 | 0.425 | 0.375 |
712 | 0.2 | 0.3125 | 0.3375 |
713 | 0.3 | 0.24999993 | 0.28333338 |
714 | 0.4 | 0.275 | 0.28125 |
715 | 0.5 | 0.26 | 0.27 |
716 | 0.6 | 0.25833338 | 0.24999993 |
717 | 0.7 | 0.24642845 | 0.23928553 |
718 | 0.8 | 0.23125 | 0.2375 |
719 | 0.9 | 0.23333323 | 0.2333332 |
720 | 1 | 0.2225 | 0.22 |
721 | 1.1 | 0.20909075 | 0.22272708 |
722 | 1.2 | 0.19999998 | 0.21458333 |
723 | 1.3 | 0.19615368 | 0.21730748 |
725 #+srcname: implementing-gnuplot
726 #+begin_src gnuplot :var data=gnuplot-data
727 set title "Implementing Gnuplot"
728 plot "data" using 1:2 with lines
732 (see [[* file result types][file result types]])
735 (see [[* file result types][file result types]])
737 ** DEFERRED use textConnection to pass tsv to R?
738 When passing args from the org buffer to R, the following route is
739 used: arg in buffer -> elisp -> tsv on file -> data frame in R. I
740 think it would be possible to avoid having to write to file by
741 constructing an R expression in org-babel-R-assign-elisp, something
744 #+begin_src emacs-lisp
745 (org-babel-R-input-command
746 (format "%s <- read.table(textConnection(\"%s\"), sep=\"\\t\", as.is=TRUE)"
747 name (orgtbl-to-tsv value '(:sep "\t" :fmt org-babel-R-quote-tsv-field))))
750 I haven't tried to implement this yet as it's basically just
751 fiddling with something that works. The only reason for it I can
752 think of would be efficiency and I haven't tested that.
754 This Didn't work after an initial test. I still think this is a
755 good idea (I also think we should try to do something similar when
756 writing out results frmo R to elisp) however as it wouldn't result
757 in any functional changes I'm bumping it down to deferred for
762 #+tblname: quick-test
765 #+srcname: quick-test-src-blk
766 #+begin_src R :var vec=quick-test
772 ** DEFERRED re-implement R evaluation using ess-command or ess-execute
773 I don't have any complaints with the current R evaluation code or
774 behaviour, but I think it would be good to use the ESS functions
775 from a political point of view. Plus of course it has the normal
776 benefits of an API (insulates us from any underlying changes etc). [DED]
778 I'll look into this. I believe that I looked at and rejected these
779 functions initially but now I can't remember why. I agree with
780 your overall point about using API's where available. I will take
781 a look back at these and either switch to using the ess commands,
782 or at least articulate under this TODO the reasons for using our
783 custom R-interaction commands. [Eric]
787 Lets just replace =org-babel-R-input-command= with =ess-execute=.
789 I tried this, and although it works in some situations, I find that
790 =ess-command= will often just hang indefinitely without returning
791 results. Also =ess-execute= will occasionally hang, and pops up
792 the buffer containing the results of the command's execution, which
793 is undesirable. For now these functions can not be used. Maybe
794 someone more familiar with the ESS code can recommend proper usage
795 of =ess-command= or some other lower-level function which could be
796 used in place of [[file:lisp/org-babel-R.el::defun%20org-babel%20R%20input%20command%20command][org-babel-R-input-command]].
800 #+begin_quote ess-command
801 (ess-command COM &optional BUF SLEEP NO-PROMPT-CHECK)
803 Send the ESS process command COM and delete the output
804 from the ESS process buffer. If an optional second argument BUF exists
805 save the output in that buffer. BUF is erased before use.
806 COM should have a terminating newline.
807 Guarantees that the value of .Last.value will be preserved.
808 When optional third arg SLEEP is non-nil, `(sleep-for (* a SLEEP))'
809 will be used in a few places where `a' is proportional to `ess-cmd-delay'.
812 #+begin_quote ess-execute
813 (ess-execute COMMAND &optional INVERT BUFF MESSAGE)
815 Send a command to the ESS process.
816 A newline is automatically added to COMMAND. Prefix arg (or second arg
817 INVERT) means invert the meaning of
818 `ess-execute-in-process-buffer'. If INVERT is 'buffer, output is
819 forced to go to the process buffer. If the output is going to a
820 buffer, name it *BUFF*. This buffer is erased before use. Optional
821 fourth arg MESSAGE is text to print at the top of the buffer (defaults
822 to the command if BUFF is not given.)
825 *** out current setup
827 1) The body of the R source code block is wrapped in a function
828 2) The function is called inside of a =write.table= function call
829 writing the results to a table
830 3) The table is read using =org-table-import=
832 ** DEFERRED Rework Interaction with Running Processes [2/5]
833 *** DONE robust to errors interrupting execution
835 #+srcname: long-runner-ruby
836 #+begin_src ruby :results silent
841 *** DEFERRED use =C-g= keyboard-quit to push processing into the background
842 This may be possible using the `run-with-timer' command.
844 I have no idea how this could work...
846 #+srcname: long-runner-ruby
847 #+begin_src ruby :results silent
852 *** TODO ability to select which of multiple sessions is being used
853 Increasingly it is looking like we're going to want to run all
854 source code blocks in comint buffer (sessions). Which will have
856 1) allowing background execution
857 2) maintaining state between source-blocks
858 - allowing inline blocks w/o header arguments
861 (like ess-switch-process in .R buffers)
863 Maybe this could be packaged into a header argument, something
864 like =:R_session= which could accept either the name of the
865 session to use, or the string =prompt=, in which case we could use
866 the =ess-switch-process= command to select a new process.
868 *** TODO evaluation of shell code as background process?
869 After C-c C-c on an R code block, the process may appear to
870 block, but C-g can be used to reclaim control of the .org buffer,
871 without interrupting the R evalution. However I believe this is not
872 true of bash/sh evaluation. [Haven't tried other languages] Perhaps
873 a solution is just to background the individual shell commands.
875 The other languages (aside from emacs lisp) are run through the
876 shell, so if we find a shell solution it should work for them as
879 Adding an ampersand seems to be a supported way to run commands in
880 the background (see [[http://www.emacswiki.org/emacs/ExecuteExternalCommand#toc4][external-commands]]). Although a more extensible
881 solution may involve the use of the [[elisp:(progn (describe-function 'call-process-region) nil)][call-process-region]] function.
883 Going to try this out in a new file [[file:lisp/org-babel-proc.el][org-babel-proc.el]]. This should
884 contain functions for asynchronously running generic shell commands
885 in the background, and then returning their input.
887 **** partial update of org-mode buffer
888 The sleekest solution to this may be using a comint buffer, and
889 then defining a filter function which would incrementally interpret
890 the results as they are returned, including insertion into the
891 org-mode buffer. This may actually cause more problems than it is
892 worth, what with the complexities of identifying the types of
893 incrementally returned results, and the need for maintenance of a
894 process marker in the org buffer.
896 **** 'working' spinner
897 It may be nice and not too difficult to place a spinner on/near the
898 evaluating source code block
900 *** TODO conversion of output from interactive shell, R (and python) sessions to org-babel buffers
901 [DED] This would be a nice feature I think. Although an org-babel
902 purist would say that it's working the wrong way round... After
903 some interactive work in a *R* buffer, you save the buffer, maybe
904 edit out some lines, and then convert it to org-babel format for
905 posterity. Same for a shell session either in a *shell* buffer, or
906 pasted from another terminal emulator. And python of course.
908 ** DEFERRED improve the source-block snippet
909 any real improvement seems somewhat beyond the ability of yasnippet
912 [[file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name%20Chapter%20title][file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name Chapter title]]
914 ,#name : Chapter title
917 ${1:$(make-string (string-width text) ?\=)}
922 [[file:snippets/org-mode/sb][sb -- snippet]]
924 waiting for guidance from those more familiar with yasnippets
925 ** DONE LoB: allow output in buffer
926 ** DONE allow default header arguments by language
927 org-babel-default-header-args:lang-name
929 An example of when this is useful is for languages which always return
930 files as their results (e.g. [[*** ditaa][ditaa]], and [[*** gnuplot][gnuplot]]).
931 ** DONE singe-function tangling and loading elisp from literate org-mode file [3/3]
933 This function should tangle the org-mode file for elisp, and then call
934 `load-file' on the resulting tangled file.
936 #+srcname: test-loading-embedded-emacs-lisp
937 #+begin_src emacs-lisp :results replace
938 (setq test-tangle-advert nil)
939 (setq test-tangle-loading nil)
940 (setq results (list :before test-tangle-loading test-tangle-advert))
941 (org-babel-load-file "test-tangle.org")
942 (setq results (list (list :after test-tangle-loading test-tangle-advert) results))
943 (delete-file "test-tangle.el")
947 #+resname: test-loading-embedded-emacs-lisp
948 | :before | nil | nil |
949 | :after | "org-babel tangles" | "use org-babel-tangle for all your emacs initialization files!!" |
951 *** DONE add optional language limiter to org-babel-tangle
952 This should check to see if there is any need to re-export
954 *** DONE ensure that org-babel-tangle returns the path to the tangled file(s)
956 #+srcname: test-return-value-of-org-babel-tangle
957 #+begin_src emacs-lisp :results replace
958 (mapcar #'file-name-nondirectory (org-babel-tangle-file "test-tangle.org" "emacs-lisp"))
964 *** DONE only tangle the file if it's actually necessary
965 ** DONE add a function to jump to a source-block by name
966 I've had an initial stab at that in org-babel-find-named-block
967 (library-of-babel branch).
969 At the same time I introduced org-babel-named-src-block-regexp, to
970 match src-blocks with srcname.
972 This is now working with the command
973 `org-babel-goto-named-source-block', all we need is a good key
976 ** DONE add =:none= session argument (for purely functional execution) [4/4]
977 This would allow source blocks to be run in their own new process
979 - These blocks could then also be run in the background (since we can
980 detach and just wait for the process to signal that it has terminated)
981 - We wouldn't be drowning in session buffers after running the tests
982 - we can re-use much of the session code to run in a more /functional/
985 While session provide a lot of cool features, like persistent
986 environments, [[* DONE function to bring up inferior-process buffer][pop-to-session]], and hints at exportation for
987 org-babel-tangle, they also have some down sides and I'm thinking that
988 session-based execution maybe shouldn't be the default behavior.
990 Down-sides to sessions
991 - *much* more complicated than functional evaluation
992 - maintaining the state of the session has weird issues
993 - waiting for evaluation to finish
994 - prompt issues like [[* TODO weird escaped characters in shell prompt break shell evaluation][shell-prompt-escapes-bug]]
995 - can't run in background
996 - litter emacs with session buffers
1000 #+srcname: ruby-task-no-session
1001 #+begin_src ruby :results replace output
1007 #+resname: ruby-task-no-session
1012 #+srcname: task-python-none-session
1013 #+begin_src python :session none :results replace value
1019 #+resname: task-python-none-session
1024 #+srcname: task-session-none-sh
1025 #+begin_src sh :results replace
1030 #+resname: task-session-none-sh
1036 #+srcname: task-no-session-R
1037 #+begin_src R :results replace output
1044 #+resname: task-no-session-R
1048 ** DONE fully purge org-babel-R of direct comint interaction
1049 try to remove all code under the [[file:lisp/org-babel-R.el::functions%20for%20evaluation%20of%20R%20code][;; functions for evaluation of R code]] line
1051 ** DONE Create objects in top level (global) environment [5/5]
1054 *** initial requirement statement [DED]
1055 At the moment, objects created by computations performed in the
1056 code block are evaluated in the scope of the
1057 code-block-function-body and therefore disappear when the code
1058 block is evaluated {unless you employ some extra trickery like
1059 assign('name', object, env=globalenv()) }. I think it will be
1060 desirable to also allow for a style wherein objects that are
1061 created in one code block persist in the R global environment and
1062 can be re-used in a separate block.
1064 This is what Sweave does, and while I'm not saying we have to be
1065 the same as Sweave, it wouldn't be hard for us to provide the same
1066 behaviour in this case; if we don't, we risk undeservedly being
1067 written off as an oddity by some.
1069 IOW one aspect of org-babel is that of a sort of functional
1070 meta-programming language. This is crazy, in a very good
1071 way. Nevertheless, wrt R I think there's going to be a lot of value
1072 in providing for a working style in which the objects are stored in
1073 the R session, rather than elisp/org buffer. This will be a very
1074 familiar working style to lots of people.
1076 There are no doubt a number of different ways of accomplishing
1077 this, the simplest being a hack like adding
1080 for(objname in ls())
1081 assign(objname, get(objname), envir=globalenv())
1084 to the source code block function body. (Maybe wrap it in an on.exit() call).
1086 However this may deserve to be thought about more carefully, perhaps
1087 with a view to having a uniform approach across languages. E.g. shell
1088 code blocks have the same semantics at the moment (no persistence of
1089 variables across code blocks), because the body is evaluated in a new
1090 bash shell process rather than a running shell. And I guess the same
1091 is true for python. However, in both these cases, you could imagine
1092 implementing the alternative in which the body is evaluated in a
1093 persistent interactive session. It's just that it's particularly
1094 natural for R, seeing as both ESS and org-babel evaluate commands in a
1095 single persistent R session.
1099 Thanks for bringing this up. I think you are absolutely correct that we
1100 should provide support for a persistent environment (maybe called a
1101 *session*) in which to evaluate code blocks. I think the current setup
1102 demonstrates my personal bias for a functional style of programming
1103 which is certainly not ideal in all contexts.
1105 While the R function you mention does look like an elegant solution, I
1106 think we should choose an implementation that would be the same across
1107 all source code types. Specifically I think we should allow the user to
1108 specify an optional *session* as a header variable (when not present we
1109 assume a default session for each language). The session name could be
1110 used to name a comint buffer (like the *R* buffer) in which all
1111 evaluation would take place (within which variables would retain their
1112 values --at least once I remove some of the functional method wrappings
1113 currently in place-- ).
1115 This would allow multiple environments to be used in the same buffer,
1116 and once this setup was implemented we should be able to fairly easily
1117 implement commands for jumping between source code blocks and the
1118 related session buffers, as well as for dumping the last N commands from
1119 a session into a new or existing source code block.
1121 Please let me know if you foresee any problems with this proposed setup,
1122 or if you think any parts might be confusing for people coming from
1123 Sweave. I'll hopefully find some time to work on this later in the
1126 *** can functional and interpreted/interactive models coexist?
1128 Even though both of these use the same =*R*= buffer the value of =a=
1129 is not preserved because it is assigned inside of a functional
1132 #+srcname: task-R-sessions
1139 #+srcname: task-R-same-session
1144 This functional wrapper was implemented in order to efficiently return
1145 the results of the execution of the entire source code block. However
1146 it inhibits the evaluation of source code blocks in the top level,
1147 which would allow for persistence of variable assignment across
1148 evaluations. How can we allow *both* evaluation in the top level, and
1149 efficient capture of the return value of an entire source code block
1150 in a language independent manner?
1152 Possible solutions...
1153 1) we can't so we will have to implement two types of evaluation
1154 depending on which is appropriate (functional or imperative)
1155 2) we remove the functional wrapper and parse the source code block
1156 into it's top level statements (most often but not always on line
1157 breaks) so that we can isolate the final segment which is our
1159 3) we add some sort of "#+return" line to the code block
1160 4) we take advantage of each languages support for meta-programming
1161 through =eval= type functions, and use said to evaluate the entire
1162 blocks in such a way that their environment can be combined with the
1163 global environment, and their results are still captured.
1164 5) I believe that most modern languages which support interactive
1165 sessions have support for a =last_result= type function, which
1166 returns the result of the last input without re-calculation. If
1167 widely enough present this would be the ideal solution to a
1168 combination of functional and imperative styles.
1170 None of these solutions seem very desirable, but for now I don't see
1171 what else would be possible.
1173 Of these options I was leaning towards (1) and (4) but now believe
1174 that if it is possible option (5) will be ideal.
1176 **** (1) both functional and imperative evaluation
1178 - can take advantage of built in functions for sending regions to the
1180 - retains the proven tested and working functional wrappers
1183 - introduces the complication of keeping track of which type of
1184 evaluation is best suited to a particular context
1185 - the current functional wrappers may require some changes in order to
1186 include the existing global context
1188 **** (4) exploit language meta-programming constructs to explicitly evaluate code
1190 - only one type of evaluation
1193 - some languages may not have sufficient meta-programming constructs
1195 **** (5) exploit some =last_value= functionality if present
1197 Need to ensure that most languages have such a function, those without
1198 will simply have to implement their own similar solution...
1200 | language | =last_value= function |
1201 |------------+-----------------------------|
1205 | shell | see [[* last command for shells][last command for shells]] |
1206 | emacs-lisp | see [[* emacs-lisp will be a special case][special-case]] |
1208 #+srcname: task-last-value
1213 ***** last command for shells
1214 Do this using the =tee= shell command, and continually pipe the output
1217 Got this idea from the following [[http://linux.derkeiler.com/Mailing-Lists/Fedora/2004-01/0898.html][email-thread]].
1219 suggested from mailing list
1221 #+srcname: bash-save-last-output-to-file
1225 bash -c "$line" | tee /tmp/last.out1
1226 mv /tmp/last.out1 /tmp/last.out
1230 another proposed solution from the above thread
1232 #+srcname: bash-save-in-variable
1235 # so - Save Output. Saves output of command in OUT shell variable.
1245 "^M": " | tee /tmp/h_lastcmd.out ^[k"
1248 export __=/tmp/h_lastcmd.out
1250 If you try it, Alt-k will stand for the old Enter; use "command $__" to
1251 access the last output.
1257 Herculano de Lima Einloft Neto
1260 ***** emacs-lisp will be a special case
1261 While it is possible for emacs-lisp to be run in a console type
1262 environment (see the =elim= function) it is *not* possible to run
1263 emacs-lisp in a different *session*. Meaning any variable set top
1264 level of the console environment will be set *everywhere* inside
1265 emacs. For this reason I think that it doesn't make any sense to
1266 worry about session support for emacs-lisp.
1268 *** Further thoughts on 'scripting' vs. functional approaches
1270 These are just thoughts, I don't know how sure I am about this.
1271 And again, perhaps I'm not saying anything very radical, just that
1272 it would be nice to have some options supporting things like
1273 receiving text output in the org buffer.
1275 I can see that you've already gone some way down the road towards
1276 the 'last value' approach, so sorry if my comments come rather
1277 late. I am concerned that we are not giving sufficient attention
1278 to stdout / the text that is returned by the interpreters. In
1279 contrast, many of our potential users will be accustomed to a
1280 'scripting' approach, where they are outputting text at various
1281 points in the code block, not just at the end. I am leaning
1282 towards thinking that we should have 2 modes of evaluation:
1283 'script' mode, and 'functional' mode.
1285 In script mode, evaluation of a code block would result in *all*
1286 text output from that code block appearing as output in the org
1287 buffer, presumably as an #+begin_example...#+end_example. There
1288 could be an :echo option controlling whether the input commands
1289 also appear in the output. [This is like Sweave].
1291 In functional mode, the *result* of the code block is available as
1292 an elisp object, and may appear in the org buffer as an org
1293 table/string, via the mechanisms you have developed already.
1295 One thing I'm wondering about is whether, in script mode, there
1296 simply should not be a return value. Perhaps this is not so
1297 different from what exists: script mode would be new, and what
1298 exists currently would be functional mode.
1300 I think it's likely that, while code evaluation will be exciting
1301 to people, a large majority of our users in a large majority of
1302 their usage will not attempt to actually use the return value from
1303 a source code block in any meaningful way. In that case, it seems
1304 rather restrictive to only allow them to see output from the end
1307 Instead I think the most accessible way to introduce org-babel to
1308 people, at least while they are learning it, is as an immensely
1309 powerful environment in which to embed their 'scripts', which now
1310 also allows them to 'run' their 'scripts'. Especially as such
1311 people are likely to be the least capable of the user-base, a
1312 possible design-rule would be to make the scripting style of usage
1313 easy (default?), perhaps requiring a special option to enable a
1314 functional style. Those who will use the functional style won't
1315 have a problem understanding what's going on, whereas the 'skript
1316 kiddies' might not even know the syntax for defining a function in
1317 their language of choice. And of course we can allow the user to
1318 set a variable in their .emacs controlling the preference, so that
1319 functional users are not inconveniennced by having to provide
1320 header args the whole time.
1322 Please don't get the impression that I am down-valuing the
1323 functional style of org-babel. I am constantly horrified at the
1324 messy 'scripts' that my colleagues produce in perl or R or
1325 whatever! Nevertheless that seems to be how a lot of people work.
1327 I think you were leaning towards the last-value approach because
1328 it offered the possibility of unified code supporting both the
1329 single evaluation environment and the functional style. If you
1330 agree with any of the above then perhaps it will impact upon this
1331 and mean that the code in the two branches has to differ a bit. In
1332 that case, functional mode could perhaps after all evaluate each
1333 code block in its own environment, thus (re)approaching 'true'
1334 functional programming (side-effects are hard to achieve).
1338 echo "There are `wc -l files` files in this directory"
1342 *** even more thoughts on evaluation, results, models and options
1344 Thanks Dan, These comments are invaluable.
1346 What do you think about this as a new list of priorities/requirements
1347 for the execution of source-code blocks.
1350 1) we want the evaluation of the source code block to take place in a
1351 session which can persist state (variables, current directory,
1353 2) source code blocks can specify their session with a header argument
1354 3) each session should correspond to an Emacs comint buffer so that the
1355 user can drop into the session and experiment with live code
1358 1) each source-code block generates some form of results which (as
1359 we have already implemented) is transfered into emacs-lisp
1360 after which it can be inserted into the org-mode buffer, or
1361 used by other source-code blocks
1362 2) when the results are translated into emacs-lisp, forced to be
1363 interpreted as a scalar (dumping their raw values into the
1364 org-mode buffer), as a vector (which is often desirable with R
1365 code blocks), or interpreted on the fly (the default option).
1366 Note that this is very nearly currently implemented through the
1367 [[* DONE results-type header (vector/file)][results-type-header]].
1368 3) there should be *two* means of collecting results from the
1369 execution of a source code block. *Either* the value of the
1370 last statement of the source code block, or the collection of
1371 all that has been passed to STDOUT during the evaluation.
1373 **** header argument or return line (*header argument*)
1375 Rather than using a header argument to specify how the return value
1376 should be passed back, I'm leaning towards the use of a =#+RETURN=
1377 line inside the block. If such a line *is not present* then we
1378 default to using STDOUT to collect results, but if such a line *is
1379 present* then we use it's value as the results of the block. I
1380 think this will allow for the most elegant specification between
1381 functional and script execution. This also cleans up some issues
1382 of implementation and finding which statement is the last
1385 Having given this more thought, I think a header argument is
1386 preferable. The =#+return:= line adds new complicating syntax for
1387 something that does little more than we would accomplish through
1388 the addition of a header argument. The only benefit being that we
1389 know where the final statement starts, which is not an issue in
1390 those languages which contain 'last value' operators.
1392 new header =:results= arguments
1393 - script :: explicitly states that we want to use STDOUT to
1394 initialize our results
1395 - return_last :: stdout is ignored instead the *value* of the final
1396 statement in the block is returned
1397 - echo :: means echo the contents of the source-code block along
1398 with the results (this implies the *script* =:results=
1401 *** DONE rework evaluation lang-by-lang [4/4]
1403 This should include...
1404 - functional results working with the comint buffer
1406 - script :: return the output of STDOUT
1407 - write a macro which runs the first redirection, executes the
1408 body, then runs the second redirection
1409 - last :: return the value of the last statement
1412 - sessions in comint buffers
1414 **** DONE Ruby [4/4]
1415 - [X] functional results working with comint
1416 - [X] script results
1417 - [X] ensure scalar/vector results args are taken into consideration
1418 - [X] ensure callable by other source block
1420 #+srcname: ruby-use-last-output
1421 #+begin_src ruby :results replace
1428 #+resname: ruby-use-last-output
1431 #+srcname: task-call-use-last-output
1432 #+begin_src ruby :var last=ruby-use-last-output :results replace
1433 last.flatten.size + 1
1436 #+resname: task-call-use-last-output
1441 #+srcname: first-ruby-session-task
1442 #+begin_src ruby :session schulte :results silent
1446 #+srcname: second-ruby-session-task
1447 #+begin_src ruby :session schulte :results silent
1451 #+srcname: without-the-right-session
1452 #+begin_src ruby :results silent
1458 - [X] functional results working with comint
1459 - [X] script results
1460 - [X] ensure scalar/vector results args are taken into consideration
1461 - [X] ensure callable by other source block
1463 To redirect output to a file, you can use the =sink()= command.
1466 #+begin_src R :results value vector silent
1473 #+srcname: task-R-use-other-output
1474 #+begin_src R :var twoentyseven=task_R_B() :results replace value
1479 #+resname: task-R-use-other-output
1482 **** DONE Python [4/4]
1483 - [X] functional results working with comint
1484 - [X] script results
1485 - [X] ensure scalar/vector results args are taken into consideration
1486 - [X] ensure callable by other source block
1488 #+srcname: task-new-eval-for-python
1489 #+begin_src python :results silent output scalar
1495 #+srcname: task-use-new-eval
1496 #+begin_src python :var tasking=task-new-eval-for-python() :results replace
1500 #+resname: task-use-new-eval
1503 **** DONE Shells [4/4]
1504 - [X] functional results working with comint
1505 - [X] script results
1506 - [X] ensure scalar/vector results args are taken into consideration
1507 - [X] ensure callable by other source block
1509 #+srcname: task-shell-new-evaluation
1510 #+begin_src sh :results silent value scalar
1515 #+srcname: task-call-other-shell
1516 #+begin_src sh :var other=task-shell-new-evaluation() :results replace scalar
1517 echo $other ' is the old date'
1520 #+resname: task-call-other-shell
1521 : $ Fri Jun 12 13:08:37 PDT 2009 is the old date
1523 *** DONE implement a *session* header argument [4/4]
1524 =:session= header argument to override the default *session* buffer
1528 #+srcname: task-ruby-named-session
1529 #+begin_src ruby :session schulte :results replace
1530 schulte = :in_schulte
1533 #+resname: task-ruby-named-session
1536 #+srcname: another-in-schulte
1537 #+begin_src ruby :session schulte
1541 #+resname: another-in-schulte
1548 #+srcname: python-session-task
1549 #+begin_src python :session what :results silent
1553 #+srcname: python-get-from-session
1554 #+begin_src python :session what :results replace
1558 #+resname: python-get-from-session
1563 #+srcname: task-shell-sessions
1564 #+begin_src sh :session what
1568 #+srcname: task-shell-sessions-what
1569 #+begin_src sh :session what :results replace
1573 #+resname: task-shell-sessions-what
1578 #+srcname: task-R-session
1579 #+begin_src R :session what :results replace
1585 #+resname: task-R-session
1588 #+srcname: another-task-R-session
1589 #+begin_src R :session what :results replace
1593 *** DONE function to bring up inferior-process buffer [4/4]
1595 This should be callable from inside of a source-code block in an
1596 org-mode buffer. It should evaluate the header arguments, then bring
1597 up the inf-proc buffer using =pop-to-buffer=.
1599 For lack of a better place, lets add this to the `org-metadown-hook'
1602 To give this a try, place the cursor on a source block with variables,
1603 (optionally git a prefix argument) then hold meta and press down.
1607 #+srcname: task-ruby-pop-to-session
1608 #+begin_src ruby :var num=9 :var another="something else"
1609 num.times{|n| puts another}
1614 #+srcname: task-python-pop-to-session
1615 #+begin_src python :var num=9 :var another="something else"
1620 #+srcname: task-R-pop-to-session
1621 #+begin_src R :var a=9 :var b=8
1627 #+srcname: task-shell-pop-sessions
1628 #+begin_src sh :var NAME="eric"
1632 *** DEFERRED function to dump last N lines from inf-proc buffer into the current source block
1634 Callable with a prefix argument to specify how many lines should be
1635 dumped into the source-code buffer.
1637 *** REJECTED comint notes
1639 Implementing comint integration in [[file:lisp/org-babel-comint.el][org-babel-comint.el]].
1642 - handling of outputs
1643 - split raw output from process by prompts
1644 - a ring of the outputs, buffer-local, `org-babel-comint-output-ring'
1645 - a switch for dumping all outputs to a buffer
1646 - inputting commands
1648 Lets drop all this language specific stuff, and just use
1649 org-babel-comint to split up our outputs, and return either the last
1650 value of an execution or the combination of values from the
1653 **** comint filter functions
1654 : ;; comint-input-filter-functions hook process-in-a-buffer
1655 : ;; comint-output-filter-functions hook function modes.
1656 : ;; comint-preoutput-filter-functions hook
1657 : ;; comint-input-filter function ...
1659 #+srcname: obc-filter-ruby
1660 #+begin_src ruby :results last
1668 ** DONE Remove protective commas from # comments before evaluating
1669 org inserts protective commas in front of ## comments in language
1670 modes that use them. We need to remove them prior to sending code
1673 #+srcname: testing-removal-of-protective-comas
1675 ,# this one might break it??
1679 ** DONE pass multiple reference arguments into R
1680 Can we do this? I wasn't sure how to supply multiple 'var' header
1681 args. Just delete this if I'm being dense.
1683 This should be working, see the following example...
1685 #+srcname: two-arg-example
1686 #+begin_src R :var n=2 :var m=8
1690 #+resname: two-arg-example
1693 ** DONE ensure that table ranges work
1694 when a table range is passed to org-babel as an argument, it should be
1695 interpreted as a vector.
1698 | 2 | 3 | Fixnum:1 |
1699 | 3 | 4 | Array:123456 |
1703 #+TBLFM: @1$3='(sbe simple-sbe-example (n 4))::@2$3='(sbe task-table-range (n @1$1..@6$1))::@3$3='(sbe task-table-range (n (@1$1..@6$1)))
1705 #+srcname: simple-sbe-example
1706 #+begin_src emacs-lisp
1710 #+srcname: task-table-range
1711 #+begin_src ruby :var n=simple-sbe-example
1715 #+srcname: simple-results
1716 #+begin_src emacs-lisp :var n=task-table-range(n=(1 2 3))
1720 #+resname: simple-results
1723 #+srcname: task-arr-referent
1724 #+begin_src ruby :var ar=(1 2 3)
1728 #+resname: task-arr-referent
1731 ** DONE global variable indicating default to vector output
1732 how about an alist... =org-babel-default-header-args= this may already
1733 exist... just execute the following and all source blocks will default
1736 #+begin_src emacs-lisp
1737 (setq org-babel-default-header-args '((:results . "vector")))
1740 ** DONE name named results if source block is named
1741 currently this isn't happening although it should be
1743 #+srcname: test-naming-named-source-blocks
1744 #+begin_src emacs-lisp
1748 #+resname: test-naming-named-source-blocks
1750 ** DONE (simple caching) check for named results before source blocks
1751 see the TODO comment in [[file:lisp/org-babel-ref.el::TODO%20This%20should%20explicitly%20look%20for%20resname%20lines%20before][org-babel-ref.el#org-babel-ref-resolve-reference]]
1752 ** DONE set =:results silent= when eval with prefix argument
1754 #+begin_src emacs-lisp
1757 ** DONE results-type header (vector/file) [3/3]
1758 In response to a point in Dan's email. We should allow the user to
1759 force scalar or vector results. This could be done with a header
1760 argument, and the default behavior could be controlled through a
1761 configuration variable.
1763 #+srcname: task-trivial-vector
1764 #+begin_src ruby :results replace vector
1771 since it doesn't make sense to turn a vector into a scalar, lets
1772 just add a two values...
1774 - vector :: forces the results to be a vector (potentially 1 dimensional)
1775 - file :: this throws an error if the result isn't a string, and
1776 tries to treat it as a path to a file.
1778 I'm just going to cram all of these into the =:results= header
1779 argument. Then if we allow multiple header arguments it should
1780 work out, for example one possible header argument string could be
1781 =:results replace vector file=, which would *replace* any existing
1782 results forcing the results into an org-mode table, and
1783 interpreting any strings as file paths.
1785 *** DONE multiple =:results= headers
1787 #+srcname: multiple-result-headers
1788 #+begin_src ruby :results replace silent
1794 *** DONE file result types
1795 When inserting into an org-mode buffer create a link with the path
1796 being the value, and optionally the display being the
1797 =file-name-nondirectory= if it exists.
1799 #+srcname: task-file-result
1800 #+begin_src python :results replace file
1805 [[something][something]]
1808 This will be useful because blocks like =ditaa= and =dot= can return
1809 the string path of their files, and can add =file= to their results
1812 *** DONE vector result types
1814 #+srcname: task-force-results
1815 #+begin_src emacs-lisp :results vector
1822 ** DONE results name
1823 In order to do this we will need to start naming our results.
1824 Since the source blocks are named with =#+srcname:= lines we can
1825 name results with =#+resname:= lines (if the source block has no
1826 name then no name is given to the =#+resname:= line on creation,
1827 otherwise the name of the source block is used).
1829 This will have the additional benefit of allowing results and
1830 source blocks to be located in different places in a buffer (and
1831 eventually in different buffers entirely).
1833 #+srcname: developing-resnames
1834 #+begin_src emacs-lisp :results silent
1838 Once source blocks are able to find their own =#+resname:= lines
1841 #+srcname: sbe-w-new-results
1842 #+begin_src emacs-lisp :results replace
1843 (sbe "developing-resnames")
1849 *** TODO change the results insertion functions to use these lines
1851 *** TODO teach references to resolve =#+resname= lines.
1853 ** DONE org-babel tests org-babel [1/1]
1854 since we are accumulating this nice collection of source-code blocks
1855 in the sandbox section we should make use of them as unit tests.
1856 What's more, we should be able to actually use org-babel to run these
1859 We would just need to cycle over every source code block under the
1860 sandbox, run it, and assert that the return value is equal to what we
1863 I have the feeling that this should be possible using only org-babel
1864 functions with minimal or no additional elisp. It would be very cool
1865 for org-babel to be able to test itself.
1867 This is now done, see [[* Tests]].
1869 *** DEFERRED org-babel assertions (may not be necessary)
1870 These could be used to make assertions about the results of a
1871 source-code block. If the assertion fails then the point could be
1872 moved to the block, and error messages and highlighting etc... could
1875 ** DONE make C-c C-c work anywhere within source code block?
1876 This seems like it would be nice to me, but perhaps it would be
1877 inefficient or ugly in implementation? I suppose you could search
1878 forward, and if you find #+end_src before you find #+begin_src,
1879 then you're inside one. [DED]
1881 Agreed, I think inside of the =#+srcname: line= would be useful as
1884 #+srcname: testing-out-cc
1885 #+begin_src emacs-lisp
1889 ** DONE integration with org tables
1890 We should make it easy to call org-babel source blocks from org-mode
1891 table formulas. This is practical now that it is possible to pass
1892 arguments to org-babel source blocks.
1894 See the related [[* (sandbox) integration w/org tables][sandbox]] header for tests/examples.
1896 *** digging in org-table.el
1897 In the past [[file:~/src/org/lisp/org-table.el::org%20table%20el%20The%20table%20editor%20for%20Org%20mode][org-table.el]] has proven difficult to work with.
1899 Should be a hook in [[file:~/src/org/lisp/org-table.el::defun%20org%20table%20eval%20formula%20optional%20arg%20equation][org-table-eval-formula]].
1901 Looks like I need to change this [[file:~/src/org/lisp/org-table.el::if%20lispp][if statement]] (line 2239) into a cond
1904 ** DONE source blocks as functions
1906 Allow source code blocks to be called like functions, with arguments
1907 specified. We are already able to call a source-code block and assign
1908 it's return result to a variable. This would just add the ability to
1909 specify the values of the arguments to the source code block assuming
1910 any exist. For an example see
1912 When a variable appears in a header argument, how do we differentiate
1913 between it's value being a reference or a literal value? I guess this
1914 could work just like a programming language. If it's escaped or in
1915 quotes, then we count it as a literal, otherwise we try to look it up
1918 ** DONE folding of code blocks? [2/2]
1919 [DED] In similar way to using outline-minor-mode for folding function
1920 bodies, can we fold code blocks? #+begin whatever statements are
1921 pretty ugly, and in any case when you're thinking about the overall
1922 game plan you don't necessarily want to see the code for each Step.
1924 *** DONE folding of source code block
1925 Sounds good, and wasn't too hard to implement. Code blocks should
1926 now be fold-able in the same manner as headlines (by pressing TAB
1929 *** REJECTED folding of results
1930 So, lets do a three-stage tab cycle... First fold the src block,
1931 then fold the results, then unfold.
1933 There's no way to tell if the results are a table or not w/o
1934 actually executing the block which would be too expensive of an
1937 ** DONE selective export of text, code, figures
1938 [DED] The org-babel buffer contains everything (code, headings and
1939 notes/prose describing what you're up to, textual/numeric/graphical
1940 code output, etc). However on export to html / LaTeX one might want
1941 to include only a subset of that content. For example you might
1942 want to create a presentation of what you've done which omits the
1945 [EMS] So I think this should be implemented as a property which can
1946 be set globally or on the outline header level (I need to review
1947 the mechanics of org-mode properties). And then as a source block
1948 header argument which will apply only to a specific source code
1949 block. A header argument of =:export= with values of
1951 - =code= :: just show the code in the source code block
1952 - =none= :: don't show the code or the results of the evaluation
1953 - =results= :: just show the results of the code evaluation (don't
1954 show the actual code)
1955 - =both= :: show both the source code, and the results
1957 this will be done in [[* (sandbox) selective export][(sandbox) selective export]].
1959 ** DONE a header argument specifying silent evaluation (no output)
1960 This would be useful across all types of source block. Currently
1961 there is a =:replace t= option to control output, this could be
1962 generalized to an =:output= option which could take the following
1963 options (maybe more)
1965 - =t= :: this would be the default, and would simply insert the
1966 results after the source block
1967 - =replace= :: to replace any results which may already be there
1968 - =silent= :: this would inhibit any insertion of the results
1970 This is now implemented see the example in the [[* silent evaluation][sandbox]]
1972 ** DONE assign variables from tables in R
1973 This is now working (see [[* (sandbox table) R][(sandbox-table)-R]]). Although it's not that
1974 impressive until we are able to print table results from R.
1976 ** DONE insert 2-D R results as tables
1977 everything is working but R and shell
1983 This has already been tackled by Dan in [[file:existing_tools/org-R.el::defconst%20org%20R%20write%20org%20table%20def][org-R:check-dimensions]]. The
1984 functions there should be useful in combination with [[http://cran.r-project.org/doc/manuals/R-data.html#Export-to-text-files][R-export-to-csv]]
1985 as a means of converting multidimensional R objects to emacs lisp.
1987 It may be as simple as first checking if the data is multidimensional,
1988 and then, if so using =write= to write the data out to a temporary
1989 file from which emacs can read the data in using =org-table-import=.
1991 Looking into this further, is seems that there is no such thing as a
1992 scalar in R [[http://tolstoy.newcastle.edu.au/R/help/03a/3733.html][R-scalar-vs-vector]] In that light I am not sure how to
1993 deal with trivial vectors (scalars) in R. I'm tempted to just treat
1994 them as vectors, but then that would lead to a proliferation of
1995 trivial 1-cell tables...
1997 ** DONE allow variable initialization from source blocks
1998 Currently it is possible to initialize a variable from an org-mode
1999 table with a block argument like =table=sandbox= (note that the
2000 variable doesn't have to named =table=) as in the following example
2006 #+begin_src emacs-lisp :var table=sandbox :results replace
2007 (message (format "table = %S" table))
2010 : "table = ((1 2 3) (4 \"schulte\" 6))"
2012 It would be good to allow initialization of variables from the results
2013 of other source blocks in the same manner. This would probably
2014 require the addition of =#+SRCNAME: example= lines for the naming of
2015 source blocks, also the =table=sandbox= syntax may have to be expanded
2016 to specify whether the target is a source code block or a table
2017 (alternately we could just match the first one with the given name
2018 whether it's a table or a source code block).
2020 At least initially I'll try to implement this so that there is no need
2021 to specify whether the reference is to a table or a source-code block.
2022 That seems to be simpler both in terms of use and implementation.
2024 This is now working for emacs-lisp, ruby and python (and mixtures of
2025 the three) source blocks. See the examples in the [[* (sandbox) referencing other source blocks][sandbox]].
2027 This is currently working only with emacs lisp as in the following
2028 example in the [[* emacs lisp source reference][emacs lisp source reference]].
2031 ** TODO Add languages [1/6]
2032 I'm sure there are many more that aren't listed here. Please add
2033 them, and bubble any that you particularly care about up to the top.
2035 Any new language should be implemented in a org-babel-lang.el file.
2036 Follow the pattern set by [[file:lisp/org-babel-script.el][org-babel-script.el]], [[file:lisp/org-babel-shell.el][org-babel-shell.el]] and
2037 [[file:lisp/org-babel-R.el][org-babel-R.el]].
2040 This could probably be added to [[file:lisp/org-babel-script.el][org-babel-script.el]]
2045 (see [[* file result types][file result types]])
2047 #+srcname: implementing-ditaa
2048 #+begin_src ditaa :results replace :file blue.png :cmdline -r
2059 [[file:blue.png][blue.png]]
2062 (see [[* file result types][file result types]])
2064 - a =file= header argument
2065 - a =cmdline= header argument
2067 - scalar variables should be replaced in the body of the gnuplot code
2068 - vector variables should be exported to tab-separated files, and
2069 the variable names should be replaced with the path to the files
2071 #+PLOT: title:"Citas" ind:1 deps:(3) type:2d with:histograms set:"yrange [0:]"
2072 #+TBLNAME: gnuplot-data
2073 | independent var | first dependent var | second dependent var |
2074 |-----------------+---------------------+----------------------|
2075 | 0.1 | 0.425 | 0.375 |
2076 | 0.2 | 0.3125 | 0.3375 |
2077 | 0.3 | 0.24999993 | 0.28333338 |
2078 | 0.4 | 0.275 | 0.28125 |
2079 | 0.5 | 0.26 | 0.27 |
2080 | 0.6 | 0.25833338 | 0.24999993 |
2081 | 0.7 | 0.24642845 | 0.23928553 |
2082 | 0.8 | 0.23125 | 0.2375 |
2083 | 0.9 | 0.23333323 | 0.2333332 |
2084 | 1 | 0.2225 | 0.22 |
2085 | 1.1 | 0.20909075 | 0.22272708 |
2086 | 1.2 | 0.19999998 | 0.21458333 |
2087 | 1.3 | 0.19615368 | 0.21730748 |
2089 #+srcname: implementing-gnuplot
2090 #+begin_src gnuplot :var data=gnuplot-data
2091 set title "Implementing Gnuplot"
2092 plot "data" using 1:2 with lines
2096 (see [[* file result types][file result types]])
2099 (see [[* file result types][file result types]])
2103 ** TODO o-b-execute-subtree overwrites some buffer contents
2104 *** Try M-x org-babel-execute-subtree with point at the beginning of this line
2109 ** TODO Allow source blocks to be recognised when #+ are not first characters on the line
2110 I think Carsten has recently altered the core so that #+ can have
2111 preceding whitespace, at least for literal/code examples. org-babel
2112 should support this.
2114 ** TODO non-orgtbl formatted lists
2117 #+srcname: this-doesn't-match-orgtbl
2118 #+begin_src emacs-lisp :results replace
2119 '((:results . "replace"))
2122 #+resname: this-doesn't-match-orgtbl
2124 ** TODO collapsing consecutive newlines in string output
2126 #+srcname: multi-line-string-output
2127 #+begin_src ruby :results output
2128 "the first line ends here
2131 and this is the second one
2136 #+resname: multi-line-string-output
2138 ** TODO cursor movement when evaluating source blocks
2139 E.g. the pie chart example. Despite the save-window-excursion in
2140 org-babel-execute:R. (I never learned how to do this properly: org-R
2141 jumps all over the place...)
2143 ** PROPOSED external shell execution can't isolate return values
2144 I have no idea how to do this as of yet. The result is that when
2145 shell functions are run w/o a session there is no difference between
2146 the =output= and =value= result arguments.
2148 Yea, I don't know how to do this either. I searched extensively on
2149 how to isolate the *last* output of a series of shell commands (see
2150 [[* last command for
2151 shells][last command for shells]]). The results of the search were basically
2152 that it was not possible (or at least not accomplish-able with a
2153 reasonable amount of effort).
2155 That fact combined with the tenancy to all ways use standard out in
2156 shell scripts led me to treat these two options (=output= and =value=)
2157 as identical in shell evaluation. Not ideal but maybe good enough for
2160 In the `results' branch I've changed this so that they're not quite
2161 identical: output results in raw stdout contents, whereas value
2162 converts it to elisp, perhaps to a table if it looks tabular. This is
2163 the same for the other languages. [Dan]
2165 ** TODO are the org-babel-trim s necessary?
2166 at the end of e.g. org-babel-R-evaluate, org-babel-python-evaluate, but
2167 not org-babel-ruby-evaluate
2168 ** TODO elisp reference fails for literal number
2169 #+srcname: elisp-test(a=4)
2170 #+begin_src emacs-lisp
2173 ** TODO use new merge function [[file:lisp/org-babel-ref.el::t%20nil%20org%20combine%20plists%20args%20nil][here]]?
2174 And at other occurrences of org-combine-plists?
2175 ** TODO LoB: with output to buffer, not working in buffers other than library-of-babel.org
2176 I haven't fixed this yet. org-babel-ref-resolve-reference moves
2177 point around, inside a save-excursion. Somehow when it comes to
2178 inserting the results (after possible further recursive calls to
2179 org-babel-ref-resolve-reference), point hasn't gone back to the
2181 ** TODO LoB: output to buffer adds creeping blank lines
2182 Compare the results of
2183 #+lob: python-add(a=5, b=17)
2185 #+resname: python-add(a=5, b=17)
2187 --------------------------------
2195 ---------------------
2197 Hmm, it's a bit confusing. I think it's to do with the fact that
2198 LoB removes the entire (#+resname and result) and starts from
2199 scratch, whereas #+begin_src only removes the result. I haven't
2200 worked out what the correct fix is yet.
2201 ** DEFERRED weird escaped characters in shell prompt break shell evaluation
2202 E.g. this doesn't work. Should the shell sessions set a sane prompt
2203 when they start up? Or is it a question of altering
2204 comint-prompt-regexp? Or altering org-babel regexps?
2207 black=30 ; red=31 ; green=32 ; yellow=33 ; blue=34 ; magenta=35 ; cyan=36 ; white=37
2210 export PS1="\[\033[${prompt_col}m\]\w${prompt_char} \[\033[0m\]"
2213 I just pushed a good amount of changes, could you see if your shell
2214 problems still exist?
2216 The problem's still there. Specifically, aIui, at [[file:lisp/langs/org-babel-sh.el::raw%20org%20babel%20comint%20with%20output%20buffer%20org%20babel%20sh%20eoe%20output%20nil%20insert%20full%20body%20comint%20send%20input%20nil%20t][this line]] of
2217 org-babel-sh.el, raw gets the value
2219 ("" "
\e[0m Sun Jun 14 19:26:24 EDT 2009\n" "
\e[0m org_babel_sh_eoe\n" "
\e[0m ")
2221 and therefore (member org-babel-sh-eoe-output ...) fails
2223 I think that `comint-prompt-regexp' needs to be altered to match
2224 the shell prompt. This shouldn't be too difficult to do by hand,
2225 using the `regexp-builder' command and should probably be part of
2226 the user's regular emacs init. I can't think of a way for us to
2227 set this automatically, and we are SOL without a regexp to match
2230 ** DONE LoB: calls fail if reference has single character name
2231 commit 21d058869df1ff23f4f8cc26f63045ac9c0190e2
2232 **** This doesn't work
2233 #+lob: R-plot(data=X)
2252 #+lob: R-plot(data=XX)
2254 ** DONE make :results replace the default?
2255 I'm tending to think that appending results to pre-existing results
2256 creates mess, and that the cleaner `replace' option should be the
2257 default. E.g. when a source block creates an image, we would want
2258 that to be updated, rather than have a new one be added.
2262 ** DONE ruby evaluation not working under ubuntu emacs 23
2263 With emacs 23.0.91.1 on ubuntu, for C-h f run-ruby I have the
2264 following, which seems to conflict with [[file:lisp/langs/org-babel-ruby.el::let%20session%20buffer%20save%20window%20excursion%20run%20ruby%20nil%20session%20current%20buffer][this line]] in org-babel-ruby.el.
2267 run-ruby is an interactive compiled Lisp function.
2271 Run an inferior Ruby process, input and output via buffer *ruby*.
2272 If there is a process already running in `*ruby*', switch to that buffer.
2273 With argument, allows you to edit the command line (default is value
2274 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
2275 (after the `comint-mode-hook' is run).
2276 (Type C-h m in the process buffer for a list of commands.)
2279 So, I may have a non-standard inf-ruby.el. Here's my version of
2283 run-ruby is an interactive Lisp function in `inf-ruby.el'.
2285 (run-ruby &optional COMMAND NAME)
2287 Run an inferior Ruby process, input and output via buffer *ruby*.
2288 If there is a process already running in `*ruby*', switch to that buffer.
2289 With argument, allows you to edit the command line (default is value
2290 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
2291 (after the `comint-mode-hook' is run).
2292 (Type C-h m in the process buffer for a list of commands.)
2295 It seems we could either bundle my version of inf-ruby.el (as it's
2296 the newest). Or we could change the use of `run-ruby' so that it
2297 is robust across multiple distributions. I think I'd prefer the
2298 former, unless the older version of inf-ruby is actually bundled
2299 with emacs, in which case maybe we should go out of our way to
2300 support it. Thoughts?
2302 I think for now I'll just include the latest [[file:util/inf-ruby.el][inf-ruby.el]] in the
2303 newly created utility directory. I doubt anyone would have a
2304 problem using the latest version of this file.
2305 ** DONE test failing forcing vector results with =test-forced-vector-results= ruby code block
2306 Note that this only seems to happen the *second* time the test table
2309 #+srcname: bug-trivial-vector
2310 #+begin_src emacs-lisp :results vector silent
2314 #+srcname: bug-forced-vector-results
2315 #+begin_src ruby :var triv=test-trivial-vector :results silent
2319 mysteriously this seems to be fixed...
2320 ** DONE defunct R sessions
2321 Sometimes an old R session will turn defunct, and newly inserted code
2322 will not be evaluated (leading to a hang).
2324 This seems to be fixed by using `inferior-ess-send-input' rather than `comint-send-input'.
2325 ** DONE ruby fails on first call to non-default session
2327 #+srcname: bug-new-session
2328 #+begin_src ruby :session is-new
2332 ** DONE when reading results from =#+resname= line
2334 Errors when trying to read from resname lines.
2336 #+resname: bug-in-resname
2339 #+srcname: bug-in-resname-reader
2340 #+begin_src emacs-lisp :var buggy=bug-in-resname() :results silent
2344 ** DONE R-code broke on "org-babel" rename
2346 #+srcname: bug-R-babels
2351 ** DONE error on trivial R results
2353 So I know it's generally not a good idea to squash error without
2354 handling them, but in this case the error almost always means that
2355 there was no file contents to be read by =org-table-import=, so I
2358 #+srcname: bug-trivial-r1
2359 #+begin_src R :results replace
2360 pie(c(1, 2, 3), labels = c(1, 2, 3))
2363 #+srcname: bug-trivial-r2
2364 #+begin_src R :results replace
2368 #+resname: bug-trivial-r2
2371 #+srcname: bug-trivial-r3
2372 #+begin_src R :results replace
2376 #+resname: bug-trivial-r3
2381 ** DONE ruby new variable creation (multi-line ruby blocks)
2382 Actually it looks like we were dropping all but the last line.
2384 #+srcname: multi-line-ruby-test
2385 #+begin_src ruby :var table=bug-numerical-table :results replace
2387 table.each{|n| total += n}
2394 ** DONE R code execution seems to choke on certain inputs
2395 Currently the R code seems to work on vertical (but not landscape)
2398 #+srcname: little-fake
2399 #+begin_src emacs-lisp
2404 #+begin_src R :var num=little-fake
2411 #+srcname: set-debug-on-error
2412 #+begin_src emacs-lisp :results silent
2413 (setq debug-on-error t)
2416 #+srcname: bug-numerical-table
2417 #+begin_src emacs-lisp :results silent
2424 #+srcname: bug-R-number-evaluation
2425 #+begin_src R :var table=bug-numerical-table
2434 #+tblname: bug-vert-table
2439 #+srcname: bug-R-vertical-table
2440 #+begin_src R :var table=bug-vert-table :results silent
2444 ** DONE org bug/request: prevent certain org behaviour within code blocks
2445 E.g. [[]] gets recognised as a link (when there's text inside the
2446 brackets). This is bad for R code at least, and more generally
2447 could be argued to be inappropriate. Is it difficult to get org to
2448 ignore text in code blocks? [DED]
2450 I believe Carsten addressed this recently on the mailing list with
2451 the comment that it was indeed a difficult issue. I believe this
2452 may be one area where we could wait for an upstream (org-mode) fix.
2454 [Dan] Carsten has fixed this now in the core.
2456 ** DONE with :results replace, non-table output doesn't replace table output
2457 And vice versa. E.g. Try this first with table and then with len(table) [DED]
2458 #+begin_src python :var table=sandbox :results replace
2463 | 4 | "schulte" | 6 |
2466 Yes, this is certainly a problem. I fear that if we begin replacing
2467 anything immediately following a source block (regardless of whether
2468 it matches the type of our current results) we may accidentally delete
2469 hand written portions of the user's org-mode buffer.
2471 I think that the best solution here would be to actually start
2472 labeling results with a line that looks something like...
2476 This would have a couple of benefits...
2477 1) we wouldn't have to worry about possibly deleting non-results
2478 (which is currently an issue)
2479 2) we could reliably replace results even if there are different types
2480 3) we could reference the results of a source-code block in variable
2481 definitions, which would be useful if for example we don't wish to
2482 re-run a source-block every time because it is long-running.
2484 Thoughts? If no-one objects, I believe I will implement the labeling
2487 ** DONE extra quotes for nested string
2488 Well R appears to be reading the tables without issue...
2490 these *should* be quoted
2492 #+begin_src sh :results replace
2497 | "README.markdown" |
2500 | "existing_tools" |
2504 | "test-export.html" |
2505 | "test-export.org" |
2507 #+srcname: test-quotes
2508 #+begin_src ruby :var tab=ls
2514 #+srcname: test-quotes
2515 #+begin_src R :var tab=ls
2521 ** DONE simple ruby arrays not working
2523 As an example eval the following. Adding a line to test
2525 #+tblname: simple-ruby-array
2528 #+srcname: ruby-array-test
2529 #+begin_src ruby :var ar = simple-ruby-array :results silent
2533 ** DONE space trailing language name
2534 fix regexp so it works when there's a space trailing the language name
2536 #+srcname: test-trailing-space
2541 ** DONE Args out of range error
2543 The following block resulted in the error below [DED]. It ran without
2544 error directly in the shell.
2547 for platf in ill aff ; do
2548 for pop in CEU YRI ASI ; do
2549 rm -f $platf/hapmap-genos-$pop-all $platf/hapmap-rs-all
2550 cat $platf/hapmap-genos-$pop-* > $platf/hapmap-genos-$pop-all
2551 cat $platf/hapmap-rs-* > $platf/hapmap-rs-all
2556 executing source block with sh...
2557 finished executing source block
2558 string-equal: Args out of range: "", -1, 0
2560 the error =string-equal: Args out of range: "", -1, 0= looks like what
2561 used to be output when the block returned an empty results string.
2562 This should be fixed in the current version, you should now see the
2563 following message =no result returned by source block=.
2565 ** DONE ruby arrays not recognized as such
2567 Something is wrong in [[file:lisp/org-babel-script.el]] related to the
2568 recognition of ruby arrays as such.
2570 #+begin_src ruby :results replace
2576 #+begin_src python :results replace
2588 Evaluate all the cells in this table for a comprehensive test of the
2589 org-babel functionality.
2591 *Note*: if you have customized =org-babel-default-header-args= then some
2592 of these tests may fail.
2594 #+TBLNAME: org-babel-tests
2595 | functionality | block | arg | expected | results | pass |
2596 |-------------------------+----------------------------+-----+-------------+-------------+------|
2597 | basic evaluation | | | | | pass |
2598 |-------------------------+----------------------------+-----+-------------+-------------+------|
2599 | emacs lisp | basic-elisp | | 5 | 5 | pass |
2600 | shell | basic-shell | | 6 | 6 | pass |
2601 | ruby | basic-ruby | | org-babel | org-babel | pass |
2602 | python | basic-python | | hello world | hello world | pass |
2603 | R | basic-R | | 13 | 13 | pass |
2604 |-------------------------+----------------------------+-----+-------------+-------------+------|
2605 | tables | | | | | pass |
2606 |-------------------------+----------------------------+-----+-------------+-------------+------|
2607 | emacs lisp | table-elisp | | 3 | 3 | pass |
2608 | ruby | table-ruby | | 1-2-3 | 1-2-3 | pass |
2609 | python | table-python | | 5 | 5 | pass |
2610 | R | table-R | | 3.5 | 3.5 | pass |
2611 |-------------------------+----------------------------+-----+-------------+-------------+------|
2612 | source block references | | | | | pass |
2613 |-------------------------+----------------------------+-----+-------------+-------------+------|
2614 | all languages | chained-ref-last | | Array | Array | pass |
2615 |-------------------------+----------------------------+-----+-------------+-------------+------|
2616 | source block functions | | | | | pass |
2617 |-------------------------+----------------------------+-----+-------------+-------------+------|
2618 | emacs lisp | defun-fibb | | fibbd | fibbd | pass |
2619 | run over | Fibonacci | 0 | 1 | 1 | pass |
2620 | a | Fibonacci | 1 | 1 | 1 | pass |
2621 | variety | Fibonacci | 2 | 2 | 2 | pass |
2622 | of | Fibonacci | 3 | 3 | 3 | pass |
2623 | different | Fibonacci | 4 | 5 | 5 | pass |
2624 | arguments | Fibonacci | 5 | 8 | 8 | pass |
2625 |-------------------------+----------------------------+-----+-------------+-------------+------|
2626 | bugs and tasks | | | | | pass |
2627 |-------------------------+----------------------------+-----+-------------+-------------+------|
2628 | simple ruby arrays | ruby-array-test | | 3 | 3 | pass |
2629 | R number evaluation | bug-R-number-evaluation | | 2 | 2 | pass |
2630 | multi-line ruby blocks | multi-line-ruby-test | | 2 | 2 | pass |
2631 | forcing vector results | test-forced-vector-results | | Array | Array | pass |
2632 |-------------------------+----------------------------+-----+-------------+-------------+------|
2633 | sessions | | | | | pass |
2634 |-------------------------+----------------------------+-----+-------------+-------------+------|
2635 | set ruby session | set-ruby-session-var | | :set | :set | pass |
2636 | get from ruby session | get-ruby-session-var | | 3 | 3 | pass |
2637 | set python session | set-python-session-var | | set | set | pass |
2638 | get from python session | get-python-session-var | | 4 | 4 | pass |
2639 | set R session | set-R-session-var | | set | set | pass |
2640 | get from R session | get-R-session-var | | 5 | 5 | pass |
2641 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
2642 #+TBLFM: $5=""::$6=""
2644 The second line (followed by replacing '[]' with '') can be used to blank out the table results, in the absence of a better method.
2648 #+srcname: basic-elisp
2649 #+begin_src emacs-lisp :results silent
2654 #+srcname: basic-shell
2655 #+begin_src sh :results silent
2660 #+srcname: date-simple
2661 #+begin_src sh :results silent
2665 #+srcname: basic-ruby
2666 #+begin_src ruby :results silent
2671 #+srcname: basic-python
2672 #+begin_src python :results silent
2678 #+begin_src R :results silent
2686 #+tblname: test-table
2690 #+srcname: table-elisp
2691 #+begin_src emacs-lisp :results silent :var table=test-table
2692 (length (car table))
2696 #+srcname: table-ruby
2697 #+begin_src ruby :results silent :var table=test-table
2698 table.first.join("-")
2702 #+srcname: table-python
2703 #+begin_src python :var table=test-table
2709 #+begin_src R :var table=test-table
2716 Lets pass a references through all of our languages...
2718 Lets start by reversing the table from the previous examples
2720 #+srcname: chained-ref-first
2721 #+begin_src python :var table = test-table
2726 #+resname: chained-ref-first
2730 Take the first part of the list
2732 #+srcname: chained-ref-second
2733 #+begin_src R :var table = chained-ref-first
2737 #+resname: chained-ref-second
2741 Turn the numbers into string
2743 #+srcname: chained-ref-third
2744 #+begin_src emacs-lisp :var table = chained-ref-second
2745 (mapcar (lambda (el) (format "%S" el)) table)
2748 #+resname: chained-ref-third
2751 and Check that it is still a list
2753 #+srcname: chained-ref-last
2754 #+begin_src ruby :var table=chained-ref-third
2759 ** source blocks as functions
2761 #+srcname: defun-fibb
2762 #+begin_src emacs-lisp :results silent
2763 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
2767 #+srcname: fibonacci
2768 #+begin_src emacs-lisp :results silent :var n=7
2778 ** sbe tests (these don't seem to be working...)
2779 Testing the insertion of results into org-mode tables.
2781 #+srcname: multi-line-output
2782 #+begin_src ruby :results replace
2783 "the first line ends here
2786 and this is the second one
2792 : the first line ends here\n\n\n and this is the second one\n\neven a third
2794 #+srcname: multi-line-error
2795 #+begin_src ruby :results replace
2796 raise "oh nooooooooooo"
2802 | the first line ends here... | -:5: warning: parenthesize argument(s) for future version... |
2803 #+TBLFM: $1='(sbe "multi-line-output")::$2='(sbe "multi-line-error")
2805 ** forcing results types tests
2807 #+srcname: test-trivial-vector
2808 #+begin_src emacs-lisp :results vector silent
2812 #+srcname: test-forced-vector-results
2813 #+begin_src ruby :var triv=test-trivial-vector :results silent
2819 #+srcname: set-ruby-session-var
2820 #+begin_src ruby :session :results silent
2825 #+srcname: get-ruby-session-var
2826 #+begin_src ruby :session :results silent
2830 #+srcname: set-python-session-var
2831 #+begin_src python :session
2836 #+srcname: get-python-session-var
2837 #+begin_src python :session
2841 #+srcname: set-R-session-var
2842 #+begin_src R :session
2847 #+srcname: get-R-session-var
2848 #+begin_src R :session
2857 To run these examples evaluate [[file:lisp/org-babel-init.el][org-babel-init.el]]
2859 ** org-babel.el beginning functionality
2861 #+begin_src sh :results replace
2866 : Sun Jul 5 18:54:39 EDT 2009
2873 : Sun Jul 05 18:54:35 -0400 2009
2885 #+begin_src R :results replace
2895 hist(rgamma(20,3,3))
2900 ** org-babel plays with tables
2901 Alright, this should demonstrate both the ability of org-babel to read
2902 tables into a lisp source code block, and to then convert the results
2903 of the source code block into an org table. It's using the classic
2904 "lisp is elegant" demonstration transpose function. To try this
2907 1. evaluate [[file:lisp/org-babel-init.el]] to load org-babel and friends
2908 2. evaluate the transpose definition =\C-c\\C-c= on the beginning of
2910 3. evaluate the next source code block, this should read in the table
2911 because of the =:var table=previous=, then transpose the table, and
2912 finally it should insert the transposed table into the buffer
2913 immediately following the block
2917 #+begin_src emacs-lisp :results silent
2918 (defun transpose (table)
2919 (apply #'mapcar* #'list table))
2927 #+begin_src emacs-lisp :var table=sandbox :results replace
2932 #+begin_src emacs-lisp
2937 | 1 | 2 | 3 | 4 | 5 |
2941 #+begin_src ruby :var table=sandbox :results replace
2942 table.first.join(" - ")
2948 #+begin_src python :var table=sandbox
2953 #+begin_src ruby :var table=sandbox :results replace
2958 : [[1, 2, 3], [4, "schulte", 6]]
2962 | 4 | "schulte" | 6 |
2964 #+begin_src python :var table=sandbox :results replace
2970 | "__add__" | "__class__" | "__contains__" | "__delattr__" | "__delitem__" | "__delslice__" | "__doc__" | "__eq__" | "__format__" | "__ge__" | "__getattribute__" | "__getitem__" | "__getslice__" | "__gt__" | "__hash__" | "__iadd__" | "__imul__" | "__init__" | "__iter__" | "__le__" | "__len__" | "__lt__" | "__mul__" | "__ne__" | "__new__" | "__reduce__" | "__reduce_ex__" | "__repr__" | "__reversed__" | "__rmul__" | "__setattr__" | "__setitem__" | "__setslice__" | "__sizeof__" | "__str__" | "__subclasshook__" | "append" | "count" | "extend" | "index" | "insert" | "pop" | "remove" | "reverse" | "sort" |
2972 *** (sandbox table) R
2974 #+TBLNAME: sandbox_r
2978 #+begin_src R :results replace
2979 x <- c(rnorm(10, mean=-3, sd=1), rnorm(10, mean=3, sd=1))
2983 | -3.35473133869346 |
2985 | -3.32819924928633 |
2986 | -2.97310212756194 |
2987 | -2.09640758369576 |
2988 | -5.06054014378736 |
2989 | -2.20713700711221 |
2990 | -1.37618039712037 |
2991 | -1.95839385821742 |
2992 | -3.90407396475502 |
2993 | 2.51168071590226 |
2994 | 3.96753011570494 |
2995 | 3.31793212627865 |
2996 | 1.99829753972341 |
2997 | 4.00403686419829 |
2998 | 4.63723764452927 |
2999 | 3.94636744261313 |
3000 | 3.58355906547775 |
3001 | 3.01563442274226 |
3004 #+begin_src R var tabel=sandbox_r :results replace
3009 | 4 | "schulte" | 6 |
3012 Now shell commands are converted to tables using =org-table-import=
3013 and if these tables are non-trivial (i.e. have multiple elements) then
3014 they are imported as org-mode tables...
3016 #+begin_src sh :results replace
3020 | "total" | 208 | "" | "" | "" | "" | "" | "" |
3021 | "-rw-r--r--" | 1 | "dan" | "dan" | 57 | 2009 | 15 | "block" |
3022 | "-rw-r--r--" | 1 | "dan" | "dan" | 35147 | 2009 | 15 | "COPYING" |
3023 | "-rw-r--r--" | 1 | "dan" | "dan" | 722 | 2009 | 18 | "examples.org" |
3024 | "drwxr-xr-x" | 4 | "dan" | "dan" | 4096 | 2009 | 19 | "existing_tools" |
3025 | "-rw-r--r--" | 1 | "dan" | "dan" | 2207 | 2009 | 14 | "intro.org" |
3026 | "drwxr-xr-x" | 2 | "dan" | "dan" | 4096 | 2009 | 18 | "org-babel" |
3027 | "-rw-r--r--" | 1 | "dan" | "dan" | 277 | 2009 | 20 | "README.markdown" |
3028 | "-rw-r--r--" | 1 | "dan" | "dan" | 11837 | 2009 | 18 | "rorg.html" |
3029 | "-rw-r--r--" | 1 | "dan" | "dan" | 61829 | 2009 | 19 | "#rorg.org#" |
3030 | "-rw-r--r--" | 1 | "dan" | "dan" | 60190 | 2009 | 19 | "rorg.org" |
3031 | "-rw-r--r--" | 1 | "dan" | "dan" | 972 | 2009 | 11 | "test-export.org" |
3034 ** silent evaluation
3042 #+begin_src ruby :results silent
3046 #+begin_src ruby :results replace
3053 ** (sandbox) referencing other source blocks
3054 Doing this in emacs-lisp first because it's trivial to convert
3055 emacs-lisp results to and from emacs-lisp.
3057 *** emacs lisp source reference
3058 This first example performs a calculation in the first source block
3059 named =top=, the results of this calculation are then saved into the
3060 variable =first= by the header argument =:var first=top=, and it is
3061 used in the calculations of the second source block.
3064 #+begin_src emacs-lisp
3068 #+begin_src emacs-lisp :var first=top :results replace
3074 This example is the same as the previous only the variable being
3075 passed through is a table rather than a number.
3077 #+begin_src emacs-lisp :results silent
3078 (defun transpose (table)
3079 (apply #'mapcar* #'list table))
3082 #+TBLNAME: top_table
3086 #+SRCNAME: second_src_example
3087 #+begin_src emacs-lisp :var table=top_table
3091 #+begin_src emacs-lisp :var table=second_src_example :results replace
3096 | 4 | "schulte" | 6 |
3098 Now working for ruby
3105 #+begin_src ruby :var other=start :results replace
3111 #+SRCNAME: start_two
3116 #+begin_src python :var another=start_two :results replace
3121 Since all variables are converted into Emacs Lisp it is no problem to
3122 reference variables specified in another language.
3124 #+SRCNAME: ruby-block
3129 #+SRCNAME: lisp_block
3130 #+begin_src emacs-lisp :var ruby-variable=ruby-block
3134 #+begin_src python :var lisp_var=lisp_block
3143 #+begin_src R :results replace
3150 #+begin_src R :var other=first_r :results replace
3157 ** (sandbox) selective export
3159 For exportation tests and examples see (including exportation of
3160 inline source code blocks) [[file:test-export.org]]
3163 ** (sandbox) source blocks as functions
3166 #+begin_src emacs-lisp :results silent
3171 #+begin_src emacs-lisp :var n=default :results replace
3177 #+begin_src emacs-lisp :var result=triple(n=3, m=98) :results replace
3183 The following just demonstrates the ability to assign variables to
3184 literal values, which was not implemented until recently.
3186 #+begin_src ruby :var num="eric" :results replace
3193 ** (sandbox) inline source blocks
3195 This is an inline source code block src_ruby{1 + 6}. And another
3196 source block with text output src_emacs-lisp{"eric"}.
3198 This is an inline source code block with header
3199 arguments. src_ruby[:var n=fibbd( n = 0 )]{n}
3202 ** (sandbox) integration w/org tables
3204 #+begin_src emacs-lisp :results silent
3205 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
3209 #+begin_src emacs-lisp :var n=4 :results silent
3213 #+begin_src emacs-lisp :results silent
3214 (mapcar #'fibbd '(0 1 2 3 4 5 6 7 8))
3217 Something is not working here. The function `sbe ' works fine when
3218 called from outside of the table (see the source block below), but
3219 produces an error when called from inside the table. I think there
3220 must be some narrowing going on during intra-table emacs-lisp
3223 | original | fibbd |
3224 |----------+-------|
3235 #+TBLFM: $2='(sbe "fibbd" (n $1))
3239 #+begin_src emacs-lisp :results silent
3240 (sbe 'fibbd (n "8"))
3245 LocalWords: DBlocks dblocks org-babel el eric fontification