1 #+TITLE: org-babel --- facilitating communication between programming languages and people
2 #+SEQ_TODO: PROPOSED TODO STARTED | DONE DEFERRED REJECTED
3 #+OPTIONS: H:3 num:nil toc:t
4 #+STARTUP: oddeven hideblocks
6 Through Org-Babel Org-Mode can communicate with programming languages.
7 Code contained in source-code blocks can be evaluated and data can
8 pass seamlessly between different programming languages, Org-Mode
9 constructs (tables, file links, example text) and interactive comint
13 - The [[* Introduction][Introduction]] :: provides a brief overview of the design and use
14 of Org-Babel including tutorials and examples.
15 - In [[* Getting started][Getting Started]] :: find instructions for installing org-babel
16 into your emacs configuration.
17 - The [[* Tasks][Tasks]] :: section contains current and past tasks roughly ordered
18 by TODO state, then importance or date-completed. This would be
19 a good place to suggest ideas for development.
20 - The [[* Bugs][Bugs]] :: section contains bug reports.
21 - The [[* Tests][Tests]] :: section consists of a large table which can be
22 evaluated to run Org-Babel's functional test suite. This
23 provides a good overview of the current functionality with
24 pointers to example source blocks.
25 - The [[* Sandbox][Sandbox]] :: demonstrates much of the early/basic functionality
26 through commented source-code blocks.
28 Also see the [[file:library-of-babel.org][Library of Babel]], an extensible collection of ready-made
29 and easily-shortcut-callable source-code blocks for handling common
34 Org-Babel enables *communication* between programming languages and
38 - communication between programs :: Data passes seamlessly between
39 different programming languages, Org-Mode constructs (tables,
40 file links, example text) and interactive comint buffers.
41 - communication between people :: Data and calculations are embedded
42 in the same document as notes explanations and reports.
44 ** communication between programs
46 Org-Mode supports embedded blocks of source code (in any language)
47 inside of Org documents. Org-Babel allows these blocks of code to be
48 executed from within Org-Mode with natural handling of their inputs
52 with both scalar, file, and table output
54 *** reading information from tables
56 *** reading information from other source blocks (disk usage in your home directory)
58 This will work for Linux and Mac users, not so sure about shell
59 commands for windows users.
61 To run place the cursor on the =#+begin_src= line of the source block
62 labeled directory-pie and press =\C-c\C-c=.
64 #+srcname: directories
65 #+begin_src bash :results replace
66 cd ~ && du -sc * |grep -v total
69 #+resname: directories
71 | 11882808 | "Documents" |
72 | 8210024 | "Downloads" |
73 | 879800 | "Library" |
76 | 5307664 | "Pictures" |
85 #+srcname: directory-pie
86 #+begin_src R :var dirs = directories
87 pie(dirs[,1], labels = dirs[,2])
90 *** operations in/on tables
92 #+tblname: grades-table
93 | student | grade | letter |
94 |---------+-------+--------|
101 #+TBLFM: $2='(sbe random-score-generator)::$3='(sbe assign-grade (score $2))
103 #+srcname: assign-grade
104 #+begin_src ruby :var score=99
115 #+srcname: random-score-generator
120 #+srcname: show-distribution
121 #+begin_src R :var grades=grades-table
126 ** communication between people
127 Quick overview of Org-Mode's exportation abilities, with links to the
128 online Org-Mode documentation, a focus on source-code blocks, and the
129 exportation options provided by Org-Babel.
131 *** Interactive tutorial
132 This would demonstrate applicability to Reproducible Research, and
133 Literate Programming.
135 *** Tests embedded in documentation
136 org-babels own functional tests are contained in a large org-mode
137 table, allowing the test suite to be run be evaluation of the table
138 and the results to be collected in the same table.
140 *** Emacs initialization files stored in Org-Mode buffers
141 Once org-babel-tangle is completed this could be a very compelling use case.
146 *** code evaluation (comint buffer sessions and external processes)
147 There are two main ways to evaluate source blocks with org-babel.
149 - external :: By default (if the =:session= header argument is not
150 present) all source code blocks are evaluated in
151 external processes. In these cases an external process
152 is used to evaluate the source-code blocks.
153 - session :: Session based evaluation uses persistent sessions in
154 comint buffers. Sessions can be used across multiple
155 source blocks setting and accessing variables in the
158 Evaluating source blocks in sessions also allows for
159 interaction with the code. To jump to the session of a
160 source block use the `org-babel-pop-to-session' command
161 or press =M-[down]= while inside of a source code block.
162 When called with a prefix argument
163 `org-babel-pop-to-session' will evaluate all header
164 arguments before jumping to the source-code block.
166 *** results (values and outputs)
167 Either the *value* or the *output* of source code blocks can be
168 collected after evaluation.
170 - value :: The default way to collect results from a source-code block
171 is to return the value of the last statement in the block.
172 This can be thought of as the return value of the block.
173 In this case any printed output of the block is ignored.
174 This can be though of a similar to a "functional" value of
176 - output :: Another way of generating results from a source-code block
177 is to collect the output generated by the execution of the
178 block. In this case all printed output is collected
179 throughout the execution of the block. This can be
180 thought of as similar to a "script" style of evaluation.
184 Add the following lines to your .emacs, replacing the path as
185 appropriate. A good place to check that things are up and running
186 would then be [[#sandbox][the sandbox]].
187 #+begin_src emacs-lisp
188 (add-to-list 'load-path "/path/to/org-babel/lisp")
189 (require 'org-babel-init)
194 ** TODO support for working with =*Org Edit Src Example*= buffers [2/4]
195 *** TODO optionally evaluate header references when we switch to =*Org Edit Src*= buffer
196 That seems to imply that the header references need to be evaluated
197 and transformed into the target language object when we hit C-c ' to
198 enter the *Org Edit Src* buffer [DED]
200 Good point, I heartily agree that this should be supported [Eric]
202 (or at least before the first time we attempt to evaluate code in that
203 buffer -- I suppose there might be an argument for lazy evaluation, in
204 case someone hits C-c ' but is "just looking" and not actually
205 evaluating anything.) Of course if evaluating the reference is
206 computationally intensive then the user might have to wait before they
207 get the *Org Edit Src* buffer. [DED]
209 I fear that it may be hard to anticipate when the references will be
210 needed, some major-modes do on-the-fly evaluation while the buffer is
211 being edited. I think that we should either do this before the buffer
212 is opened or not at all, specifically I think we should resolve
213 references if the user calls C-c ' with a prefix argument. Does that
214 sound reasonable? [Eric]
218 [Dan] So now that we have org-src-mode and org-src-mode-hook, I guess
219 org-babel should do this by using the hook to make sure that, when C-c
220 C-' is issued on a source block, any references are resolved and
221 assignments are made in the appropriate session.
222 *** TODO set buffer-local-process variables appropriately [DED]
223 I think something like this would be great. You've probably
224 already thought of this, but just to note it down: it would be really
225 nice if org-babel's notion of a buffer's 'session/process' played
226 nicely with ESS's notion of the buffer's session/process. ESS keeps
227 the current process name for a buffer in a buffer-local variable
228 ess-local-process-name. So one thing we will probably want to do is
229 make sure that the *Org Edit Src Example* buffer sets that variable
232 I had not thought of that, but I agree whole heartedly. [Eric]
234 Once this is done every variable should be able to dump regions into
235 their inferior-process buffer using major-mode functions.
236 *** DEFERRED send code to inferior process
237 Another thought on this topic: I think we will want users to send
238 chunks of code to the interpreter from within the *Org Edit Src*
239 buffer, and I think that's what you have in mind already. In ESS that
240 is done using the ess-eval-* functions. [DED]
242 I think we can leave this up to the major-mode in the source code
243 buffer, as almost every source-code major mode will have functions for
244 doing things like sending regions to the inferior process. If
245 anything we might need to set the value of the buffer local inferior
246 process variable. [Eric]
248 *** DONE some possible requests/proposed changes for Carsten [4/4]
249 While I remember, some possible requests/proposed changes for Carsten
250 come to mind in that regard:
252 **** DONE Remap C-x C-s to save the source to the org buffer?
253 I've done this personally and I find it essential. I'm using
254 #+begin_src emacs-lisp
255 (defun org-edit-src-save ()
256 "Update the parent org buffer with the edited source code, save
257 the parent org-buffer, and return to the source code edit
266 (define-key org-exit-edit-mode-map "\C-x\C-s" 'org-edit-src-save)
270 I think this is great, but I think it should be implemented in the
273 **** DEFERRED Rename buffer and minor mode?
274 Something shorter than *Org Edit Src Example* for the buffer
275 name. org-babel is bringing org's source code interaction to a
276 level of maturity where the 'example' is no longer
277 appropriate. And if further keybindings are going to be added to
278 the minor mode then maybe org-edit-src-mode is a better name than
281 Maybe we should name the buffer with a combination of the source
282 code and the session. I think that makes sense.
284 [ES] Are you also suggesting a new org-edit-src minor mode?
285 [DED] org-exit-edit-mode is a minor mode that already exists:
287 Minor mode installing a single key binding, "C-c '" to exit special edit.
289 org-edit-src-save now has a binding in that mode, so I guess all
290 I'm saying at this stage is that it's a bit of a misnomer. But
291 perhaps we will also have more functionality to add to that minor
292 mode, making it even more of a misnomer. Perhaps something like
293 org-src-mode would be better.
294 **** DONE Changed minor mode name and added hooks
296 **** DEFERRED a hook called when the src edit buffer is created
297 This should be implemented in the org-mode core
299 ** TODO resolve references to other org buffers/files
300 This would allow source blocks to call upon tables, source-blocks,
301 and results in other org buffers/files.
304 - [[file:lisp/org-babel-ref.el::TODO%20allow%20searching%20for%20names%20in%20other%20buffers][org-babel-ref.el:searching-in-other-buffers]]
305 - [[file:lisp/org-babel.el::defun%20org-babel%20find%20named%20result%20name][org-babel.el#org-babel-find-named-result]]
306 ** TODO resolve references to other non-org files
307 - tabular data in .csv, .tsv etc format
308 - files of interpreted code: anything stopping us giving such files
309 similar status to a source code block?
310 - Would be nice to allow org and non-org files to be remote
311 ** TODO figure out how to handle errors during evaluation
312 R has a try function, with error handling, along the lines of
313 python. I bet ruby does too. Maybe more of an issue for functional
314 style; in my proposed scripting style the error just gets dumped to
315 the org buffer and the user is thus alerted.
316 ** TODO figure out how to handle graphic output
317 This is listed under [[* graphical output][graphical output]] in out objectives.
319 This should take advantage of the =:results file= option, and
320 languages which almost always produce graphical output should set
321 =:results file= to true by default. That would handle placing these
322 results in the buffer. Then if there is a combination of =silent= and
323 =file= =:results= headers we could drop the results to a temp buffer
324 and pop open that buffer...
325 ** TODO =\C-c \C-o= to open results of source block
326 by adding a =defadvice= to =org-open-at-point= we can use the common
327 =\C-c \C-o= keybinding to open the results of a source-code block.
328 This would be especially useful for source-code blocks which generate
329 graphical results and insert a file link as the results in the
330 org-mode buffer. (see [[* figure out how to handle graphic output][TODO figure out how to handle graphic output]]).
331 This could also act reasonably with other results types...
333 - file :: use org-open-at-point to open the file
334 - scalar :: open results unquoted in a new buffer
335 - tabular :: export the table to a new buffer and open that buffer
337 ** TODO Finalise behaviour regarding vector/scalar output
338 *** DONE Stop spaces causing vector output
339 This simple example of multilingual chaining produces vector output if
340 there are spaces in the message and scalar otherwise.
344 #+begin_src R :var msg=msg-from-python
345 paste(msg, "und R", sep=" ")
349 : org-babel speaks elisp y python und R
351 #+srcname: msg-from-python
352 #+begin_src python :var msg=msg-from-elisp
356 #+srcname: msg-from-elisp
357 #+begin_src emacs-lisp :var msg="org-babel speaks"
358 (concat msg " elisp")
360 ** STARTED share org-babel [1/4]
361 how should we share org-babel?
363 *** DONE post to org-mode
364 *** TODO post to ess mailing list
365 *** TODO create a org-babel page on worg
366 *** TODO create a short screencast demonstrating org-babel in action
369 we need to think up some good examples
371 **** interactive tutorials
372 This could be a place to use [[* org-babel assertions][org-babel assertions]].
374 for example the first step of a tutorial could assert that the version
375 of the software-package (or whatever) is equal to some value, then
376 source-code blocks could be used with confidence (and executed
377 directly from) the rest of the tutorial.
379 **** answering a text-book question w/code example
380 org-babel is an ideal environment enabling both the development and
381 demonstrationg of the code snippets required as answers to many
384 **** something using tables
385 maybe something along the lines of calculations from collected grades
388 Maybe something like the following which outputs sizes of directories
389 under the home directory, and then instead of the trivial =emacs-lisp=
390 block we could use an R block to create a nice pie chart of the
394 #+begin_src bash :results replace
398 #+begin_src emacs-lisp :var sizes=sizes :results replace
402 ** TODO command line execution
403 Allow source code blocks to be called form the command line. This
404 will be easy using the =sbe= function in [[file:lisp/org-babel-table.el][org-babel-table.el]].
406 This will rely upon [[* resolve references to other buffers][resolve references to other buffers]].
408 ** TODO inline source code blocks [3/5]
409 Like the =\R{ code }= blocks
411 not sure what the format should be, maybe just something simple
412 like =src_lang[]{}= where lang is the name of the source code
413 language to be evaluated, =[]= is optional and contains any header
414 arguments and ={}= contains the code.
416 (see [[* (sandbox) inline source blocks][the-sandbox]])
418 *** DONE evaluation with \C-c\C-c
419 Putting aside the header argument issue for now we can just run these
420 with the following default header arguments
421 - =:results= :: silent
422 - =:exports= :: results
424 *** DONE inline exportation
425 Need to add an interblock hook (or some such) through org-exp-blocks
426 *** DONE header arguments
427 We should make it possible to use header arguments.
429 *** TODO fontification
430 we should color these blocks differently
432 *** TODO refine html exportation
433 should use a span class, and should show original source in tool-tip
435 ** TODO formulate general rules for handling vectors and tables / matrices with names
436 This is non-trivial, but may be worth doing, in particular to
437 develop a nice framework for sending data to/from R.
439 In R, indexing vector elements, and rows and columns, using
440 strings rather than integers is an important part of the
442 - elements of a vector may have names
443 - matrices and data.frames may have "column names" and "row names"
444 which can be used for indexing
445 - In a data frame, row names *must* be unique
453 > mat <- matrix(1:4, nrow=2, ncol=2, dimnames=list(c("r1","r2"), c("c1","c2")))
458 > # The names are separate from the data: they do not interfere with operations on the data
465 > df <- data.frame(var1=1:26, var2=26:1, row.names=letters)
467 [1] 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
473 So it's tempting to try to provide support for this in org-babel. For example
474 - allow R to refer to columns of a :var reference by their names
475 - When appropriate, results from R appear in the org buffer with "named
478 However none (?) of the other languages we are currently supporting
479 really have a native matrix type, let alone "column names" or "row
480 names". Names are used in e.g. python and perl to refer to entries
483 It currently seems to me that support for this in org-babel would
484 require setting rules about when org tables are considered to have
485 named columns/fields, and ensuring that (a) languages with a notion
486 of named columns/fields use them appropriately and (b) languages
487 with no such notion do not treat then as data.
489 - Org allows something that *looks* like column names to be separated
491 - Org also allows a row to *function* as column names when special
492 markers are placed in the first column. An hline is unnecessary
493 (indeed hlines are purely cosmetic in org [correct?]
494 - Org does not have a notion of "row names" [correct?]
496 The full org table functionality exeplified [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]] has features that
497 we would not support in e.g. R (like names for the row below).
499 *** Initial statement: allow tables with hline to be passed as args into R
500 This doesn't seem to work at the moment (example below). It would
501 also be nice to have a natural way for the column names of the org
502 table to become the column names of the R data frame, and to have
503 the option to specify that the first column is to be used as row
504 names in R (these must be unique). But this might require a bit of
509 | col1 | col2 | col3 |
510 |------+---------+------|
518 #+begin_src R var tabel=egtable
523 | "col1" | "col2" | "col3" |
524 |--------+-----------+--------|
526 | 4 | "schulte" | 6 |
529 Another example is in the [[*operations%20in%20on%20tables][grades example]].
531 ** TODO re-implement helper functions from org-R
532 *** Initial statement [Eric]
533 Much of the power of org-R seems to be in it's helper functions for
534 the quick graphing of tables. Should we try to re-implement these
535 functions on top of org-babel?
537 I'm thinking this may be useful both to add features to org-babel-R and
538 also to potentially suggest extensions of the framework. For example
539 one that comes to mind is the ability to treat a source-code block
540 like a function which accepts arguments and returns results. Actually
541 this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]).
543 - We want to provide convenient off-the-shelf actions
544 (e.g. plotting data) that make use of our new code evaluation
545 environment but do not require any actual coding.
546 *** Initial Design proposal [Dan]
547 - *Input data* will be specified using the same mechanism as :var
548 references, thus the input data may come from a table, or
549 another source block, and it is initially available as an elisp
551 - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that
552 line will apply an *action* to the referenced data.
553 - *Actions correspond to source blocks*: our library of available
554 actions will be a library of org-babel source blocks. Thus the
555 code for executing an action, and the code for dealing with the
556 output of the action will be the same code as for executing
557 source blocks in general
558 - Optionally, the user can have the relevant source block inserted
559 into the org buffer after the (say) #+BABELDO line. This will
560 allow the user to fine tune the action by modifying the code
561 (especially useful for plots).
562 - So maybe a #+BABELDO line will have header args
563 - :data (a reference to a table or source code block)
564 - :action (or should that be :srcname?) which will be something
565 like :action pie-chart, referring to a source block which will
566 be executed with the :data referent passed in using a :var arg.
567 - :showcode or something controlling whether to show the code
569 *** Modification to design
570 I'm implementing this, at least initially, as a new interpreter
571 named 'babel', which has an empty body. 'babel' blocks take
572 a :srcname header arg, and look for the source-code block with
573 that name. They then execute the referenced block, after first
574 appending their own header args on to the target block's header
577 If the target block is in the library of babel (a.o.t. e.g. the
578 current buffer), then the code in the block will refer to the
579 input data with a name dictated by convention (e.g. __data__
580 (something which is syntactically legal in all languages...). Thus
581 the babel block will use a :var __data__ = whatever header arg to
582 reference the data to be plotted.
585 This is covered by the [[file:library-of-babel.org][Library of Babel]], which will contain
586 ready-made source blocks designed to carry out useful common tasks.
588 ** PROPOSED Are we happy with current behaviour regarding vector/scalar output?
589 This simple example of multilingual chaining produces vector output if
590 there are spaces in the message and scalar otherwise.
592 #+begin_src R :var msg=msg-from-python
593 paste(msg, "und_R", sep="_")
596 #+srcname: msg-from-python
597 #+begin_src python :var msg=msg-from-elisp
601 #+srcname: msg-from-elisp
602 #+begin_src emacs-lisp :var msg="org-babel_speaks"
603 (concat msg "_elisp")
606 ** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format
607 I haven't thought about this properly. Just noting it down. What
608 Sweave uses is called "R noweb" (.Rnw).
610 I found a good description of noweb in the following article (see
611 the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]).
613 I think there are two parts to noweb, the construction of
614 documentation and the extraction of source-code (with notangle).
616 *documentation*: org-mode handles all of our documentation needs in
617 a manner that I believe is superior to noweb.
619 *source extraction* At this point I don't see anyone writing large
620 applications with 100% of the source code contained in org-babel
621 files, rather I see org-babel files containing things like
622 - notes with active code chunks
623 - interactive tutorials
624 - requirements documents with code running test suites
625 - and of course experimental reports with the code to run the
626 experiment, and perform analysis
628 Basically I think the scope of the programs written in org-babel
629 (at least initially) will be small enough that it wont require the
630 addition of a tangle type program to extract all of the source code
631 into a running application.
633 On the other hand, since we already have named blocks of source
634 code which reference other blocks on which they rely, this
635 shouldn't be too hard to implement either on our own, or possibly
636 relying on something like noweb/notangle.
638 ** PROPOSED support for passing paths to files between source blocks
639 Maybe this should be it's own result type (in addition to scalars and
640 vectors). The reason being that some source-code blocks (for example
641 ditaa or anything that results in the creation of a file) may want to
642 pass a file path back to org-mode which could then be inserted into
643 the org-mode buffer as a link to the file...
645 This would allow for display of images upon export providing
646 functionality similar to =org-exp-blocks= only in a more general
649 ** DEFERRED use textConnection to pass tsv to R?
650 When passing args from the org buffer to R, the following route is
651 used: arg in buffer -> elisp -> tsv on file -> data frame in R. I
652 think it would be possible to avoid having to write to file by
653 constructing an R expression in org-babel-R-assign-elisp, something
656 #+begin_src emacs-lisp
657 (org-babel-R-input-command
658 (format "%s <- read.table(textConnection(\"%s\"), sep=\"\\t\", as.is=TRUE)"
659 name (orgtbl-to-tsv value '(:sep "\t" :fmt org-babel-R-quote-tsv-field))))
662 I haven't tried to implement this yet as it's basically just
663 fiddling with something that works. The only reason for it I can
664 think of would be efficiency and I haven't tested that.
666 This Didn't work after an initial test. I still think this is a
667 good idea (I also think we should try to do something similar when
668 writing out results frmo R to elisp) however as it wouldn't result
669 in any functional changes I'm bumping it down to deferred for
674 #+tblname: quick-test
677 #+srcname: quick-test-src-blk
678 #+begin_src R :var vec=quick-test
684 ** DEFERRED re-implement R evaluation using ess-command or ess-execute
685 I don't have any complaints with the current R evaluation code or
686 behaviour, but I think it would be good to use the ESS functions
687 from a political point of view. Plus of course it has the normal
688 benefits of an API (insulates us from any underlying changes etc). [DED]
690 I'll look into this. I believe that I looked at and rejected these
691 functions initially but now I can't remember why. I agree with
692 your overall point about using API's where available. I will take
693 a look back at these and either switch to using the ess commands,
694 or at least articulate under this TODO the reasons for using our
695 custom R-interaction commands. [Eric]
699 Lets just replace =org-babel-R-input-command= with =ess-execute=.
701 I tried this, and although it works in some situations, I find that
702 =ess-command= will often just hang indefinitely without returning
703 results. Also =ess-execute= will occasionally hang, and pops up
704 the buffer containing the results of the command's execution, which
705 is undesirable. For now these functions can not be used. Maybe
706 someone more familiar with the ESS code can recommend proper usage
707 of =ess-command= or some other lower-level function which could be
708 used in place of [[file:lisp/org-babel-R.el::defun%20org-babel%20R%20input%20command%20command][org-babel-R-input-command]].
712 #+begin_quote ess-command
713 (ess-command COM &optional BUF SLEEP NO-PROMPT-CHECK)
715 Send the ESS process command COM and delete the output
716 from the ESS process buffer. If an optional second argument BUF exists
717 save the output in that buffer. BUF is erased before use.
718 COM should have a terminating newline.
719 Guarantees that the value of .Last.value will be preserved.
720 When optional third arg SLEEP is non-nil, `(sleep-for (* a SLEEP))'
721 will be used in a few places where `a' is proportional to `ess-cmd-delay'.
724 #+begin_quote ess-execute
725 (ess-execute COMMAND &optional INVERT BUFF MESSAGE)
727 Send a command to the ESS process.
728 A newline is automatically added to COMMAND. Prefix arg (or second arg
729 INVERT) means invert the meaning of
730 `ess-execute-in-process-buffer'. If INVERT is 'buffer, output is
731 forced to go to the process buffer. If the output is going to a
732 buffer, name it *BUFF*. This buffer is erased before use. Optional
733 fourth arg MESSAGE is text to print at the top of the buffer (defaults
734 to the command if BUFF is not given.)
737 *** out current setup
739 1) The body of the R source code block is wrapped in a function
740 2) The function is called inside of a =write.table= function call
741 writing the results to a table
742 3) The table is read using =org-table-import=
744 ** DEFERRED Rework Interaction with Running Processes [2/5]
745 *** DONE robust to errors interrupting execution
747 #+srcname: long-runner-ruby
748 #+begin_src ruby :results silent
753 *** DEFERRED use =C-g= keyboard-quit to push processing into the background
754 This may be possible using the `run-with-timer' command.
756 I have no idea how this could work...
758 #+srcname: long-runner-ruby
759 #+begin_src ruby :results silent
764 *** TODO ability to select which of multiple sessions is being used
765 Increasingly it is looking like we're going to want to run all
766 source code blocks in comint buffer (sessions). Which will have
768 1) allowing background execution
769 2) maintaining state between source-blocks
770 - allowing inline blocks w/o header arguments
773 (like ess-switch-process in .R buffers)
775 Maybe this could be packaged into a header argument, something
776 like =:R_session= which could accept either the name of the
777 session to use, or the string =prompt=, in which case we could use
778 the =ess-switch-process= command to select a new process.
780 *** TODO evaluation of shell code as background process?
781 After C-c C-c on an R code block, the process may appear to
782 block, but C-g can be used to reclaim control of the .org buffer,
783 without interrupting the R evalution. However I believe this is not
784 true of bash/sh evaluation. [Haven't tried other languages] Perhaps
785 a solution is just to background the individual shell commands.
787 The other languages (aside from emacs lisp) are run through the
788 shell, so if we find a shell solution it should work for them as
791 Adding an ampersand seems to be a supported way to run commands in
792 the background (see [[http://www.emacswiki.org/emacs/ExecuteExternalCommand#toc4][external-commands]]). Although a more extensible
793 solution may involve the use of the [[elisp:(progn (describe-function 'call-process-region) nil)][call-process-region]] function.
795 Going to try this out in a new file [[file:lisp/org-babel-proc.el][org-babel-proc.el]]. This should
796 contain functions for asynchronously running generic shell commands
797 in the background, and then returning their input.
799 **** partial update of org-mode buffer
800 The sleekest solution to this may be using a comint buffer, and
801 then defining a filter function which would incrementally interpret
802 the results as they are returned, including insertion into the
803 org-mode buffer. This may actually cause more problems than it is
804 worth, what with the complexities of identifying the types of
805 incrementally returned results, and the need for maintenance of a
806 process marker in the org buffer.
808 **** 'working' spinner
809 It may be nice and not too difficult to place a spinner on/near the
810 evaluating source code block
812 *** TODO conversion of output from interactive shell, R (and python) sessions to org-babel buffers
813 [DED] This would be a nice feature I think. Although an org-babel
814 purist would say that it's working the wrong way round... After
815 some interactive work in a *R* buffer, you save the buffer, maybe
816 edit out some lines, and then convert it to org-babel format for
817 posterity. Same for a shell session either in a *shell* buffer, or
818 pasted from another terminal emulator. And python of course.
820 ** DEFERRED improve the source-block snippet
821 any real improvement seems somewhat beyond the ability of yasnippet
824 [[file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name%20Chapter%20title][file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name Chapter title]]
826 ,#name : Chapter title
829 ${1:$(make-string (string-width text) ?\=)}
834 [[file:snippets/org-mode/sb][sb -- snippet]]
836 waiting for guidance from those more familiar with yasnippets
837 ** DONE add a function to jump to a source-block by name
838 I've had an initial stab at that in org-babel-find-named-block
839 (library-of-babel branch).
841 At the same time I introduced org-babel-named-src-block-regexp, to
842 match src-blocks with srcname.
844 This is now working with the command
845 `org-babel-goto-named-source-block', all we need is a good key
848 ** DONE add =:none= session argument (for purely functional execution) [4/4]
849 This would allow source blocks to be run in their own new process
851 - These blocks could then also be run in the background (since we can
852 detach and just wait for the process to signal that it has terminated)
853 - We wouldn't be drowning in session buffers after running the tests
854 - we can re-use much of the session code to run in a more /functional/
857 While session provide a lot of cool features, like persistent
858 environments, [[* DONE function to bring up inferior-process buffer][pop-to-session]], and hints at exportation for
859 org-babel-tangle, they also have some down sides and I'm thinking that
860 session-based execution maybe shouldn't be the default behavior.
862 Down-sides to sessions
863 - *much* more complicated than functional evaluation
864 - maintaining the state of the session has weird issues
865 - waiting for evaluation to finish
866 - prompt issues like [[* TODO weird escaped characters in shell prompt break shell evaluation][shell-prompt-escapes-bug]]
867 - can't run in background
868 - litter emacs with session buffers
872 #+srcname: ruby-task-no-session
873 #+begin_src ruby :results replace output
879 #+resname: ruby-task-no-session
884 #+srcname: task-python-none-session
885 #+begin_src python :session none :results replace value
891 #+resname: task-python-none-session
896 #+srcname: task-session-none-sh
897 #+begin_src sh :results replace
902 #+resname: task-session-none-sh
908 #+srcname: task-no-session-R
909 #+begin_src R :results replace output
916 #+resname: task-no-session-R
920 ** DONE fully purge org-babel-R of direct comint interaction
921 try to remove all code under the [[file:lisp/org-babel-R.el::functions%20for%20evaluation%20of%20R%20code][;; functions for evaluation of R code]] line
923 ** DONE Create objects in top level (global) environment [5/5]
926 *** initial requirement statement [DED]
927 At the moment, objects created by computations performed in the
928 code block are evaluated in the scope of the
929 code-block-function-body and therefore disappear when the code
930 block is evaluated {unless you employ some extra trickery like
931 assign('name', object, env=globalenv()) }. I think it will be
932 desirable to also allow for a style wherein objects that are
933 created in one code block persist in the R global environment and
934 can be re-used in a separate block.
936 This is what Sweave does, and while I'm not saying we have to be
937 the same as Sweave, it wouldn't be hard for us to provide the same
938 behaviour in this case; if we don't, we risk undeservedly being
939 written off as an oddity by some.
941 IOW one aspect of org-babel is that of a sort of functional
942 meta-programming language. This is crazy, in a very good
943 way. Nevertheless, wrt R I think there's going to be a lot of value
944 in providing for a working style in which the objects are stored in
945 the R session, rather than elisp/org buffer. This will be a very
946 familiar working style to lots of people.
948 There are no doubt a number of different ways of accomplishing
949 this, the simplest being a hack like adding
953 assign(objname, get(objname), envir=globalenv())
956 to the source code block function body. (Maybe wrap it in an on.exit() call).
958 However this may deserve to be thought about more carefully, perhaps
959 with a view to having a uniform approach across languages. E.g. shell
960 code blocks have the same semantics at the moment (no persistence of
961 variables across code blocks), because the body is evaluated in a new
962 bash shell process rather than a running shell. And I guess the same
963 is true for python. However, in both these cases, you could imagine
964 implementing the alternative in which the body is evaluated in a
965 persistent interactive session. It's just that it's particularly
966 natural for R, seeing as both ESS and org-babel evaluate commands in a
967 single persistent R session.
971 Thanks for bringing this up. I think you are absolutely correct that we
972 should provide support for a persistent environment (maybe called a
973 *session*) in which to evaluate code blocks. I think the current setup
974 demonstrates my personal bias for a functional style of programming
975 which is certainly not ideal in all contexts.
977 While the R function you mention does look like an elegant solution, I
978 think we should choose an implementation that would be the same across
979 all source code types. Specifically I think we should allow the user to
980 specify an optional *session* as a header variable (when not present we
981 assume a default session for each language). The session name could be
982 used to name a comint buffer (like the *R* buffer) in which all
983 evaluation would take place (within which variables would retain their
984 values --at least once I remove some of the functional method wrappings
985 currently in place-- ).
987 This would allow multiple environments to be used in the same buffer,
988 and once this setup was implemented we should be able to fairly easily
989 implement commands for jumping between source code blocks and the
990 related session buffers, as well as for dumping the last N commands from
991 a session into a new or existing source code block.
993 Please let me know if you foresee any problems with this proposed setup,
994 or if you think any parts might be confusing for people coming from
995 Sweave. I'll hopefully find some time to work on this later in the
998 *** can functional and interpreted/interactive models coexist?
1000 Even though both of these use the same =*R*= buffer the value of =a=
1001 is not preserved because it is assigned inside of a functional
1004 #+srcname: task-R-sessions
1011 #+srcname: task-R-same-session
1016 This functional wrapper was implemented in order to efficiently return
1017 the results of the execution of the entire source code block. However
1018 it inhibits the evaluation of source code blocks in the top level,
1019 which would allow for persistence of variable assignment across
1020 evaluations. How can we allow *both* evaluation in the top level, and
1021 efficient capture of the return value of an entire source code block
1022 in a language independent manner?
1024 Possible solutions...
1025 1) we can't so we will have to implement two types of evaluation
1026 depending on which is appropriate (functional or imperative)
1027 2) we remove the functional wrapper and parse the source code block
1028 into it's top level statements (most often but not always on line
1029 breaks) so that we can isolate the final segment which is our
1031 3) we add some sort of "#+return" line to the code block
1032 4) we take advantage of each languages support for meta-programming
1033 through =eval= type functions, and use said to evaluate the entire
1034 blocks in such a way that their environment can be combined with the
1035 global environment, and their results are still captured.
1036 5) I believe that most modern languages which support interactive
1037 sessions have support for a =last_result= type function, which
1038 returns the result of the last input without re-calculation. If
1039 widely enough present this would be the ideal solution to a
1040 combination of functional and imperative styles.
1042 None of these solutions seem very desirable, but for now I don't see
1043 what else would be possible.
1045 Of these options I was leaning towards (1) and (4) but now believe
1046 that if it is possible option (5) will be ideal.
1048 **** (1) both functional and imperative evaluation
1050 - can take advantage of built in functions for sending regions to the
1052 - retains the proven tested and working functional wrappers
1055 - introduces the complication of keeping track of which type of
1056 evaluation is best suited to a particular context
1057 - the current functional wrappers may require some changes in order to
1058 include the existing global context
1060 **** (4) exploit language meta-programming constructs to explicitly evaluate code
1062 - only one type of evaluation
1065 - some languages may not have sufficient meta-programming constructs
1067 **** (5) exploit some =last_value= functionality if present
1069 Need to ensure that most languages have such a function, those without
1070 will simply have to implement their own similar solution...
1072 | language | =last_value= function |
1073 |------------+-----------------------------|
1077 | shell | see [[* last command for shells][last command for shells]] |
1078 | emacs-lisp | see [[* emacs-lisp will be a special case][special-case]] |
1080 #+srcname: task-last-value
1085 ***** last command for shells
1086 Do this using the =tee= shell command, and continually pipe the output
1089 Got this idea from the following [[http://linux.derkeiler.com/Mailing-Lists/Fedora/2004-01/0898.html][email-thread]].
1091 suggested from mailing list
1093 #+srcname: bash-save-last-output-to-file
1097 bash -c "$line" | tee /tmp/last.out1
1098 mv /tmp/last.out1 /tmp/last.out
1102 another proposed solution from the above thread
1104 #+srcname: bash-save-in-variable
1107 # so - Save Output. Saves output of command in OUT shell variable.
1117 "^M": " | tee /tmp/h_lastcmd.out ^[k"
1120 export __=/tmp/h_lastcmd.out
1122 If you try it, Alt-k will stand for the old Enter; use "command $__" to
1123 access the last output.
1129 Herculano de Lima Einloft Neto
1132 ***** emacs-lisp will be a special case
1133 While it is possible for emacs-lisp to be run in a console type
1134 environment (see the =elim= function) it is *not* possible to run
1135 emacs-lisp in a different *session*. Meaning any variable set top
1136 level of the console environment will be set *everywhere* inside
1137 emacs. For this reason I think that it doesn't make any sense to
1138 worry about session support for emacs-lisp.
1140 *** Further thoughts on 'scripting' vs. functional approaches
1142 These are just thoughts, I don't know how sure I am about this.
1143 And again, perhaps I'm not saying anything very radical, just that
1144 it would be nice to have some options supporting things like
1145 receiving text output in the org buffer.
1147 I can see that you've already gone some way down the road towards
1148 the 'last value' approach, so sorry if my comments come rather
1149 late. I am concerned that we are not giving sufficient attention
1150 to stdout / the text that is returned by the interpreters. In
1151 contrast, many of our potential users will be accustomed to a
1152 'scripting' approach, where they are outputting text at various
1153 points in the code block, not just at the end. I am leaning
1154 towards thinking that we should have 2 modes of evaluation:
1155 'script' mode, and 'functional' mode.
1157 In script mode, evaluation of a code block would result in *all*
1158 text output from that code block appearing as output in the org
1159 buffer, presumably as an #+begin_example...#+end_example. There
1160 could be an :echo option controlling whether the input commands
1161 also appear in the output. [This is like Sweave].
1163 In functional mode, the *result* of the code block is available as
1164 an elisp object, and may appear in the org buffer as an org
1165 table/string, via the mechanisms you have developed already.
1167 One thing I'm wondering about is whether, in script mode, there
1168 simply should not be a return value. Perhaps this is not so
1169 different from what exists: script mode would be new, and what
1170 exists currently would be functional mode.
1172 I think it's likely that, while code evaluation will be exciting
1173 to people, a large majority of our users in a large majority of
1174 their usage will not attempt to actually use the return value from
1175 a source code block in any meaningful way. In that case, it seems
1176 rather restrictive to only allow them to see output from the end
1179 Instead I think the most accessible way to introduce org-babel to
1180 people, at least while they are learning it, is as an immensely
1181 powerful environment in which to embed their 'scripts', which now
1182 also allows them to 'run' their 'scripts'. Especially as such
1183 people are likely to be the least capable of the user-base, a
1184 possible design-rule would be to make the scripting style of usage
1185 easy (default?), perhaps requiring a special option to enable a
1186 functional style. Those who will use the functional style won't
1187 have a problem understanding what's going on, whereas the 'skript
1188 kiddies' might not even know the syntax for defining a function in
1189 their language of choice. And of course we can allow the user to
1190 set a variable in their .emacs controlling the preference, so that
1191 functional users are not inconveniennced by having to provide
1192 header args the whole time.
1194 Please don't get the impression that I am down-valuing the
1195 functional style of org-babel. I am constantly horrified at the
1196 messy 'scripts' that my colleagues produce in perl or R or
1197 whatever! Nevertheless that seems to be how a lot of people work.
1199 I think you were leaning towards the last-value approach because
1200 it offered the possibility of unified code supporting both the
1201 single evaluation environment and the functional style. If you
1202 agree with any of the above then perhaps it will impact upon this
1203 and mean that the code in the two branches has to differ a bit. In
1204 that case, functional mode could perhaps after all evaluate each
1205 code block in its own environment, thus (re)approaching 'true'
1206 functional programming (side-effects are hard to achieve).
1210 echo "There are `wc -l files` files in this directory"
1214 *** even more thoughts on evaluation, results, models and options
1216 Thanks Dan, These comments are invaluable.
1218 What do you think about this as a new list of priorities/requirements
1219 for the execution of source-code blocks.
1222 1) we want the evaluation of the source code block to take place in a
1223 session which can persist state (variables, current directory,
1225 2) source code blocks can specify their session with a header argument
1226 3) each session should correspond to an Emacs comint buffer so that the
1227 user can drop into the session and experiment with live code
1230 1) each source-code block generates some form of results which (as
1231 we have already implemented) is transfered into emacs-lisp
1232 after which it can be inserted into the org-mode buffer, or
1233 used by other source-code blocks
1234 2) when the results are translated into emacs-lisp, forced to be
1235 interpreted as a scalar (dumping their raw values into the
1236 org-mode buffer), as a vector (which is often desirable with R
1237 code blocks), or interpreted on the fly (the default option).
1238 Note that this is very nearly currently implemented through the
1239 [[* DONE results-type header (vector/file)][results-type-header]].
1240 3) there should be *two* means of collecting results from the
1241 execution of a source code block. *Either* the value of the
1242 last statement of the source code block, or the collection of
1243 all that has been passed to STDOUT during the evaluation.
1245 **** header argument or return line (*header argument*)
1247 Rather than using a header argument to specify how the return value
1248 should be passed back, I'm leaning towards the use of a =#+RETURN=
1249 line inside the block. If such a line *is not present* then we
1250 default to using STDOUT to collect results, but if such a line *is
1251 present* then we use it's value as the results of the block. I
1252 think this will allow for the most elegant specification between
1253 functional and script execution. This also cleans up some issues
1254 of implementation and finding which statement is the last
1257 Having given this more thought, I think a header argument is
1258 preferable. The =#+return:= line adds new complicating syntax for
1259 something that does little more than we would accomplish through
1260 the addition of a header argument. The only benefit being that we
1261 know where the final statement starts, which is not an issue in
1262 those languages which contain 'last value' operators.
1264 new header =:results= arguments
1265 - script :: explicitly states that we want to use STDOUT to
1266 initialize our results
1267 - return_last :: stdout is ignored instead the *value* of the final
1268 statement in the block is returned
1269 - echo :: means echo the contents of the source-code block along
1270 with the results (this implies the *script* =:results=
1273 *** DONE rework evaluation lang-by-lang [4/4]
1275 This should include...
1276 - functional results working with the comint buffer
1278 - script :: return the output of STDOUT
1279 - write a macro which runs the first redirection, executes the
1280 body, then runs the second redirection
1281 - last :: return the value of the last statement
1284 - sessions in comint buffers
1286 **** DONE Ruby [4/4]
1287 - [X] functional results working with comint
1288 - [X] script results
1289 - [X] ensure scalar/vector results args are taken into consideration
1290 - [X] ensure callable by other source block
1292 #+srcname: ruby-use-last-output
1293 #+begin_src ruby :results replace
1300 #+resname: ruby-use-last-output
1303 #+srcname: task-call-use-last-output
1304 #+begin_src ruby :var last=ruby-use-last-output :results replace
1305 last.flatten.size + 1
1308 #+resname: task-call-use-last-output
1313 #+srcname: first-ruby-session-task
1314 #+begin_src ruby :session schulte :results silent
1318 #+srcname: second-ruby-session-task
1319 #+begin_src ruby :session schulte :results silent
1323 #+srcname: without-the-right-session
1324 #+begin_src ruby :results silent
1330 - [X] functional results working with comint
1331 - [X] script results
1332 - [X] ensure scalar/vector results args are taken into consideration
1333 - [X] ensure callable by other source block
1335 To redirect output to a file, you can use the =sink()= command.
1338 #+begin_src R :results value vector silent
1345 #+srcname: task-R-use-other-output
1346 #+begin_src R :var twoentyseven=task_R_B() :results replace value
1351 #+resname: task-R-use-other-output
1354 **** DONE Python [4/4]
1355 - [X] functional results working with comint
1356 - [X] script results
1357 - [X] ensure scalar/vector results args are taken into consideration
1358 - [X] ensure callable by other source block
1360 #+srcname: task-new-eval-for-python
1361 #+begin_src python :results silent output scalar
1367 #+srcname: task-use-new-eval
1368 #+begin_src python :var tasking=task-new-eval-for-python() :results replace
1372 #+resname: task-use-new-eval
1375 **** DONE Shells [4/4]
1376 - [X] functional results working with comint
1377 - [X] script results
1378 - [X] ensure scalar/vector results args are taken into consideration
1379 - [X] ensure callable by other source block
1381 #+srcname: task-shell-new-evaluation
1382 #+begin_src sh :results silent value scalar
1387 #+srcname: task-call-other-shell
1388 #+begin_src sh :var other=task-shell-new-evaluation() :results replace scalar
1389 echo $other ' is the old date'
1392 #+resname: task-call-other-shell
1393 : $ Fri Jun 12 13:08:37 PDT 2009 is the old date
1395 *** DONE implement a *session* header argument [4/4]
1396 =:session= header argument to override the default *session* buffer
1400 #+srcname: task-ruby-named-session
1401 #+begin_src ruby :session schulte :results replace
1402 schulte = :in_schulte
1405 #+resname: task-ruby-named-session
1408 #+srcname: another-in-schulte
1409 #+begin_src ruby :session schulte
1413 #+resname: another-in-schulte
1420 #+srcname: python-session-task
1421 #+begin_src python :session what :results silent
1425 #+srcname: python-get-from-session
1426 #+begin_src python :session what :results replace
1430 #+resname: python-get-from-session
1435 #+srcname: task-shell-sessions
1436 #+begin_src sh :session what
1440 #+srcname: task-shell-sessions-what
1441 #+begin_src sh :session what :results replace
1445 #+resname: task-shell-sessions-what
1450 #+srcname: task-R-session
1451 #+begin_src R :session what :results replace
1457 #+resname: task-R-session
1460 #+srcname: another-task-R-session
1461 #+begin_src R :session what :results replace
1465 *** DONE function to bring up inferior-process buffer [4/4]
1467 This should be callable from inside of a source-code block in an
1468 org-mode buffer. It should evaluate the header arguments, then bring
1469 up the inf-proc buffer using =pop-to-buffer=.
1471 For lack of a better place, lets add this to the `org-metadown-hook'
1474 To give this a try, place the cursor on a source block with variables,
1475 (optionally git a prefix argument) then hold meta and press down.
1479 #+srcname: task-ruby-pop-to-session
1480 #+begin_src ruby :var num=9 :var another="something else"
1481 num.times{|n| puts another}
1486 #+srcname: task-python-pop-to-session
1487 #+begin_src python :var num=9 :var another="something else"
1492 #+srcname: task-R-pop-to-session
1493 #+begin_src R :var a=9 :var b=8
1499 #+srcname: task-shell-pop-sessions
1500 #+begin_src sh :var NAME="eric"
1504 *** DEFERRED function to dump last N lines from inf-proc buffer into the current source block
1506 Callable with a prefix argument to specify how many lines should be
1507 dumped into the source-code buffer.
1509 *** REJECTED comint notes
1511 Implementing comint integration in [[file:lisp/org-babel-comint.el][org-babel-comint.el]].
1514 - handling of outputs
1515 - split raw output from process by prompts
1516 - a ring of the outputs, buffer-local, `org-babel-comint-output-ring'
1517 - a switch for dumping all outputs to a buffer
1518 - inputting commands
1520 Lets drop all this language specific stuff, and just use
1521 org-babel-comint to split up our outputs, and return either the last
1522 value of an execution or the combination of values from the
1525 **** comint filter functions
1526 : ;; comint-input-filter-functions hook process-in-a-buffer
1527 : ;; comint-output-filter-functions hook function modes.
1528 : ;; comint-preoutput-filter-functions hook
1529 : ;; comint-input-filter function ...
1531 #+srcname: obc-filter-ruby
1532 #+begin_src ruby :results last
1540 ** DONE Remove protective commas from # comments before evaluating
1541 org inserts protective commas in front of ## comments in language
1542 modes that use them. We need to remove them prior to sending code
1545 #+srcname: testing-removal-of-protective-comas
1547 ,# this one might break it??
1551 ** DONE pass multiple reference arguments into R
1552 Can we do this? I wasn't sure how to supply multiple 'var' header
1553 args. Just delete this if I'm being dense.
1555 This should be working, see the following example...
1557 #+srcname: two-arg-example
1558 #+begin_src R :var n=2 :var m=8
1562 #+resname: two-arg-example
1565 ** DONE ensure that table ranges work
1566 when a table range is passed to org-babel as an argument, it should be
1567 interpreted as a vector.
1570 | 2 | 3 | Fixnum:1 |
1571 | 3 | 4 | Array:123456 |
1575 #+TBLFM: @1$3='(sbe simple-sbe-example (n 4))::@2$3='(sbe task-table-range (n @1$1..@6$1))::@3$3='(sbe task-table-range (n (@1$1..@6$1)))
1577 #+srcname: simple-sbe-example
1578 #+begin_src emacs-lisp
1582 #+srcname: task-table-range
1583 #+begin_src ruby :var n=simple-sbe-example
1587 #+srcname: simple-results
1588 #+begin_src emacs-lisp :var n=task-table-range(n=(1 2 3))
1592 #+resname: simple-results
1595 #+srcname: task-arr-referent
1596 #+begin_src ruby :var ar=(1 2 3)
1600 #+resname: task-arr-referent
1603 ** DONE global variable indicating default to vector output
1604 how about an alist... =org-babel-default-header-args= this may already
1605 exist... just execute the following and all source blocks will default
1608 #+begin_src emacs-lisp
1609 (setq org-babel-default-header-args '((:results . "vector")))
1612 ** DONE name named results if source block is named
1613 currently this isn't happening although it should be
1615 #+srcname: test-naming-named-source-blocks
1616 #+begin_src emacs-lisp
1620 #+resname: test-naming-named-source-blocks
1622 ** DONE (simple caching) check for named results before source blocks
1623 see the TODO comment in [[file:lisp/org-babel-ref.el::TODO%20This%20should%20explicitly%20look%20for%20resname%20lines%20before][org-babel-ref.el#org-babel-ref-resolve-reference]]
1624 ** DONE set =:results silent= when eval with prefix argument
1626 #+begin_src emacs-lisp
1629 ** DONE results-type header (vector/file) [3/3]
1630 In response to a point in Dan's email. We should allow the user to
1631 force scalar or vector results. This could be done with a header
1632 argument, and the default behavior could be controlled through a
1633 configuration variable.
1635 #+srcname: task-trivial-vector
1636 #+begin_src ruby :results replace vector
1643 since it doesn't make sense to turn a vector into a scalar, lets
1644 just add a two values...
1646 - vector :: forces the results to be a vector (potentially 1 dimensional)
1647 - file :: this throws an error if the result isn't a string, and
1648 tries to treat it as a path to a file.
1650 I'm just going to cram all of these into the =:results= header
1651 argument. Then if we allow multiple header arguments it should
1652 work out, for example one possible header argument string could be
1653 =:results replace vector file=, which would *replace* any existing
1654 results forcing the results into an org-mode table, and
1655 interpreting any strings as file paths.
1657 *** DONE multiple =:results= headers
1659 #+srcname: multiple-result-headers
1660 #+begin_src ruby :results replace silent
1666 *** DONE file result types
1667 When inserting into an org-mode buffer create a link with the path
1668 being the value, and optionally the display being the
1669 =file-name-nondirectory= if it exists.
1671 #+srcname: task-file-result
1672 #+begin_src python :results replace file
1677 [[something][something]]
1680 This will be useful because blocks like =ditaa= and =dot= can return
1681 the string path of their files, and can add =file= to their results
1684 *** DONE vector result types
1686 #+srcname: task-force-results
1687 #+begin_src emacs-lisp :results vector
1694 ** DONE results name
1695 In order to do this we will need to start naming our results.
1696 Since the source blocks are named with =#+srcname:= lines we can
1697 name results with =#+resname:= lines (if the source block has no
1698 name then no name is given to the =#+resname:= line on creation,
1699 otherwise the name of the source block is used).
1701 This will have the additional benefit of allowing results and
1702 source blocks to be located in different places in a buffer (and
1703 eventually in different buffers entirely).
1705 #+srcname: developing-resnames
1706 #+begin_src emacs-lisp :results silent
1710 Once source blocks are able to find their own =#+resname:= lines
1713 #+srcname: sbe-w-new-results
1714 #+begin_src emacs-lisp :results replace
1715 (sbe "developing-resnames")
1721 *** TODO change the results insertion functions to use these lines
1723 *** TODO teach references to resolve =#+resname= lines.
1725 ** DONE org-babel tests org-babel [1/1]
1726 since we are accumulating this nice collection of source-code blocks
1727 in the sandbox section we should make use of them as unit tests.
1728 What's more, we should be able to actually use org-babel to run these
1731 We would just need to cycle over every source code block under the
1732 sandbox, run it, and assert that the return value is equal to what we
1735 I have the feeling that this should be possible using only org-babel
1736 functions with minimal or no additional elisp. It would be very cool
1737 for org-babel to be able to test itself.
1739 This is now done, see [[* Tests]].
1741 *** DEFERRED org-babel assertions (may not be necessary)
1742 These could be used to make assertions about the results of a
1743 source-code block. If the assertion fails then the point could be
1744 moved to the block, and error messages and highlighting etc... could
1747 ** DONE make C-c C-c work anywhere within source code block?
1748 This seems like it would be nice to me, but perhaps it would be
1749 inefficient or ugly in implementation? I suppose you could search
1750 forward, and if you find #+end_src before you find #+begin_src,
1751 then you're inside one. [DED]
1753 Agreed, I think inside of the =#+srcname: line= would be useful as
1756 #+srcname: testing-out-cc
1757 #+begin_src emacs-lisp
1761 ** DONE integration with org tables
1762 We should make it easy to call org-babel source blocks from org-mode
1763 table formulas. This is practical now that it is possible to pass
1764 arguments to org-babel source blocks.
1766 See the related [[* (sandbox) integration w/org tables][sandbox]] header for tests/examples.
1768 *** digging in org-table.el
1769 In the past [[file:~/src/org/lisp/org-table.el::org%20table%20el%20The%20table%20editor%20for%20Org%20mode][org-table.el]] has proven difficult to work with.
1771 Should be a hook in [[file:~/src/org/lisp/org-table.el::defun%20org%20table%20eval%20formula%20optional%20arg%20equation][org-table-eval-formula]].
1773 Looks like I need to change this [[file:~/src/org/lisp/org-table.el::if%20lispp][if statement]] (line 2239) into a cond
1776 ** DONE source blocks as functions
1778 Allow source code blocks to be called like functions, with arguments
1779 specified. We are already able to call a source-code block and assign
1780 it's return result to a variable. This would just add the ability to
1781 specify the values of the arguments to the source code block assuming
1782 any exist. For an example see
1784 When a variable appears in a header argument, how do we differentiate
1785 between it's value being a reference or a literal value? I guess this
1786 could work just like a programming language. If it's escaped or in
1787 quotes, then we count it as a literal, otherwise we try to look it up
1790 ** DONE folding of code blocks? [2/2]
1791 [DED] In similar way to using outline-minor-mode for folding function
1792 bodies, can we fold code blocks? #+begin whatever statements are
1793 pretty ugly, and in any case when you're thinking about the overall
1794 game plan you don't necessarily want to see the code for each Step.
1796 *** DONE folding of source code block
1797 Sounds good, and wasn't too hard to implement. Code blocks should
1798 now be fold-able in the same manner as headlines (by pressing TAB
1801 *** REJECTED folding of results
1802 So, lets do a three-stage tab cycle... First fold the src block,
1803 then fold the results, then unfold.
1805 There's no way to tell if the results are a table or not w/o
1806 actually executing the block which would be too expensive of an
1809 ** DONE selective export of text, code, figures
1810 [DED] The org-babel buffer contains everything (code, headings and
1811 notes/prose describing what you're up to, textual/numeric/graphical
1812 code output, etc). However on export to html / LaTeX one might want
1813 to include only a subset of that content. For example you might
1814 want to create a presentation of what you've done which omits the
1817 [EMS] So I think this should be implemented as a property which can
1818 be set globally or on the outline header level (I need to review
1819 the mechanics of org-mode properties). And then as a source block
1820 header argument which will apply only to a specific source code
1821 block. A header argument of =:export= with values of
1823 - =code= :: just show the code in the source code block
1824 - =none= :: don't show the code or the results of the evaluation
1825 - =results= :: just show the results of the code evaluation (don't
1826 show the actual code)
1827 - =both= :: show both the source code, and the results
1829 this will be done in [[* (sandbox) selective export][(sandbox) selective export]].
1831 ** DONE a header argument specifying silent evaluation (no output)
1832 This would be useful across all types of source block. Currently
1833 there is a =:replace t= option to control output, this could be
1834 generalized to an =:output= option which could take the following
1835 options (maybe more)
1837 - =t= :: this would be the default, and would simply insert the
1838 results after the source block
1839 - =replace= :: to replace any results which may already be there
1840 - =silent= :: this would inhibit any insertion of the results
1842 This is now implemented see the example in the [[* silent evaluation][sandbox]]
1844 ** DONE assign variables from tables in R
1845 This is now working (see [[* (sandbox table) R][(sandbox-table)-R]]). Although it's not that
1846 impressive until we are able to print table results from R.
1848 ** DONE insert 2-D R results as tables
1849 everything is working but R and shell
1855 This has already been tackled by Dan in [[file:existing_tools/org-R.el::defconst%20org%20R%20write%20org%20table%20def][org-R:check-dimensions]]. The
1856 functions there should be useful in combination with [[http://cran.r-project.org/doc/manuals/R-data.html#Export-to-text-files][R-export-to-csv]]
1857 as a means of converting multidimensional R objects to emacs lisp.
1859 It may be as simple as first checking if the data is multidimensional,
1860 and then, if so using =write= to write the data out to a temporary
1861 file from which emacs can read the data in using =org-table-import=.
1863 Looking into this further, is seems that there is no such thing as a
1864 scalar in R [[http://tolstoy.newcastle.edu.au/R/help/03a/3733.html][R-scalar-vs-vector]] In that light I am not sure how to
1865 deal with trivial vectors (scalars) in R. I'm tempted to just treat
1866 them as vectors, but then that would lead to a proliferation of
1867 trivial 1-cell tables...
1869 ** DONE allow variable initialization from source blocks
1870 Currently it is possible to initialize a variable from an org-mode
1871 table with a block argument like =table=sandbox= (note that the
1872 variable doesn't have to named =table=) as in the following example
1878 #+begin_src emacs-lisp :var table=sandbox :results replace
1879 (message (format "table = %S" table))
1882 : "table = ((1 2 3) (4 \"schulte\" 6))"
1884 It would be good to allow initialization of variables from the results
1885 of other source blocks in the same manner. This would probably
1886 require the addition of =#+SRCNAME: example= lines for the naming of
1887 source blocks, also the =table=sandbox= syntax may have to be expanded
1888 to specify whether the target is a source code block or a table
1889 (alternately we could just match the first one with the given name
1890 whether it's a table or a source code block).
1892 At least initially I'll try to implement this so that there is no need
1893 to specify whether the reference is to a table or a source-code block.
1894 That seems to be simpler both in terms of use and implementation.
1896 This is now working for emacs-lisp, ruby and python (and mixtures of
1897 the three) source blocks. See the examples in the [[* (sandbox) referencing other source blocks][sandbox]].
1899 This is currently working only with emacs lisp as in the following
1900 example in the [[* emacs lisp source reference][emacs lisp source reference]].
1903 ** TODO Add languages [0/5]
1904 I'm sure there are many more that aren't listed here. Please add
1905 them, and bubble any that you particularly care about up to the top.
1907 Any new language should be implemented in a org-babel-lang.el file.
1908 Follow the pattern set by [[file:lisp/org-babel-script.el][org-babel-script.el]], [[file:lisp/org-babel-shell.el][org-babel-shell.el]] and
1909 [[file:lisp/org-babel-R.el][org-babel-R.el]].
1912 This could probably be added to [[file:lisp/org-babel-script.el][org-babel-script.el]]
1917 (see [[* file result types][file result types]])
1920 (see [[* file result types][file result types]])
1923 (see [[* file result types][file result types]])
1927 ** TODO Allow source blocks to be recognised when #+ are not first characters on the line
1928 I think Carsten has recently altered the core so that #+ can have
1929 preceding whitespace, at least for literal/code examples. org-babel
1930 should support this.
1931 ** TODO make :results replace the default?
1932 I'm tending to think that appending results to pre-existing results
1933 creates mess, and that the cleaner `replace' option should be the
1934 default. E.g. when a source block creates an image, we would want
1935 that to be updated, rather than have a new one be added.
1939 ** TODO non-orgtbl formatted lists
1942 #+srcname: this-doesn't-match-orgtbl
1943 #+begin_src emacs-lisp :results replace
1944 '((:results . "replace"))
1947 #+resname: this-doesn't-match-orgtbl
1949 ** TODO collapsing consecutive newlines in string output
1951 #+srcname: multi-line-string-output
1952 #+begin_src ruby :results replace
1953 "the first line ends here
1956 and this is the second one
1962 : the first line ends here
1963 : and this is the second one
1964 : return even a third
1966 ** TODO cursor movement when evaluating source blocks
1967 E.g. the pie chart example. Despite the save-window-excursion in
1968 org-babel-execute:R. (I never learned how to do this properly: org-R
1969 jumps all over the place...)
1971 ** PROPOSED external shell execution can't isolate return values
1972 I have no idea how to do this as of yet. The result is that when
1973 shell functions are run w/o a session there is no difference between
1974 the =output= and =value= result arguments.
1976 Yea, I don't know how to do this either. I searched extensively on
1977 how to isolate the *last* output of a series of shell commands (see
1978 [[* last command for
1979 shells][last command for shells]]). The results of the search were basically
1980 that it was not possible (or at least not accomplish-able with a
1981 reasonable amount of effort).
1983 That fact combined with the tenancy to all ways use standard out in
1984 shell scripts led me to treat these two options (=output= and =value=)
1985 as identical in shell evaluation. Not ideal but maybe good enough for
1988 ** DEFERRED weird escaped characters in shell prompt break shell evaluation
1989 E.g. this doesn't work. Should the shell sessions set a sane prompt
1990 when they start up? Or is it a question of altering
1991 comint-prompt-regexp? Or altering org-babel regexps?
1994 black=30 ; red=31 ; green=32 ; yellow=33 ; blue=34 ; magenta=35 ; cyan=36 ; white=37
1997 export PS1="\[\033[${prompt_col}m\]\w${prompt_char} \[\033[0m\]"
2000 I just pushed a good amount of changes, could you see if your shell
2001 problems still exist?
2003 The problem's still there. Specifically, aIui, at [[file:lisp/langs/org-babel-sh.el::raw%20org%20babel%20comint%20with%20output%20buffer%20org%20babel%20sh%20eoe%20output%20nil%20insert%20full%20body%20comint%20send%20input%20nil%20t][this line]] of
2004 org-babel-sh.el, raw gets the value
2006 ("" "
\e[0m Sun Jun 14 19:26:24 EDT 2009\n" "
\e[0m org_babel_sh_eoe\n" "
\e[0m ")
2008 and therefore (member org-babel-sh-eoe-output ...) fails
2010 I think that `comint-prompt-regexp' needs to be altered to match
2011 the shell prompt. This shouldn't be too difficult to do by hand,
2012 using the `regexp-builder' command and should probably be part of
2013 the user's regular emacs init. I can't think of a way for us to
2014 set this automatically, and we are SOL without a regexp to match
2016 ** DONE ruby evaluation not working under ubuntu emacs 23
2017 With emacs 23.0.91.1 on ubuntu, for C-h f run-ruby I have the
2018 following, which seems to conflict with [[file:lisp/langs/org-babel-ruby.el::let%20session%20buffer%20save%20window%20excursion%20run%20ruby%20nil%20session%20current%20buffer][this line]] in org-babel-ruby.el.
2021 run-ruby is an interactive compiled Lisp function.
2025 Run an inferior Ruby process, input and output via buffer *ruby*.
2026 If there is a process already running in `*ruby*', switch to that buffer.
2027 With argument, allows you to edit the command line (default is value
2028 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
2029 (after the `comint-mode-hook' is run).
2030 (Type C-h m in the process buffer for a list of commands.)
2033 So, I may have a non-standard inf-ruby.el. Here's my version of
2037 run-ruby is an interactive Lisp function in `inf-ruby.el'.
2039 (run-ruby &optional COMMAND NAME)
2041 Run an inferior Ruby process, input and output via buffer *ruby*.
2042 If there is a process already running in `*ruby*', switch to that buffer.
2043 With argument, allows you to edit the command line (default is value
2044 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
2045 (after the `comint-mode-hook' is run).
2046 (Type C-h m in the process buffer for a list of commands.)
2049 It seems we could either bundle my version of inf-ruby.el (as it's
2050 the newest). Or we could change the use of `run-ruby' so that it
2051 is robust across multiple distributions. I think I'd prefer the
2052 former, unless the older version of inf-ruby is actually bundled
2053 with emacs, in which case maybe we should go out of our way to
2054 support it. Thoughts?
2056 I think for now I'll just include the latest [[file:util/inf-ruby.el][inf-ruby.el]] in the
2057 newly created utility directory. I doubt anyone would have a
2058 problem using the latest version of this file.
2059 ** DONE test failing forcing vector results with =test-forced-vector-results= ruby code block
2060 Note that this only seems to happen the *second* time the test table
2063 #+srcname: bug-trivial-vector
2064 #+begin_src emacs-lisp :results vector silent
2068 #+srcname: bug-forced-vector-results
2069 #+begin_src ruby :var triv=test-trivial-vector :results silent
2073 mysteriously this seems to be fixed...
2074 ** DONE defunct R sessions
2075 Sometimes an old R session will turn defunct, and newly inserted code
2076 will not be evaluated (leading to a hang).
2078 This seems to be fixed by using `inferior-ess-send-input' rather than `comint-send-input'.
2079 ** DONE ruby fails on first call to non-default session
2081 #+srcname: bug-new-session
2082 #+begin_src ruby :session is-new
2086 ** DONE when reading results from =#+resname= line
2088 Errors when trying to read from resname lines.
2090 #+resname: bug-in-resname
2093 #+srcname: bug-in-resname-reader
2094 #+begin_src emacs-lisp :var buggy=bug-in-resname() :results silent
2098 ** DONE R-code broke on "org-babel" rename
2100 #+srcname: bug-R-babels
2105 ** DONE error on trivial R results
2107 So I know it's generally not a good idea to squash error without
2108 handling them, but in this case the error almost always means that
2109 there was no file contents to be read by =org-table-import=, so I
2112 #+srcname: bug-trivial-r1
2113 #+begin_src R :results replace
2114 pie(c(1, 2, 3), labels = c(1, 2, 3))
2117 #+srcname: bug-trivial-r2
2118 #+begin_src R :results replace
2122 #+resname: bug-trivial-r2
2125 #+srcname: bug-trivial-r3
2126 #+begin_src R :results replace
2130 #+resname: bug-trivial-r3
2135 ** DONE ruby new variable creation (multi-line ruby blocks)
2136 Actually it looks like we were dropping all but the last line.
2138 #+srcname: multi-line-ruby-test
2139 #+begin_src ruby :var table=bug-numerical-table :results replace
2141 table.each{|n| total += n}
2148 ** DONE R code execution seems to choke on certain inputs
2149 Currently the R code seems to work on vertical (but not landscape)
2152 #+srcname: little-fake
2153 #+begin_src emacs-lisp
2157 #+begin_src R :var num=little-fake
2171 #+srcname: set-debug-on-error
2172 #+begin_src emacs-lisp :results silent
2173 (setq debug-on-error t)
2176 #+srcname: bug-numerical-table
2177 #+begin_src emacs-lisp :results silent
2181 #+srcname: bug-R-number-evaluation
2182 #+begin_src R :var table=bug-numerical-table :results replace
2189 #+tblname: bug-vert-table
2194 #+srcname: bug-R-vertical-table
2195 #+begin_src R :var table=bug-vert-table :results silent
2199 ** DONE org bug/request: prevent certain org behaviour within code blocks
2200 E.g. [[]] gets recognised as a link (when there's text inside the
2201 brackets). This is bad for R code at least, and more generally
2202 could be argued to be inappropriate. Is it difficult to get org to
2203 ignore text in code blocks? [DED]
2205 I believe Carsten addressed this recently on the mailing list with
2206 the comment that it was indeed a difficult issue. I believe this
2207 may be one area where we could wait for an upstream (org-mode) fix.
2209 [Dan] Carsten has fixed this now in the core.
2211 ** DONE with :results replace, non-table output doesn't replace table output
2212 And vice versa. E.g. Try this first with table and then with len(table) [DED]
2213 #+begin_src python :var table=sandbox :results replace
2218 | 4 | "schulte" | 6 |
2221 Yes, this is certainly a problem. I fear that if we begin replacing
2222 anything immediately following a source block (regardless of whether
2223 it matches the type of our current results) we may accidentally delete
2224 hand written portions of the user's org-mode buffer.
2226 I think that the best solution here would be to actually start
2227 labeling results with a line that looks something like...
2231 This would have a couple of benefits...
2232 1) we wouldn't have to worry about possibly deleting non-results
2233 (which is currently an issue)
2234 2) we could reliably replace results even if there are different types
2235 3) we could reference the results of a source-code block in variable
2236 definitions, which would be useful if for example we don't wish to
2237 re-run a source-block every time because it is long-running.
2239 Thoughts? If no-one objects, I believe I will implement the labeling
2242 ** DONE extra quotes for nested string
2243 Well R appears to be reading the tables without issue...
2245 these *should* be quoted
2247 #+begin_src sh :results replace
2252 | "README.markdown" |
2255 | "existing_tools" |
2259 | "test-export.html" |
2260 | "test-export.org" |
2262 #+srcname: test-quotes
2263 #+begin_src ruby :var tab=ls
2269 #+srcname: test-quotes
2270 #+begin_src R :var tab=ls
2276 ** DONE simple ruby arrays not working
2278 As an example eval the following. Adding a line to test
2280 #+tblname: simple-ruby-array
2283 #+srcname: ruby-array-test
2284 #+begin_src ruby :var ar = simple-ruby-array :results silent
2288 ** DONE space trailing language name
2289 fix regexp so it works when there's a space trailing the language name
2291 #+srcname: test-trailing-space
2296 ** DONE Args out of range error
2298 The following block resulted in the error below [DED]. It ran without
2299 error directly in the shell.
2302 for platf in ill aff ; do
2303 for pop in CEU YRI ASI ; do
2304 rm -f $platf/hapmap-genos-$pop-all $platf/hapmap-rs-all
2305 cat $platf/hapmap-genos-$pop-* > $platf/hapmap-genos-$pop-all
2306 cat $platf/hapmap-rs-* > $platf/hapmap-rs-all
2311 executing source block with sh...
2312 finished executing source block
2313 string-equal: Args out of range: "", -1, 0
2315 the error =string-equal: Args out of range: "", -1, 0= looks like what
2316 used to be output when the block returned an empty results string.
2317 This should be fixed in the current version, you should now see the
2318 following message =no result returned by source block=.
2320 ** DONE ruby arrays not recognized as such
2322 Something is wrong in [[file:lisp/org-babel-script.el]] related to the
2323 recognition of ruby arrays as such.
2325 #+begin_src ruby :results replace
2331 #+begin_src python :results replace
2339 Evaluate all the cells in this table for a comprehensive test of the
2340 org-babel functionality.
2342 *Note*: if you have customized =org-babel-default-header-args= then some
2343 of these tests may fail.
2345 #+TBLNAME: org-babel-tests
2346 | functionality | block | arg | expected | results | pass |
2347 |-------------------------+----------------------------+-----+-------------+-------------+------|
2348 | basic evaluation | | | | | |
2349 |-------------------------+----------------------------+-----+-------------+-------------+------|
2350 | emacs lisp | basic-elisp | | 5 | | |
2351 | shell | basic-shell | | 6 | | |
2352 | ruby | basic-ruby | | org-babel | | |
2353 | python | basic-python | | hello world | | |
2354 | R | basic-R | | 13 | | |
2355 |-------------------------+----------------------------+-----+-------------+-------------+------|
2356 | tables | | | | | |
2357 |-------------------------+----------------------------+-----+-------------+-------------+------|
2358 | emacs lisp | table-elisp | | 3 | | |
2359 | ruby | table-ruby | | 1-2-3 | | |
2360 | python | table-python | | 5 | | |
2361 | R | table-R | | 3.5 | | |
2362 |-------------------------+----------------------------+-----+-------------+-------------+------|
2363 | source block references | | | | | |
2364 |-------------------------+----------------------------+-----+-------------+-------------+------|
2365 | all languages | chained-ref-last | | Array | | |
2366 |-------------------------+----------------------------+-----+-------------+-------------+------|
2367 | source block functions | | | | | |
2368 |-------------------------+----------------------------+-----+-------------+-------------+------|
2369 | emacs lisp | defun-fibb | | fibbd | | |
2370 | run over | Fibonacci | 0 | 1 | | |
2371 | a | Fibonacci | 1 | 1 | | |
2372 | variety | Fibonacci | 2 | 2 | | |
2373 | of | Fibonacci | 3 | 3 | | |
2374 | different | Fibonacci | 4 | 5 | | |
2375 | arguments | Fibonacci | 5 | 8 | | |
2376 |-------------------------+----------------------------+-----+-------------+-------------+------|
2377 | bugs and tasks | | | | | |
2378 |-------------------------+----------------------------+-----+-------------+-------------+------|
2379 | simple ruby arrays | ruby-array-test | | 3 | | |
2380 | R number evaluation | bug-R-number-evaluation | | 2 | | |
2381 | multi-line ruby blocks | multi-line-ruby-test | | 2 | | |
2382 | forcing vector results | test-forced-vector-results | | Array | | |
2383 |-------------------------+----------------------------+-----+-------------+-------------+------|
2384 | sessions | | | | | |
2385 |-------------------------+----------------------------+-----+-------------+-------------+------|
2386 | set ruby session | set-ruby-session-var | | :set | | |
2387 | get from ruby session | get-ruby-session-var | | 3 | | |
2388 | set python session | set-python-session-var | | set | | |
2389 | get from python session | get-python-session-var | | 4 | | |
2390 | set R session | set-R-session-var | | set | | |
2391 | get from R session | get-R-session-var | | 5 | | |
2392 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
2396 #+srcname: basic-elisp
2397 #+begin_src emacs-lisp :results silent
2401 #+srcname: basic-shell
2402 #+begin_src sh :results silent
2406 #+srcname: date-simple
2407 #+begin_src sh :results silent
2411 #+srcname: basic-ruby
2412 #+begin_src ruby :results silent
2416 #+srcname: basic-python
2417 #+begin_src python :results silent
2422 #+begin_src R :results silent
2429 #+tblname: test-table
2433 #+srcname: table-elisp
2434 #+begin_src emacs-lisp :results silent :var table=test-table
2435 (length (car table))
2438 #+srcname: table-ruby
2439 #+begin_src ruby :results silent :var table=test-table
2440 table.first.join("-")
2443 #+srcname: table-python
2444 #+begin_src python :var table=test-table
2449 #+begin_src R :var table=test-table
2455 Lets pass a references through all of our languages...
2457 Lets start by reversing the table from the previous examples
2459 #+srcname: chained-ref-first
2460 #+begin_src python :var table = test-table
2465 #+resname: chained-ref-first
2469 Take the first part of the list
2471 #+srcname: chained-ref-second
2472 #+begin_src R :var table = chained-ref-first
2476 #+resname: chained-ref-second
2480 Turn the numbers into string
2482 #+srcname: chained-ref-third
2483 #+begin_src emacs-lisp :var table = chained-ref-second
2484 (mapcar (lambda (el) (format "%S" el)) table)
2487 #+resname: chained-ref-third
2490 and Check that it is still a list
2492 #+srcname: chained-ref-last
2493 #+begin_src ruby :var table=chained-ref-third
2497 ** source blocks as functions
2499 #+srcname: defun-fibb
2500 #+begin_src emacs-lisp :results silent
2501 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
2504 #+srcname: fibonacci
2505 #+begin_src emacs-lisp :results silent :var n=7
2509 ** sbe tests (these don't seem to be working...)
2510 Testing the insertion of results into org-mode tables.
2512 #+srcname: multi-line-output
2513 #+begin_src ruby :results replace
2514 "the first line ends here
2517 and this is the second one
2523 : the first line ends here\n\n\n and this is the second one\n\neven a third
2525 #+srcname: multi-line-error
2526 #+begin_src ruby :results replace
2527 raise "oh nooooooooooo"
2533 | the first line ends here... | -:5: warning: parenthesize argument(s) for future version... |
2534 #+TBLFM: $1='(sbe "multi-line-output")::$2='(sbe "multi-line-error")
2536 ** forcing results types tests
2538 #+srcname: test-trivial-vector
2539 #+begin_src emacs-lisp :results vector silent
2543 #+srcname: test-forced-vector-results
2544 #+begin_src ruby :var triv=test-trivial-vector :results silent
2550 #+srcname: set-ruby-session-var
2551 #+begin_src ruby :session :results silent
2556 #+srcname: get-ruby-session-var
2557 #+begin_src ruby :session :results silent
2561 #+srcname: set-python-session-var
2562 #+begin_src python :session
2567 #+srcname: get-python-session-var
2568 #+begin_src python :session
2572 #+srcname: set-R-session-var
2573 #+begin_src R :session
2578 #+srcname: get-R-session-var
2579 #+begin_src R :session
2588 To run these examples evaluate [[file:lisp/org-babel-init.el][org-babel-init.el]]
2590 ** org-babel.el beginning functionality
2592 #+begin_src sh :results replace
2596 : Thu May 14 18:52:25 EDT 2009
2602 : Thu May 14 18:59:09 -0400 2009
2613 #+begin_src R :results replace
2626 hist(rgamma(20,3,3))
2630 ** org-babel plays with tables
2631 Alright, this should demonstrate both the ability of org-babel to read
2632 tables into a lisp source code block, and to then convert the results
2633 of the source code block into an org table. It's using the classic
2634 "lisp is elegant" demonstration transpose function. To try this
2637 1. evaluate [[file:lisp/org-babel-init.el]] to load org-babel and friends
2638 2. evaluate the transpose definition =\C-c\\C-c= on the beginning of
2640 3. evaluate the next source code block, this should read in the table
2641 because of the =:var table=previous=, then transpose the table, and
2642 finally it should insert the transposed table into the buffer
2643 immediately following the block
2647 #+begin_src emacs-lisp :results silent
2648 (defun transpose (table)
2649 (apply #'mapcar* #'list table))
2657 #+begin_src emacs-lisp :var table=sandbox :results replace
2662 #+begin_src emacs-lisp
2666 | 1 | 2 | 3 | 4 | 5 |
2670 #+begin_src ruby :var table=sandbox :results replace
2671 table.first.join(" - ")
2677 #+begin_src python :var table=sandbox
2684 #+begin_src ruby :var table=sandbox :results replace
2689 | 4 | "schulte" | 6 |
2691 #+begin_src python :var table=sandbox :results replace
2697 | "__add__" | "__class__" | "__contains__" | "__delattr__" | "__delitem__" | "__delslice__" | "__doc__" | "__eq__" | "__format__" | "__ge__" | "__getattribute__" | "__getitem__" | "__getslice__" | "__gt__" | "__hash__" | "__iadd__" | "__imul__" | "__init__" | "__iter__" | "__le__" | "__len__" | "__lt__" | "__mul__" | "__ne__" | "__new__" | "__reduce__" | "__reduce_ex__" | "__repr__" | "__reversed__" | "__rmul__" | "__setattr__" | "__setitem__" | "__setslice__" | "__sizeof__" | "__str__" | "__subclasshook__" | "append" | "count" | "extend" | "index" | "insert" | "pop" | "remove" | "reverse" | "sort" |
2699 *** (sandbox table) R
2701 #+TBLNAME: sandbox_r
2705 #+begin_src R :results replace
2706 x <- c(rnorm(10, mean=-3, sd=1), rnorm(10, mean=3, sd=1))
2710 | -3.35473133869346 |
2712 | -3.32819924928633 |
2713 | -2.97310212756194 |
2714 | -2.09640758369576 |
2715 | -5.06054014378736 |
2716 | -2.20713700711221 |
2717 | -1.37618039712037 |
2718 | -1.95839385821742 |
2719 | -3.90407396475502 |
2720 | 2.51168071590226 |
2721 | 3.96753011570494 |
2722 | 3.31793212627865 |
2723 | 1.99829753972341 |
2724 | 4.00403686419829 |
2725 | 4.63723764452927 |
2726 | 3.94636744261313 |
2727 | 3.58355906547775 |
2728 | 3.01563442274226 |
2731 #+begin_src R var tabel=sandbox_r :results replace
2736 | 4 | "schulte" | 6 |
2739 Now shell commands are converted to tables using =org-table-import=
2740 and if these tables are non-trivial (i.e. have multiple elements) then
2741 they are imported as org-mode tables...
2743 #+begin_src sh :results replace
2747 | "total" | 208 | "" | "" | "" | "" | "" | "" |
2748 | "-rw-r--r--" | 1 | "dan" | "dan" | 57 | 2009 | 15 | "block" |
2749 | "-rw-r--r--" | 1 | "dan" | "dan" | 35147 | 2009 | 15 | "COPYING" |
2750 | "-rw-r--r--" | 1 | "dan" | "dan" | 722 | 2009 | 18 | "examples.org" |
2751 | "drwxr-xr-x" | 4 | "dan" | "dan" | 4096 | 2009 | 19 | "existing_tools" |
2752 | "-rw-r--r--" | 1 | "dan" | "dan" | 2207 | 2009 | 14 | "intro.org" |
2753 | "drwxr-xr-x" | 2 | "dan" | "dan" | 4096 | 2009 | 18 | "org-babel" |
2754 | "-rw-r--r--" | 1 | "dan" | "dan" | 277 | 2009 | 20 | "README.markdown" |
2755 | "-rw-r--r--" | 1 | "dan" | "dan" | 11837 | 2009 | 18 | "rorg.html" |
2756 | "-rw-r--r--" | 1 | "dan" | "dan" | 61829 | 2009 | 19 | "#rorg.org#" |
2757 | "-rw-r--r--" | 1 | "dan" | "dan" | 60190 | 2009 | 19 | "rorg.org" |
2758 | "-rw-r--r--" | 1 | "dan" | "dan" | 972 | 2009 | 11 | "test-export.org" |
2761 ** silent evaluation
2769 #+begin_src ruby :results silent
2773 #+begin_src ruby :results replace
2780 ** (sandbox) referencing other source blocks
2781 Doing this in emacs-lisp first because it's trivial to convert
2782 emacs-lisp results to and from emacs-lisp.
2784 *** emacs lisp source reference
2785 This first example performs a calculation in the first source block
2786 named =top=, the results of this calculation are then saved into the
2787 variable =first= by the header argument =:var first=top=, and it is
2788 used in the calculations of the second source block.
2791 #+begin_src emacs-lisp
2795 #+begin_src emacs-lisp :var first=top :results replace
2801 This example is the same as the previous only the variable being
2802 passed through is a table rather than a number.
2804 #+begin_src emacs-lisp :results silent
2805 (defun transpose (table)
2806 (apply #'mapcar* #'list table))
2809 #+TBLNAME: top_table
2813 #+SRCNAME: second_src_example
2814 #+begin_src emacs-lisp :var table=top_table
2818 #+begin_src emacs-lisp :var table=second_src_example :results replace
2823 | 4 | "schulte" | 6 |
2825 Now working for ruby
2832 #+begin_src ruby :var other=start :results replace
2838 #+SRCNAME: start_two
2843 #+begin_src python :var another=start_two :results replace
2848 Since all variables are converted into Emacs Lisp it is no problem to
2849 reference variables specified in another language.
2851 #+SRCNAME: ruby-block
2856 #+SRCNAME: lisp_block
2857 #+begin_src emacs-lisp :var ruby-variable=ruby-block
2861 #+begin_src python :var lisp_var=lisp_block
2870 #+begin_src R :results replace
2877 #+begin_src R :var other=first_r :results replace
2884 ** (sandbox) selective export
2886 For exportation tests and examples see (including exportation of
2887 inline source code blocks) [[file:test-export.org]]
2890 ** (sandbox) source blocks as functions
2893 #+begin_src emacs-lisp :results silent
2898 #+begin_src emacs-lisp :var n=default :results replace
2904 #+begin_src emacs-lisp :var result=triple(n=3, m=98) :results replace
2910 The following just demonstrates the ability to assign variables to
2911 literal values, which was not implemented until recently.
2913 #+begin_src ruby :var num="eric" :results replace
2920 ** (sandbox) inline source blocks
2922 This is an inline source code block src_ruby{1 + 6}. And another
2923 source block with text output src_emacs-lisp{"eric"}.
2925 This is an inline source code block with header
2926 arguments. src_ruby[:var n=fibbd( n = 0 )]{n}
2929 ** (sandbox) integration w/org tables
2931 #+begin_src emacs-lisp :results silent
2932 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
2936 #+begin_src emacs-lisp :var n=4 :results silent
2940 #+begin_src emacs-lisp :results silent
2941 (mapcar #'fibbd '(0 1 2 3 4 5 6 7 8))
2944 Something is not working here. The function `sbe ' works fine when
2945 called from outside of the table (see the source block below), but
2946 produces an error when called from inside the table. I think there
2947 must be some narrowing going on during intra-table emacs-lisp
2950 | original | fibbd |
2951 |----------+-------|
2962 #+TBLFM: $2='(sbe "fibbd" (n $1))
2966 #+begin_src emacs-lisp :results silent
2967 (sbe 'fibbd (n "8"))
2972 LocalWords: DBlocks dblocks org-babel el eric fontification