1 #+TITLE: org-babel --- facilitating communication between programming languages and people
2 #+SEQ_TODO: PROPOSED TODO STARTED | DONE DEFERRED REJECTED
3 #+OPTIONS: H:3 num:nil toc:t
4 #+STARTUP: oddeven hideblocks
6 Through Org-Babel Org-Mode can communicate with programming languages.
7 Code contained in source-code blocks can be evaluated and data can
8 pass seamlessly between different programming languages, Org-Mode
9 constructs (tables, file links, example text) and interactive comint
13 - The [[* Introduction][Introduction]] :: provides a brief overview of the design and use
14 of Org-Babel including tutorials and examples.
15 - In [[* Getting started][Getting Started]] :: find instructions for installing org-babel
16 into your emacs configuration.
17 - The [[* Tasks][Tasks]] :: section contains current and past tasks roughly ordered
18 by TODO state, then importance or date-completed. This would be
19 a good place to suggest ideas for development.
20 - The [[* Bugs][Bugs]] :: section contains bug reports.
21 - The [[* Tests][Tests]] :: section consists of a large table which can be
22 evaluated to run Org-Babel's functional test suite. This
23 provides a good overview of the current functionality with
24 pointers to example source blocks.
25 - The [[* Sandbox][Sandbox]] :: demonstrates much of the early/basic functionality
26 through commented source-code blocks.
28 Also see the [[file:library-of-babel.org][Library of Babel]], an extensible collection of ready-made
29 and easily-shortcut-callable source-code blocks for handling common
34 Org-Babel enables *communication* between programming languages and
38 - communication between programs :: Data passes seamlessly between
39 different programming languages, Org-Mode constructs (tables,
40 file links, example text) and interactive comint buffers.
41 - communication between people :: Data and calculations are embedded
42 in the same document as notes explanations and reports.
44 ** communication between programs
46 Org-Mode supports embedded blocks of source code (in any language)
47 inside of Org documents. Org-Babel allows these blocks of code to be
48 executed from within Org-Mode with natural handling of their inputs
52 with both scalar, file, and table output
54 *** reading information from tables
56 *** reading information from other source blocks (disk usage in your home directory)
58 This will work for Linux and Mac users, not so sure about shell
59 commands for windows users.
61 To run place the cursor on the =#+begin_src= line of the source block
62 labeled directory-pie and press =\C-c\C-c=.
64 #+srcname: directories
65 #+begin_src bash :results replace
66 cd ~ && du -sc * |grep -v total
69 #+resname: directories
71 | 11882808 | "Documents" |
72 | 8210024 | "Downloads" |
73 | 879800 | "Library" |
76 | 5307664 | "Pictures" |
85 #+srcname: directory-pie
86 #+begin_src R :var dirs = directories :session R-pie-example
87 pie(dirs[,1], labels = dirs[,2])
95 *** operations in/on tables
97 #+tblname: grades-table
98 | student | grade | letter |
99 |---------+-------+--------|
106 #+TBLFM: $2='(sbe random-score-generator)::$3='(sbe assign-grade (score $2))
108 #+srcname: assign-grade
109 #+begin_src ruby :var score=99
120 #+srcname: random-score-generator
125 #+srcname: show-distribution
126 #+begin_src R :var grades=grades-table :session *R*
134 ** communication between people
135 Quick overview of Org-Mode's exportation abilities, with links to the
136 online Org-Mode documentation, a focus on source-code blocks, and the
137 exportation options provided by Org-Babel.
139 *** Interactive tutorial
140 This would demonstrate applicability to Reproducible Research, and
141 Literate Programming.
143 *** Tests embedded in documentation
144 org-babels own functional tests are contained in a large org-mode
145 table, allowing the test suite to be run be evaluation of the table
146 and the results to be collected in the same table.
148 *** Emacs initialization files stored in Org-Mode buffers
149 Using `org-babel-tangle' it is possible to embed your Emacs
150 initialization into org-mode files. This allows for folding,
151 note-taking, todo's etc... embedded with the source-code of your Emacs
152 initialization, and through org-mode's publishing features aids in
153 sharing your customizations with others.
155 It may be worthwhile to create a fork of Phil Hagelberg's
156 [[http://github.com/technomancy/emacs-starter-kit/tree/master][emacs-starter-kit]] which uses literate org-mode files for all of the
157 actual elisp customization. These org-mode files could then be
158 exported to html and used to populate the repositories wiki on [[http://github.com/][github]].
163 *** code evaluation (comint buffer sessions and external processes)
164 There are two main ways to evaluate source blocks with org-babel.
166 - external :: By default (if the =:session= header argument is not
167 present) all source code blocks are evaluated in
168 external processes. In these cases an external process
169 is used to evaluate the source-code blocks.
170 - session :: Session based evaluation uses persistent sessions in
171 comint buffers. Sessions can be used across multiple
172 source blocks setting and accessing variables in the
175 Evaluating source blocks in sessions also allows for
176 interaction with the code. To jump to the session of a
177 source block use the `org-babel-pop-to-session' command
178 or press =M-[down]= while inside of a source code block.
179 When called with a prefix argument
180 `org-babel-pop-to-session' will evaluate all header
181 arguments before jumping to the source-code block.
183 *** results (values and outputs)
184 Either the *value* or the *output* of source code blocks can be
185 collected after evaluation.
187 - value :: The default way to collect results from a source-code block
188 is to return the value of the last statement in the block.
189 This can be thought of as the return value of the block.
190 In this case any printed output of the block is ignored.
191 This can be though of a similar to a "functional" value of
193 - output :: Another way of generating results from a source-code block
194 is to collect the output generated by the execution of the
195 block. In this case all printed output is collected
196 throughout the execution of the block. This can be
197 thought of as similar to a "script" style of evaluation.
201 Add the following lines to your .emacs, replacing the path as
202 appropriate. A good place to check that things are up and running
203 would then be [[#sandbox][the sandbox]].
204 #+begin_src emacs-lisp
205 (add-to-list 'load-path "/path/to/org-babel/lisp")
206 (require 'org-babel-init)
211 ** PROPOSED raise elisp error when source-blocks return errors
212 Not sure how/if this would work, but it may be desirable.
214 ** PROPOSED allow `anonymous' function block with function call args?
215 My question here is simply whether we're going to allow
216 #+begin_src python(arg=ref)
220 but with preference given to
221 #+srcname blockname(arg=ref)
222 ** PROPOSED allow :result as synonym for :results?
223 ** PROPOSED allow 'output mode to return stdout as value?
224 Maybe we should allow this. In fact, if block x is called
225 with :results output, and it references blocks y and z, then
226 shouldn't the output of x contain a concatenation of the outputs of
227 y and z, together with x's own output? That would raise the
228 question of what happens if y is defined with :results output and z
229 with :results value. I guess z's (possibly vector/tabular) output
230 would be inside a literal example block containing the whole lot.
231 ** PROPOSED optional timestamp for output
232 Add option to place an (inactive) timestamp at the #+resname, to
233 record when that output was generated.
235 *** source code block timestamps (optional addition)
236 [Eric] If we did this would we then want to place a timestamp on the
237 source-code block, so that we would know if the results are
238 current or out of date? This would have the effect of caching the
239 results of calculations and then only re-running if the
240 source-code has changed. For the caching to work we would need to
241 check not only the timestamp on a source-code block, but also the
242 timestamps of any tables or source-code blocks referenced by the
243 original source-code block.
245 [Dan] I do remember getting frustrated by Sweave always having to
246 re-do everything, so this could be desirable, as long as it's easy
247 to over-ride of course. I'm not sure it should be the default
248 behaviour unless we are very confident that it works well.
250 **** maintaining source-code block timestamps
251 It may make sense to add a hook to `org-edit-special' which could
252 update the source-code blocks timestamp. If the user edits the
253 contents of a source-code block directly I can think of no
254 efficient way of maintaining the timestamp.
256 ** TODO make tangle files read-only?
257 With a file-local variable setting, yea that makes sense. Maybe
258 the header should reference the related org-mode file.
260 ** TODO take default values for header args from properties
261 Use file-wide and subtree wide properties to set default values for
263 ** TODO support for working with =*Org Edit Src Example*= buffers [2/4]
264 *** TODO optionally evaluate header references when we switch to =*Org Edit Src*= buffer
265 That seems to imply that the header references need to be evaluated
266 and transformed into the target language object when we hit C-c ' to
267 enter the *Org Edit Src* buffer [DED]
269 Good point, I heartily agree that this should be supported [Eric]
271 (or at least before the first time we attempt to evaluate code in that
272 buffer -- I suppose there might be an argument for lazy evaluation, in
273 case someone hits C-c ' but is "just looking" and not actually
274 evaluating anything.) Of course if evaluating the reference is
275 computationally intensive then the user might have to wait before they
276 get the *Org Edit Src* buffer. [DED]
278 I fear that it may be hard to anticipate when the references will be
279 needed, some major-modes do on-the-fly evaluation while the buffer is
280 being edited. I think that we should either do this before the buffer
281 is opened or not at all, specifically I think we should resolve
282 references if the user calls C-c ' with a prefix argument. Does that
283 sound reasonable? [Eric]
287 [Dan] So now that we have org-src-mode and org-src-mode-hook, I guess
288 org-babel should do this by using the hook to make sure that, when C-c
289 C-' is issued on a source block, any references are resolved and
290 assignments are made in the appropriate session.
291 *** TODO set buffer-local-process variables appropriately [DED]
292 I think something like this would be great. You've probably
293 already thought of this, but just to note it down: it would be really
294 nice if org-babel's notion of a buffer's 'session/process' played
295 nicely with ESS's notion of the buffer's session/process. ESS keeps
296 the current process name for a buffer in a buffer-local variable
297 ess-local-process-name. So one thing we will probably want to do is
298 make sure that the *Org Edit Src Example* buffer sets that variable
301 I had not thought of that, but I agree whole heartedly. [Eric]
303 Once this is done every variable should be able to dump regions into
304 their inferior-process buffer using major-mode functions.
305 *** DEFERRED send code to inferior process
306 Another thought on this topic: I think we will want users to send
307 chunks of code to the interpreter from within the *Org Edit Src*
308 buffer, and I think that's what you have in mind already. In ESS that
309 is done using the ess-eval-* functions. [DED]
311 I think we can leave this up to the major-mode in the source code
312 buffer, as almost every source-code major mode will have functions for
313 doing things like sending regions to the inferior process. If
314 anything we might need to set the value of the buffer local inferior
315 process variable. [Eric]
317 *** DONE some possible requests/proposed changes for Carsten [4/4]
318 While I remember, some possible requests/proposed changes for Carsten
319 come to mind in that regard:
321 **** DONE Remap C-x C-s to save the source to the org buffer?
322 I've done this personally and I find it essential. I'm using
323 #+begin_src emacs-lisp
324 (defun org-edit-src-save ()
325 "Update the parent org buffer with the edited source code, save
326 the parent org-buffer, and return to the source code edit
335 (define-key org-exit-edit-mode-map "\C-x\C-s" 'org-edit-src-save)
339 I think this is great, but I think it should be implemented in the
342 **** DEFERRED Rename buffer and minor mode?
343 Something shorter than *Org Edit Src Example* for the buffer
344 name. org-babel is bringing org's source code interaction to a
345 level of maturity where the 'example' is no longer
346 appropriate. And if further keybindings are going to be added to
347 the minor mode then maybe org-edit-src-mode is a better name than
350 Maybe we should name the buffer with a combination of the source
351 code and the session. I think that makes sense.
353 [ES] Are you also suggesting a new org-edit-src minor mode?
354 [DED] org-exit-edit-mode is a minor mode that already exists:
356 Minor mode installing a single key binding, "C-c '" to exit special edit.
358 org-edit-src-save now has a binding in that mode, so I guess all
359 I'm saying at this stage is that it's a bit of a misnomer. But
360 perhaps we will also have more functionality to add to that minor
361 mode, making it even more of a misnomer. Perhaps something like
362 org-src-mode would be better.
363 **** DONE Changed minor mode name and added hooks
365 **** DONE a hook called when the src edit buffer is created
366 This should be implemented in the org-mode core
368 ** TODO resolve references to other org buffers/files
369 This would allow source blocks to call upon tables, source-blocks,
370 and results in other org buffers/files.
373 - [[file:lisp/org-babel-ref.el::TODO%20allow%20searching%20for%20names%20in%20other%20buffers][org-babel-ref.el:searching-in-other-buffers]]
374 - [[file:lisp/org-babel.el::defun%20org-babel%20find%20named%20result%20name][org-babel.el#org-babel-find-named-result]]
375 ** TODO resolve references to other non-org files
376 - tabular data in .csv, .tsv etc format
377 - files of interpreted code: anything stopping us giving such files
378 similar status to a source code block?
379 - Would be nice to allow org and non-org files to be remote
380 ** TODO figure out how to handle errors during evaluation
381 I expect it will be hard to do this properly, but ultimately it
382 would be nice to be able to specify somewhere to receive STDERR,
383 and to be warned if it is non-empty.
385 Probably simpler in non-session evaluation than session? At least
386 the mechanism will be different I guess.
388 R has a try function, with error handling, along the lines of
389 python. I bet ruby does too. Maybe more of an issue for functional
390 style; in my proposed scripting style the error just gets dumped to
391 the org buffer and the user is thus alerted.
392 ** TODO figure out how to handle graphic output
393 This is listed under [[* graphical output][graphical output]] in out objectives.
395 This should take advantage of the =:results file= option, and
396 languages which almost always produce graphical output should set
397 =:results file= to true by default (this is currently done for the
398 gnuplot and ditaa languages). That would handle placing these results
399 in the buffer. Then if there is a combination of =silent= and =file=
400 =:results= headers we could drop the results to a temp buffer and pop
403 Display of file results is addressed in the [[* =\C-c \C-o= to open results of source block][open-results-task]].
405 *** TODO R graphics to screen means session evaluation
406 If R graphical output is going to screen then evaluation must be
407 in a session, otherwise the graphics will disappear as soon as the
410 *** Adding to a discussion started in email
411 I'm not deeply wedded to these ideas, just noting them down. I'm
412 probably just thinking of R and haven't really thought about how
413 this fits with the other graphics-generating languages.
415 > I used the approach below to get graphical file output
416 > today, which is one idea at least. Maybe it could be linked up with
417 > your :results file variable. (Or do we need a :results image for R?)
420 I don't think we need a special image results variable, but I may be
421 missing what the code below accomplishes. Would the task I added about
422 adding org-open-at-point functionality to source code blocks take care
425 Dan: I'm not sure. I think the ability for a script to generate both
426 text and graphical output might be a natural expectation, at least for
432 > #+srcname: cohort-scatter-plots-2d(org_babel_graphical_output_file="cohort-scatter-plots-2d.png")
434 > if(exists("org_babel_output_file"))
435 > png(filename=org_babel_graphical_output_file, width=1000, height=1000)
436 > ## plotting code in here
437 > if(exists("org_babel_graphical_output_file")) dev.off()
440 Dan: Yes, the results :file option is nice for dealing with graphical
441 output, and that could well be enough. Something based on the scheme
442 above would have a couple of points in its favour:
443 1. It's easy to switch between output going to on-screen graphics and
444 output going to file: Output will go to screen unless a string variable
445 with a standard name (e.g. ""org_babel_graphical_output_file"")
446 exists in which case it will go to the file indicated by the value
448 2. The block can return a result / script output, as well as produce
451 In interactive use we might want to allow the user to choose between
452 screen and file output. In non-interactive use such as export, it
453 would be file output (subject to the :exports directives).
455 ** TODO =\C-c \C-o= to open results of source block
456 by adding a =defadvice= to =org-open-at-point= we can use the common
457 =\C-c \C-o= keybinding to open the results of a source-code block.
458 This would be especially useful for source-code blocks which generate
459 graphical results and insert a file link as the results in the
460 org-mode buffer. (see [[* figure out how to handle graphic output][TODO figure out how to handle graphic output]]).
461 This could also act reasonably with other results types...
463 - file :: use org-open-at-point to open the file
464 - scalar :: open results unquoted in a new buffer
465 - tabular :: export the table to a new buffer and open that buffer
466 ** TODO Finalise behaviour regarding vector/scalar output
467 *** DONE Stop spaces causing vector output
468 This simple example of multilingual chaining produces vector output if
469 there are spaces in the message and scalar otherwise.
473 #+srcname: msg-from-R(msg=msg-from-python)
475 paste(msg, "und R", sep=" ")
479 : org-babel speaks elisp y python und R
481 #+srcname: msg-from-python(msg=msg-from-elisp)
486 #+srcname: msg-from-elisp(msg="org-babel speaks")
487 #+begin_src emacs-lisp
488 (concat msg " elisp")
490 ** STARTED share org-babel [1/4]
491 how should we share org-babel?
493 *** DONE post to org-mode
494 *** TODO post to ess mailing list
495 *** TODO create a org-babel page on worg
496 *** TODO create a short screencast demonstrating org-babel in action
499 we need to think up some good examples
501 **** interactive tutorials
502 This could be a place to use [[* org-babel assertions][org-babel assertions]].
504 for example the first step of a tutorial could assert that the version
505 of the software-package (or whatever) is equal to some value, then
506 source-code blocks could be used with confidence (and executed
507 directly from) the rest of the tutorial.
509 **** answering a text-book question w/code example
510 org-babel is an ideal environment enabling both the development and
511 demonstrationg of the code snippets required as answers to many
514 **** something using tables
515 maybe something along the lines of calculations from collected grades
518 Maybe something like the following which outputs sizes of directories
519 under the home directory, and then instead of the trivial =emacs-lisp=
520 block we could use an R block to create a nice pie chart of the
524 #+begin_src bash :results replace
528 #+begin_src emacs-lisp :var sizes=sizes :results replace
532 ** TODO command line execution
533 Allow source code blocks to be called form the command line. This
534 will be easy using the =sbe= function in [[file:lisp/org-babel-table.el][org-babel-table.el]].
536 This will rely upon [[* resolve references to other buffers][resolve references to other buffers]].
538 ** TODO inline source code blocks [3/5]
539 Like the =\R{ code }= blocks
541 not sure what the format should be, maybe just something simple
542 like =src_lang[]{}= where lang is the name of the source code
543 language to be evaluated, =[]= is optional and contains any header
544 arguments and ={}= contains the code.
546 (see [[* (sandbox) inline source blocks][the-sandbox]])
548 *** DONE evaluation with \C-c\C-c
549 Putting aside the header argument issue for now we can just run these
550 with the following default header arguments
551 - =:results= :: silent
552 - =:exports= :: results
554 *** DONE inline exportation
555 Need to add an interblock hook (or some such) through org-exp-blocks
556 *** DONE header arguments
557 We should make it possible to use header arguments.
559 *** TODO fontification
560 we should color these blocks differently
562 *** TODO refine html exportation
563 should use a span class, and should show original source in tool-tip
565 ** TODO LoB: re-implement plotting and analysis functions from org-R
566 I'll do this soon, now that we things are a bit more settled and we
567 have column names in R.
568 ** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format
569 I haven't thought about this properly. Just noting it down. What
570 Sweave uses is called "R noweb" (.Rnw).
572 I found a good description of noweb in the following article (see
573 the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]).
575 I think there are two parts to noweb, the construction of
576 documentation and the extraction of source-code (with notangle).
578 *documentation*: org-mode handles all of our documentation needs in
579 a manner that I believe is superior to noweb.
581 *source extraction* At this point I don't see anyone writing large
582 applications with 100% of the source code contained in org-babel
583 files, rather I see org-babel files containing things like
584 - notes with active code chunks
585 - interactive tutorials
586 - requirements documents with code running test suites
587 - and of course experimental reports with the code to run the
588 experiment, and perform analysis
590 Basically I think the scope of the programs written in org-babel
591 (at least initially) will be small enough that it wont require the
592 addition of a tangle type program to extract all of the source code
593 into a running application.
595 On the other hand, since we already have named blocks of source
596 code which reference other blocks on which they rely, this
597 shouldn't be too hard to implement either on our own, or possibly
598 relying on something like noweb/notangle.
600 ** PROPOSED support for passing paths to files between source blocks
601 Maybe this should be it's own result type (in addition to scalars and
602 vectors). The reason being that some source-code blocks (for example
603 ditaa or anything that results in the creation of a file) may want to
604 pass a file path back to org-mode which could then be inserted into
605 the org-mode buffer as a link to the file...
607 This would allow for display of images upon export providing
608 functionality similar to =org-exp-blocks= only in a more general
610 ** DEFERRED Support rownames and other org babel table features?
612 The full org table features are detailed in the manual [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]].
615 Perhaps add a :rownames header arg. This would be an integer
616 (usually 1) which would have the effect of post-processing all the
617 variables created in the R session in the following way: if the
618 integer is j, set the row names to the contents of column j and
619 delete column j. Perhaps it is artificial to allow this integer to
620 take any value other than 1. The default would be nil which would
621 mean no such behaviour.
623 Actually I don't know about that. If multiple variables are passed
624 in, it's not appropriate to alter them all in the same way. The
625 rownames specification would normally refer to just one of the
626 variables. For now maybe just say this has to be done in R. E.g.
628 #+TBLNAME: sample-sizes
629 | collection | size | exclude | include | exclude2 | include2 |
630 |-----------------+------+---------+---------+----------+----------|
631 | 58C | 2936 | 8 | 2928 | 256 | 2680 |
632 | MS | 5852 | 771 | 5081 | 771 | 5081 |
633 | NBS | 2929 | 64 | 2865 | 402 | 2527 |
634 | POBI | 2717 | 1 | 2716 | 1 | 2716 |
635 | 58C+MS+NBS+POBI | | | 13590 | | 13004 |
636 #+TBLFM: @2$4=@2$2 - @2$3::@2$6=@2$2 - @2$5::@3$4=@3$2-@3$3::@3$6=@3$2 - @3$5::@4$4=@4$2 - @4$3::@4$6=@4$2 - @4$5::@5$4=@5$2-@5$3::@5$6=@5$2 - @5$5::@6$4=vsum(@2$4..@5$4)::@6$6=vsum(@2$6..@5$6)
638 #+srcname: make-size-table(size=sample-sizes)
640 rownames(size) <- size[,1]
646 [I don't think it's as problematic as this makes out]
647 This is non-trivial, but may be worth doing, in particular to
648 develop a nice framework for sending data to/from R.
650 In R, indexing vector elements, and rows and columns, using
651 strings rather than integers is an important part of the
653 - elements of a vector may have names
654 - matrices and data.frames may have "column names" and "row names"
655 which can be used for indexing
656 - In a data frame, row names *must* be unique
664 > mat <- matrix(1:4, nrow=2, ncol=2, dimnames=list(c("r1","r2"), c("c1","c2")))
669 > # The names are separate from the data: they do not interfere with operations on the data
676 > df <- data.frame(var1=1:26, var2=26:1, row.names=letters)
678 [1] 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
684 So it's tempting to try to provide support for this in org-babel. For example
685 - allow R to refer to columns of a :var reference by their names
686 - When appropriate, results from R appear in the org buffer with "named
689 However none (?) of the other languages we are currently supporting
690 really have a native matrix type, let alone "column names" or "row
691 names". Names are used in e.g. python and perl to refer to entries
694 It currently seems to me that support for this in org-babel would
695 require setting rules about when org tables are considered to have
696 named columns/fields, and ensuring that (a) languages with a notion
697 of named columns/fields use them appropriately and (b) languages
698 with no such notion do not treat then as data.
700 - Org allows something that *looks* like column names to be separated
702 - Org also allows a row to *function* as column names when special
703 markers are placed in the first column. An hline is unnecessary
704 (indeed hlines are purely cosmetic in org [correct?]
705 - Org does not have a notion of "row names" [correct?]
707 The full org table functionality exeplified [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]] has features that
708 we would not support in e.g. R (like names for the row below).
710 **** Initial statement: allow tables with hline to be passed as args into R
711 This doesn't seem to work at the moment (example below). It would
712 also be nice to have a natural way for the column names of the org
713 table to become the column names of the R data frame, and to have
714 the option to specify that the first column is to be used as row
715 names in R (these must be unique). But this might require a bit of
720 | col1 | col2 | col3 |
721 |------+---------+------|
729 #+begin_src R :var tabel=egtable :colnames t
734 | "col1" | "col2" | "col3" |
735 |--------+-----------+--------|
737 | 4 | "schulte" | 6 |
740 Another example is in the [[*operations%20in%20on%20tables][grades example]].
742 ** DEFERRED use textConnection to pass tsv to R?
743 When passing args from the org buffer to R, the following route is
744 used: arg in buffer -> elisp -> tsv on file -> data frame in R. I
745 think it would be possible to avoid having to write to file by
746 constructing an R expression in org-babel-R-assign-elisp, something
749 #+begin_src emacs-lisp
750 (org-babel-R-input-command
751 (format "%s <- read.table(textConnection(\"%s\"), sep=\"\\t\", as.is=TRUE)"
752 name (orgtbl-to-tsv value '(:sep "\t" :fmt org-babel-R-quote-tsv-field))))
755 I haven't tried to implement this yet as it's basically just
756 fiddling with something that works. The only reason for it I can
757 think of would be efficiency and I haven't tested that.
759 This Didn't work after an initial test. I still think this is a
760 good idea (I also think we should try to do something similar when
761 writing out results frmo R to elisp) however as it wouldn't result
762 in any functional changes I'm bumping it down to deferred for
767 #+tblname: quick-test
770 #+srcname: quick-test-src-blk
771 #+begin_src R :var vec=quick-test
781 ** DEFERRED Rework Interaction with Running Processes [2/5]
782 *** DONE robust to errors interrupting execution
784 #+srcname: long-runner-ruby
785 #+begin_src ruby :results silent
790 *** DEFERRED use =C-g= keyboard-quit to push processing into the background
791 This may be possible using the `run-with-timer' command.
793 I have no idea how this could work...
795 #+srcname: long-runner-ruby
796 #+begin_src ruby :results silent
801 *** TODO ability to select which of multiple sessions is being used
802 Increasingly it is looking like we're going to want to run all
803 source code blocks in comint buffer (sessions). Which will have
805 1) allowing background execution
806 2) maintaining state between source-blocks
807 - allowing inline blocks w/o header arguments
810 (like ess-switch-process in .R buffers)
812 Maybe this could be packaged into a header argument, something
813 like =:R_session= which could accept either the name of the
814 session to use, or the string =prompt=, in which case we could use
815 the =ess-switch-process= command to select a new process.
817 *** TODO evaluation of shell code as background process?
818 After C-c C-c on an R code block, the process may appear to
819 block, but C-g can be used to reclaim control of the .org buffer,
820 without interrupting the R evalution. However I believe this is not
821 true of bash/sh evaluation. [Haven't tried other languages] Perhaps
822 a solution is just to background the individual shell commands.
824 The other languages (aside from emacs lisp) are run through the
825 shell, so if we find a shell solution it should work for them as
828 Adding an ampersand seems to be a supported way to run commands in
829 the background (see [[http://www.emacswiki.org/emacs/ExecuteExternalCommand#toc4][external-commands]]). Although a more extensible
830 solution may involve the use of the [[elisp:(progn (describe-function 'call-process-region) nil)][call-process-region]] function.
832 Going to try this out in a new file [[file:lisp/org-babel-proc.el][org-babel-proc.el]]. This should
833 contain functions for asynchronously running generic shell commands
834 in the background, and then returning their input.
836 **** partial update of org-mode buffer
837 The sleekest solution to this may be using a comint buffer, and
838 then defining a filter function which would incrementally interpret
839 the results as they are returned, including insertion into the
840 org-mode buffer. This may actually cause more problems than it is
841 worth, what with the complexities of identifying the types of
842 incrementally returned results, and the need for maintenance of a
843 process marker in the org buffer.
845 **** 'working' spinner
846 It may be nice and not too difficult to place a spinner on/near the
847 evaluating source code block
849 *** TODO conversion of output from interactive shell, R (and python) sessions to org-babel buffers
850 [DED] This would be a nice feature I think. Although an org-babel
851 purist would say that it's working the wrong way round... After
852 some interactive work in a *R* buffer, you save the buffer, maybe
853 edit out some lines, and then convert it to org-babel format for
854 posterity. Same for a shell session either in a *shell* buffer, or
855 pasted from another terminal emulator. And python of course.
857 ** DEFERRED improve the source-block snippet
858 any real improvement seems somewhat beyond the ability of yasnippet
861 [[file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name%20Chapter%20title][file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name Chapter title]]
863 ,#name : Chapter title
866 ${1:$(make-string (string-width text) ?\=)}
871 [[file:snippets/org-mode/sb][sb -- snippet]]
873 waiting for guidance from those more familiar with yasnippets
875 ** REJECTED re-implement R evaluation using ess-command or ess-execute
876 I don't have any complaints with the current R evaluation code or
877 behaviour, but I think it would be good to use the ESS functions
878 from a political point of view. Plus of course it has the normal
879 benefits of an API (insulates us from any underlying changes etc). [DED]
881 I'll look into this. I believe that I looked at and rejected these
882 functions initially but now I can't remember why. I agree with
883 your overall point about using API's where available. I will take
884 a look back at these and either switch to using the ess commands,
885 or at least articulate under this TODO the reasons for using our
886 custom R-interaction commands. [Eric]
890 Lets just replace =org-babel-R-input-command= with =ess-execute=.
892 I tried this, and although it works in some situations, I find that
893 =ess-command= will often just hang indefinitely without returning
894 results. Also =ess-execute= will occasionally hang, and pops up
895 the buffer containing the results of the command's execution, which
896 is undesirable. For now these functions can not be used. Maybe
897 someone more familiar with the ESS code can recommend proper usage
898 of =ess-command= or some other lower-level function which could be
899 used in place of [[file:lisp/org-babel-R.el::defun%20org-babel%20R%20input%20command%20command][org-babel-R-input-command]].
903 #+begin_quote ess-command
904 (ess-command COM &optional BUF SLEEP NO-PROMPT-CHECK)
906 Send the ESS process command COM and delete the output
907 from the ESS process buffer. If an optional second argument BUF exists
908 save the output in that buffer. BUF is erased before use.
909 COM should have a terminating newline.
910 Guarantees that the value of .Last.value will be preserved.
911 When optional third arg SLEEP is non-nil, `(sleep-for (* a SLEEP))'
912 will be used in a few places where `a' is proportional to `ess-cmd-delay'.
915 #+begin_quote ess-execute
916 (ess-execute COMMAND &optional INVERT BUFF MESSAGE)
918 Send a command to the ESS process.
919 A newline is automatically added to COMMAND. Prefix arg (or second arg
920 INVERT) means invert the meaning of
921 `ess-execute-in-process-buffer'. If INVERT is 'buffer, output is
922 forced to go to the process buffer. If the output is going to a
923 buffer, name it *BUFF*. This buffer is erased before use. Optional
924 fourth arg MESSAGE is text to print at the top of the buffer (defaults
925 to the command if BUFF is not given.)
928 *** out current setup
930 1) The body of the R source code block is wrapped in a function
931 2) The function is called inside of a =write.table= function call
932 writing the results to a table
933 3) The table is read using =org-table-import=
934 ** DONE add =:tangle= family of header arguments
936 - no :: don't include source-code block when tangling
937 - yes :: do include source-code block when tangling
939 this is tested in [[file:test-tangle.org::*Emacs%20Lisp%20initialization%20stuff][test-tangle.org]]
942 This would be good thing to address soon. I'm imagining that
943 e.g. here, the 'caller' block would return the answer 30. I believe
944 there's a few issues here: i.e. the naked 'a' without a reference
945 is not understood; the default arg b=6 is not understood.
947 #+srcname: adder(a, b=6)
952 #+srcname: caller(var=adder(a=24))
953 #+begin_src python :results silent
957 ** DONE extensible library of callable source blocks
959 This is covered by the [[file:library-of-babel.org][Library of Babel]], which will contain
960 ready-made source blocks designed to carry out useful common tasks.
961 *** Initial statement [Eric]
962 Much of the power of org-R seems to be in it's helper functions for
963 the quick graphing of tables. Should we try to re-implement these
964 functions on top of org-babel?
966 I'm thinking this may be useful both to add features to org-babel-R and
967 also to potentially suggest extensions of the framework. For example
968 one that comes to mind is the ability to treat a source-code block
969 like a function which accepts arguments and returns results. Actually
970 this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]).
972 - We want to provide convenient off-the-shelf actions
973 (e.g. plotting data) that make use of our new code evaluation
974 environment but do not require any actual coding.
975 *** Initial Design proposal [Dan]
976 - *Input data* will be specified using the same mechanism as :var
977 references, thus the input data may come from a table, or
978 another source block, and it is initially available as an elisp
980 - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that
981 line will apply an *action* to the referenced data.
982 - *Actions correspond to source blocks*: our library of available
983 actions will be a library of org-babel source blocks. Thus the
984 code for executing an action, and the code for dealing with the
985 output of the action will be the same code as for executing
986 source blocks in general
987 - Optionally, the user can have the relevant source block inserted
988 into the org buffer after the (say) #+BABELDO line. This will
989 allow the user to fine tune the action by modifying the code
990 (especially useful for plots).
991 - So maybe a #+BABELDO line will have header args
992 - :data (a reference to a table or source code block)
993 - :action (or should that be :srcname?) which will be something
994 like :action pie-chart, referring to a source block which will
995 be executed with the :data referent passed in using a :var arg.
996 - :showcode or something controlling whether to show the code
998 *** Modification to design
999 I'm implementing this, at least initially, as a new interpreter
1000 named 'babel', which has an empty body. 'babel' blocks take
1001 a :srcname header arg, and look for the source-code block with
1002 that name. They then execute the referenced block, after first
1003 appending their own header args on to the target block's header
1006 If the target block is in the library of babel (a.o.t. e.g. the
1007 current buffer), then the code in the block will refer to the
1008 input data with a name dictated by convention (e.g. __data__
1009 (something which is syntactically legal in all languages...). Thus
1010 the babel block will use a :var __data__ = whatever header arg to
1011 reference the data to be plotted.
1013 ** DONE Column names in R input/output
1014 This has been implemented: Automatic on input to R; optional in
1015 output. Note that this equates column names with the header row in
1016 an org table; whereas org actually has a mechanism whereby a row
1017 with a '!' in the first field defines column names. I have not
1018 attempted to support these org table mechanisms yet. See [[*Support%20rownames%20and%20other%20org%20babel%20table%20features][this
1019 DEFERRED todo item]].
1020 ** DONE use example block for large amounts of stdout output?
1021 We're currently `examplizing' with : at the beginning of the line,
1022 but should larger amounts of output be in a
1023 \#+begin_example...\#+end_example block? What's the cutoff? > 1
1024 line? This would be nice as it would allow folding of lengthy
1025 output. Sometimes one will want to see stdout just to check
1026 everything looks OK, and then fold it away.
1028 I'm addressing this in branch 'examplizing-output'.
1029 Yea, that makes sense. (either that or allow folding of large
1030 blocks escaped with =:=).
1032 Proposed cutoff of 10 lines, we can save this value in a user
1033 customizable variable.
1034 *** DONE add ability to remove such results
1035 ** DONE exclusive =exports= params
1037 #+srcname: implement-export-exclusivity
1044 ** DONE LoB: allow output in buffer
1045 ** DONE allow default header arguments by language
1046 org-babel-default-header-args:lang-name
1048 An example of when this is useful is for languages which always return
1049 files as their results (e.g. [[*** ditaa][ditaa]], and [[*** gnuplot][gnuplot]]).
1050 ** DONE singe-function tangling and loading elisp from literate org-mode file [3/3]
1052 This function should tangle the org-mode file for elisp, and then call
1053 `load-file' on the resulting tangled file.
1055 #+srcname: test-loading-embedded-emacs-lisp
1056 #+begin_src emacs-lisp :results replace
1057 (setq test-tangle-advert nil)
1058 (setq test-tangle-loading nil)
1059 (setq results (list :before test-tangle-loading test-tangle-advert))
1060 (org-babel-load-file "test-tangle.org")
1061 (setq results (list (list :after test-tangle-loading test-tangle-advert) results))
1062 (delete-file "test-tangle.el")
1066 #+resname: test-loading-embedded-emacs-lisp
1067 | :before | nil | nil |
1068 | :after | "org-babel tangles" | "use org-babel-tangle for all your emacs initialization files!!" |
1070 *** DONE add optional language limiter to org-babel-tangle
1071 This should check to see if there is any need to re-export
1073 *** DONE ensure that org-babel-tangle returns the path to the tangled file(s)
1075 #+srcname: test-return-value-of-org-babel-tangle
1076 #+begin_src emacs-lisp :results replace
1077 (mapcar #'file-name-nondirectory (org-babel-tangle-file "test-tangle.org" "emacs-lisp"))
1081 | "test-tangle.el" |
1083 *** DONE only tangle the file if it's actually necessary
1084 ** DONE add a function to jump to a source-block by name
1085 I've had an initial stab at that in org-babel-find-named-block
1086 (library-of-babel branch).
1088 At the same time I introduced org-babel-named-src-block-regexp, to
1089 match src-blocks with srcname.
1091 This is now working with the command
1092 `org-babel-goto-named-source-block', all we need is a good key
1095 ** DONE add =:none= session argument (for purely functional execution) [4/4]
1096 This would allow source blocks to be run in their own new process
1098 - These blocks could then also be run in the background (since we can
1099 detach and just wait for the process to signal that it has terminated)
1100 - We wouldn't be drowning in session buffers after running the tests
1101 - we can re-use much of the session code to run in a more /functional/
1104 While session provide a lot of cool features, like persistent
1105 environments, [[* DONE function to bring up inferior-process buffer][pop-to-session]], and hints at exportation for
1106 org-babel-tangle, they also have some down sides and I'm thinking that
1107 session-based execution maybe shouldn't be the default behavior.
1109 Down-sides to sessions
1110 - *much* more complicated than functional evaluation
1111 - maintaining the state of the session has weird issues
1112 - waiting for evaluation to finish
1113 - prompt issues like [[* TODO weird escaped characters in shell prompt break shell evaluation][shell-prompt-escapes-bug]]
1114 - can't run in background
1115 - litter emacs with session buffers
1119 #+srcname: ruby-task-no-session
1120 #+begin_src ruby :results replace output
1126 #+resname: ruby-task-no-session
1131 #+srcname: task-python-none-session
1132 #+begin_src python :session none :results replace value
1138 #+resname: task-python-none-session
1143 #+srcname: task-session-none-sh
1144 #+begin_src sh :results replace
1149 #+resname: task-session-none-sh
1155 #+srcname: task-no-session-R
1156 #+begin_src R :results replace output
1163 #+resname: task-no-session-R
1167 ** DONE fully purge org-babel-R of direct comint interaction
1168 try to remove all code under the [[file:lisp/org-babel-R.el::functions%20for%20evaluation%20of%20R%20code][;; functions for evaluation of R code]] line
1170 ** DONE Create objects in top level (global) environment [5/5]
1173 *** initial requirement statement [DED]
1174 At the moment, objects created by computations performed in the
1175 code block are evaluated in the scope of the
1176 code-block-function-body and therefore disappear when the code
1177 block is evaluated {unless you employ some extra trickery like
1178 assign('name', object, env=globalenv()) }. I think it will be
1179 desirable to also allow for a style wherein objects that are
1180 created in one code block persist in the R global environment and
1181 can be re-used in a separate block.
1183 This is what Sweave does, and while I'm not saying we have to be
1184 the same as Sweave, it wouldn't be hard for us to provide the same
1185 behaviour in this case; if we don't, we risk undeservedly being
1186 written off as an oddity by some.
1188 IOW one aspect of org-babel is that of a sort of functional
1189 meta-programming language. This is crazy, in a very good
1190 way. Nevertheless, wrt R I think there's going to be a lot of value
1191 in providing for a working style in which the objects are stored in
1192 the R session, rather than elisp/org buffer. This will be a very
1193 familiar working style to lots of people.
1195 There are no doubt a number of different ways of accomplishing
1196 this, the simplest being a hack like adding
1199 for(objname in ls())
1200 assign(objname, get(objname), envir=globalenv())
1203 to the source code block function body. (Maybe wrap it in an on.exit() call).
1205 However this may deserve to be thought about more carefully, perhaps
1206 with a view to having a uniform approach across languages. E.g. shell
1207 code blocks have the same semantics at the moment (no persistence of
1208 variables across code blocks), because the body is evaluated in a new
1209 bash shell process rather than a running shell. And I guess the same
1210 is true for python. However, in both these cases, you could imagine
1211 implementing the alternative in which the body is evaluated in a
1212 persistent interactive session. It's just that it's particularly
1213 natural for R, seeing as both ESS and org-babel evaluate commands in a
1214 single persistent R session.
1218 Thanks for bringing this up. I think you are absolutely correct that we
1219 should provide support for a persistent environment (maybe called a
1220 *session*) in which to evaluate code blocks. I think the current setup
1221 demonstrates my personal bias for a functional style of programming
1222 which is certainly not ideal in all contexts.
1224 While the R function you mention does look like an elegant solution, I
1225 think we should choose an implementation that would be the same across
1226 all source code types. Specifically I think we should allow the user to
1227 specify an optional *session* as a header variable (when not present we
1228 assume a default session for each language). The session name could be
1229 used to name a comint buffer (like the *R* buffer) in which all
1230 evaluation would take place (within which variables would retain their
1231 values --at least once I remove some of the functional method wrappings
1232 currently in place-- ).
1234 This would allow multiple environments to be used in the same buffer,
1235 and once this setup was implemented we should be able to fairly easily
1236 implement commands for jumping between source code blocks and the
1237 related session buffers, as well as for dumping the last N commands from
1238 a session into a new or existing source code block.
1240 Please let me know if you foresee any problems with this proposed setup,
1241 or if you think any parts might be confusing for people coming from
1242 Sweave. I'll hopefully find some time to work on this later in the
1245 *** can functional and interpreted/interactive models coexist?
1247 Even though both of these use the same =*R*= buffer the value of =a=
1248 is not preserved because it is assigned inside of a functional
1251 #+srcname: task-R-sessions
1258 #+srcname: task-R-same-session
1263 This functional wrapper was implemented in order to efficiently return
1264 the results of the execution of the entire source code block. However
1265 it inhibits the evaluation of source code blocks in the top level,
1266 which would allow for persistence of variable assignment across
1267 evaluations. How can we allow *both* evaluation in the top level, and
1268 efficient capture of the return value of an entire source code block
1269 in a language independent manner?
1271 Possible solutions...
1272 1) we can't so we will have to implement two types of evaluation
1273 depending on which is appropriate (functional or imperative)
1274 2) we remove the functional wrapper and parse the source code block
1275 into it's top level statements (most often but not always on line
1276 breaks) so that we can isolate the final segment which is our
1278 3) we add some sort of "#+return" line to the code block
1279 4) we take advantage of each languages support for meta-programming
1280 through =eval= type functions, and use said to evaluate the entire
1281 blocks in such a way that their environment can be combined with the
1282 global environment, and their results are still captured.
1283 5) I believe that most modern languages which support interactive
1284 sessions have support for a =last_result= type function, which
1285 returns the result of the last input without re-calculation. If
1286 widely enough present this would be the ideal solution to a
1287 combination of functional and imperative styles.
1289 None of these solutions seem very desirable, but for now I don't see
1290 what else would be possible.
1292 Of these options I was leaning towards (1) and (4) but now believe
1293 that if it is possible option (5) will be ideal.
1295 **** (1) both functional and imperative evaluation
1297 - can take advantage of built in functions for sending regions to the
1299 - retains the proven tested and working functional wrappers
1302 - introduces the complication of keeping track of which type of
1303 evaluation is best suited to a particular context
1304 - the current functional wrappers may require some changes in order to
1305 include the existing global context
1307 **** (4) exploit language meta-programming constructs to explicitly evaluate code
1309 - only one type of evaluation
1312 - some languages may not have sufficient meta-programming constructs
1314 **** (5) exploit some =last_value= functionality if present
1316 Need to ensure that most languages have such a function, those without
1317 will simply have to implement their own similar solution...
1319 | language | =last_value= function |
1320 |------------+-----------------------------|
1324 | shell | see [[* last command for shells][last command for shells]] |
1325 | emacs-lisp | see [[* emacs-lisp will be a special case][special-case]] |
1327 #+srcname: task-last-value
1332 ***** last command for shells
1333 Do this using the =tee= shell command, and continually pipe the output
1336 Got this idea from the following [[http://linux.derkeiler.com/Mailing-Lists/Fedora/2004-01/0898.html][email-thread]].
1338 suggested from mailing list
1340 #+srcname: bash-save-last-output-to-file
1344 bash -c "$line" | tee /tmp/last.out1
1345 mv /tmp/last.out1 /tmp/last.out
1349 another proposed solution from the above thread
1351 #+srcname: bash-save-in-variable
1354 # so - Save Output. Saves output of command in OUT shell variable.
1364 "^M": " | tee /tmp/h_lastcmd.out ^[k"
1367 export __=/tmp/h_lastcmd.out
1369 If you try it, Alt-k will stand for the old Enter; use "command $__" to
1370 access the last output.
1376 Herculano de Lima Einloft Neto
1379 ***** emacs-lisp will be a special case
1380 While it is possible for emacs-lisp to be run in a console type
1381 environment (see the =elim= function) it is *not* possible to run
1382 emacs-lisp in a different *session*. Meaning any variable set top
1383 level of the console environment will be set *everywhere* inside
1384 emacs. For this reason I think that it doesn't make any sense to
1385 worry about session support for emacs-lisp.
1387 *** Further thoughts on 'scripting' vs. functional approaches
1389 These are just thoughts, I don't know how sure I am about this.
1390 And again, perhaps I'm not saying anything very radical, just that
1391 it would be nice to have some options supporting things like
1392 receiving text output in the org buffer.
1394 I can see that you've already gone some way down the road towards
1395 the 'last value' approach, so sorry if my comments come rather
1396 late. I am concerned that we are not giving sufficient attention
1397 to stdout / the text that is returned by the interpreters. In
1398 contrast, many of our potential users will be accustomed to a
1399 'scripting' approach, where they are outputting text at various
1400 points in the code block, not just at the end. I am leaning
1401 towards thinking that we should have 2 modes of evaluation:
1402 'script' mode, and 'functional' mode.
1404 In script mode, evaluation of a code block would result in *all*
1405 text output from that code block appearing as output in the org
1406 buffer, presumably as an #+begin_example...#+end_example. There
1407 could be an :echo option controlling whether the input commands
1408 also appear in the output. [This is like Sweave].
1410 In functional mode, the *result* of the code block is available as
1411 an elisp object, and may appear in the org buffer as an org
1412 table/string, via the mechanisms you have developed already.
1414 One thing I'm wondering about is whether, in script mode, there
1415 simply should not be a return value. Perhaps this is not so
1416 different from what exists: script mode would be new, and what
1417 exists currently would be functional mode.
1419 I think it's likely that, while code evaluation will be exciting
1420 to people, a large majority of our users in a large majority of
1421 their usage will not attempt to actually use the return value from
1422 a source code block in any meaningful way. In that case, it seems
1423 rather restrictive to only allow them to see output from the end
1426 Instead I think the most accessible way to introduce org-babel to
1427 people, at least while they are learning it, is as an immensely
1428 powerful environment in which to embed their 'scripts', which now
1429 also allows them to 'run' their 'scripts'. Especially as such
1430 people are likely to be the least capable of the user-base, a
1431 possible design-rule would be to make the scripting style of usage
1432 easy (default?), perhaps requiring a special option to enable a
1433 functional style. Those who will use the functional style won't
1434 have a problem understanding what's going on, whereas the 'skript
1435 kiddies' might not even know the syntax for defining a function in
1436 their language of choice. And of course we can allow the user to
1437 set a variable in their .emacs controlling the preference, so that
1438 functional users are not inconveniennced by having to provide
1439 header args the whole time.
1441 Please don't get the impression that I am down-valuing the
1442 functional style of org-babel. I am constantly horrified at the
1443 messy 'scripts' that my colleagues produce in perl or R or
1444 whatever! Nevertheless that seems to be how a lot of people work.
1446 I think you were leaning towards the last-value approach because
1447 it offered the possibility of unified code supporting both the
1448 single evaluation environment and the functional style. If you
1449 agree with any of the above then perhaps it will impact upon this
1450 and mean that the code in the two branches has to differ a bit. In
1451 that case, functional mode could perhaps after all evaluate each
1452 code block in its own environment, thus (re)approaching 'true'
1453 functional programming (side-effects are hard to achieve).
1457 echo "There are `wc -l files` files in this directory"
1461 *** even more thoughts on evaluation, results, models and options
1463 Thanks Dan, These comments are invaluable.
1465 What do you think about this as a new list of priorities/requirements
1466 for the execution of source-code blocks.
1469 1) we want the evaluation of the source code block to take place in a
1470 session which can persist state (variables, current directory,
1472 2) source code blocks can specify their session with a header argument
1473 3) each session should correspond to an Emacs comint buffer so that the
1474 user can drop into the session and experiment with live code
1477 1) each source-code block generates some form of results which (as
1478 we have already implemented) is transfered into emacs-lisp
1479 after which it can be inserted into the org-mode buffer, or
1480 used by other source-code blocks
1481 2) when the results are translated into emacs-lisp, forced to be
1482 interpreted as a scalar (dumping their raw values into the
1483 org-mode buffer), as a vector (which is often desirable with R
1484 code blocks), or interpreted on the fly (the default option).
1485 Note that this is very nearly currently implemented through the
1486 [[* DONE results-type header (vector/file)][results-type-header]].
1487 3) there should be *two* means of collecting results from the
1488 execution of a source code block. *Either* the value of the
1489 last statement of the source code block, or the collection of
1490 all that has been passed to STDOUT during the evaluation.
1492 **** header argument or return line (*header argument*)
1494 Rather than using a header argument to specify how the return value
1495 should be passed back, I'm leaning towards the use of a =#+RETURN=
1496 line inside the block. If such a line *is not present* then we
1497 default to using STDOUT to collect results, but if such a line *is
1498 present* then we use it's value as the results of the block. I
1499 think this will allow for the most elegant specification between
1500 functional and script execution. This also cleans up some issues
1501 of implementation and finding which statement is the last
1504 Having given this more thought, I think a header argument is
1505 preferable. The =#+return:= line adds new complicating syntax for
1506 something that does little more than we would accomplish through
1507 the addition of a header argument. The only benefit being that we
1508 know where the final statement starts, which is not an issue in
1509 those languages which contain 'last value' operators.
1511 new header =:results= arguments
1512 - script :: explicitly states that we want to use STDOUT to
1513 initialize our results
1514 - return_last :: stdout is ignored instead the *value* of the final
1515 statement in the block is returned
1516 - echo :: means echo the contents of the source-code block along
1517 with the results (this implies the *script* =:results=
1520 *** DONE rework evaluation lang-by-lang [4/4]
1522 This should include...
1523 - functional results working with the comint buffer
1525 - script :: return the output of STDOUT
1526 - write a macro which runs the first redirection, executes the
1527 body, then runs the second redirection
1528 - last :: return the value of the last statement
1531 - sessions in comint buffers
1533 **** DONE Ruby [4/4]
1534 - [X] functional results working with comint
1535 - [X] script results
1536 - [X] ensure scalar/vector results args are taken into consideration
1537 - [X] ensure callable by other source block
1539 #+srcname: ruby-use-last-output
1540 #+begin_src ruby :results replace
1547 #+resname: ruby-use-last-output
1550 #+srcname: task-call-use-last-output
1551 #+begin_src ruby :var last=ruby-use-last-output :results replace
1552 last.flatten.size + 1
1555 #+resname: task-call-use-last-output
1560 #+srcname: first-ruby-session-task
1561 #+begin_src ruby :session schulte :results silent
1565 #+srcname: second-ruby-session-task
1566 #+begin_src ruby :session schulte :results silent
1570 #+srcname: without-the-right-session
1571 #+begin_src ruby :results silent
1577 - [X] functional results working with comint
1578 - [X] script results
1579 - [X] ensure scalar/vector results args are taken into consideration
1580 - [X] ensure callable by other source block
1582 To redirect output to a file, you can use the =sink()= command.
1585 #+begin_src R :results value vector silent
1592 #+srcname: task-R-use-other-output
1593 #+begin_src R :var twoentyseven=task_R_B() :results replace value
1598 #+resname: task-R-use-other-output
1601 **** DONE Python [4/4]
1602 - [X] functional results working with comint
1603 - [X] script results
1604 - [X] ensure scalar/vector results args are taken into consideration
1605 - [X] ensure callable by other source block
1607 #+srcname: task-new-eval-for-python
1608 #+begin_src python :results silent output scalar
1614 #+srcname: task-use-new-eval
1615 #+begin_src python :var tasking=task-new-eval-for-python() :results replace
1619 #+resname: task-use-new-eval
1622 **** DONE Shells [4/4]
1623 - [X] functional results working with comint
1624 - [X] script results
1625 - [X] ensure scalar/vector results args are taken into consideration
1626 - [X] ensure callable by other source block
1628 #+srcname: task-shell-new-evaluation
1629 #+begin_src sh :results silent value scalar
1634 #+srcname: task-call-other-shell
1635 #+begin_src sh :var other=task-shell-new-evaluation() :results replace scalar
1636 echo $other ' is the old date'
1639 #+resname: task-call-other-shell
1640 : $ Fri Jun 12 13:08:37 PDT 2009 is the old date
1642 *** DONE implement a *session* header argument [4/4]
1643 =:session= header argument to override the default *session* buffer
1647 #+srcname: task-ruby-named-session
1648 #+begin_src ruby :session schulte :results replace
1649 schulte = :in_schulte
1652 #+resname: task-ruby-named-session
1655 #+srcname: another-in-schulte
1656 #+begin_src ruby :session schulte
1660 #+resname: another-in-schulte
1667 #+srcname: python-session-task
1668 #+begin_src python :session what :results silent
1672 #+srcname: python-get-from-session
1673 #+begin_src python :session what :results replace
1677 #+resname: python-get-from-session
1682 #+srcname: task-shell-sessions
1683 #+begin_src sh :session what
1687 #+srcname: task-shell-sessions-what
1688 #+begin_src sh :session what :results replace
1692 #+resname: task-shell-sessions-what
1697 #+srcname: task-R-session
1698 #+begin_src R :session what :results replace
1704 #+resname: task-R-session
1707 #+srcname: another-task-R-session
1708 #+begin_src R :session what :results replace
1712 *** DONE function to bring up inferior-process buffer [4/4]
1714 This should be callable from inside of a source-code block in an
1715 org-mode buffer. It should evaluate the header arguments, then bring
1716 up the inf-proc buffer using =pop-to-buffer=.
1718 For lack of a better place, lets add this to the `org-metadown-hook'
1721 To give this a try, place the cursor on a source block with variables,
1722 (optionally git a prefix argument) then hold meta and press down.
1726 #+srcname: task-ruby-pop-to-session
1727 #+begin_src ruby :var num=9 :var another="something else"
1728 num.times{|n| puts another}
1733 #+srcname: task-python-pop-to-session
1734 #+begin_src python :var num=9 :var another="something else"
1739 #+srcname: task-R-pop-to-session
1740 #+begin_src R :var a=9 :var b=8
1746 #+srcname: task-shell-pop-sessions
1747 #+begin_src sh :var NAME="eric"
1751 *** DEFERRED function to dump last N lines from inf-proc buffer into the current source block
1753 Callable with a prefix argument to specify how many lines should be
1754 dumped into the source-code buffer.
1756 *** REJECTED comint notes
1758 Implementing comint integration in [[file:lisp/org-babel-comint.el][org-babel-comint.el]].
1761 - handling of outputs
1762 - split raw output from process by prompts
1763 - a ring of the outputs, buffer-local, `org-babel-comint-output-ring'
1764 - a switch for dumping all outputs to a buffer
1765 - inputting commands
1767 Lets drop all this language specific stuff, and just use
1768 org-babel-comint to split up our outputs, and return either the last
1769 value of an execution or the combination of values from the
1772 **** comint filter functions
1773 : ;; comint-input-filter-functions hook process-in-a-buffer
1774 : ;; comint-output-filter-functions hook function modes.
1775 : ;; comint-preoutput-filter-functions hook
1776 : ;; comint-input-filter function ...
1778 #+srcname: obc-filter-ruby
1779 #+begin_src ruby :results last
1787 ** DONE Remove protective commas from # comments before evaluating
1788 org inserts protective commas in front of ## comments in language
1789 modes that use them. We need to remove them prior to sending code
1792 #+srcname: testing-removal-of-protective-comas
1794 ,# this one might break it??
1798 ** DONE pass multiple reference arguments into R
1799 Can we do this? I wasn't sure how to supply multiple 'var' header
1800 args. Just delete this if I'm being dense.
1802 This should be working, see the following example...
1804 #+srcname: two-arg-example
1805 #+begin_src R :var n=2 :var m=8
1809 #+resname: two-arg-example
1812 ** DONE ensure that table ranges work
1813 when a table range is passed to org-babel as an argument, it should be
1814 interpreted as a vector.
1817 | 2 | 3 | Fixnum:1 |
1818 | 3 | 4 | Array:123456 |
1822 #+TBLFM: @1$3='(sbe simple-sbe-example (n 4))::@2$3='(sbe task-table-range (n @1$1..@6$1))::@3$3='(sbe task-table-range (n (@1$1..@6$1)))
1824 #+srcname: simple-sbe-example
1825 #+begin_src emacs-lisp
1829 #+srcname: task-table-range
1830 #+begin_src ruby :var n=simple-sbe-example
1834 #+srcname: simple-results
1835 #+begin_src emacs-lisp :var n=task-table-range(n=(1 2 3))
1839 #+resname: simple-results
1842 #+srcname: task-arr-referent
1843 #+begin_src ruby :var ar=(1 2 3)
1847 #+resname: task-arr-referent
1850 ** DONE global variable indicating default to vector output
1851 how about an alist... =org-babel-default-header-args= this may already
1852 exist... just execute the following and all source blocks will default
1855 #+begin_src emacs-lisp
1856 (setq org-babel-default-header-args '((:results . "vector")))
1859 ** DONE name named results if source block is named
1860 currently this isn't happening although it should be
1862 #+srcname: test-naming-named-source-blocks
1863 #+begin_src emacs-lisp
1867 #+resname: test-naming-named-source-blocks
1869 ** DONE (simple caching) check for named results before source blocks
1870 see the TODO comment in [[file:lisp/org-babel-ref.el::TODO%20This%20should%20explicitly%20look%20for%20resname%20lines%20before][org-babel-ref.el#org-babel-ref-resolve-reference]]
1871 ** DONE set =:results silent= when eval with prefix argument
1873 #+begin_src emacs-lisp
1876 ** DONE results-type header (vector/file) [3/3]
1877 In response to a point in Dan's email. We should allow the user to
1878 force scalar or vector results. This could be done with a header
1879 argument, and the default behavior could be controlled through a
1880 configuration variable.
1882 #+srcname: task-trivial-vector
1883 #+begin_src ruby :results replace vector
1890 since it doesn't make sense to turn a vector into a scalar, lets
1891 just add a two values...
1893 - vector :: forces the results to be a vector (potentially 1 dimensional)
1894 - file :: this throws an error if the result isn't a string, and
1895 tries to treat it as a path to a file.
1897 I'm just going to cram all of these into the =:results= header
1898 argument. Then if we allow multiple header arguments it should
1899 work out, for example one possible header argument string could be
1900 =:results replace vector file=, which would *replace* any existing
1901 results forcing the results into an org-mode table, and
1902 interpreting any strings as file paths.
1904 *** DONE multiple =:results= headers
1906 #+srcname: multiple-result-headers
1907 #+begin_src ruby :results replace silent
1913 *** DONE file result types
1914 When inserting into an org-mode buffer create a link with the path
1915 being the value, and optionally the display being the
1916 =file-name-nondirectory= if it exists.
1918 #+srcname: task-file-result
1919 #+begin_src python :results replace file
1924 [[something][something]]
1927 This will be useful because blocks like =ditaa= and =dot= can return
1928 the string path of their files, and can add =file= to their results
1931 *** DONE vector result types
1933 #+srcname: task-force-results
1934 #+begin_src emacs-lisp :results vector
1941 ** DONE results name
1942 In order to do this we will need to start naming our results.
1943 Since the source blocks are named with =#+srcname:= lines we can
1944 name results with =#+resname:= lines (if the source block has no
1945 name then no name is given to the =#+resname:= line on creation,
1946 otherwise the name of the source block is used).
1948 This will have the additional benefit of allowing results and
1949 source blocks to be located in different places in a buffer (and
1950 eventually in different buffers entirely).
1952 #+srcname: developing-resnames
1953 #+begin_src emacs-lisp :results silent
1957 Once source blocks are able to find their own =#+resname:= lines
1960 #+srcname: sbe-w-new-results
1961 #+begin_src emacs-lisp :results replace
1962 (sbe "developing-resnames")
1968 *** TODO change the results insertion functions to use these lines
1970 *** TODO teach references to resolve =#+resname= lines.
1972 ** DONE org-babel tests org-babel [1/1]
1973 since we are accumulating this nice collection of source-code blocks
1974 in the sandbox section we should make use of them as unit tests.
1975 What's more, we should be able to actually use org-babel to run these
1978 We would just need to cycle over every source code block under the
1979 sandbox, run it, and assert that the return value is equal to what we
1982 I have the feeling that this should be possible using only org-babel
1983 functions with minimal or no additional elisp. It would be very cool
1984 for org-babel to be able to test itself.
1986 This is now done, see [[* Tests]].
1988 *** DEFERRED org-babel assertions (may not be necessary)
1989 These could be used to make assertions about the results of a
1990 source-code block. If the assertion fails then the point could be
1991 moved to the block, and error messages and highlighting etc... could
1994 ** DONE make C-c C-c work anywhere within source code block?
1995 This seems like it would be nice to me, but perhaps it would be
1996 inefficient or ugly in implementation? I suppose you could search
1997 forward, and if you find #+end_src before you find #+begin_src,
1998 then you're inside one. [DED]
2000 Agreed, I think inside of the =#+srcname: line= would be useful as
2003 #+srcname: testing-out-cc
2004 #+begin_src emacs-lisp
2008 ** DONE integration with org tables
2009 We should make it easy to call org-babel source blocks from org-mode
2010 table formulas. This is practical now that it is possible to pass
2011 arguments to org-babel source blocks.
2013 See the related [[* (sandbox) integration w/org tables][sandbox]] header for tests/examples.
2015 *** digging in org-table.el
2016 In the past [[file:~/src/org/lisp/org-table.el::org%20table%20el%20The%20table%20editor%20for%20Org%20mode][org-table.el]] has proven difficult to work with.
2018 Should be a hook in [[file:~/src/org/lisp/org-table.el::defun%20org%20table%20eval%20formula%20optional%20arg%20equation][org-table-eval-formula]].
2020 Looks like I need to change this [[file:~/src/org/lisp/org-table.el::if%20lispp][if statement]] (line 2239) into a cond
2023 ** DONE source blocks as functions
2025 Allow source code blocks to be called like functions, with arguments
2026 specified. We are already able to call a source-code block and assign
2027 it's return result to a variable. This would just add the ability to
2028 specify the values of the arguments to the source code block assuming
2029 any exist. For an example see
2031 When a variable appears in a header argument, how do we differentiate
2032 between it's value being a reference or a literal value? I guess this
2033 could work just like a programming language. If it's escaped or in
2034 quotes, then we count it as a literal, otherwise we try to look it up
2037 ** DONE folding of code blocks? [2/2]
2038 [DED] In similar way to using outline-minor-mode for folding function
2039 bodies, can we fold code blocks? #+begin whatever statements are
2040 pretty ugly, and in any case when you're thinking about the overall
2041 game plan you don't necessarily want to see the code for each Step.
2043 *** DONE folding of source code block
2044 Sounds good, and wasn't too hard to implement. Code blocks should
2045 now be fold-able in the same manner as headlines (by pressing TAB
2048 *** REJECTED folding of results
2049 So, lets do a three-stage tab cycle... First fold the src block,
2050 then fold the results, then unfold.
2052 There's no way to tell if the results are a table or not w/o
2053 actually executing the block which would be too expensive of an
2056 ** DONE selective export of text, code, figures
2057 [DED] The org-babel buffer contains everything (code, headings and
2058 notes/prose describing what you're up to, textual/numeric/graphical
2059 code output, etc). However on export to html / LaTeX one might want
2060 to include only a subset of that content. For example you might
2061 want to create a presentation of what you've done which omits the
2064 [EMS] So I think this should be implemented as a property which can
2065 be set globally or on the outline header level (I need to review
2066 the mechanics of org-mode properties). And then as a source block
2067 header argument which will apply only to a specific source code
2068 block. A header argument of =:export= with values of
2070 - =code= :: just show the code in the source code block
2071 - =none= :: don't show the code or the results of the evaluation
2072 - =results= :: just show the results of the code evaluation (don't
2073 show the actual code)
2074 - =both= :: show both the source code, and the results
2076 this will be done in [[* (sandbox) selective export][(sandbox) selective export]].
2078 ** DONE a header argument specifying silent evaluation (no output)
2079 This would be useful across all types of source block. Currently
2080 there is a =:replace t= option to control output, this could be
2081 generalized to an =:output= option which could take the following
2082 options (maybe more)
2084 - =t= :: this would be the default, and would simply insert the
2085 results after the source block
2086 - =replace= :: to replace any results which may already be there
2087 - =silent= :: this would inhibit any insertion of the results
2089 This is now implemented see the example in the [[* silent evaluation][sandbox]]
2091 ** DONE assign variables from tables in R
2092 This is now working (see [[* (sandbox table) R][(sandbox-table)-R]]). Although it's not that
2093 impressive until we are able to print table results from R.
2095 ** DONE insert 2-D R results as tables
2096 everything is working but R and shell
2102 This has already been tackled by Dan in [[file:existing_tools/org-R.el::defconst%20org%20R%20write%20org%20table%20def][org-R:check-dimensions]]. The
2103 functions there should be useful in combination with [[http://cran.r-project.org/doc/manuals/R-data.html#Export-to-text-files][R-export-to-csv]]
2104 as a means of converting multidimensional R objects to emacs lisp.
2106 It may be as simple as first checking if the data is multidimensional,
2107 and then, if so using =write= to write the data out to a temporary
2108 file from which emacs can read the data in using =org-table-import=.
2110 Looking into this further, is seems that there is no such thing as a
2111 scalar in R [[http://tolstoy.newcastle.edu.au/R/help/03a/3733.html][R-scalar-vs-vector]] In that light I am not sure how to
2112 deal with trivial vectors (scalars) in R. I'm tempted to just treat
2113 them as vectors, but then that would lead to a proliferation of
2114 trivial 1-cell tables...
2116 ** DONE allow variable initialization from source blocks
2117 Currently it is possible to initialize a variable from an org-mode
2118 table with a block argument like =table=sandbox= (note that the
2119 variable doesn't have to named =table=) as in the following example
2125 #+begin_src emacs-lisp :var table=sandbox :results replace
2126 (message (format "table = %S" table))
2129 : "table = ((1 2 3) (4 \"schulte\" 6))"
2131 It would be good to allow initialization of variables from the results
2132 of other source blocks in the same manner. This would probably
2133 require the addition of =#+SRCNAME: example= lines for the naming of
2134 source blocks, also the =table=sandbox= syntax may have to be expanded
2135 to specify whether the target is a source code block or a table
2136 (alternately we could just match the first one with the given name
2137 whether it's a table or a source code block).
2139 At least initially I'll try to implement this so that there is no need
2140 to specify whether the reference is to a table or a source-code block.
2141 That seems to be simpler both in terms of use and implementation.
2143 This is now working for emacs-lisp, ruby and python (and mixtures of
2144 the three) source blocks. See the examples in the [[* (sandbox) referencing other source blocks][sandbox]].
2146 This is currently working only with emacs lisp as in the following
2147 example in the [[* emacs lisp source reference][emacs lisp source reference]].
2150 ** TODO Add languages [2/6]
2151 I'm sure there are many more that aren't listed here. Please add
2152 them, and bubble any that you particularly care about up to the top.
2154 Any new language should be implemented in a org-babel-lang.el file.
2155 Follow the pattern set by [[file:lisp/org-babel-script.el][org-babel-script.el]], [[file:lisp/org-babel-shell.el][org-babel-shell.el]] and
2156 [[file:lisp/org-babel-R.el][org-babel-R.el]].
2159 This could probably be added to [[file:lisp/org-babel-script.el][org-babel-script.el]]
2164 (see [[* file result types][file result types]])
2166 #+srcname: implementing-ditaa
2167 #+begin_src ditaa :results replace :file blue.png :cmdline -r
2177 #+resname: implementing-ditaa
2178 [[file:blue.png][blue.png]]
2180 *** DONE gnuplot [7/7]
2181 (see [[* file result types][file result types]])
2183 #+PLOT: title:"Citas" ind:1 deps:(3) type:2d with:histograms set:"yrange [0:]"
2184 #+TBLNAME: gnuplot-data
2185 | independent var | first dependent var | second dependent var |
2186 |-----------------+---------------------+----------------------|
2187 | 0.1 | 0.425 | 0.375 |
2188 | 0.2 | 0.3125 | 0.3375 |
2189 | 0.3 | 0.24999993 | 0.28333338 |
2190 | 0.4 | 0.275 | 0.28125 |
2191 | 0.5 | 0.26 | 0.27 |
2192 | 0.6 | 0.25833338 | 0.24999993 |
2193 | 0.7 | 0.24642845 | 0.23928553 |
2194 | 0.8 | 0.23125 | 0.2375 |
2195 | 0.9 | 0.23333323 | 0.2333332 |
2196 | 1 | 0.2225 | 0.22 |
2197 | 1.1 | 0.20909075 | 0.22272708 |
2198 | 1.2 | 0.19999998 | 0.21458333 |
2199 | 1.3 | 0.19615368 | 0.21730748 |
2201 #+srcname: implementing-gnuplot
2202 #+begin_src gnuplot :var data=gnuplot-data :results silent
2203 set title "Implementing Gnuplot"
2204 plot data using 1:2 with lines
2207 **** DONE add variables
2208 gnuplot 4.2 and up support user defined variables. This is how
2209 we will handle variables with org-babel (meaning we will need to
2210 require gnuplot 4.2 and up for variable support, which can be
2211 install using [[http://www.macports.org/install.php][macports]] on Mac OSX).
2213 - scalar variables should be replaced in the body of the gnuplot code
2214 - vector variables should be exported to tab-separated files, and
2215 the variable names should be replaced with the path to the files
2217 **** DONE direct plotting w/o session
2218 **** DEFERRED gnuplot support for column/row names
2219 This should be implemented along the lines of the [[* STARTED Column (and row) names of tables in R input/output][R-colname-support]].
2221 We can do something similar to the :labels param in org-plot, we just
2222 have to be careful to ensure that each label is aligned with the
2225 This may be walking too close to an entirely prebuilt plotting tool
2226 rather than straight gnuplot code evaluation. For now I think this
2229 **** DONE a =file= header argument
2230 to specify a file holding the results
2232 #+srcname: gnuplot-to-file-implementation
2233 #+begin_src gnuplot :file plot.png :var data=gnuplot-data
2234 plot data using 1:2, data using 1:3 with lines
2238 [[file:plot.png][plot.png]]
2240 **** DONE helpers from org-plot.el
2241 There are a variety of helpers in org-plot which can be fit nicely
2242 into custom gnuplot header arguments.
2244 These should all be in place by now.
2246 **** DEFERRED header argument specifying 3D data
2249 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2250 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2251 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2252 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 |
2253 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2254 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2255 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 1 | 0 |
2256 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
2257 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
2258 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2259 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2260 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 0 |
2261 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2262 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2264 #+srcname: implementing-gnuplot-grid-plots
2265 #+begin_src gnuplot :vars data=org-grid
2269 **** DONE gnuplot sessions
2270 Working on this, we won't support multiple sessions as `gnuplot-mode'
2271 isn't setup for such things.
2273 Also we can't display results with the default :none session, so for
2274 gnuplot we really want the default behavior to be :default, and to
2275 only run a :none session when explicitly specified.
2277 #+srcname: implementing-gnuplot-sessions
2278 #+begin_src gnuplot :var data=gnuplot-data :session none :file session.png
2279 set title "Implementing Gnuplot Sessions"
2280 plot data using 1:2 with lines
2284 [[file:session.png][session.png]]
2287 (see [[* file result types][file result types]])
2290 (see [[* file result types][file result types]])
2294 ** TODO allow srcname to omit function call parentheses
2295 Someone needs to revisit those regexps. Is there an argument for
2296 moving some of the regexps used to match function calls into
2297 defvars? (i.e. in o-b.el and o-b-ref.el)
2299 ** TODO creeping blank lines
2300 There's still inappropriate addition of blank lines in some circumstances.
2302 Hmm, it's a bit confusing. It's to do with o-b-remove-result. LoB
2303 removes the entire (#+resname and result) and starts from scratch,
2304 whereas #+begin_src only removes the result. I haven't worked out
2305 what the correct fix is yet. Maybe the right thing to do is to make
2306 sure that those functions (o-b-remove-result et al.) are neutral
2307 with respect to newlines. Sounds easy, but...
2317 Compare the results of
2318 #+lob: python-add(a=5, b=17)
2320 #+resname: python-add(a=5, b=17)
2322 --------------------------------
2330 ---------------------
2331 ** TODO problem with newlines in output when :results value
2332 #+begin_src python :results value
2333 '\n'.join(map(str, range(4)))
2339 Whereas I was hoping for
2346 This is some sort of non-printing char / quoting issue I think. Note
2349 #+begin_src python :results value
2350 '\\n'.join(map(str, range(4)))
2357 #+begin_src python :results output
2358 print('\n'.join(map(str, range(4))))
2367 *** collapsing consecutive newlines in string output
2369 This is an example of the same bug
2371 #+srcname: multi-line-string-output
2372 #+begin_src ruby :results output
2373 "the first line ends here
2376 and this is the second one
2381 This doesn't produce anything at all now. I believe that's because
2382 I've changed things so that :results output really does *not* get the
2383 value of the block, only the STDOUT. So if we add a print statement
2386 #+srcname: multi-line-string-output
2387 #+begin_src ruby :results output
2388 print "the first line ends here
2391 and this is the second one
2397 : the first line ends here
2400 : and this is the second one
2404 However, the behaviour with :results value is wrong
2406 #+srcname: multi-line-string-value
2408 "the first line ends here
2411 and this is the second one
2419 ** TODO prompt characters appearing in output with R
2420 #+begin_src R :session *R* :results output
2429 ** TODO o-b-execute-subtree overwrites heading when subtree is folded
2431 Try M-x org-babel-execute-subtree with the subtree folded and
2432 point at the beginning of the heading line.
2437 ** TODO Allow source blocks to be recognised when #+ are not first characters on the line
2438 I think Carsten has recently altered the core so that #+ can have
2439 preceding whitespace, at least for literal/code examples. org-babel
2440 should support this.
2442 ** TODO non-orgtbl formatted lists
2445 #+srcname: this-doesn't-match-orgtbl
2446 #+begin_src emacs-lisp :results replace
2447 '((:results . "replace"))
2450 #+resname: this-doesn't-match-orgtbl
2452 ** PROPOSED external shell execution can't isolate return values
2453 I have no idea how to do this as of yet. The result is that when
2454 shell functions are run w/o a session there is no difference between
2455 the =output= and =value= result arguments.
2457 Yea, I don't know how to do this either. I searched extensively on
2458 how to isolate the *last* output of a series of shell commands (see
2459 [[* last command for
2460 shells][last command for shells]]). The results of the search were basically
2461 that it was not possible (or at least not accomplish-able with a
2462 reasonable amount of effort).
2464 That fact combined with the tenancy to all ways use standard out in
2465 shell scripts led me to treat these two options (=output= and =value=)
2466 as identical in shell evaluation. Not ideal but maybe good enough for
2469 In the `results' branch I've changed this so that they're not quite
2470 identical: output results in raw stdout contents, whereas value
2471 converts it to elisp, perhaps to a table if it looks tabular. This is
2472 the same for the other languages. [Dan]
2474 ** TODO are the org-babel-trim s necessary?
2475 at the end of e.g. org-babel-R-evaluate, org-babel-python-evaluate, but
2476 not org-babel-ruby-evaluate
2477 ** TODO use new merge function [[file:lisp/org-babel-ref.el::t%20nil%20org%20combine%20plists%20args%20nil][here]]?
2478 And at other occurrences of org-combine-plists?
2479 ** TODO LoB is not populated on startup
2480 org-babel-library-of-babel is nil for me on startup. I have to
2481 evaluate the [[file:lisp/org-babel-lob.el::][org-babel-lob-ingest]] line manually.
2482 ** DONE Fix nested evaluation
2483 The current parser / evaluator fails with greater levels of nested
2484 function block calls (example below).
2486 *** Initial statement [ded]
2487 If we want to overcome this I think we'd have to redesign some of
2488 the evaluation mechanism. Seeing as we are also facing issues like
2489 dealing with default argument values, and seeing as we now know
2490 how we want the library of babel to behave in addition to the
2491 source blocks, now might be a good time to think about this. It
2492 would be nice to do the full thing at some point, but otoh we may
2493 not consider it a massive priority.
2495 AIui, there are two stages: (i) construct a parse tree, and (ii)
2496 evaluate it and return the value at the root. In the parse tree
2497 each node represents an unevaluated value (either a literal value
2498 or a reference). Node v may have descendent nodes, which represent
2499 values upon which node v's evaluation depends. Once that tree is
2500 constructed, then we evaluate the nodes from the tips towards the
2501 root (a post-order traversal).
2503 [This would also provide a solution for concatenating the STDOUTs
2504 of called blocks, which is a [[*allow%20output%20mode%20to%20return%20stdout%20as%20value][task below]]; we concatenate them in
2505 whatever order the traversal is done in.]
2507 In addition to the variable references (i.e. daughter nodes), each
2508 node would contain the information needed to evaluate that node
2509 (e.g. lang body). Then we would pass a function postorder over the
2510 tree which would call o-b-execute-src-block at each node, finally
2511 returning the value at the root.
2513 Fwiw I made a very tentative small start at stubbing this out in
2514 org-babel-call.el in the 'evaluation' branch. And I've made a start
2515 at sketching a parsing algorithm below.
2516 **** Parse tree algorithm
2517 Seeing as we're just trying to parse a string like
2518 f(a=1,b=g(c=2,d=3)) it shouldn't be too hard. But of course there
2519 are 'proper' parsers written in elisp out there,
2520 e.g. [[http://cedet.sourceforge.net/semantic.shtml][Semantic]]. Perhaps we can find what we need -- our syntax is
2521 pretty much the same as python and R isn't it?
2523 Or, a complete hack, but maybe it would be we easy to transform it
2524 to XML and then parse that with some existing tool?
2526 But if we're doing it ourselves, something very vaguely like this?
2527 (I'm sure there're lots of problems with this)
2529 #+srcname: org-babel-call-parse(call)
2531 ## we are currently reading a reference name: the name of the root function
2532 whereami = "refname"
2533 node = root = Node()
2534 for c in call_string:
2537 whereami = "varname" # now we're reading a variable name
2540 node.daughters = [node.daughters, new]
2543 whereami = "refname"
2545 whereami = "varname"
2552 if whereami = "varname":
2553 node.varnames[varnum] += c
2554 elif whereami = "refname":
2558 *** discussion / investigation
2559 I believe that this issue should be addressed as a bug rather than as
2560 a point for new development. The code in [[file:lisp/org-babel-ref.el][org-babel-ref.el]] already
2561 resolves variable references in a recursive manner which *should* work
2562 in the same manner regardless of the depth of the number of nested
2563 function calls. This recursive evaluation has the effect of
2564 implicitly constructing the parse tree that your are thinking of
2565 constructing explicitly.
2567 Through using some of the commented out debugging statements in
2568 [[file:lisp/org-babel-ref.el][org-babel-ref.el]] I have looked at what may be going wrong in the
2569 current evaluation setup, and it seems that nested variables are being
2570 set using the =:var= header argument, and these variables are being
2571 overridden by the *default* variables which are being entered through
2572 the new functional syntax (see the demonstration header below).
2574 I believe that once this bug is fixed we should be back to fully
2575 resolution of nested arguments. We should capture this functionality
2576 in a test to ensure that we continue to test it as we move forward. I
2577 can take a look at implementing this once I get a chance.
2579 Looks like the problem may be in [[file:lisp/org-babel.el::defun%20org%20babel%20merge%20params%20rest%20plists][org-babel-merge-params]], which seems
2580 to be trampling the provided :vars values.
2582 Nope, now it seems that we are actually looking up the results line,
2583 rather than the actual source-code block, which would make sense given
2584 that the results-line will return the same value regardless of the
2585 arguments supplied. See the output of this [[file:lisp/org-babel-ref.el::message%20type%20S%20type%20debugging][debug-statement]].
2587 We need to be sure that we don't read from a =#+resname:= line when we
2588 have a non-nil set of arguments.
2591 After uncommenting the debugging statements located [[file:lisp/org-babel-ref.el::message%20format%20first%20second%20S%20S%20new%20refere%20new%20referent%20debugging][here]] and more
2592 importantly [[file:lisp/org-babel-ref.el::message%20nested%20args%20S%20args%20debugging][here]], we can see that the current reference code does
2593 evaluate the references correctly, and it uses the =:var= header
2594 argument to set =a=8=, however the default variables specified using
2595 the functional syntax in =adder(a=3, b=2)= is overriding this
2598 ***** doesn't work with functional syntax
2600 #+srcname: adder-func(a=3, b=2)
2605 #+resname: adder-func
2608 #+srcname: after-adder-func(arg=adder-func(a=8))
2613 #+resname: after-adder-func
2616 ***** still does work with =:var= syntax
2618 so it looks like regardless of the syntax used we're not overriding
2619 the default argument values.
2621 #+srcname: adder-header
2622 #+begin_src python :var a=3 :var b=2
2626 #+resname: adder-header
2629 #+srcname: after-adder-header
2630 #+begin_src python :var arg=adder-header(a=8, b=0)
2634 #+resname: after-adder-header
2637 *** Set of test cases
2638 **** Both defaults provided in definition
2639 #+srcname: adder1(a=10,b=20)
2647 ****** DONE Rely on defaults
2655 ******* DONE empty parens () not recognised as lob call
2656 E.g. remove spaces between parens above
2658 updated [[file:lisp/org-babel-lob.el::defvar%20org%20babel%20lob%20one%20liner%20regexp%20lob%20t%20n%20n%20t%20n][org-babel-lob-one-liner-regexp]]
2660 ****** DONE One supplied, one default
2663 #+resname: adder1(a=0)
2669 #+resname: adder1(b=0)
2673 ****** DONE Both supplied
2674 #+lob: adder1(a=1,b=2)
2676 #+resname: adder1(a=1,b=2)
2680 **** One arg lacks default in definition
2681 #+srcname: adder2(a=10,b)
2685 ****** DEFERRED Rely on defaults (one of which is missing)
2690 ## should be error: b has no default
2692 Maybe we should let the programming language handle this case. For
2693 example python spits out an error in the =#+lob= line above. Maybe
2694 rather than catching these errors our-selves we should raise an error
2695 when the source-block returns an error. I'll propose a [[* PROPOSED raise elisp error when source-blocks return errors][task]] for this
2696 idea, I'm not sure how/if it would work...
2698 ****** DEFERRED Default over-ridden
2701 See the above [[* DEFERRED Rely on defaults (one of which is missing)][deferred]] and the new proposed [[* PROPOSED raise elisp error when source-blocks return errors][task]], I think it may be
2702 more flexible to allow the source block language to handle the error.
2705 ## should be error: b has no default
2707 ****** DONE Missing default supplied
2710 #+resname: adder2(b=1)
2716 ****** DONE One over-ridden, one supplied
2717 #+lob: adder2(a=1,b=2)
2719 #+resname: adder2(a=1,b=2)
2723 *** Example that fails
2725 #+srcname: adder(a=0, b=99)
2735 #+begin_src python :results silent
2740 #+srcname: level-one-nesting
2741 #+begin_src python :var arg=adder(a=one(),b=one())
2748 #+srcname: level-one-nesting
2749 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=one()))
2756 *** DONE deeply nested arguments still fails
2758 #+srcname: deeply-nested-args-bug
2759 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=one()))
2766 results in this error
2767 : supplied params=nil
2768 : new-refere="adder", new-referent="a=adder(a=one(),b=one()),b=adder(a=one(),b=one())"
2769 : args=((:var . "a=adder(a=one()") (:var . "b=one())") (:var . "b=adder(a=one()") (:var . "b=one())"))
2771 : supplied params=((:var . "a=adder(a=one()") (:var . "b=one())") (:var . "b=adder(a=one()") (:var . "b=one())"))
2772 : new-refere="adder", new-referent="a=one("
2773 : args=((:var . "a=one("))
2775 : supplied params=((:var . "a=one("))
2776 : reference 'one(' not found in this buffer
2778 Need to change the regexp in [[file:lisp/org-babel-ref.el::assign%20any%20arguments%20to%20pass%20to%20source%20block][org-babel-ref-resolve-reference]] so that
2779 it only matches when the parenthesis are balanced. Maybe look at
2780 [[http://www.gnu.org/software/emacs/elisp/html_node/List-Motion.html][this]].
2782 ** DONE avoid stripping whitespace from output when :results output
2783 This may be partly solved by using o-b-chomp rather than o-b-trim
2784 in the o-b-LANG-evaluate functions.
2785 ** DEFERRED weird escaped characters in shell prompt break shell evaluation
2786 E.g. this doesn't work. Should the shell sessions set a sane prompt
2787 when they start up? Or is it a question of altering
2788 comint-prompt-regexp? Or altering org-babel regexps?
2791 black=30 ; red=31 ; green=32 ; yellow=33 ; blue=34 ; magenta=35 ; cyan=36 ; white=37
2794 export PS1="\[\033[${prompt_col}m\]\w${prompt_char} \[\033[0m\]"
2797 I just pushed a good amount of changes, could you see if your shell
2798 problems still exist?
2800 The problem's still there. Specifically, aIui, at [[file:lisp/langs/org-babel-sh.el::raw%20org%20babel%20comint%20with%20output%20buffer%20org%20babel%20sh%20eoe%20output%20nil%20insert%20full%20body%20comint%20send%20input%20nil%20t][this line]] of
2801 org-babel-sh.el, raw gets the value
2803 ("" "
\e[0m Sun Jun 14 19:26:24 EDT 2009\n" "
\e[0m org_babel_sh_eoe\n" "
\e[0m ")
2805 and therefore (member org-babel-sh-eoe-output ...) fails
2807 I think that `comint-prompt-regexp' needs to be altered to match
2808 the shell prompt. This shouldn't be too difficult to do by hand,
2809 using the `regexp-builder' command and should probably be part of
2810 the user's regular emacs init. I can't think of a way for us to
2811 set this automatically, and we are SOL without a regexp to match
2813 ** DONE function calls in #+srcname: refs
2815 My srcname references don't seem to be working for function
2816 calls. This needs fixing.
2823 srcname function call doesn't work for calling a source block
2824 #+srcname: caller(var1=called())
2837 They do work for a simple reference
2838 #+srcname: caller2(var1=56)
2847 and they do work for :var header arg
2849 #+begin_src python :var var1=called()
2855 ** DONE LoB: with output to buffer, not working in buffers other than library-of-babel.org
2857 I haven't fixed this yet. org-babel-ref-resolve-reference moves
2858 point around, inside a save-excursion. Somehow when it comes to
2859 inserting the results (after possible further recursive calls to
2860 org-babel-ref-resolve-reference), point hasn't gone back to the
2863 #+tblname: test-data
2868 #+lob: R-plot(data=test-data)
2870 #+lob: python-add(a=2, b=9)
2872 #+resname: python-add(a=2, b=9)
2876 I think this got fixed in the bugfixes before merging results into master.
2878 ** DONE cursor movement when evaluating source blocks
2879 E.g. the pie chart example. Despite the save-window-excursion in
2880 org-babel-execute:R. (I never learned how to do this properly: org-R
2881 jumps all over the place...)
2883 I don't see this now [ded]
2885 ** DONE LoB: calls fail if reference has single character name
2886 commit 21d058869df1ff23f4f8cc26f63045ac9c0190e2
2887 **** This doesn't work
2888 #+lob: R-plot(data=X)
2907 #+lob: R-plot(data=XX)
2909 ** DONE make :results replace the default?
2910 I'm tending to think that appending results to pre-existing results
2911 creates mess, and that the cleaner `replace' option should be the
2912 default. E.g. when a source block creates an image, we would want
2913 that to be updated, rather than have a new one be added.
2917 ** DONE ruby evaluation not working under ubuntu emacs 23
2918 With emacs 23.0.91.1 on ubuntu, for C-h f run-ruby I have the
2919 following, which seems to conflict with [[file:lisp/langs/org-babel-ruby.el::let%20session%20buffer%20save%20window%20excursion%20run%20ruby%20nil%20session%20current%20buffer][this line]] in org-babel-ruby.el.
2922 run-ruby is an interactive compiled Lisp function.
2926 Run an inferior Ruby process, input and output via buffer *ruby*.
2927 If there is a process already running in `*ruby*', switch to that buffer.
2928 With argument, allows you to edit the command line (default is value
2929 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
2930 (after the `comint-mode-hook' is run).
2931 (Type C-h m in the process buffer for a list of commands.)
2934 So, I may have a non-standard inf-ruby.el. Here's my version of
2938 run-ruby is an interactive Lisp function in `inf-ruby.el'.
2940 (run-ruby &optional COMMAND NAME)
2942 Run an inferior Ruby process, input and output via buffer *ruby*.
2943 If there is a process already running in `*ruby*', switch to that buffer.
2944 With argument, allows you to edit the command line (default is value
2945 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
2946 (after the `comint-mode-hook' is run).
2947 (Type C-h m in the process buffer for a list of commands.)
2950 It seems we could either bundle my version of inf-ruby.el (as it's
2951 the newest). Or we could change the use of `run-ruby' so that it
2952 is robust across multiple distributions. I think I'd prefer the
2953 former, unless the older version of inf-ruby is actually bundled
2954 with emacs, in which case maybe we should go out of our way to
2955 support it. Thoughts?
2957 I think for now I'll just include the latest [[file:util/inf-ruby.el][inf-ruby.el]] in the
2958 newly created utility directory. I doubt anyone would have a
2959 problem using the latest version of this file.
2960 ** DONE test failing forcing vector results with =test-forced-vector-results= ruby code block
2961 Note that this only seems to happen the *second* time the test table
2964 #+srcname: bug-trivial-vector
2965 #+begin_src emacs-lisp :results vector silent
2969 #+srcname: bug-forced-vector-results
2970 #+begin_src ruby :var triv=test-trivial-vector :results silent
2974 mysteriously this seems to be fixed...
2975 ** DONE defunct R sessions
2976 Sometimes an old R session will turn defunct, and newly inserted code
2977 will not be evaluated (leading to a hang).
2979 This seems to be fixed by using `inferior-ess-send-input' rather than `comint-send-input'.
2980 ** DONE ruby fails on first call to non-default session
2982 #+srcname: bug-new-session
2983 #+begin_src ruby :session is-new
2987 ** DONE when reading results from =#+resname= line
2989 Errors when trying to read from resname lines.
2991 #+resname: bug-in-resname
2994 #+srcname: bug-in-resname-reader
2995 #+begin_src emacs-lisp :var buggy=bug-in-resname() :results silent
2999 ** DONE R-code broke on "org-babel" rename
3001 #+srcname: bug-R-babels
3006 ** DONE error on trivial R results
3008 So I know it's generally not a good idea to squash error without
3009 handling them, but in this case the error almost always means that
3010 there was no file contents to be read by =org-table-import=, so I
3013 #+srcname: bug-trivial-r1
3014 #+begin_src R :results replace
3015 pie(c(1, 2, 3), labels = c(1, 2, 3))
3018 #+srcname: bug-trivial-r2
3019 #+begin_src R :results replace
3023 #+resname: bug-trivial-r2
3026 #+srcname: bug-trivial-r3
3027 #+begin_src R :results replace
3031 #+resname: bug-trivial-r3
3036 ** DONE ruby new variable creation (multi-line ruby blocks)
3037 Actually it looks like we were dropping all but the last line.
3039 #+srcname: multi-line-ruby-test
3040 #+begin_src ruby :var table=bug-numerical-table :results replace
3042 table.each{|n| total += n}
3049 ** DONE R code execution seems to choke on certain inputs
3050 Currently the R code seems to work on vertical (but not landscape)
3053 #+srcname: little-fake
3054 #+begin_src emacs-lisp
3059 #+begin_src R :var num=little-fake
3066 #+srcname: set-debug-on-error
3067 #+begin_src emacs-lisp :results silent
3068 (setq debug-on-error t)
3071 #+srcname: bug-numerical-table
3072 #+begin_src emacs-lisp :results silent
3079 #+srcname: bug-R-number-evaluation
3080 #+begin_src R :var table=bug-numerical-table
3089 #+tblname: bug-vert-table
3094 #+srcname: bug-R-vertical-table
3095 #+begin_src R :var table=bug-vert-table :results silent
3099 ** DONE org bug/request: prevent certain org behaviour within code blocks
3100 E.g. [[]] gets recognised as a link (when there's text inside the
3101 brackets). This is bad for R code at least, and more generally
3102 could be argued to be inappropriate. Is it difficult to get org to
3103 ignore text in code blocks? [DED]
3105 I believe Carsten addressed this recently on the mailing list with
3106 the comment that it was indeed a difficult issue. I believe this
3107 may be one area where we could wait for an upstream (org-mode) fix.
3109 [Dan] Carsten has fixed this now in the core.
3111 ** DONE with :results replace, non-table output doesn't replace table output
3112 And vice versa. E.g. Try this first with table and then with len(table) [DED]
3113 #+begin_src python :var table=sandbox :results replace
3118 | 4 | "schulte" | 6 |
3121 Yes, this is certainly a problem. I fear that if we begin replacing
3122 anything immediately following a source block (regardless of whether
3123 it matches the type of our current results) we may accidentally delete
3124 hand written portions of the user's org-mode buffer.
3126 I think that the best solution here would be to actually start
3127 labeling results with a line that looks something like...
3131 This would have a couple of benefits...
3132 1) we wouldn't have to worry about possibly deleting non-results
3133 (which is currently an issue)
3134 2) we could reliably replace results even if there are different types
3135 3) we could reference the results of a source-code block in variable
3136 definitions, which would be useful if for example we don't wish to
3137 re-run a source-block every time because it is long-running.
3139 Thoughts? If no-one objects, I believe I will implement the labeling
3142 ** DONE extra quotes for nested string
3143 Well R appears to be reading the tables without issue...
3145 these *should* be quoted
3147 #+begin_src sh :results replace
3152 | "README.markdown" |
3155 | "existing_tools" |
3159 | "test-export.html" |
3160 | "test-export.org" |
3162 #+srcname: test-quotes
3163 #+begin_src ruby :var tab=ls
3169 #+srcname: test-quotes
3170 #+begin_src R :var tab=ls
3176 ** DONE simple ruby arrays not working
3178 As an example eval the following. Adding a line to test
3180 #+tblname: simple-ruby-array
3183 #+srcname: ruby-array-test
3184 #+begin_src ruby :var ar = simple-ruby-array :results silent
3188 ** DONE space trailing language name
3189 fix regexp so it works when there's a space trailing the language name
3191 #+srcname: test-trailing-space
3196 ** DONE Args out of range error
3198 The following block resulted in the error below [DED]. It ran without
3199 error directly in the shell.
3202 for platf in ill aff ; do
3203 for pop in CEU YRI ASI ; do
3204 rm -f $platf/hapmap-genos-$pop-all $platf/hapmap-rs-all
3205 cat $platf/hapmap-genos-$pop-* > $platf/hapmap-genos-$pop-all
3206 cat $platf/hapmap-rs-* > $platf/hapmap-rs-all
3211 executing source block with sh...
3212 finished executing source block
3213 string-equal: Args out of range: "", -1, 0
3215 the error =string-equal: Args out of range: "", -1, 0= looks like what
3216 used to be output when the block returned an empty results string.
3217 This should be fixed in the current version, you should now see the
3218 following message =no result returned by source block=.
3220 ** DONE ruby arrays not recognized as such
3222 Something is wrong in [[file:lisp/org-babel-script.el]] related to the
3223 recognition of ruby arrays as such.
3225 #+begin_src ruby :results replace
3231 #+begin_src python :results replace
3236 ** REJECTED elisp reference fails for literal number
3237 That's a bug in Dan's elisp, not in org-babel.
3238 #+srcname: elisp-test(a=4)
3239 #+begin_src emacs-lisp
3251 Evaluate all the cells in this table for a comprehensive test of the
3252 org-babel functionality.
3254 *Note*: if you have customized =org-babel-default-header-args= then some
3255 of these tests may fail.
3257 #+TBLNAME: org-babel-tests
3258 | functionality | block | arg | expected | results | pass |
3259 |-------------------------+----------------------------+-----+-------------+-------------+------|
3260 | basic evaluation | | | | | pass |
3261 |-------------------------+----------------------------+-----+-------------+-------------+------|
3262 | emacs lisp | basic-elisp | | 5 | 5 | pass |
3263 | shell | basic-shell | | 6 | 6 | pass |
3264 | ruby | basic-ruby | | org-babel | org-babel | pass |
3265 | python | basic-python | | hello world | hello world | pass |
3266 | R | basic-R | | 13 | 13 | pass |
3267 |-------------------------+----------------------------+-----+-------------+-------------+------|
3268 | tables | | | | | pass |
3269 |-------------------------+----------------------------+-----+-------------+-------------+------|
3270 | emacs lisp | table-elisp | | 3 | 3 | pass |
3271 | ruby | table-ruby | | 1-2-3 | 1-2-3 | pass |
3272 | python | table-python | | 5 | 5 | pass |
3273 | R | table-R | | 3.5 | 3.5 | pass |
3274 | R: col names in R | table-R-colnames | | -3 | -3 | pass |
3275 | R: col names in org | table-R-colnames-org | | 169 | 169 | pass |
3276 |-------------------------+----------------------------+-----+-------------+-------------+------|
3277 | source block references | | | | | pass |
3278 |-------------------------+----------------------------+-----+-------------+-------------+------|
3279 | all languages | chained-ref-last | | Array | Array | pass |
3280 |-------------------------+----------------------------+-----+-------------+-------------+------|
3281 | source block functions | | | | | pass |
3282 |-------------------------+----------------------------+-----+-------------+-------------+------|
3283 | emacs lisp | defun-fibb | | fibbd | fibbd | pass |
3284 | run over | Fibonacci | 0 | 1 | 1 | pass |
3285 | a | Fibonacci | 1 | 1 | 1 | pass |
3286 | variety | Fibonacci | 2 | 2 | 2 | pass |
3287 | of | Fibonacci | 3 | 3 | 3 | pass |
3288 | different | Fibonacci | 4 | 5 | 5 | pass |
3289 | arguments | Fibonacci | 5 | 8 | 8 | pass |
3290 |-------------------------+----------------------------+-----+-------------+-------------+------|
3291 | bugs and tasks | | | | | pass |
3292 |-------------------------+----------------------------+-----+-------------+-------------+------|
3293 | simple ruby arrays | ruby-array-test | | 3 | 3 | pass |
3294 | R number evaluation | bug-R-number-evaluation | | 2 | 2 | pass |
3295 | multi-line ruby blocks | multi-line-ruby-test | | 2 | 2 | pass |
3296 | forcing vector results | test-forced-vector-results | | Array | Array | pass |
3297 | deeply nested arguments | deeply-nested-args-bug | | 8 | 8 | pass |
3298 |-------------------------+----------------------------+-----+-------------+-------------+------|
3299 | sessions | | | | | pass |
3300 |-------------------------+----------------------------+-----+-------------+-------------+------|
3301 | set ruby session | set-ruby-session-var | | :set | :set | pass |
3302 | get from ruby session | get-ruby-session-var | | 3 | 3 | pass |
3303 | set python session | set-python-session-var | | set | set | pass |
3304 | get from python session | get-python-session-var | | 4 | 4 | pass |
3305 | set R session | set-R-session-var | | set | set | pass |
3306 | get from R session | get-R-session-var | | 5 | 5 | pass |
3307 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
3308 #+TBLFM: $5=""::$6=""
3311 The second TBLFM line (followed by replacing '[]' with '') can be used
3312 to blank out the table results, in the absence of a better method.
3316 #+srcname: basic-elisp
3317 #+begin_src emacs-lisp :results silent
3322 #+srcname: basic-shell
3323 #+begin_src sh :results silent
3328 #+srcname: date-simple
3329 #+begin_src sh :results silent
3333 #+srcname: basic-ruby
3334 #+begin_src ruby :results silent
3339 #+srcname: basic-python
3340 #+begin_src python :results silent
3346 #+begin_src R :results silent
3354 #+tblname: test-table
3358 #+tblname: test-table-colnames
3359 | var1 | var2 | var3 |
3360 |------+------+------|
3364 #+srcname: table-elisp
3365 #+begin_src emacs-lisp :results silent :var table=test-table
3366 (length (car table))
3370 #+srcname: table-ruby
3371 #+begin_src ruby :results silent :var table=test-table
3372 table.first.join("-")
3376 #+srcname: table-python
3377 #+begin_src python :var table=test-table
3381 #+srcname: table-R(table=test-table)
3386 #+srcname: table-R-colnames(table=test-table-colnames)
3387 #+begin_src R :results silent
3388 sum(table$var2 - table$var3)
3391 #+srcname: R-square(x=default-name-doesnt-exist)
3392 #+begin_src R :colnames t
3396 This should return 169. The fact that R is able to use the column name
3397 to index the data frame (x$var3) proves that a table with column names
3398 (a header row) has been recognised as input for the R-square function
3399 block, and that the R-square block has output an elisp table with
3400 column names, and that the colnames have again been recognised when
3401 creating the R variables in this block.
3402 #+srcname: table-R-colnames-org(x = R-square(x=test-table-colnames))
3412 Lets pass a references through all of our languages...
3414 Lets start by reversing the table from the previous examples
3416 #+srcname: chained-ref-first
3417 #+begin_src python :var table = test-table
3422 #+resname: chained-ref-first
3426 Take the first part of the list
3428 #+srcname: chained-ref-second
3429 #+begin_src R :var table = chained-ref-first
3433 #+resname: chained-ref-second
3437 Turn the numbers into string
3439 #+srcname: chained-ref-third
3440 #+begin_src emacs-lisp :var table = chained-ref-second
3441 (mapcar (lambda (el) (format "%S" el)) table)
3444 #+resname: chained-ref-third
3447 and Check that it is still a list
3449 #+srcname: chained-ref-last
3450 #+begin_src ruby :var table=chained-ref-third
3455 ** source blocks as functions
3457 #+srcname: defun-fibb
3458 #+begin_src emacs-lisp :results silent
3459 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
3463 #+srcname: fibonacci
3464 #+begin_src emacs-lisp :results silent :var n=7
3474 ** sbe tests (these don't seem to be working...)
3475 Testing the insertion of results into org-mode tables.
3477 #+srcname: multi-line-output
3478 #+begin_src ruby :results replace
3479 "the first line ends here
3482 and this is the second one
3488 : the first line ends here\n\n\n and this is the second one\n\neven a third
3490 #+srcname: multi-line-error
3491 #+begin_src ruby :results replace
3492 raise "oh nooooooooooo"
3498 | the first line ends here... | -:5: warning: parenthesize argument(s) for future version... |
3499 #+TBLFM: $1='(sbe "multi-line-output")::$2='(sbe "multi-line-error")
3501 ** forcing results types tests
3503 #+srcname: test-trivial-vector
3504 #+begin_src emacs-lisp :results vector silent
3508 #+srcname: test-forced-vector-results
3509 #+begin_src ruby :var triv=test-trivial-vector :results silent
3515 #+srcname: set-ruby-session-var
3516 #+begin_src ruby :session :results silent
3521 #+srcname: get-ruby-session-var
3522 #+begin_src ruby :session :results silent
3526 #+srcname: set-python-session-var
3527 #+begin_src python :session
3532 #+srcname: get-python-session-var
3533 #+begin_src python :session
3537 #+srcname: set-R-session-var
3538 #+begin_src R :session
3543 #+srcname: get-R-session-var
3544 #+begin_src R :session
3553 To run these examples evaluate [[file:lisp/org-babel-init.el][org-babel-init.el]]
3555 ** org-babel.el beginning functionality
3557 #+begin_src sh :results replace
3562 : Sun Jul 5 18:54:39 EDT 2009
3569 : Sun Jul 05 18:54:35 -0400 2009
3581 #+begin_src R :results replace
3591 hist(rgamma(20,3,3))
3596 ** org-babel plays with tables
3597 Alright, this should demonstrate both the ability of org-babel to read
3598 tables into a lisp source code block, and to then convert the results
3599 of the source code block into an org table. It's using the classic
3600 "lisp is elegant" demonstration transpose function. To try this
3603 1. evaluate [[file:lisp/org-babel-init.el]] to load org-babel and friends
3604 2. evaluate the transpose definition =\C-c\\C-c= on the beginning of
3606 3. evaluate the next source code block, this should read in the table
3607 because of the =:var table=previous=, then transpose the table, and
3608 finally it should insert the transposed table into the buffer
3609 immediately following the block
3613 #+begin_src emacs-lisp :results silent
3614 (defun transpose (table)
3615 (apply #'mapcar* #'list table))
3623 #+begin_src emacs-lisp :var table=sandbox :results replace
3628 #+begin_src emacs-lisp
3633 | 1 | 2 | 3 | 4 | 5 |
3637 #+begin_src ruby :var table=sandbox :results replace
3638 table.first.join(" - ")
3644 #+begin_src python :var table=sandbox
3649 #+begin_src ruby :var table=sandbox :results replace
3654 : [[1, 2, 3], [4, "schulte", 6]]
3658 | 4 | "schulte" | 6 |
3660 #+begin_src python :var table=sandbox :results replace
3666 | "__add__" | "__class__" | "__contains__" | "__delattr__" | "__delitem__" | "__delslice__" | "__doc__" | "__eq__" | "__format__" | "__ge__" | "__getattribute__" | "__getitem__" | "__getslice__" | "__gt__" | "__hash__" | "__iadd__" | "__imul__" | "__init__" | "__iter__" | "__le__" | "__len__" | "__lt__" | "__mul__" | "__ne__" | "__new__" | "__reduce__" | "__reduce_ex__" | "__repr__" | "__reversed__" | "__rmul__" | "__setattr__" | "__setitem__" | "__setslice__" | "__sizeof__" | "__str__" | "__subclasshook__" | "append" | "count" | "extend" | "index" | "insert" | "pop" | "remove" | "reverse" | "sort" |
3668 *** (sandbox table) R
3670 #+TBLNAME: sandbox_r
3674 #+begin_src R :results replace
3675 x <- c(rnorm(10, mean=-3, sd=1), rnorm(10, mean=3, sd=1))
3679 | -3.35473133869346 |
3681 | -3.32819924928633 |
3682 | -2.97310212756194 |
3683 | -2.09640758369576 |
3684 | -5.06054014378736 |
3685 | -2.20713700711221 |
3686 | -1.37618039712037 |
3687 | -1.95839385821742 |
3688 | -3.90407396475502 |
3689 | 2.51168071590226 |
3690 | 3.96753011570494 |
3691 | 3.31793212627865 |
3692 | 1.99829753972341 |
3693 | 4.00403686419829 |
3694 | 4.63723764452927 |
3695 | 3.94636744261313 |
3696 | 3.58355906547775 |
3697 | 3.01563442274226 |
3700 #+begin_src R var tabel=sandbox_r :results replace
3705 | 4 | "schulte" | 6 |
3708 Now shell commands are converted to tables using =org-table-import=
3709 and if these tables are non-trivial (i.e. have multiple elements) then
3710 they are imported as org-mode tables...
3712 #+begin_src sh :results replace
3716 | "total" | 208 | "" | "" | "" | "" | "" | "" |
3717 | "-rw-r--r--" | 1 | "dan" | "dan" | 57 | 2009 | 15 | "block" |
3718 | "-rw-r--r--" | 1 | "dan" | "dan" | 35147 | 2009 | 15 | "COPYING" |
3719 | "-rw-r--r--" | 1 | "dan" | "dan" | 722 | 2009 | 18 | "examples.org" |
3720 | "drwxr-xr-x" | 4 | "dan" | "dan" | 4096 | 2009 | 19 | "existing_tools" |
3721 | "-rw-r--r--" | 1 | "dan" | "dan" | 2207 | 2009 | 14 | "intro.org" |
3722 | "drwxr-xr-x" | 2 | "dan" | "dan" | 4096 | 2009 | 18 | "org-babel" |
3723 | "-rw-r--r--" | 1 | "dan" | "dan" | 277 | 2009 | 20 | "README.markdown" |
3724 | "-rw-r--r--" | 1 | "dan" | "dan" | 11837 | 2009 | 18 | "rorg.html" |
3725 | "-rw-r--r--" | 1 | "dan" | "dan" | 61829 | 2009 | 19 | "#rorg.org#" |
3726 | "-rw-r--r--" | 1 | "dan" | "dan" | 60190 | 2009 | 19 | "rorg.org" |
3727 | "-rw-r--r--" | 1 | "dan" | "dan" | 972 | 2009 | 11 | "test-export.org" |
3730 ** silent evaluation
3738 #+begin_src ruby :results silent
3742 #+begin_src ruby :results replace
3749 ** (sandbox) referencing other source blocks
3750 Doing this in emacs-lisp first because it's trivial to convert
3751 emacs-lisp results to and from emacs-lisp.
3753 *** emacs lisp source reference
3754 This first example performs a calculation in the first source block
3755 named =top=, the results of this calculation are then saved into the
3756 variable =first= by the header argument =:var first=top=, and it is
3757 used in the calculations of the second source block.
3760 #+begin_src emacs-lisp
3764 #+begin_src emacs-lisp :var first=top :results replace
3770 This example is the same as the previous only the variable being
3771 passed through is a table rather than a number.
3773 #+begin_src emacs-lisp :results silent
3774 (defun transpose (table)
3775 (apply #'mapcar* #'list table))
3778 #+TBLNAME: top_table
3782 #+SRCNAME: second_src_example
3783 #+begin_src emacs-lisp :var table=top_table
3787 #+begin_src emacs-lisp :var table=second_src_example :results replace
3792 | 4 | "schulte" | 6 |
3794 Now working for ruby
3801 #+begin_src ruby :var other=start :results replace
3807 #+SRCNAME: start_two
3812 #+begin_src python :var another=start_two :results replace
3817 Since all variables are converted into Emacs Lisp it is no problem to
3818 reference variables specified in another language.
3820 #+SRCNAME: ruby-block
3825 #+SRCNAME: lisp_block
3826 #+begin_src emacs-lisp :var ruby-variable=ruby-block
3830 #+begin_src python :var lisp_var=lisp_block
3839 #+begin_src R :results replace
3846 #+begin_src R :var other=first_r :results replace
3853 ** (sandbox) selective export
3855 For exportation tests and examples see (including exportation of
3856 inline source code blocks) [[file:test-export.org]]
3859 ** (sandbox) source blocks as functions
3862 #+begin_src emacs-lisp :results silent
3867 #+begin_src emacs-lisp :var n=default :results replace
3873 #+begin_src emacs-lisp :var result=triple(n=3, m=98) :results replace
3879 The following just demonstrates the ability to assign variables to
3880 literal values, which was not implemented until recently.
3882 #+begin_src ruby :var num="eric" :results replace
3889 ** (sandbox) inline source blocks
3891 This is an inline source code block src_ruby{1 + 6}. And another
3892 source block with text output src_emacs-lisp{"eric"}.
3894 This is an inline source code block with header
3895 arguments. src_ruby[:var n=fibbd( n = 0 )]{n}
3898 ** (sandbox) integration w/org tables
3900 #+begin_src emacs-lisp :results silent
3901 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
3905 #+begin_src emacs-lisp :var n=4 :results silent
3909 #+begin_src emacs-lisp :results silent
3910 (mapcar #'fibbd '(0 1 2 3 4 5 6 7 8))
3913 Something is not working here. The function `sbe ' works fine when
3914 called from outside of the table (see the source block below), but
3915 produces an error when called from inside the table. I think there
3916 must be some narrowing going on during intra-table emacs-lisp
3919 | original | fibbd |
3920 |----------+-------|
3931 #+TBLFM: $2='(sbe "fibbd" (n $1))
3935 #+begin_src emacs-lisp :results silent
3936 (sbe 'fibbd (n "8"))
3941 LocalWords: DBlocks dblocks org-babel el eric fontification