1 #+TITLE: org-babel --- facilitating communication between programming languages and people
2 #+SEQ_TODO: PROPOSED TODO STARTED | DONE DEFERRED REJECTED
3 #+OPTIONS: H:3 num:nil toc:2 \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:(HIDE) tags:not-in-toc
4 #+STARTUP: oddeven hideblocks
8 <p>executable source code blocks in org-mode</p>
12 <img src="images/tower-of-babel.png" alt="images/tower-of-babel.png" />
13 <div id="attr">from <a href="http://www.flickr.com/photos/23379658@N05/" title=""><b>Martijn Streefkerk</b></a></div>
18 Org-Babel makes source-code blocks in Org-Mode executable and allows
19 data to pass seamlessly between different programming languages,
20 Org-Mode constructs (tables, file links, example text) and interactive
24 - The [[* Introduction][Introduction]] :: provides a brief overview of the design and use
25 of Org-Babel including tutorials and examples.
26 - In [[* Getting started][Getting Started]] :: find instructions for installing org-babel
27 into your emacs configuration.
28 - The [[* Tasks][Tasks]] :: section contains current and past tasks roughly ordered
29 by TODO state, then importance or date-completed. This would be
30 a good place to suggest ideas for development.
31 - The [[* Bugs][Bugs]] :: section contains bug reports.
32 - The [[* Tests][Tests]] :: section consists of a large table which can be
33 evaluated to run Org-Babel's functional test suite. This
34 provides a good overview of the current functionality with
35 pointers to example source blocks.
36 - The [[* Sandbox][Sandbox]] :: demonstrates much of the early/basic functionality
37 through commented source-code blocks.
39 Also see the [[file:library-of-babel.org][Library of Babel]], an extensible collection of ready-made
40 and easily-shortcut-callable source-code blocks for handling common
45 Org-Babel enables *communication* between programming languages and
49 - communication between programs :: Data passes seamlessly between
50 different programming languages, Org-Mode constructs (tables,
51 file links, example text) and interactive comint buffers.
52 - communication between people :: Data and calculations are embedded
53 in the same document as notes explanations and reports.
55 ** communication between programs
57 Org-Mode supports embedded blocks of source code (in any language)
58 inside of Org documents. Org-Babel allows these blocks of code to be
59 executed from within Org-Mode with natural handling of their inputs
63 with both scalar, file, and table output
65 *** reading information from tables
67 *** reading information from other source blocks (disk usage in your home directory)
69 This will work for Linux and Mac users, not so sure about shell
70 commands for windows users.
72 To run place the cursor on the =#+begin_src= line of the source block
73 labeled directory-pie and press =\C-c\C-c=.
75 #+srcname: directories
76 #+begin_src bash :results replace
77 cd ~ && du -sc * |grep -v total
80 #+resname: directories
82 | 11882808 | "Documents" |
83 | 8210024 | "Downloads" |
84 | 879800 | "Library" |
87 | 5307664 | "Pictures" |
96 #+srcname: directory-pie
97 #+begin_src R :var dirs = directories :session R-pie-example
98 pie(dirs[,1], labels = dirs[,2])
106 *** operations in/on tables
108 #+tblname: grades-table
109 | student | grade | letter |
110 |---------+-------+--------|
117 #+TBLFM: $2='(sbe random-score-generator)::$3='(sbe assign-grade (score $2))
119 #+srcname: assign-grade
120 #+begin_src ruby :var score=99
131 #+srcname: random-score-generator
136 #+srcname: show-distribution
137 #+begin_src R :var grades=grades-table :session *R*
145 ** communication between people
146 Quick overview of Org-Mode's exportation abilities, with links to the
147 online Org-Mode documentation, a focus on source-code blocks, and the
148 exportation options provided by Org-Babel.
150 *** Interactive tutorial
151 This would demonstrate applicability to Reproducible Research, and
152 Literate Programming.
154 *** Tests embedded in documentation
155 org-babels own functional tests are contained in a large org-mode
156 table, allowing the test suite to be run be evaluation of the table
157 and the results to be collected in the same table.
159 *** Emacs initialization files stored in Org-Mode buffers
160 Using `org-babel-tangle' it is possible to embed your Emacs
161 initialization into org-mode files. This allows for folding,
162 note-taking, todo's etc... embedded with the source-code of your Emacs
163 initialization, and through org-mode's publishing features aids in
164 sharing your customizations with others.
166 It may be worthwhile to create a fork of Phil Hagelberg's
167 [[http://github.com/technomancy/emacs-starter-kit/tree/master][emacs-starter-kit]] which uses literate org-mode files for all of the
168 actual elisp customization. These org-mode files could then be
169 exported to html and used to populate the repositories wiki on [[http://github.com/][github]].
174 *** code evaluation (comint buffer sessions and external processes)
175 There are two main ways to evaluate source blocks with org-babel.
177 - external :: By default (if the =:session= header argument is not
178 present) all source code blocks are evaluated in
179 external processes. In these cases an external process
180 is used to evaluate the source-code blocks.
181 - session :: Session based evaluation uses persistent sessions in
182 comint buffers. Sessions can be used across multiple
183 source blocks setting and accessing variables in the
186 Evaluating source blocks in sessions also allows for
187 interaction with the code. To jump to the session of a
188 source block use the `org-babel-pop-to-session' command
189 or press =M-[down]= while inside of a source code block.
190 When called with a prefix argument
191 `org-babel-pop-to-session' will evaluate all header
192 arguments before jumping to the source-code block.
194 *** results (values and outputs)
195 Either the *value* or the *output* of source code blocks can be
196 collected after evaluation.
198 - value :: The default way to collect results from a source-code block
199 is to return the value of the last statement in the block.
200 This can be thought of as the return value of the block.
201 In this case any printed output of the block is ignored.
202 This can be though of a similar to a "functional" value of
204 - output :: Another way of generating results from a source-code block
205 is to collect the output generated by the execution of the
206 block. In this case all printed output is collected
207 throughout the execution of the block. This can be
208 thought of as similar to a "script" style of evaluation.
212 Add the following lines to your .emacs, replacing the path as
213 appropriate. A good place to check that things are up and running
214 would then be [[#sandbox][the sandbox]].
215 #+begin_src emacs-lisp
216 (add-to-list 'load-path "/path/to/org-babel/lisp")
217 (require 'org-babel-init)
222 ** STARTED share org-babel [1/6]
223 how should we share org-babel?
224 *** DONE post to org-mode
225 *** TODO post to ess mailing list
226 *** TODO create a org-babel page on worg
227 *** TODO create a short screencast demonstrating org-babel in action
228 *** PROPOSED a peer-reviewed publication?
230 The following notes are biased towards statistics-oriented
231 journals because ESS and Sweave are written by people associated
232 with / in statistics departments. But I am sure there are suitable
233 journals out there for an article on using org mode for
234 reproducible research (and literate programming etc).
236 Clearly, we would invite Carsten to be involved with this.
238 ESS is described in a peer-reviewed journal article:
239 Emacs Speaks Statistics: A Multiplatform, Multipackage Development Environment for Statistical Analysis [Abstract]
240 Journal of Computational & Graphical Statistics 13(1), 247-261
241 Rossini, A.J, Heiberger, R.M., Sparapani, R.A., Maechler, M., Hornik, K. (2004)
242 [[http://www.amstat.org/publications/jcgs.cfm][Journal of Computational and Graphical Statistics]]
244 Also [[http://www.amstat.org/publications/jss.cfm][Journal of Statistical Software]] Established in 1996, the
245 Journal of Statistical Software publishes articles, book reviews,
246 code snippets, and software reviews. The contents are freely
247 available online. For both articles and code snippets, the source
248 code is published along with the paper.
252 Friedrich Leisch and Anthony J. Rossini. Reproducible statistical
253 research. Chance, 16(2):46-50, 2003. [ bib ]
257 Friedrich Leisch. Sweave: Dynamic generation of statistical reports
258 using literate data analysis. In Wolfgang Härdle and Bernd Rönz,
259 editors, Compstat 2002 - Proceedings in Computational Statistics,
260 pages 575-580. Physica Verlag, Heidelberg, 2002. ISBN 3-7908-1517-9.
264 We could also look at the Journals publishing these [[http://www.reproducibleresearch.net/index.php/RR_links#Articles_about_RR_.28chronologically.29][Reproducible
267 *** PROPOSED an article in [[http://journal.r-project.org/][The R Journal]]
268 This looks good. It seems that their main topic to software tools for
269 use by R programmers, and Org-babel is certainly that.
271 *** existing similar tools
272 try to collect pointers to similar tools
274 Reproducible Research
275 - [[http://en.wikipedia.org/wiki/Sweave][Sweave]]
278 - [[http://www.cs.tufts.edu/~nr/noweb/][Noweb]]
279 - [[http://www-cs-faculty.stanford.edu/~knuth/cweb.html][Cweb]]
280 - [[http://www.lri.fr/~filliatr/ocamlweb/][OCamlWeb]]
282 Meta Functional Programming
285 Programmable Spreadsheet
289 we need to think up some good examples
291 **** interactive tutorials
292 This could be a place to use [[* org-babel assertions][org-babel assertions]].
294 for example the first step of a tutorial could assert that the version
295 of the software-package (or whatever) is equal to some value, then
296 source-code blocks could be used with confidence (and executed
297 directly from) the rest of the tutorial.
299 **** answering a text-book question w/code example
300 org-babel is an ideal environment enabling both the development and
301 demonstrationg of the code snippets required as answers to many
304 **** something using tables
305 maybe something along the lines of calculations from collected grades
308 Maybe something like the following which outputs sizes of directories
309 under the home directory, and then instead of the trivial =emacs-lisp=
310 block we could use an R block to create a nice pie chart of the
314 #+begin_src bash :results replace
318 #+begin_src emacs-lisp :var sizes=sizes :results replace
321 *** Answer to question on list
322 From: Hector Villafuerte <hectorvd@gmail.com>
323 Subject: [Orgmode] Merge tables
324 Date: Wed, 19 Aug 2009 10:08:40 -0600
325 To: emacs-orgmode@gnu.org
328 I've just discovered Org and are truly impressed with it; using it for
331 Here's what I want to do: I have 2 tables with the same number of rows
332 (one row per subject). I would like to make just one big table by
333 copying the second table to the right of the first one. This is a
334 no-brainer in a spreadsheet but my attempts in Org have failed. Any
337 By the way, thanks for this great piece of software!
341 **** Suppose the tables are as follows
354 **** Here is an answer using R in org-babel
356 #+srcname: column-bind(a=tab1, b=tab2)
357 #+begin_src R :colnames t
361 #+resname: column-bind
362 | "a" | "b" | "c" | "d" | "e" | "f" |
363 |-----+-----+-----+-----+-----+-----|
364 | 1 | 2 | 3 | 4 | 5 | 6 |
365 | 7 | 8 | 9 | 10 | 11 | 12 |
369 Use org-table-export, do it in external spreadsheet software,
370 then org-table-import
371 ** TODO sha1 hash based caching
373 :CUSTOM_ID: sha1-caching
377 I wonder if we should consider some cashing of images, also for
378 export. I think we could have an alist with sha1 hashes as keys and
379 image files as values. The sha1 hash could be made from the entire
380 code and the command that is used to create the image..
387 (sha1 stuff) seems to work.
389 org-feed.el has a (require 'sha1) and org-publish.el uses it too.
394 ** TODO support for working with =*Org Edit Src Example*= buffers [4/7]
395 *** STARTED Patch against org source.
396 I've worked on several related changes to source code edit buffer
397 behaviour in the org core. My current patch (below) does the
398 following. Detailed explanation / working notes are below.
399 - C-x s offers to save edit buffers
400 - C-x C-c offers to save edit buffers
401 - C-x k warns that you're killing an edit buffer
402 - If you do kill an edit buffer, the overlay in the parent buffer is removed
403 - Edit buffers are named *Org Src <orgbuf>[<lang>]*, where
404 <orgbuf> is the name of the org-mode buffer containing this
405 source code block, and lang is the language major mode. The
406 latter might be unnecessary?
408 These changes were added to the main org repository in commit
409 4b6988bf36cb458c9d113ee4332e016990c1eb04
411 **** Detailed working notes to go with that patch
412 ***** Recap of current org-src-mode
414 If you use C-c ' to work on code in a begin_source block, the code
415 buffer is put in minor mode org-src-mode, which features the
416 following two useful key-bindings:
418 | C-x s | org-edit-src-save | save the code in the source code block in the parent org file |
419 | C-c ' | org-edit-src-exit | return to the parent org file with new code |
421 Furthermore, while the edit buffer is alive, the originating code
422 block is subject to a special overlay which links to the edit
423 buffer when you click on it.
425 This is all excellent, and I use it daily, but I think there's
426 still a couple of improvements that we should make.
429 C-x k kills the buffer without questions; the overlay remains, but
430 now links to a deleted buffer.
431 ***** Proposed bug II
432 C-x C-c kills a modified edit buffer silently, without offering to
433 save your work. I have lost work like that a number of times
435 ***** Proposed bug III
436 C-x s does not offer to save a modified edit buffer
437 ***** Notes on solution
438 ****** write-contents-functions
439 A good start seems to be to use org-src-mode-hook to add
440 org-edit-src-save to the write-contents-functions list. This
441 means that when it comes to saving, org-edit-src-save will be
442 called and no subsequent attempt will be made to save the buffer
443 in the normal way. (This should obviate the remapping of C-x C-s
444 to org-edit-src-save in org-src.el)
445 ****** buffer-offer-save
446 We also want to set this to t.
448 ****** Where does this get us?
450 - C-x s still does *not* offer to save the edit buffer. That's
451 because buffer-file-name is nil.
453 - C-x C-c does ask us whether we want to save the
454 edit buffer. However, since buffer-file-name is nil it asks us
455 for a file name. The check in org-edit-src-exit throws an error
456 unless the buffer is named '* Org Edit '...
458 - C-x k kills the buffer silently, leaving a broken overlay
459 link. If buffer-file-name were set, it would have warned that
460 the buffer was modified.
462 ****** buffer-file-name
463 So, that all suggests that we need to set buffer-file-name, even
464 though we don't really want to associate this buffer with a file
465 in the normal way. As for the file name, my current suggestion
466 is parent-org-filename[edit-buffer-name].
468 [I had to move the (org-src-mode) call to the end of
469 org-edit-src-code to make sure that the required variables were
470 defined when the hook was called.]
472 ****** And so where are we now?
473 - C-x s *does* offer to save the edit buffer, but in saving
474 produces a warning that the edit buffer is modified.
475 - C-x k now gives a warning that the edit buffer is modified
477 - C-x C-c is working as desired, except that again we get
478 warnings that the edit buffer is modified, once when we save,
479 and again just before exiting emacs.
480 - And C-c ' now issues a warning that the edit buffer is
481 modified when we leave it, which we don't want.
482 ****** So, we need to get rid of the buffer modification warnings.
483 I've made buffer-file-name nil inside the let binding in
486 - C-x s behaves as desired, except that as was already the case,
487 the edit buffer is always considered modified, and so repeated
488 invocations keep saving it.
489 - As was already the case, C-x k always gives a warning that the
490 edit buffer has been modified.
491 - C-x C-c is as desired (offers to save the edit buffer) except
492 that it warns of the modified buffer just before exiting.
493 - C-c ' is as it should be (silent)
495 We've got the desired behaviour, at the cost of being forced to
496 assign a buffer-file-name to the edit buffer. The consequence is
497 that the edit buffer is considered to always be modified, since
498 a file of that name is never actually written to (doesn't even
499 exist). I couldn't see a way to trick emacs into believing that
500 the buffer was unmodified since last save. But in any case, I
501 think there's an argument that these modifications warnings are
502 a good thing, because one should not leave active edit buffers
503 around: you should always have exited with C-c ' first.
505 **** TODO Doesn't currently work with ess-load-file
506 ess-load-file contains these two lines
507 #+begin_src emacs-lisp
508 (let ((source-buffer (get-file-buffer filename)))
509 (if (ess-check-source filename)
510 (error "Buffer %s has not been saved" (buffer-name source-buffer)))
513 which have the effect of, in the course of saving, deleting the buffer
514 `source-buffer', and then attempting to use it subsequently. The only
515 solution I have thought of so far is submitting a patch to ess which
516 would, e.g. reverse the order of those two lines (perform the error
517 check outside the let binding).
519 In fact, even after doing that there are further problems generated by
520 the fact that the edit buffer has an associated filename for which the
521 file doesn't exist. I think this worked OK in the past when the edit
522 buffer had no associated filename. So this is a problem which needs
523 addressing. Maybe defadvice could be used on ess functions where
524 necessary to make org/org-babel play nicely with ess?
526 *** DONE name edit buffer according to #+srcname (and language?)
527 See above patch agains org.
528 *** DONE optionally evaluate header references when we switch to =*Org Edit Src*= buffer
529 That seems to imply that the header references need to be evaluated
530 and transformed into the target language object when we hit C-c ' to
531 enter the *Org Edit Src* buffer [DED]
533 Good point, I heartily agree that this should be supported [Eric]
535 (or at least before the first time we attempt to evaluate code in that
536 buffer -- I suppose there might be an argument for lazy evaluation, in
537 case someone hits C-c ' but is "just looking" and not actually
538 evaluating anything.) Of course if evaluating the reference is
539 computationally intensive then the user might have to wait before they
540 get the *Org Edit Src* buffer. [DED]
542 I fear that it may be hard to anticipate when the references will be
543 needed, some major-modes do on-the-fly evaluation while the buffer is
544 being edited. I think that we should either do this before the buffer
545 is opened or not at all, specifically I think we should resolve
546 references if the user calls C-c ' with a prefix argument. Does that
547 sound reasonable? [Eric]
551 [Dan] So now that we have org-src-mode and org-src-mode-hook, I guess
552 org-babel should do this by using the hook to make sure that, when C-c
553 C-' is issued on a source block, any references are resolved and
554 assignments are made in the appropriate session.
556 #+tblname: my-little-table
560 #+srcname: resolve-vars-on-edit
561 #+begin_src ruby :var table=my-little-table :results silent :session test
562 table.size.times.do |n|
567 *** TODO set buffer-local-process variables appropriately [DED]
568 I think something like this would be great. You've probably
569 already thought of this, but just to note it down: it would be really
570 nice if org-babel's notion of a buffer's 'session/process' played
571 nicely with ESS's notion of the buffer's session/process. ESS keeps
572 the current process name for a buffer in a buffer-local variable
573 ess-local-process-name. So one thing we will probably want to do is
574 make sure that the *Org Edit Src Example* buffer sets that variable
577 I had not thought of that, but I agree whole heartedly. [Eric]
579 Once this is done every variable should be able to dump regions into
580 their inferior-process buffer using major-mode functions.
581 *** REJECTED send code to inferior process
582 Another thought on this topic: I think we will want users to send
583 chunks of code to the interpreter from within the *Org Edit Src*
584 buffer, and I think that's what you have in mind already. In ESS that
585 is done using the ess-eval-* functions. [DED]
587 I think we can leave this up to the major-mode in the source code
588 buffer, as almost every source-code major mode will have functions for
589 doing things like sending regions to the inferior process. If
590 anything we might need to set the value of the buffer local inferior
591 process variable. [Eric]
593 *** DONE some possible requests/proposed changes for Carsten [4/4]
594 While I remember, some possible requests/proposed changes for Carsten
595 come to mind in that regard:
597 **** DONE Remap C-x C-s to save the source to the org buffer?
598 I've done this personally and I find it essential. I'm using
599 #+begin_src emacs-lisp
600 (defun org-edit-src-save ()
601 "Update the parent org buffer with the edited source code, save
602 the parent org-buffer, and return to the source code edit
611 (define-key org-exit-edit-mode-map "\C-x\C-s" 'org-edit-src-save)
615 I think this is great, but I think it should be implemented in the
618 **** DEFERRED Rename buffer and minor mode?
619 Something shorter than *Org Edit Src Example* for the buffer
620 name. org-babel is bringing org's source code interaction to a
621 level of maturity where the 'example' is no longer
622 appropriate. And if further keybindings are going to be added to
623 the minor mode then maybe org-edit-src-mode is a better name than
626 Maybe we should name the buffer with a combination of the source
627 code and the session. I think that makes sense.
629 [ES] Are you also suggesting a new org-edit-src minor mode?
630 [DED] org-exit-edit-mode is a minor mode that already exists:
632 Minor mode installing a single key binding, "C-c '" to exit special edit.
634 org-edit-src-save now has a binding in that mode, so I guess all
635 I'm saying at this stage is that it's a bit of a misnomer. But
636 perhaps we will also have more functionality to add to that minor
637 mode, making it even more of a misnomer. Perhaps something like
638 org-src-mode would be better.
639 **** DONE Changed minor mode name and added hooks
641 **** DONE a hook called when the src edit buffer is created
642 This should be implemented in the org-mode core
643 ** TODO resolve references to other org buffers/files
644 This would allow source blocks to call upon tables, source-blocks,
645 and results in other org buffers/files.
648 - [[file:lisp/org-babel-ref.el::TODO%20allow%20searching%20for%20names%20in%20other%20buffers][org-babel-ref.el:searching-in-other-buffers]]
649 - [[file:lisp/org-babel.el::defun%20org-babel%20find%20named%20result%20name][org-babel.el#org-babel-find-named-result]]
650 ** TODO resolve references to other non-org files
651 - tabular data in .csv, .tsv etc format
652 - files of interpreted code: anything stopping us giving such files
653 similar status to a source code block?
654 - Would be nice to allow org and non-org files to be remote
655 ** TODO Finalise behaviour regarding vector/scalar output
656 *** DONE Stop spaces causing vector output
657 This simple example of multilingual chaining produces vector output if
658 there are spaces in the message and scalar otherwise.
662 #+srcname: msg-from-R(msg=msg-from-python)
664 paste(msg, "und R", sep=" ")
668 : org-babel speaks elisp y python und R
670 #+srcname: msg-from-python(msg=msg-from-elisp)
675 #+srcname: msg-from-elisp(msg="org-babel speaks")
676 #+begin_src emacs-lisp
677 (concat msg " elisp")
679 ** TODO command line execution
680 Allow source code blocks to be called form the command line. This
681 will be easy using the =sbe= function in [[file:lisp/org-babel-table.el][org-babel-table.el]].
683 This will rely upon [[* resolve references to other buffers][resolve references to other buffers]].
685 ** TODO inline source code blocks [3/5]
686 Like the =\R{ code }= blocks
688 not sure what the format should be, maybe just something simple
689 like =src_lang[]{}= where lang is the name of the source code
690 language to be evaluated, =[]= is optional and contains any header
691 arguments and ={}= contains the code.
693 (see [[* (sandbox) inline source blocks][the-sandbox]])
695 *** DONE evaluation with \C-c\C-c
696 Putting aside the header argument issue for now we can just run these
697 with the following default header arguments
698 - =:results= :: silent
699 - =:exports= :: results
701 *** DONE inline exportation
702 Need to add an interblock hook (or some such) through org-exp-blocks
703 *** DONE header arguments
704 We should make it possible to use header arguments.
706 *** TODO fontification
707 we should color these blocks differently
709 *** TODO refine html exportation
710 should use a span class, and should show original source in tool-tip
711 ** TODO LoB: re-implement plotting and analysis functions from org-R
712 I'll do this soon, now that we things are a bit more settled and we
713 have column names in R.
714 ** PROPOSED allow `anonymous' function block with function call args?
715 My question here is simply whether we're going to allow
716 #+begin_src python(arg=ref)
720 but with preference given to
721 #+srcname blockname(arg=ref)
722 ** PROPOSED allow :result as synonym for :results?
723 ** PROPOSED allow 'output mode to return stdout as value?
724 Maybe we should allow this. In fact, if block x is called
725 with :results output, and it references blocks y and z, then
726 shouldn't the output of x contain a concatenation of the outputs of
727 y and z, together with x's own output? That would raise the
728 question of what happens if y is defined with :results output and z
729 with :results value. I guess z's (possibly vector/tabular) output
730 would be inside a literal example block containing the whole lot.
731 ** PROPOSED make tangle files read-only?
732 With a file-local variable setting, yea that makes sense. Maybe
733 the header should reference the related org-mode file.
734 ** PROPOSED Creating presentations
735 The [[mairix:t:@@9854.1246500519@gamaville.dokosmarshall.org][recent thread]] containing posts by Nick Dokos and Sebastian
736 Vaubán on exporting to beamer looked very interesting, but I
737 haven't had time to try it out yet. I would really like it if,
738 eventually, we can generate a presentation (with graphics generated
739 by code blocks) from the same org file that contains all the notes
740 and code etc. I just wanted that to be on record in this document;
741 I don't have anything more profound to say about it at the moment,
742 and I'm not sure to what extent it is an org-babel issue.
743 ** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format
744 I haven't thought about this properly. Just noting it down. What
745 Sweave uses is called "R noweb" (.Rnw).
747 I found a good description of noweb in the following article (see
748 the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]).
750 I think there are two parts to noweb, the construction of
751 documentation and the extraction of source-code (with notangle).
753 *documentation*: org-mode handles all of our documentation needs in
754 a manner that I believe is superior to noweb.
756 *source extraction* At this point I don't see anyone writing large
757 applications with 100% of the source code contained in org-babel
758 files, rather I see org-babel files containing things like
759 - notes with active code chunks
760 - interactive tutorials
761 - requirements documents with code running test suites
762 - and of course experimental reports with the code to run the
763 experiment, and perform analysis
765 Basically I think the scope of the programs written in org-babel
766 (at least initially) will be small enough that it wont require the
767 addition of a tangle type program to extract all of the source code
768 into a running application.
770 On the other hand, since we already have named blocks of source
771 code which reference other blocks on which they rely, this
772 shouldn't be too hard to implement either on our own, or possibly
773 relying on something like noweb/notangle.
775 ** PROPOSED support for passing paths to files between source blocks
776 Maybe this should be it's own result type (in addition to scalars and
777 vectors). The reason being that some source-code blocks (for example
778 ditaa or anything that results in the creation of a file) may want to
779 pass a file path back to org-mode which could then be inserted into
780 the org-mode buffer as a link to the file...
782 This would allow for display of images upon export providing
783 functionality similar to =org-exp-blocks= only in a more general
785 ** DEFERRED optional timestamp for output
786 *DEFERRED*: I'm deferring this in deference to the better caching
787 system proposed by Carsten. (see [[sha1-caching]])
789 Add option to place an (inactive) timestamp at the #+resname, to
790 record when that output was generated.
792 *** source code block timestamps (optional addition)
793 [Eric] If we did this would we then want to place a timestamp on the
794 source-code block, so that we would know if the results are
795 current or out of date? This would have the effect of caching the
796 results of calculations and then only re-running if the
797 source-code has changed. For the caching to work we would need to
798 check not only the timestamp on a source-code block, but also the
799 timestamps of any tables or source-code blocks referenced by the
800 original source-code block.
802 [Dan] I do remember getting frustrated by Sweave always having to
803 re-do everything, so this could be desirable, as long as it's easy
804 to over-ride of course. I'm not sure it should be the default
805 behaviour unless we are very confident that it works well.
807 **** maintaining source-code block timestamps
808 It may make sense to add a hook to `org-edit-special' which could
809 update the source-code blocks timestamp. If the user edits the
810 contents of a source-code block directly I can think of no
811 efficient way of maintaining the timestamp.
812 ** DEFERRED figure out how to handle errors during evaluation
813 I expect it will be hard to do this properly, but ultimately it
814 would be nice to be able to specify somewhere to receive STDERR,
815 and to be warned if it is non-empty.
817 Probably simpler in non-session evaluation than session? At least
818 the mechanism will be different I guess.
820 R has a try function, with error handling, along the lines of
821 python. I bet ruby does too. Maybe more of an issue for functional
822 style; in my proposed scripting style the error just gets dumped to
823 the org buffer and the user is thus alerted.
825 For now I think the current behavior of returning any error
826 messages generated by the source language is sufficient.
827 ** DEFERRED source-name visible in LaTeX and html exports
828 Maybe this should be done in backend specific manners.
830 The listings package may provide for naming a source-code block...
832 Actually there is no obvious simple and attractive way to implement
833 this. Closing this issue for now.
834 ** DEFERRED Support rownames and other org babel table features?
836 The full org table features are detailed in the manual [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]].
839 Perhaps add a :rownames header arg. This would be an integer
840 (usually 1) which would have the effect of post-processing all the
841 variables created in the R session in the following way: if the
842 integer is j, set the row names to the contents of column j and
843 delete column j. Perhaps it is artificial to allow this integer to
844 take any value other than 1. The default would be nil which would
845 mean no such behaviour.
847 Actually I don't know about that. If multiple variables are passed
848 in, it's not appropriate to alter them all in the same way. The
849 rownames specification would normally refer to just one of the
850 variables. For now maybe just say this has to be done in R. E.g.
852 #+TBLNAME: sample-sizes
853 | collection | size | exclude | include | exclude2 | include2 |
854 |-----------------+------+---------+---------+----------+----------|
855 | 58C | 2936 | 8 | 2928 | 256 | 2680 |
856 | MS | 5852 | 771 | 5081 | 771 | 5081 |
857 | NBS | 2929 | 64 | 2865 | 402 | 2527 |
858 | POBI | 2717 | 1 | 2716 | 1 | 2716 |
859 | 58C+MS+NBS+POBI | | | 13590 | | 13004 |
860 #+TBLFM: @2$4=@2$2 - @2$3::@2$6=@2$2 - @2$5::@3$4=@3$2-@3$3::@3$6=@3$2 - @3$5::@4$4=@4$2 - @4$3::@4$6=@4$2 - @4$5::@5$4=@5$2-@5$3::@5$6=@5$2 - @5$5::@6$4=vsum(@2$4..@5$4)::@6$6=vsum(@2$6..@5$6)
862 #+srcname: make-size-table(size=sample-sizes)
864 rownames(size) <- size[,1]
870 [I don't think it's as problematic as this makes out]
871 This is non-trivial, but may be worth doing, in particular to
872 develop a nice framework for sending data to/from R.
874 In R, indexing vector elements, and rows and columns, using
875 strings rather than integers is an important part of the
877 - elements of a vector may have names
878 - matrices and data.frames may have "column names" and "row names"
879 which can be used for indexing
880 - In a data frame, row names *must* be unique
888 > mat <- matrix(1:4, nrow=2, ncol=2, dimnames=list(c("r1","r2"), c("c1","c2")))
893 > # The names are separate from the data: they do not interfere with operations on the data
900 > df <- data.frame(var1=1:26, var2=26:1, row.names=letters)
902 [1] 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
908 So it's tempting to try to provide support for this in org-babel. For example
909 - allow R to refer to columns of a :var reference by their names
910 - When appropriate, results from R appear in the org buffer with "named
913 However none (?) of the other languages we are currently supporting
914 really have a native matrix type, let alone "column names" or "row
915 names". Names are used in e.g. python and perl to refer to entries
918 It currently seems to me that support for this in org-babel would
919 require setting rules about when org tables are considered to have
920 named columns/fields, and ensuring that (a) languages with a notion
921 of named columns/fields use them appropriately and (b) languages
922 with no such notion do not treat then as data.
924 - Org allows something that *looks* like column names to be separated
926 - Org also allows a row to *function* as column names when special
927 markers are placed in the first column. An hline is unnecessary
928 (indeed hlines are purely cosmetic in org [correct?]
929 - Org does not have a notion of "row names" [correct?]
931 The full org table functionality exeplified [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]] has features that
932 we would not support in e.g. R (like names for the row below).
934 **** Initial statement: allow tables with hline to be passed as args into R
935 This doesn't seem to work at the moment (example below). It would
936 also be nice to have a natural way for the column names of the org
937 table to become the column names of the R data frame, and to have
938 the option to specify that the first column is to be used as row
939 names in R (these must be unique). But this might require a bit of
944 | col1 | col2 | col3 |
945 |------+---------+------|
953 #+begin_src R :var tabel=egtable :colnames t
958 | "col1" | "col2" | "col3" |
959 |--------+-----------+--------|
961 | 4 | "schulte" | 6 |
964 Another example is in the [[*operations%20in%20on%20tables][grades example]].
965 ** DEFERRED use textConnection to pass tsv to R?
966 When passing args from the org buffer to R, the following route is
967 used: arg in buffer -> elisp -> tsv on file -> data frame in R. I
968 think it would be possible to avoid having to write to file by
969 constructing an R expression in org-babel-R-assign-elisp, something
972 #+begin_src emacs-lisp
973 (org-babel-R-input-command
974 (format "%s <- read.table(textConnection(\"%s\"), sep=\"\\t\", as.is=TRUE)"
975 name (orgtbl-to-tsv value '(:sep "\t" :fmt org-babel-R-quote-tsv-field))))
978 I haven't tried to implement this yet as it's basically just
979 fiddling with something that works. The only reason for it I can
980 think of would be efficiency and I haven't tested that.
982 This Didn't work after an initial test. I still think this is a
983 good idea (I also think we should try to do something similar when
984 writing out results frmo R to elisp) however as it wouldn't result
985 in any functional changes I'm bumping it down to deferred for
990 #+tblname: quick-test
993 #+srcname: quick-test-src-blk
994 #+begin_src R :var vec=quick-test
1004 ** DEFERRED Rework Interaction with Running Processes [2/5]
1005 *** DONE robust to errors interrupting execution
1007 #+srcname: long-runner-ruby
1008 #+begin_src ruby :results silent
1010 :patton_is_an_grumpy
1013 *** DEFERRED use =C-g= keyboard-quit to push processing into the background
1014 This may be possible using the `run-with-timer' command.
1016 I have no idea how this could work...
1018 #+srcname: long-runner-ruby
1019 #+begin_src ruby :results silent
1021 :patton_is_an_grumpy
1024 *** TODO ability to select which of multiple sessions is being used
1025 Increasingly it is looking like we're going to want to run all
1026 source code blocks in comint buffer (sessions). Which will have
1028 1) allowing background execution
1029 2) maintaining state between source-blocks
1030 - allowing inline blocks w/o header arguments
1033 (like ess-switch-process in .R buffers)
1035 Maybe this could be packaged into a header argument, something
1036 like =:R_session= which could accept either the name of the
1037 session to use, or the string =prompt=, in which case we could use
1038 the =ess-switch-process= command to select a new process.
1040 *** TODO evaluation of shell code as background process?
1041 After C-c C-c on an R code block, the process may appear to
1042 block, but C-g can be used to reclaim control of the .org buffer,
1043 without interrupting the R evalution. However I believe this is not
1044 true of bash/sh evaluation. [Haven't tried other languages] Perhaps
1045 a solution is just to background the individual shell commands.
1047 The other languages (aside from emacs lisp) are run through the
1048 shell, so if we find a shell solution it should work for them as
1051 Adding an ampersand seems to be a supported way to run commands in
1052 the background (see [[http://www.emacswiki.org/emacs/ExecuteExternalCommand#toc4][external-commands]]). Although a more extensible
1053 solution may involve the use of the [[elisp:(progn (describe-function 'call-process-region) nil)][call-process-region]] function.
1055 Going to try this out in a new file [[file:lisp/org-babel-proc.el][org-babel-proc.el]]. This should
1056 contain functions for asynchronously running generic shell commands
1057 in the background, and then returning their input.
1059 **** partial update of org-mode buffer
1060 The sleekest solution to this may be using a comint buffer, and
1061 then defining a filter function which would incrementally interpret
1062 the results as they are returned, including insertion into the
1063 org-mode buffer. This may actually cause more problems than it is
1064 worth, what with the complexities of identifying the types of
1065 incrementally returned results, and the need for maintenance of a
1066 process marker in the org buffer.
1068 **** 'working' spinner
1069 It may be nice and not too difficult to place a spinner on/near the
1070 evaluating source code block
1072 *** TODO conversion of output from interactive shell, R (and python) sessions to org-babel buffers
1073 [DED] This would be a nice feature I think. Although an org-babel
1074 purist would say that it's working the wrong way round... After
1075 some interactive work in a *R* buffer, you save the buffer, maybe
1076 edit out some lines, and then convert it to org-babel format for
1077 posterity. Same for a shell session either in a *shell* buffer, or
1078 pasted from another terminal emulator. And python of course.
1079 ** DEFERRED improve the source-block snippet
1080 any real improvement seems somewhat beyond the ability of yasnippet
1083 [[file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name%20Chapter%20title][file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name Chapter title]]
1085 ,#name : Chapter title
1088 ${1:$(make-string (string-width text) ?\=)}
1093 [[file:snippets/org-mode/sb][sb -- snippet]]
1095 waiting for guidance from those more familiar with yasnippets
1097 ** REJECTED re-implement R evaluation using ess-command or ess-execute
1098 I don't have any complaints with the current R evaluation code or
1099 behaviour, but I think it would be good to use the ESS functions
1100 from a political point of view. Plus of course it has the normal
1101 benefits of an API (insulates us from any underlying changes etc). [DED]
1103 I'll look into this. I believe that I looked at and rejected these
1104 functions initially but now I can't remember why. I agree with
1105 your overall point about using API's where available. I will take
1106 a look back at these and either switch to using the ess commands,
1107 or at least articulate under this TODO the reasons for using our
1108 custom R-interaction commands. [Eric]
1112 Lets just replace =org-babel-R-input-command= with =ess-execute=.
1114 I tried this, and although it works in some situations, I find that
1115 =ess-command= will often just hang indefinitely without returning
1116 results. Also =ess-execute= will occasionally hang, and pops up
1117 the buffer containing the results of the command's execution, which
1118 is undesirable. For now these functions can not be used. Maybe
1119 someone more familiar with the ESS code can recommend proper usage
1120 of =ess-command= or some other lower-level function which could be
1121 used in place of [[file:lisp/org-babel-R.el::defun%20org-babel%20R%20input%20command%20command][org-babel-R-input-command]].
1125 #+begin_quote ess-command
1126 (ess-command COM &optional BUF SLEEP NO-PROMPT-CHECK)
1128 Send the ESS process command COM and delete the output
1129 from the ESS process buffer. If an optional second argument BUF exists
1130 save the output in that buffer. BUF is erased before use.
1131 COM should have a terminating newline.
1132 Guarantees that the value of .Last.value will be preserved.
1133 When optional third arg SLEEP is non-nil, `(sleep-for (* a SLEEP))'
1134 will be used in a few places where `a' is proportional to `ess-cmd-delay'.
1137 #+begin_quote ess-execute
1138 (ess-execute COMMAND &optional INVERT BUFF MESSAGE)
1140 Send a command to the ESS process.
1141 A newline is automatically added to COMMAND. Prefix arg (or second arg
1142 INVERT) means invert the meaning of
1143 `ess-execute-in-process-buffer'. If INVERT is 'buffer, output is
1144 forced to go to the process buffer. If the output is going to a
1145 buffer, name it *BUFF*. This buffer is erased before use. Optional
1146 fourth arg MESSAGE is text to print at the top of the buffer (defaults
1147 to the command if BUFF is not given.)
1150 *** out current setup
1152 1) The body of the R source code block is wrapped in a function
1153 2) The function is called inside of a =write.table= function call
1154 writing the results to a table
1155 3) The table is read using =org-table-import=
1156 ** DONE figure out how to handle graphic output
1158 This is listed under [[* graphical output][graphical output]] in out objectives.
1160 This should take advantage of the =:results file= option, and
1161 languages which almost always produce graphical output should set
1162 =:results file= to true by default (this is currently done for the
1163 gnuplot and ditaa languages). That would handle placing these results
1164 in the buffer. Then if there is a combination of =silent= and =file=
1165 =:results= headers we could drop the results to a temp buffer and pop
1168 Display of file results is addressed in the [[* =\C-c \C-o= to open results of source block][open-results-task]].
1170 I think this is done for now. With the ability of the file option it
1171 is now possible to save images directly to a file. Then calling
1172 =\C-c\C-o= with point on the source block will open the related
1175 *** R graphics to screen means session evaluation
1176 If R graphical output is going to screen then evaluation must be
1177 in a session, otherwise the graphics will disappear as soon as the
1180 *** Adding to a discussion started in email
1181 I'm not deeply wedded to these ideas, just noting them down. I'm
1182 probably just thinking of R and haven't really thought about how
1183 this fits with the other graphics-generating languages.
1185 > I used the approach below to get graphical file output
1186 > today, which is one idea at least. Maybe it could be linked up with
1187 > your :results file variable. (Or do we need a :results image for R?)
1190 I don't think we need a special image results variable, but I may be
1191 missing what the code below accomplishes. Would the task I added about
1192 adding org-open-at-point functionality to source code blocks take care
1195 Dan: I'm not sure. I think the ability for a script to generate both
1196 text and graphical output might be a natural expectation, at least for
1202 > #+srcname: cohort-scatter-plots-2d(org_babel_graphical_output_file="cohort-scatter-plots-2d.png")
1204 > if(exists("org_babel_output_file"))
1205 > png(filename=org_babel_graphical_output_file, width=1000, height=1000)
1206 > ## plotting code in here
1207 > if(exists("org_babel_graphical_output_file")) dev.off()
1210 Dan: Yes, the results :file option is nice for dealing with graphical
1211 output, and that could well be enough. Something based on the scheme
1212 above would have a couple of points in its favour:
1213 1. It's easy to switch between output going to on-screen graphics and
1214 output going to file: Output will go to screen unless a string variable
1215 with a standard name (e.g. ""org_babel_graphical_output_file"")
1216 exists in which case it will go to the file indicated by the value
1218 2. The block can return a result / script output, as well as produce
1221 In interactive use we might want to allow the user to choose between
1222 screen and file output. In non-interactive use such as export, it
1223 would be file output (subject to the :exports directives).
1224 ** DONE new results types (org, html, latex)
1225 Thanks to Tom Short for this recommendation.
1227 - raw or org :: in which case the results are implemented raw, unquoted
1228 into the org-mode file. This would also handle links as
1229 source block output.
1230 - html :: the results are inserted inside of a #+BEGIN_HTML block
1231 - latex :: the results are inserted inside of a #+BEGIN_LATEX block
1234 : #+begin_src R :session *R* :results org
1235 : cat("***** This is a table\n")
1236 : cat("| 1 | 2 | 3 |\n")
1237 : cat("[[http://google.com][Google it here]]\n"
1241 : ***** This is a table
1243 [[http://google.com][: Google it here]]
1245 We actually might want to remove the =#+resname= line if the results
1246 type is org-mode, not sure... Either way I don't think there is a
1247 good way to capture/remove org type results.
1250 #+srcname: latex-results
1251 #+begin_src emacs-lisp :results latex
1252 "this should be inside of a LaTeX block"
1257 this should be inside of a LaTeX block
1261 #+srcname: html-results
1262 #+begin_src emacs-lisp :results html
1263 "this should be inside of a HTML block
1276 this should be inside of a HTML block
1289 Added a =raw= results header argument, which will insert the results
1290 of a source-code block into an org buffer un-escaped. Also, if the
1291 results look like a table, then the table will be aligned.
1293 #+srcname: raw-table-demonstration
1294 #+begin_src ruby :results output raw
1295 puts "| root | square |"
1298 puts "| #{n} | #{n*n} |"
1316 Not sure how/if this would work, but it may be desirable.
1317 ** DONE org-bable-tangle: no default extension if one already exists
1318 ** DONE take default values for header args from properties
1319 Use file-wide and subtree wide properties to set default values for
1322 [DED] One thing I'm finding when working with R is that an org file
1323 may contain many source blocks, but that I just want to evaluate a
1324 subset of them. Typically this is in order to take up where I left
1325 off: I need to recreate a bunch of variables in the session
1326 environment. I'm thinking maybe we want to use a tag-based
1327 mechanism similar to :export: and :noexport: to control evaluation
1328 on a per-subtree basis.
1330 *** test-header with properties
1336 Ahh... as is so often the case, just had to wrap
1337 `org-babel-params-from-properties' in a `save-match-data' form.
1339 #+tblname: why-def-props-cause-probs
1342 #+srcname: default-props-implementation
1343 #+begin_src emacs-lisp :tangle no :var my-lis=why-def-props-cause-probs :results silent
1344 (+ (length my-lis) def)
1347 ** DONE new reference syntax *inside* source code blocks
1348 This is from an email discussion on the org-mode mailing list with
1349 Sébastien. The goal here is to mimic the source-block reference style
1350 of Noweb. Upon export and/or tangle these references could be
1351 replaced with the actual body of the referenced source-code block.
1353 See the following for an example.
1355 #+srcname: ems-ruby-print-header
1357 puts "---------------------------header---------------------------"
1360 #+srcname: emacs-ruby-print-footer
1362 puts "---------------------------footer---------------------------"
1365 #+srcname: ems-ruby-print-message
1366 #+begin_src ruby :file ruby-noweb.rb
1367 # <<ems-ruby-print-header>>
1369 # <<ems-ruby-print-footer>>
1372 Upon export the previous source-code block would result in a file
1373 being generated at =ruby-noweb.rb= with the following contents
1375 : puts "---------------------------header---------------------------"
1377 : puts "---------------------------footer---------------------------"
1379 the body of a source-code block with all =<<src-name>>= references
1380 expanded can now be returned by `org-babel-expand-noweb-references'.
1381 This function is now called by default on all source-code blocks on
1384 ** DONE re-work tangling system
1385 Sometimes when tangling a file (e.g. when extracting elisp from a
1386 org-mode file) we want to get nearly every source-code block.
1388 Sometimes we want to only extract those source-code blocks which
1389 reference a indicate that they should be extracted (e.g. traditional
1390 literate programming along the Noweb model)
1392 I'm not sure how we can devise a single simple tangling system that
1393 naturally fits both of these use cases.
1396 the =tangle= header argument will default to =no= meaning source-code
1397 blocks will *not* be exported by default. In order for a source-code
1398 block to be tangled it needs to have an output file specified. This
1399 can happen in two ways...
1401 1) a file-wide default output file can be passed to `org-babel-tangle'
1402 which will then be used for all blocks
1403 2) if the value of the =tangle= header argument is anything other than
1404 =no= or =yes= then it is used as the file name
1406 #+srcname: test-new-tangling
1407 #+begin_src emacs-lisp
1408 (org-babel-load-file "test-tangle.org")
1409 (if (string= test-tangle-advert "use org-babel-tangle for all your emacs initialization files!!")
1417 ** DONE =\C-c \C-o= to open results of source block
1418 by adding a =defadvice= to =org-open-at-point= we can use the common
1419 =\C-c \C-o= keybinding to open the results of a source-code block.
1420 This would be especially useful for source-code blocks which generate
1421 graphical results and insert a file link as the results in the
1422 org-mode buffer. (see [[* figure out how to handle graphic output][TODO figure out how to handle graphic output]]).
1423 This could also act reasonably with other results types...
1425 - file :: use org-open-at-point to open the file
1426 - scalar :: open results unquoted in a new buffer
1427 - tabular :: export the table to a new buffer and open that buffer
1429 when called with a prefix argument the block is re-run
1431 #+srcname: task-opening-results-of-blocks
1432 #+begin_src ditaa :results replace :file blue.png :cmdline -r
1443 [[file:blue.png][blue.png]]
1445 #+srcname: task-open-vector
1446 #+begin_src emacs-lisp
1454 #+srcname: task-open-scalar
1455 #+begin_src ruby :results output
1471 ** DONE Stop spaces causing vector output
1472 This simple example of multilingual chaining produces vector output if
1473 there are spaces in the message and scalar otherwise.
1477 #+srcname: msg-from-R(msg=msg-from-python)
1479 paste(msg, "und R", sep=" ")
1483 : org-babel speaks elisp y python und R
1485 #+srcname: msg-from-python(msg=msg-from-elisp)
1490 #+srcname: msg-from-elisp(msg="org-babel speaks")
1491 #+begin_src emacs-lisp
1492 (concat msg " elisp")
1495 ** DONE add =:tangle= family of header arguments
1497 - no :: don't include source-code block when tangling
1498 - yes :: do include source-code block when tangling
1500 this is tested in [[file:test-tangle.org::*Emacs%20Lisp%20initialization%20stuff][test-tangle.org]]
1502 ** DONE extensible library of callable source blocks
1504 This is covered by the [[file:library-of-babel.org][Library of Babel]], which will contain
1505 ready-made source blocks designed to carry out useful common tasks.
1506 *** Initial statement [Eric]
1507 Much of the power of org-R seems to be in it's helper functions for
1508 the quick graphing of tables. Should we try to re-implement these
1509 functions on top of org-babel?
1511 I'm thinking this may be useful both to add features to org-babel-R and
1512 also to potentially suggest extensions of the framework. For example
1513 one that comes to mind is the ability to treat a source-code block
1514 like a function which accepts arguments and returns results. Actually
1515 this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]).
1516 *** Objectives [Dan]
1517 - We want to provide convenient off-the-shelf actions
1518 (e.g. plotting data) that make use of our new code evaluation
1519 environment but do not require any actual coding.
1520 *** Initial Design proposal [Dan]
1521 - *Input data* will be specified using the same mechanism as :var
1522 references, thus the input data may come from a table, or
1523 another source block, and it is initially available as an elisp
1525 - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that
1526 line will apply an *action* to the referenced data.
1527 - *Actions correspond to source blocks*: our library of available
1528 actions will be a library of org-babel source blocks. Thus the
1529 code for executing an action, and the code for dealing with the
1530 output of the action will be the same code as for executing
1531 source blocks in general
1532 - Optionally, the user can have the relevant source block inserted
1533 into the org buffer after the (say) #+BABELDO line. This will
1534 allow the user to fine tune the action by modifying the code
1535 (especially useful for plots).
1536 - So maybe a #+BABELDO line will have header args
1537 - :data (a reference to a table or source code block)
1538 - :action (or should that be :srcname?) which will be something
1539 like :action pie-chart, referring to a source block which will
1540 be executed with the :data referent passed in using a :var arg.
1541 - :showcode or something controlling whether to show the code
1543 *** Modification to design
1544 I'm implementing this, at least initially, as a new interpreter
1545 named 'babel', which has an empty body. 'babel' blocks take
1546 a :srcname header arg, and look for the source-code block with
1547 that name. They then execute the referenced block, after first
1548 appending their own header args on to the target block's header
1551 If the target block is in the library of babel (a.o.t. e.g. the
1552 current buffer), then the code in the block will refer to the
1553 input data with a name dictated by convention (e.g. __data__
1554 (something which is syntactically legal in all languages...). Thus
1555 the babel block will use a :var __data__ = whatever header arg to
1556 reference the data to be plotted.
1558 ** DONE Column names in R input/output
1559 This has been implemented: Automatic on input to R; optional in
1560 output. Note that this equates column names with the header row in
1561 an org table; whereas org actually has a mechanism whereby a row
1562 with a '!' in the first field defines column names. I have not
1563 attempted to support these org table mechanisms yet. See [[*Support%20rownames%20and%20other%20org%20babel%20table%20features][this
1564 DEFERRED todo item]].
1565 ** DONE use example block for large amounts of stdout output?
1566 We're currently `examplizing' with : at the beginning of the line,
1567 but should larger amounts of output be in a
1568 \#+begin_example...\#+end_example block? What's the cutoff? > 1
1569 line? This would be nice as it would allow folding of lengthy
1570 output. Sometimes one will want to see stdout just to check
1571 everything looks OK, and then fold it away.
1573 I'm addressing this in branch 'examplizing-output'.
1574 Yea, that makes sense. (either that or allow folding of large
1575 blocks escaped with =:=).
1577 Proposed cutoff of 10 lines, we can save this value in a user
1578 customizable variable.
1579 *** DONE add ability to remove such results
1580 ** DONE exclusive =exports= params
1582 #+srcname: implement-export-exclusivity
1589 ** DONE LoB: allow output in buffer
1590 ** DONE allow default header arguments by language
1591 org-babel-default-header-args:lang-name
1593 An example of when this is useful is for languages which always return
1594 files as their results (e.g. [[*** ditaa][ditaa]], and [[*** gnuplot][gnuplot]]).
1595 ** DONE singe-function tangling and loading elisp from literate org-mode file [3/3]
1597 This function should tangle the org-mode file for elisp, and then call
1598 `load-file' on the resulting tangled file.
1600 #+srcname: test-loading-embedded-emacs-lisp
1601 #+begin_src emacs-lisp :results replace
1602 (setq test-tangle-advert nil)
1603 (setq test-tangle-loading nil)
1604 (setq results (list :before test-tangle-loading test-tangle-advert))
1605 (org-babel-load-file "test-tangle.org")
1606 (setq results (list (list :after test-tangle-loading test-tangle-advert) results))
1607 (delete-file "test-tangle.el")
1611 #+resname: test-loading-embedded-emacs-lisp
1612 | :before | nil | nil |
1613 | :after | "org-babel tangles" | "use org-babel-tangle for all your emacs initialization files!!" |
1615 *** DONE add optional language limiter to org-babel-tangle
1616 This should check to see if there is any need to re-export
1618 *** DONE ensure that org-babel-tangle returns the path to the tangled file(s)
1620 #+srcname: test-return-value-of-org-babel-tangle
1621 #+begin_src emacs-lisp :results replace
1622 (mapcar #'file-name-nondirectory (org-babel-tangle-file "test-tangle.org" "emacs-lisp"))
1626 | "test-tangle.el" |
1628 *** DONE only tangle the file if it's actually necessary
1629 ** DONE add a function to jump to a source-block by name
1630 I've had an initial stab at that in org-babel-find-named-block
1631 (library-of-babel branch).
1633 At the same time I introduced org-babel-named-src-block-regexp, to
1634 match src-blocks with srcname.
1636 This is now working with the command
1637 `org-babel-goto-named-source-block', all we need is a good key
1640 ** DONE add =:none= session argument (for purely functional execution) [4/4]
1641 This would allow source blocks to be run in their own new process
1643 - These blocks could then also be run in the background (since we can
1644 detach and just wait for the process to signal that it has terminated)
1645 - We wouldn't be drowning in session buffers after running the tests
1646 - we can re-use much of the session code to run in a more /functional/
1649 While session provide a lot of cool features, like persistent
1650 environments, [[* DONE function to bring up inferior-process buffer][pop-to-session]], and hints at exportation for
1651 org-babel-tangle, they also have some down sides and I'm thinking that
1652 session-based execution maybe shouldn't be the default behavior.
1654 Down-sides to sessions
1655 - *much* more complicated than functional evaluation
1656 - maintaining the state of the session has weird issues
1657 - waiting for evaluation to finish
1658 - prompt issues like [[* TODO weird escaped characters in shell prompt break shell evaluation][shell-prompt-escapes-bug]]
1659 - can't run in background
1660 - litter emacs with session buffers
1664 #+srcname: ruby-task-no-session
1665 #+begin_src ruby :results replace output
1671 #+resname: ruby-task-no-session
1676 #+srcname: task-python-none-session
1677 #+begin_src python :session none :results replace value
1683 #+resname: task-python-none-session
1688 #+srcname: task-session-none-sh
1689 #+begin_src sh :results replace
1694 #+resname: task-session-none-sh
1700 #+srcname: task-no-session-R
1701 #+begin_src R :results replace output
1708 #+resname: task-no-session-R
1712 ** DONE fully purge org-babel-R of direct comint interaction
1713 try to remove all code under the [[file:lisp/org-babel-R.el::functions%20for%20evaluation%20of%20R%20code][;; functions for evaluation of R code]] line
1715 ** DONE Create objects in top level (global) environment [5/5]
1718 *** initial requirement statement [DED]
1719 At the moment, objects created by computations performed in the
1720 code block are evaluated in the scope of the
1721 code-block-function-body and therefore disappear when the code
1722 block is evaluated {unless you employ some extra trickery like
1723 assign('name', object, env=globalenv()) }. I think it will be
1724 desirable to also allow for a style wherein objects that are
1725 created in one code block persist in the R global environment and
1726 can be re-used in a separate block.
1728 This is what Sweave does, and while I'm not saying we have to be
1729 the same as Sweave, it wouldn't be hard for us to provide the same
1730 behaviour in this case; if we don't, we risk undeservedly being
1731 written off as an oddity by some.
1733 IOW one aspect of org-babel is that of a sort of functional
1734 meta-programming language. This is crazy, in a very good
1735 way. Nevertheless, wrt R I think there's going to be a lot of value
1736 in providing for a working style in which the objects are stored in
1737 the R session, rather than elisp/org buffer. This will be a very
1738 familiar working style to lots of people.
1740 There are no doubt a number of different ways of accomplishing
1741 this, the simplest being a hack like adding
1744 for(objname in ls())
1745 assign(objname, get(objname), envir=globalenv())
1748 to the source code block function body. (Maybe wrap it in an on.exit() call).
1750 However this may deserve to be thought about more carefully, perhaps
1751 with a view to having a uniform approach across languages. E.g. shell
1752 code blocks have the same semantics at the moment (no persistence of
1753 variables across code blocks), because the body is evaluated in a new
1754 bash shell process rather than a running shell. And I guess the same
1755 is true for python. However, in both these cases, you could imagine
1756 implementing the alternative in which the body is evaluated in a
1757 persistent interactive session. It's just that it's particularly
1758 natural for R, seeing as both ESS and org-babel evaluate commands in a
1759 single persistent R session.
1763 Thanks for bringing this up. I think you are absolutely correct that we
1764 should provide support for a persistent environment (maybe called a
1765 *session*) in which to evaluate code blocks. I think the current setup
1766 demonstrates my personal bias for a functional style of programming
1767 which is certainly not ideal in all contexts.
1769 While the R function you mention does look like an elegant solution, I
1770 think we should choose an implementation that would be the same across
1771 all source code types. Specifically I think we should allow the user to
1772 specify an optional *session* as a header variable (when not present we
1773 assume a default session for each language). The session name could be
1774 used to name a comint buffer (like the *R* buffer) in which all
1775 evaluation would take place (within which variables would retain their
1776 values --at least once I remove some of the functional method wrappings
1777 currently in place-- ).
1779 This would allow multiple environments to be used in the same buffer,
1780 and once this setup was implemented we should be able to fairly easily
1781 implement commands for jumping between source code blocks and the
1782 related session buffers, as well as for dumping the last N commands from
1783 a session into a new or existing source code block.
1785 Please let me know if you foresee any problems with this proposed setup,
1786 or if you think any parts might be confusing for people coming from
1787 Sweave. I'll hopefully find some time to work on this later in the
1790 *** can functional and interpreted/interactive models coexist?
1792 Even though both of these use the same =*R*= buffer the value of =a=
1793 is not preserved because it is assigned inside of a functional
1796 #+srcname: task-R-sessions
1803 #+srcname: task-R-same-session
1808 This functional wrapper was implemented in order to efficiently return
1809 the results of the execution of the entire source code block. However
1810 it inhibits the evaluation of source code blocks in the top level,
1811 which would allow for persistence of variable assignment across
1812 evaluations. How can we allow *both* evaluation in the top level, and
1813 efficient capture of the return value of an entire source code block
1814 in a language independent manner?
1816 Possible solutions...
1817 1) we can't so we will have to implement two types of evaluation
1818 depending on which is appropriate (functional or imperative)
1819 2) we remove the functional wrapper and parse the source code block
1820 into it's top level statements (most often but not always on line
1821 breaks) so that we can isolate the final segment which is our
1823 3) we add some sort of "#+return" line to the code block
1824 4) we take advantage of each languages support for meta-programming
1825 through =eval= type functions, and use said to evaluate the entire
1826 blocks in such a way that their environment can be combined with the
1827 global environment, and their results are still captured.
1828 5) I believe that most modern languages which support interactive
1829 sessions have support for a =last_result= type function, which
1830 returns the result of the last input without re-calculation. If
1831 widely enough present this would be the ideal solution to a
1832 combination of functional and imperative styles.
1834 None of these solutions seem very desirable, but for now I don't see
1835 what else would be possible.
1837 Of these options I was leaning towards (1) and (4) but now believe
1838 that if it is possible option (5) will be ideal.
1840 **** (1) both functional and imperative evaluation
1842 - can take advantage of built in functions for sending regions to the
1844 - retains the proven tested and working functional wrappers
1847 - introduces the complication of keeping track of which type of
1848 evaluation is best suited to a particular context
1849 - the current functional wrappers may require some changes in order to
1850 include the existing global context
1852 **** (4) exploit language meta-programming constructs to explicitly evaluate code
1854 - only one type of evaluation
1857 - some languages may not have sufficient meta-programming constructs
1859 **** (5) exploit some =last_value= functionality if present
1861 Need to ensure that most languages have such a function, those without
1862 will simply have to implement their own similar solution...
1864 | language | =last_value= function |
1865 |------------+-----------------------------|
1869 | shell | see [[* last command for shells][last command for shells]] |
1870 | emacs-lisp | see [[* emacs-lisp will be a special case][special-case]] |
1872 #+srcname: task-last-value
1877 ***** last command for shells
1878 Do this using the =tee= shell command, and continually pipe the output
1881 Got this idea from the following [[http://linux.derkeiler.com/Mailing-Lists/Fedora/2004-01/0898.html][email-thread]].
1883 suggested from mailing list
1885 #+srcname: bash-save-last-output-to-file
1889 bash -c "$line" | tee /tmp/last.out1
1890 mv /tmp/last.out1 /tmp/last.out
1894 another proposed solution from the above thread
1896 #+srcname: bash-save-in-variable
1899 # so - Save Output. Saves output of command in OUT shell variable.
1909 "^M": " | tee /tmp/h_lastcmd.out ^[k"
1912 export __=/tmp/h_lastcmd.out
1914 If you try it, Alt-k will stand for the old Enter; use "command $__" to
1915 access the last output.
1921 Herculano de Lima Einloft Neto
1924 ***** emacs-lisp will be a special case
1925 While it is possible for emacs-lisp to be run in a console type
1926 environment (see the =elim= function) it is *not* possible to run
1927 emacs-lisp in a different *session*. Meaning any variable set top
1928 level of the console environment will be set *everywhere* inside
1929 emacs. For this reason I think that it doesn't make any sense to
1930 worry about session support for emacs-lisp.
1932 *** Further thoughts on 'scripting' vs. functional approaches
1934 These are just thoughts, I don't know how sure I am about this.
1935 And again, perhaps I'm not saying anything very radical, just that
1936 it would be nice to have some options supporting things like
1937 receiving text output in the org buffer.
1939 I can see that you've already gone some way down the road towards
1940 the 'last value' approach, so sorry if my comments come rather
1941 late. I am concerned that we are not giving sufficient attention
1942 to stdout / the text that is returned by the interpreters. In
1943 contrast, many of our potential users will be accustomed to a
1944 'scripting' approach, where they are outputting text at various
1945 points in the code block, not just at the end. I am leaning
1946 towards thinking that we should have 2 modes of evaluation:
1947 'script' mode, and 'functional' mode.
1949 In script mode, evaluation of a code block would result in *all*
1950 text output from that code block appearing as output in the org
1951 buffer, presumably as an #+begin_example...#+end_example. There
1952 could be an :echo option controlling whether the input commands
1953 also appear in the output. [This is like Sweave].
1955 In functional mode, the *result* of the code block is available as
1956 an elisp object, and may appear in the org buffer as an org
1957 table/string, via the mechanisms you have developed already.
1959 One thing I'm wondering about is whether, in script mode, there
1960 simply should not be a return value. Perhaps this is not so
1961 different from what exists: script mode would be new, and what
1962 exists currently would be functional mode.
1964 I think it's likely that, while code evaluation will be exciting
1965 to people, a large majority of our users in a large majority of
1966 their usage will not attempt to actually use the return value from
1967 a source code block in any meaningful way. In that case, it seems
1968 rather restrictive to only allow them to see output from the end
1971 Instead I think the most accessible way to introduce org-babel to
1972 people, at least while they are learning it, is as an immensely
1973 powerful environment in which to embed their 'scripts', which now
1974 also allows them to 'run' their 'scripts'. Especially as such
1975 people are likely to be the least capable of the user-base, a
1976 possible design-rule would be to make the scripting style of usage
1977 easy (default?), perhaps requiring a special option to enable a
1978 functional style. Those who will use the functional style won't
1979 have a problem understanding what's going on, whereas the 'skript
1980 kiddies' might not even know the syntax for defining a function in
1981 their language of choice. And of course we can allow the user to
1982 set a variable in their .emacs controlling the preference, so that
1983 functional users are not inconveniennced by having to provide
1984 header args the whole time.
1986 Please don't get the impression that I am down-valuing the
1987 functional style of org-babel. I am constantly horrified at the
1988 messy 'scripts' that my colleagues produce in perl or R or
1989 whatever! Nevertheless that seems to be how a lot of people work.
1991 I think you were leaning towards the last-value approach because
1992 it offered the possibility of unified code supporting both the
1993 single evaluation environment and the functional style. If you
1994 agree with any of the above then perhaps it will impact upon this
1995 and mean that the code in the two branches has to differ a bit. In
1996 that case, functional mode could perhaps after all evaluate each
1997 code block in its own environment, thus (re)approaching 'true'
1998 functional programming (side-effects are hard to achieve).
2002 echo "There are `wc -l files` files in this directory"
2006 *** even more thoughts on evaluation, results, models and options
2008 Thanks Dan, These comments are invaluable.
2010 What do you think about this as a new list of priorities/requirements
2011 for the execution of source-code blocks.
2014 1) we want the evaluation of the source code block to take place in a
2015 session which can persist state (variables, current directory,
2017 2) source code blocks can specify their session with a header argument
2018 3) each session should correspond to an Emacs comint buffer so that the
2019 user can drop into the session and experiment with live code
2022 1) each source-code block generates some form of results which (as
2023 we have already implemented) is transfered into emacs-lisp
2024 after which it can be inserted into the org-mode buffer, or
2025 used by other source-code blocks
2026 2) when the results are translated into emacs-lisp, forced to be
2027 interpreted as a scalar (dumping their raw values into the
2028 org-mode buffer), as a vector (which is often desirable with R
2029 code blocks), or interpreted on the fly (the default option).
2030 Note that this is very nearly currently implemented through the
2031 [[* DONE results-type header (vector/file)][results-type-header]].
2032 3) there should be *two* means of collecting results from the
2033 execution of a source code block. *Either* the value of the
2034 last statement of the source code block, or the collection of
2035 all that has been passed to STDOUT during the evaluation.
2037 **** header argument or return line (*header argument*)
2039 Rather than using a header argument to specify how the return value
2040 should be passed back, I'm leaning towards the use of a =#+RETURN=
2041 line inside the block. If such a line *is not present* then we
2042 default to using STDOUT to collect results, but if such a line *is
2043 present* then we use it's value as the results of the block. I
2044 think this will allow for the most elegant specification between
2045 functional and script execution. This also cleans up some issues
2046 of implementation and finding which statement is the last
2049 Having given this more thought, I think a header argument is
2050 preferable. The =#+return:= line adds new complicating syntax for
2051 something that does little more than we would accomplish through
2052 the addition of a header argument. The only benefit being that we
2053 know where the final statement starts, which is not an issue in
2054 those languages which contain 'last value' operators.
2056 new header =:results= arguments
2057 - script :: explicitly states that we want to use STDOUT to
2058 initialize our results
2059 - return_last :: stdout is ignored instead the *value* of the final
2060 statement in the block is returned
2061 - echo :: means echo the contents of the source-code block along
2062 with the results (this implies the *script* =:results=
2065 *** DONE rework evaluation lang-by-lang [4/4]
2067 This should include...
2068 - functional results working with the comint buffer
2070 - script :: return the output of STDOUT
2071 - write a macro which runs the first redirection, executes the
2072 body, then runs the second redirection
2073 - last :: return the value of the last statement
2076 - sessions in comint buffers
2078 **** DONE Ruby [4/4]
2079 - [X] functional results working with comint
2080 - [X] script results
2081 - [X] ensure scalar/vector results args are taken into consideration
2082 - [X] ensure callable by other source block
2084 #+srcname: ruby-use-last-output
2085 #+begin_src ruby :results replace
2092 #+resname: ruby-use-last-output
2095 #+srcname: task-call-use-last-output
2096 #+begin_src ruby :var last=ruby-use-last-output :results replace
2097 last.flatten.size + 1
2100 #+resname: task-call-use-last-output
2105 #+srcname: first-ruby-session-task
2106 #+begin_src ruby :session schulte :results silent
2110 #+srcname: second-ruby-session-task
2111 #+begin_src ruby :session schulte :results silent
2115 #+srcname: without-the-right-session
2116 #+begin_src ruby :results silent
2122 - [X] functional results working with comint
2123 - [X] script results
2124 - [X] ensure scalar/vector results args are taken into consideration
2125 - [X] ensure callable by other source block
2127 To redirect output to a file, you can use the =sink()= command.
2130 #+begin_src R :results value vector silent
2137 #+srcname: task-R-use-other-output
2138 #+begin_src R :var twoentyseven=task_R_B() :results replace value
2143 #+resname: task-R-use-other-output
2146 **** DONE Python [4/4]
2147 - [X] functional results working with comint
2148 - [X] script results
2149 - [X] ensure scalar/vector results args are taken into consideration
2150 - [X] ensure callable by other source block
2152 #+srcname: task-new-eval-for-python
2153 #+begin_src python :results silent output scalar
2159 #+srcname: task-use-new-eval
2160 #+begin_src python :var tasking=task-new-eval-for-python() :results replace
2164 #+resname: task-use-new-eval
2167 **** DONE Shells [4/4]
2168 - [X] functional results working with comint
2169 - [X] script results
2170 - [X] ensure scalar/vector results args are taken into consideration
2171 - [X] ensure callable by other source block
2173 #+srcname: task-shell-new-evaluation
2174 #+begin_src sh :results silent value scalar
2179 #+srcname: task-call-other-shell
2180 #+begin_src sh :var other=task-shell-new-evaluation() :results replace scalar
2181 echo $other ' is the old date'
2184 #+resname: task-call-other-shell
2185 : $ Fri Jun 12 13:08:37 PDT 2009 is the old date
2187 *** DONE implement a *session* header argument [4/4]
2188 =:session= header argument to override the default *session* buffer
2192 #+srcname: task-ruby-named-session
2193 #+begin_src ruby :session schulte :results replace
2194 schulte = :in_schulte
2197 #+resname: task-ruby-named-session
2200 #+srcname: another-in-schulte
2201 #+begin_src ruby :session schulte
2205 #+resname: another-in-schulte
2212 #+srcname: python-session-task
2213 #+begin_src python :session what :results silent
2217 #+srcname: python-get-from-session
2218 #+begin_src python :session what :results replace
2222 #+resname: python-get-from-session
2227 #+srcname: task-shell-sessions
2228 #+begin_src sh :session what
2232 #+srcname: task-shell-sessions-what
2233 #+begin_src sh :session what :results replace
2237 #+resname: task-shell-sessions-what
2242 #+srcname: task-R-session
2243 #+begin_src R :session what :results replace
2249 #+resname: task-R-session
2252 #+srcname: another-task-R-session
2253 #+begin_src R :session what :results replace
2257 *** DONE function to bring up inferior-process buffer [4/4]
2259 This should be callable from inside of a source-code block in an
2260 org-mode buffer. It should evaluate the header arguments, then bring
2261 up the inf-proc buffer using =pop-to-buffer=.
2263 For lack of a better place, lets add this to the `org-metadown-hook'
2266 To give this a try, place the cursor on a source block with variables,
2267 (optionally git a prefix argument) then hold meta and press down.
2271 #+srcname: task-ruby-pop-to-session
2272 #+begin_src ruby :var num=9 :var another="something else"
2273 num.times{|n| puts another}
2278 #+srcname: task-python-pop-to-session
2279 #+begin_src python :var num=9 :var another="something else"
2284 #+srcname: task-R-pop-to-session
2285 #+begin_src R :var a=9 :var b=8
2291 #+srcname: task-shell-pop-sessions
2292 #+begin_src sh :var NAME="eric"
2296 *** DEFERRED function to dump last N lines from inf-proc buffer into the current source block
2298 Callable with a prefix argument to specify how many lines should be
2299 dumped into the source-code buffer.
2301 *** REJECTED comint notes
2303 Implementing comint integration in [[file:lisp/org-babel-comint.el][org-babel-comint.el]].
2306 - handling of outputs
2307 - split raw output from process by prompts
2308 - a ring of the outputs, buffer-local, `org-babel-comint-output-ring'
2309 - a switch for dumping all outputs to a buffer
2310 - inputting commands
2312 Lets drop all this language specific stuff, and just use
2313 org-babel-comint to split up our outputs, and return either the last
2314 value of an execution or the combination of values from the
2317 **** comint filter functions
2318 : ;; comint-input-filter-functions hook process-in-a-buffer
2319 : ;; comint-output-filter-functions hook function modes.
2320 : ;; comint-preoutput-filter-functions hook
2321 : ;; comint-input-filter function ...
2323 #+srcname: obc-filter-ruby
2324 #+begin_src ruby :results last
2332 ** DONE Remove protective commas from # comments before evaluating
2333 org inserts protective commas in front of ## comments in language
2334 modes that use them. We need to remove them prior to sending code
2337 #+srcname: testing-removal-of-protective-comas
2339 ,# this one might break it??
2343 ** DONE pass multiple reference arguments into R
2344 Can we do this? I wasn't sure how to supply multiple 'var' header
2345 args. Just delete this if I'm being dense.
2347 This should be working, see the following example...
2349 #+srcname: two-arg-example
2350 #+begin_src R :var n=2 :var m=8
2354 #+resname: two-arg-example
2357 ** DONE ensure that table ranges work
2358 when a table range is passed to org-babel as an argument, it should be
2359 interpreted as a vector.
2362 | 2 | 3 | Fixnum:1 |
2363 | 3 | 4 | Array:123456 |
2367 #+TBLFM: @1$3='(sbe simple-sbe-example (n 4))::@2$3='(sbe task-table-range (n @1$1..@6$1))::@3$3='(sbe task-table-range (n (@1$1..@6$1)))
2369 #+srcname: simple-sbe-example
2370 #+begin_src emacs-lisp
2374 #+srcname: task-table-range
2375 #+begin_src ruby :var n=simple-sbe-example
2379 #+srcname: simple-results
2380 #+begin_src emacs-lisp :var n=task-table-range(n=(1 2 3))
2384 #+resname: simple-results
2387 #+srcname: task-arr-referent
2388 #+begin_src ruby :var ar=(1 2 3)
2392 #+resname: task-arr-referent
2395 ** DONE global variable indicating default to vector output
2396 how about an alist... =org-babel-default-header-args= this may already
2397 exist... just execute the following and all source blocks will default
2400 #+begin_src emacs-lisp
2401 (setq org-babel-default-header-args '((:results . "vector")))
2404 ** DONE name named results if source block is named
2405 currently this isn't happening although it should be
2407 #+srcname: test-naming-named-source-blocks
2408 #+begin_src emacs-lisp
2412 #+resname: test-naming-named-source-blocks
2414 ** DONE (simple caching) check for named results before source blocks
2415 see the TODO comment in [[file:lisp/org-babel-ref.el::TODO%20This%20should%20explicitly%20look%20for%20resname%20lines%20before][org-babel-ref.el#org-babel-ref-resolve-reference]]
2416 ** DONE set =:results silent= when eval with prefix argument
2418 #+begin_src emacs-lisp
2421 ** DONE results-type header (vector/file) [3/3]
2422 In response to a point in Dan's email. We should allow the user to
2423 force scalar or vector results. This could be done with a header
2424 argument, and the default behavior could be controlled through a
2425 configuration variable.
2427 #+srcname: task-trivial-vector
2428 #+begin_src ruby :results replace vector
2435 since it doesn't make sense to turn a vector into a scalar, lets
2436 just add a two values...
2438 - vector :: forces the results to be a vector (potentially 1 dimensional)
2439 - file :: this throws an error if the result isn't a string, and
2440 tries to treat it as a path to a file.
2442 I'm just going to cram all of these into the =:results= header
2443 argument. Then if we allow multiple header arguments it should
2444 work out, for example one possible header argument string could be
2445 =:results replace vector file=, which would *replace* any existing
2446 results forcing the results into an org-mode table, and
2447 interpreting any strings as file paths.
2449 *** DONE multiple =:results= headers
2451 #+srcname: multiple-result-headers
2452 #+begin_src ruby :results replace silent
2458 *** DONE file result types
2459 When inserting into an org-mode buffer create a link with the path
2460 being the value, and optionally the display being the
2461 =file-name-nondirectory= if it exists.
2463 #+srcname: task-file-result
2464 #+begin_src python :results replace file
2469 [[something][something]]
2472 This will be useful because blocks like =ditaa= and =dot= can return
2473 the string path of their files, and can add =file= to their results
2476 *** DONE vector result types
2478 #+srcname: task-force-results
2479 #+begin_src emacs-lisp :results vector
2486 ** DONE results name
2487 In order to do this we will need to start naming our results.
2488 Since the source blocks are named with =#+srcname:= lines we can
2489 name results with =#+resname:= lines (if the source block has no
2490 name then no name is given to the =#+resname:= line on creation,
2491 otherwise the name of the source block is used).
2493 This will have the additional benefit of allowing results and
2494 source blocks to be located in different places in a buffer (and
2495 eventually in different buffers entirely).
2497 #+srcname: developing-resnames
2498 #+begin_src emacs-lisp :results silent
2502 Once source blocks are able to find their own =#+resname:= lines
2505 #+srcname: sbe-w-new-results
2506 #+begin_src emacs-lisp :results replace
2507 (sbe "developing-resnames")
2513 *** TODO change the results insertion functions to use these lines
2515 *** TODO teach references to resolve =#+resname= lines.
2517 ** DONE org-babel tests org-babel [1/1]
2518 since we are accumulating this nice collection of source-code blocks
2519 in the sandbox section we should make use of them as unit tests.
2520 What's more, we should be able to actually use org-babel to run these
2523 We would just need to cycle over every source code block under the
2524 sandbox, run it, and assert that the return value is equal to what we
2527 I have the feeling that this should be possible using only org-babel
2528 functions with minimal or no additional elisp. It would be very cool
2529 for org-babel to be able to test itself.
2531 This is now done, see [[* Tests]].
2533 *** DEFERRED org-babel assertions (may not be necessary)
2534 These could be used to make assertions about the results of a
2535 source-code block. If the assertion fails then the point could be
2536 moved to the block, and error messages and highlighting etc... could
2539 ** DONE make C-c C-c work anywhere within source code block?
2540 This seems like it would be nice to me, but perhaps it would be
2541 inefficient or ugly in implementation? I suppose you could search
2542 forward, and if you find #+end_src before you find #+begin_src,
2543 then you're inside one. [DED]
2545 Agreed, I think inside of the =#+srcname: line= would be useful as
2548 #+srcname: testing-out-cc
2549 #+begin_src emacs-lisp
2553 ** DONE integration with org tables
2554 We should make it easy to call org-babel source blocks from org-mode
2555 table formulas. This is practical now that it is possible to pass
2556 arguments to org-babel source blocks.
2558 See the related [[* (sandbox) integration w/org tables][sandbox]] header for tests/examples.
2560 *** digging in org-table.el
2561 In the past [[file:~/src/org/lisp/org-table.el::org%20table%20el%20The%20table%20editor%20for%20Org%20mode][org-table.el]] has proven difficult to work with.
2563 Should be a hook in [[file:~/src/org/lisp/org-table.el::defun%20org%20table%20eval%20formula%20optional%20arg%20equation][org-table-eval-formula]].
2565 Looks like I need to change this [[file:~/src/org/lisp/org-table.el::if%20lispp][if statement]] (line 2239) into a cond
2568 ** DONE source blocks as functions
2570 Allow source code blocks to be called like functions, with arguments
2571 specified. We are already able to call a source-code block and assign
2572 it's return result to a variable. This would just add the ability to
2573 specify the values of the arguments to the source code block assuming
2574 any exist. For an example see
2576 When a variable appears in a header argument, how do we differentiate
2577 between it's value being a reference or a literal value? I guess this
2578 could work just like a programming language. If it's escaped or in
2579 quotes, then we count it as a literal, otherwise we try to look it up
2582 ** DONE folding of code blocks? [2/2]
2583 [DED] In similar way to using outline-minor-mode for folding function
2584 bodies, can we fold code blocks? #+begin whatever statements are
2585 pretty ugly, and in any case when you're thinking about the overall
2586 game plan you don't necessarily want to see the code for each Step.
2588 *** DONE folding of source code block
2589 Sounds good, and wasn't too hard to implement. Code blocks should
2590 now be fold-able in the same manner as headlines (by pressing TAB
2593 *** REJECTED folding of results
2594 So, lets do a three-stage tab cycle... First fold the src block,
2595 then fold the results, then unfold.
2597 There's no way to tell if the results are a table or not w/o
2598 actually executing the block which would be too expensive of an
2601 ** DONE selective export of text, code, figures
2602 [DED] The org-babel buffer contains everything (code, headings and
2603 notes/prose describing what you're up to, textual/numeric/graphical
2604 code output, etc). However on export to html / LaTeX one might want
2605 to include only a subset of that content. For example you might
2606 want to create a presentation of what you've done which omits the
2609 [EMS] So I think this should be implemented as a property which can
2610 be set globally or on the outline header level (I need to review
2611 the mechanics of org-mode properties). And then as a source block
2612 header argument which will apply only to a specific source code
2613 block. A header argument of =:export= with values of
2615 - =code= :: just show the code in the source code block
2616 - =none= :: don't show the code or the results of the evaluation
2617 - =results= :: just show the results of the code evaluation (don't
2618 show the actual code)
2619 - =both= :: show both the source code, and the results
2621 this will be done in [[* (sandbox) selective export][(sandbox) selective export]].
2623 ** DONE a header argument specifying silent evaluation (no output)
2624 This would be useful across all types of source block. Currently
2625 there is a =:replace t= option to control output, this could be
2626 generalized to an =:output= option which could take the following
2627 options (maybe more)
2629 - =t= :: this would be the default, and would simply insert the
2630 results after the source block
2631 - =replace= :: to replace any results which may already be there
2632 - =silent= :: this would inhibit any insertion of the results
2634 This is now implemented see the example in the [[* silent evaluation][sandbox]]
2636 ** DONE assign variables from tables in R
2637 This is now working (see [[* (sandbox table) R][(sandbox-table)-R]]). Although it's not that
2638 impressive until we are able to print table results from R.
2640 ** DONE insert 2-D R results as tables
2641 everything is working but R and shell
2647 This has already been tackled by Dan in [[file:existing_tools/org-R.el::defconst%20org%20R%20write%20org%20table%20def][org-R:check-dimensions]]. The
2648 functions there should be useful in combination with [[http://cran.r-project.org/doc/manuals/R-data.html#Export-to-text-files][R-export-to-csv]]
2649 as a means of converting multidimensional R objects to emacs lisp.
2651 It may be as simple as first checking if the data is multidimensional,
2652 and then, if so using =write= to write the data out to a temporary
2653 file from which emacs can read the data in using =org-table-import=.
2655 Looking into this further, is seems that there is no such thing as a
2656 scalar in R [[http://tolstoy.newcastle.edu.au/R/help/03a/3733.html][R-scalar-vs-vector]] In that light I am not sure how to
2657 deal with trivial vectors (scalars) in R. I'm tempted to just treat
2658 them as vectors, but then that would lead to a proliferation of
2659 trivial 1-cell tables...
2661 ** DONE allow variable initialization from source blocks
2662 Currently it is possible to initialize a variable from an org-mode
2663 table with a block argument like =table=sandbox= (note that the
2664 variable doesn't have to named =table=) as in the following example
2670 #+begin_src emacs-lisp :var table=sandbox :results replace
2671 (message (format "table = %S" table))
2674 : "table = ((1 2 3) (4 \"schulte\" 6))"
2676 It would be good to allow initialization of variables from the results
2677 of other source blocks in the same manner. This would probably
2678 require the addition of =#+SRCNAME: example= lines for the naming of
2679 source blocks, also the =table=sandbox= syntax may have to be expanded
2680 to specify whether the target is a source code block or a table
2681 (alternately we could just match the first one with the given name
2682 whether it's a table or a source code block).
2684 At least initially I'll try to implement this so that there is no need
2685 to specify whether the reference is to a table or a source-code block.
2686 That seems to be simpler both in terms of use and implementation.
2688 This is now working for emacs-lisp, ruby and python (and mixtures of
2689 the three) source blocks. See the examples in the [[* (sandbox) referencing other source blocks][sandbox]].
2691 This is currently working only with emacs lisp as in the following
2692 example in the [[* emacs lisp source reference][emacs lisp source reference]].
2695 ** TODO Add languages [11/16]
2696 I'm sure there are many more that aren't listed here. Please add
2697 them, and bubble any that you particularly care about up to the top.
2699 Any new language should be implemented in a org-babel-lang.el file.
2700 Follow the pattern set by [[file:lisp/org-babel-script.el][org-babel-script.el]], [[file:lisp/org-babel-shell.el][org-babel-shell.el]] and
2701 [[file:lisp/org-babel-R.el][org-babel-R.el]].
2712 let fac n = if n == 0 then 1 else n * fac (n - 1)
2724 | 1 | 2 | 3 | 4 | 5 |
2726 **** allow non-interactive evaluation
2728 *** STARTED ocaml [2/3]
2730 - [X] Working for the simple case (no arguments, simple output)
2731 - [X] correct handling of vector/list output
2732 - [ ] ability to import arguments
2739 | n -> fib(n - 1) + fib(n - 2) in
2761 [|"ocaml"; "array"|]
2765 | "ocaml" | "array" |
2768 This could probably be added to [[file:lisp/org-babel-script.el][org-babel-script.el]]
2772 - support for sessions
2773 - add more useful header arguments (user, passwd, database, etc...)
2774 - support for more engines (currently only supports mysql)
2775 - what's a reasonable way to drop table data into SQL?
2777 #+srcname: sql-example
2778 #+begin_src sql :engine mysql
2784 | "information_schema" |
2788 Sass is a very nice extension of CSS, which is much nicer to read and
2789 write (see [[http://sass-lang.com/][sass-lang]]).
2791 #+srcname: sass-example
2792 #+begin_src sass :file stylesheet.css :results file
2802 [[file:stylesheet.css][stylesheet.css]]
2805 trivial [[file:lisp/langs/org-babel-css.el][org-babel-css.el]]
2808 (see [[* file result types][file result types]])
2810 #+srcname: implementing-ditaa
2811 #+begin_src ditaa :results replace :file blue.png :cmdline -r
2821 #+resname: implementing-ditaa
2822 [[file:blue.png][blue.png]]
2824 *** DONE gnuplot [7/7]
2825 (see [[* file result types][file result types]])
2827 #+PLOT: title:"Citas" ind:1 deps:(3) type:2d with:histograms set:"yrange [0:]"
2828 #+TBLNAME: gnuplot-data
2829 | independent var | first dependent var | second dependent var |
2830 |-----------------+---------------------+----------------------|
2831 | 0.1 | 0.425 | 0.375 |
2832 | 0.2 | 0.3125 | 0.3375 |
2833 | 0.3 | 0.24999993 | 0.28333338 |
2834 | 0.4 | 0.275 | 0.28125 |
2835 | 0.5 | 0.26 | 0.27 |
2836 | 0.6 | 0.25833338 | 0.24999993 |
2837 | 0.7 | 0.24642845 | 0.23928553 |
2838 | 0.8 | 0.23125 | 0.2375 |
2839 | 0.9 | 0.23333323 | 0.2333332 |
2840 | 1 | 0.2225 | 0.22 |
2841 | 1.1 | 0.20909075 | 0.22272708 |
2842 | 1.2 | 0.19999998 | 0.21458333 |
2843 | 1.3 | 0.19615368 | 0.21730748 |
2845 #+srcname: implementing-gnuplot
2846 #+begin_src gnuplot :var data=gnuplot-data :results silent
2847 set title "Implementing Gnuplot"
2848 plot data using 1:2 with lines
2851 **** DONE add variables
2852 gnuplot 4.2 and up support user defined variables. This is how
2853 we will handle variables with org-babel (meaning we will need to
2854 require gnuplot 4.2 and up for variable support, which can be
2855 install using [[http://www.macports.org/install.php][macports]] on Mac OSX).
2857 - scalar variables should be replaced in the body of the gnuplot code
2858 - vector variables should be exported to tab-separated files, and
2859 the variable names should be replaced with the path to the files
2861 **** DONE direct plotting w/o session
2862 **** DEFERRED gnuplot support for column/row names
2863 This should be implemented along the lines of the [[* STARTED Column (and row) names of tables in R input/output][R-colname-support]].
2865 We can do something similar to the :labels param in org-plot, we just
2866 have to be careful to ensure that each label is aligned with the
2869 This may be walking too close to an entirely prebuilt plotting tool
2870 rather than straight gnuplot code evaluation. For now I think this
2873 **** DONE a =file= header argument
2874 to specify a file holding the results
2876 #+srcname: gnuplot-to-file-implementation
2877 #+begin_src gnuplot :file plot.png :var data=gnuplot-data
2878 plot data using 1:2, data using 1:3 with lines
2882 [[file:plot.png][plot.png]]
2884 **** DONE helpers from org-plot.el
2885 There are a variety of helpers in org-plot which can be fit nicely
2886 into custom gnuplot header arguments.
2888 These should all be in place by now.
2890 **** DEFERRED header argument specifying 3D data
2893 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2894 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2895 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2896 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 |
2897 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2898 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2899 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 1 | 0 |
2900 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
2901 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
2902 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2903 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2904 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 0 |
2905 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2906 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2908 #+srcname: implementing-gnuplot-grid-plots
2909 #+begin_src gnuplot :vars data=org-grid
2913 **** DONE gnuplot sessions
2914 Working on this, we won't support multiple sessions as `gnuplot-mode'
2915 isn't setup for such things.
2917 Also we can't display results with the default :none session, so for
2918 gnuplot we really want the default behavior to be :default, and to
2919 only run a :none session when explicitly specified.
2921 #+srcname: implementing-gnuplot-sessions
2922 #+begin_src gnuplot :var data=gnuplot-data :session none :file session.png
2923 set title "Implementing Gnuplot Sessions"
2924 plot data using 1:2 with lines
2928 [[file:session.png][session.png]]
2931 (see [[* file result types][file result types]])
2933 #+srcname: implementing-dot-support
2934 #+begin_src dot :file test-dot.png :cmdline -Tpng
2935 digraph data_relationships {
2936 "data_requirement" [shape=Mrecord, label="{DataRequirement|description\lformat\l}"]
2937 "data_product" [shape=Mrecord, label="{DataProduct|name\lversion\lpoc\lformat\l}"]
2938 "data_requirement" -> "data_product"
2943 [[file:test-dot.png][test-dot.png]]
2946 (see [[* file result types][file result types]])
2948 for information on asymptote see http://asymptote.sourceforge.net
2950 #+begin_src asymptote :file asymptote-test.png
2955 real f(real t) {return 1+cos(t);}
2957 path g=polargraph(f,0,2pi,operator ..)--cycle;
2960 xaxis("$x$",above=true);
2961 yaxis("$y$",above=true);
2963 dot("$(a,0)$",(1,0),N);
2964 dot("$(2a,0)$",(2,0),N+E);
2968 [[file:asymptote-test.png][asymptote-test.png]]
2978 ** DONE stripping indentation from source-code blocks
2979 This is a problem in [[file:lisp/org-babel-exp.el][org-babel-exp.el]].
2981 ** DONE failing to write srcname to resname when evaluating a named block
2983 #+srcname: please-name-my-result
2984 #+begin_src emacs-lisp
2988 #+resname: please-name-my-result
2992 ** DONE Python session evaluation bug
2993 The following block evaluates correctly with :session none
2994 (set :results to output), but fails with session-based evaluation
2995 (with :results value, as below, you see the error message)
2997 I edebug'ed it and it seems fine until [[file:lisp/langs/org-babel-python.el::comint%20session%20evaluation%20org%20babel%20comint%20in%20buffer%20buffer%20let%20full%20body][we go to comint]].
2999 #+begin_src python :session pysession :results value
3001 from subprocess import *
3006 cc = [('58C','NBS'),
3010 for outdir in ['none', 'noscots', 'lax', 'strict']:
3011 outdir = os.path.join('exclusion-study', outdir)
3012 for case, control in cc:
3013 outfile = os.path.join(outdir, '%s-vs-%s-direct' % (case, control))
3014 cmd = 'snptest %s -frequentist 1 -hwe ' % ('-gen_gz' if format == '.gen.gz' else '')
3015 cmd += '-cases %s %s ' % (case + format, case + '.sample')
3017 cmd += '-controls %s %s ' % (control + format, control + '.sample')
3018 cmd += '-exclude_samples %s ' % os.path.join(outdir, 'exclusions')
3019 cmd += '-o %s ' % outfile
3020 cmd += '-chunk %d ' % chunk
3021 cmd += '> %s' % outfile + '.log'
3028 snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls NBS.gen.gz NBS.sample -exclude_samples exclusion-study/none/exclusions -o exclusion-study/none/58C-vs-NBS-direct -chunk 10000 > exclusion-study/none/58C-vs-NBS-direct.log
3029 snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/none/exclusions -o exclusion-study/none/58C-vs-POBI-direct -chunk 10000 > exclusion-study/none/58C-vs-POBI-direct.log
3030 snptest -gen_gz -frequentist 1 -hwe -cases NBS.gen.gz NBS.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/none/exclusions -o exclusion-study/none/NBS-vs-POBI-direct -chunk 10000 > exclusion-study/none/NBS-vs-POBI-direct.log
3031 snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls NBS.gen.gz NBS.sample -exclude_samples exclusion-study/noscots/exclusions -o exclusion-study/noscots/58C-vs-NBS-direct -chunk 10000 > exclusion-study/noscots/58C-vs-NBS-direct.log
3032 snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/noscots/exclusions -o exclusion-study/noscots/58C-vs-POBI-direct -chunk 10000 > exclusion-study/noscots/58C-vs-POBI-direct.log
3033 snptest -gen_gz -frequentist 1 -hwe -cases NBS.gen.gz NBS.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/noscots/exclusions -o exclusion-study/noscots/NBS-vs-POBI-direct -chunk 10000 > exclusion-study/noscots/NBS-vs-POBI-direct.log
3034 snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls NBS.gen.gz NBS.sample -exclude_samples exclusion-study/lax/exclusions -o exclusion-study/lax/58C-vs-NBS-direct -chunk 10000 > exclusion-study/lax/58C-vs-NBS-direct.log
3035 snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/lax/exclusions -o exclusion-study/lax/58C-vs-POBI-direct -chunk 10000 > exclusion-study/lax/58C-vs-POBI-direct.log
3036 snptest -gen_gz -frequentist 1 -hwe -cases NBS.gen.gz NBS.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/lax/exclusions -o exclusion-study/lax/NBS-vs-POBI-direct -chunk 10000 > exclusion-study/lax/NBS-vs-POBI-direct.log
3037 snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls NBS.gen.gz NBS.sample -exclude_samples exclusion-study/strict/exclusions -o exclusion-study/strict/58C-vs-NBS-direct -chunk 10000 > exclusion-study/strict/58C-vs-NBS-direct.log
3038 snptest -gen_gz -frequentist 1 -hwe -cases 58C.gen.gz 58C.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/strict/exclusions -o exclusion-study/strict/58C-vs-POBI-direct -chunk 10000 > exclusion-study/strict/58C-vs-POBI-direct.log
3039 snptest -gen_gz -frequentist 1 -hwe -cases NBS.gen.gz NBS.sample -controls POBI.gen.gz POBI.sample -exclude_samples exclusion-study/strict/exclusions -o exclusion-study/strict/NBS-vs-POBI-direct -chunk 10000 > exclusion-study/strict/NBS-vs-POBI-direct.log
3042 ** DONE require users to explicitly turn on each language
3043 As we continue to add more languages to org-babel, many of which will
3044 require new major-modes we need to re-think how languages are added to
3047 Currently we are requiring all available languages in the
3048 [[file:lisp/org-babel-init.el][org-babel-init.el]] file. I think we need to change this to a user
3049 setting so that only the language which have their requirements met
3050 (in terms of system executables and emacs major modes) are loaded. It
3051 is one more step for install, but it seems to me to be the only
3056 we add something like the following to the instillation instructions
3058 #+begin_src emacs-lisp
3059 ;; Uncomment each of the following require lines if you want org-babel
3060 ;; to support that language. Each language has a comment explaining
3061 ;; it's dependencies. See the related files in lisp/langs for more
3062 ;; detailed explanations of requirements.
3064 ;; (require 'org-babel-R) ;; ess-mode
3065 ;; (require 'org-babel-asymptote) ;; asymptote be installed on your system
3066 ;; (require 'org-babel-css) ;; none
3067 ;; (require 'org-babel-ditaa) ;; ditaa be installed on your system
3068 ;; (require 'org-babel-dot) ;; dot be installed on your system
3069 ;; (require 'org-babel-gnuplot) ;; gnuplot-mode
3070 ;; (require 'org-babel-python) ;; python-mode
3071 ;; (require 'org-babel-ruby) ;; inf-ruby mode, ruby and irb must be installed on your system
3072 ;; (require 'org-babel-sql) ;; none
3075 note that =org-babel-sh=, =org-babel-emacs-lisp= are not included in
3076 the list as they can safely be assumed to work on any system.
3079 we should come up with a way to gracefully degrade when support for a
3080 specific language is missing
3082 > To demonstrate creation of documents, open the "test-export.org" file in
3083 > the base of the org-babel directory, and export it as you would any
3084 > other org-mode file. The "exports" header argument controls how
3085 > source-code blocks are exported, with the following options
3087 > - none :: no part of the source-code block is exported in the document
3088 > - results :: only the output of the evaluated block is exported
3089 > - code :: the code itself is exported
3090 > - both :: both the code and results are exported
3092 I have this error showing up:
3094 executing Ruby source code block
3095 apply: Searching for program: no such file or directory, irb
3097 ** DONE problem with newlines in output when :results value
3099 #+begin_src python :results value
3100 '\n'.join(map(str, range(4)))
3109 Whereas I was hoping for
3116 *Note*: to generate the above you can try using the new =raw= results
3119 #+begin_src python :results value raw
3120 '|'+'|\n|'.join(map(str, range(4)))+'|'
3129 This is now working, it doesn't return as a table because the value
3130 returned is technically a string. To return the table mentioned above
3131 try something like the following.
3134 [[0], [1], [2], [3]]
3143 This is some sort of non-printing char / quoting issue I think. Note
3146 #+begin_src python :results value
3147 '\\n'.join(map(str, range(4)))
3154 #+begin_src python :results output
3155 print('\n'.join(map(str, range(4))))
3164 *** collapsing consecutive newlines in string output
3166 This is an example of the same bug
3168 #+srcname: multi-line-string-output
3169 #+begin_src ruby :results output
3170 "the first line ends here
3173 and this is the second one
3178 This doesn't produce anything at all now. I believe that's because
3179 I've changed things so that :results output really does *not* get the
3180 value of the block, only the STDOUT. So if we add a print statement
3183 #+srcname: multi-line-string-output
3184 #+begin_src ruby :results output
3185 print "the first line ends here
3188 and this is the second one
3194 : the first line ends here
3197 : and this is the second one
3201 However, the behaviour with :results value is wrong
3203 #+srcname: multi-line-string-value
3205 "the first line ends here
3208 and this is the second one
3216 ** TODO prompt characters appearing in output with R
3217 #+begin_src R :session *R* :results output
3226 ** TODO o-b-execute-subtree overwrites heading when subtree is folded
3228 Try M-x org-babel-execute-subtree with the subtree folded and
3229 point at the beginning of the heading line.
3233 ** DONE non-orgtbl formatted lists
3236 #+srcname: this-doesn't-match-orgtbl
3237 #+begin_src emacs-lisp :results replace
3238 '((:results . "replace"))
3242 | (:results . "replace") |
3244 #+srcname: this-probably-also-wont-work
3245 #+begin_src emacs-lisp :results replace
3252 ** PROPOSED allow un-named arguments
3261 ## produces no output
3263 It's not essential but would be nice for this to work. To do it
3264 properly, would mean that we'd have to specify rules for how a string
3265 of supplied arguments (some possibly named) interact with the
3266 arguments in the definition (some possibly with defaults) to give
3267 values to the variables in the funbction body.
3268 ** PROPOSED external shell execution can't isolate return values
3269 I have no idea how to do this as of yet. The result is that when
3270 shell functions are run w/o a session there is no difference between
3271 the =output= and =value= result arguments.
3273 Yea, I don't know how to do this either. I searched extensively on
3274 how to isolate the *last* output of a series of shell commands (see
3275 [[* last command for
3276 shells][last command for shells]]). The results of the search were basically
3277 that it was not possible (or at least not accomplish-able with a
3278 reasonable amount of effort).
3280 That fact combined with the tenancy to all ways use standard out in
3281 shell scripts led me to treat these two options (=output= and =value=)
3282 as identical in shell evaluation. Not ideal but maybe good enough for
3285 In the `results' branch I've changed this so that they're not quite
3286 identical: output results in raw stdout contents, whereas value
3287 converts it to elisp, perhaps to a table if it looks tabular. This is
3288 the same for the other languages. [Dan]
3290 ** DONE adding blank line when source-block produces no output
3292 #+srcname: show-org-babel-trim
3294 find . \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o -type f \( -iname \*.el \) -exec grep -i -nH -e org-babel-trim {} \;
3297 ** DONE Allow source blocks to be recognised when #+ are not first characters on the line
3298 I think Carsten has recently altered the core so that #+ can have
3299 preceding whitespace, at least for literal/code examples. org-babel
3300 should support this.
3302 #+srcname: testing-indentation
3303 #+begin_src emacs-lisp :results silent
3304 (message "i'm indented")
3307 #+srcname: testing-non-indentation
3308 #+begin_src emacs-lisp :results silent
3309 (message "I'm not indented")
3312 #+srcname: i-resolve-references-to-the-indented
3313 #+begin_src emacs-lisp :var speech=testing-indentation :results silent
3314 (message "I said %s" speech)
3317 ** DONE are the org-babel-trim s necessary?
3318 at the end of e.g. org-babel-R-evaluate, org-babel-python-evaluate, but
3319 not org-babel-ruby-evaluate
3321 I think it depends on the language, if we find that extra blank
3322 lines are being inserted in a particular language that is a good
3323 indication that the trim or chomp functions may be appropriate.
3325 org-babel-trim and the related org-babel-chomp are use throughout
3328 #+srcname: show-org-babel-trim-usage
3329 #+begin_src sh :results output
3330 find lisp/ \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o -type f \( -iname \*.el \) -exec grep -i -nH org-babel-trim {} \;
3335 lisp//langs/org-babel-python.el:49: vars "\n") "\n" (org-babel-trim body) "\n")) ;; then the source block body
3336 lisp//langs/org-babel-python.el:143: (org-remove-indentation (org-babel-trim body)) "[\r\n]")))
3337 lisp//langs/org-babel-python.el:158: #'org-babel-trim
3338 lisp//langs/org-babel-python.el:166: (reverse (mapcar #'org-babel-trim raw)))))))
3339 lisp//langs/org-babel-python.el:169: (output (org-babel-trim (mapconcat #'identity (reverse (cdr results)) "\n")))
3340 lisp//langs/org-babel-python.el:170: (value (org-babel-python-table-or-string (org-babel-trim (car results)))))))))
3341 lisp//langs/org-babel-ruby.el:149: (mapcar #'org-babel-trim raw)))))))
3342 lisp//langs/org-babel-sh.el:148: (mapcar #'org-babel-trim raw)))))))
3343 lisp//langs/org-babel-sh.el:151: (output (org-babel-trim (mapconcat #'org-babel-trim (reverse results) "\n")))
3344 lisp//org-babel-ref.el:161: (mapcar #'org-babel-trim (reverse (cons buffer return)))))
3345 lisp//org-babel.el:198: (with-temp-buffer (insert (org-babel-trim body)) (copy-region-as-kill (point-min) (point-max)))
3346 lisp//org-babel.el:465: (org-babel-trim
3347 lisp//org-babel.el:706:(defun org-babel-trim (string &optional regexp)
3350 #+srcname: show-org-babel-chomp-usage
3351 #+begin_src sh :results output
3352 find lisp/ \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o -type f \( -iname \*.el \) -exec grep -i -nH org-babel-chomp {} \;
3357 lisp//langs/org-babel-R.el:122: (full-body (mapconcat #'org-babel-chomp
3358 lisp//langs/org-babel-R.el:143: (delete nil (mapcar #'extractor (mapcar #'org-babel-chomp raw))) "\n"))))))))
3359 lisp//langs/org-babel-ruby.el:143: #'org-babel-chomp
3360 lisp//langs/org-babel-sh.el:142: (full-body (mapconcat #'org-babel-chomp
3361 lisp//org-babel-tangle.el:163: (insert (format "\n%s\n" (org-babel-chomp body)))
3362 lisp//org-babel.el:362: (org-babel-chomp (match-string 2 arg)))
3363 lisp//org-babel.el:698:(defun org-babel-chomp (string &optional regexp)
3364 lisp//org-babel.el:707: "Like `org-babel-chomp' only it runs on both the front and back of the string"
3365 lisp//org-babel.el:708: (org-babel-chomp (org-babel-reverse-string
3366 lisp//org-babel.el:709: (org-babel-chomp (org-babel-reverse-string string) regexp)) regexp))
3369 ** DONE LoB is not populated on startup
3370 org-babel-library-of-babel is nil for me on startup. I have to
3371 evaluate the [[file:lisp/org-babel-lob.el::][org-babel-lob-ingest]] line manually.
3373 #+tblname: R-plot-example-data
3380 #+lob: R-plot(data=R-plot-example-data)
3384 I've added a section to [[file:lisp/org-babel-init.el][org-babel-init.el]] which will load the
3385 library of babel on startup.
3387 Note that this needs to be done in [[file:lisp/org-babel-init.el][org-babel-init.el]] rather than in
3388 [[file:lisp/org-babel-lob.el][org-babel-lob.el]], not entirely sure why, something about it being
3391 Also, I'm now having the file closed if it wasn't being visited by
3392 a buffer before being loaded.
3394 ** DONE use new merge function [[file:lisp/org-babel-ref.el::t%20nil%20org%20combine%20plists%20args%20nil][here]]?
3395 And at other occurrences of org-combine-plists?
3396 ** DONE creeping blank lines
3397 There's still inappropriate addition of blank lines in some circumstances.
3399 Hmm, it's a bit confusing. It's to do with o-b-remove-result. LoB
3400 removes the entire (#+resname and result) and starts from scratch,
3401 whereas #+begin_src only removes the result. I haven't worked out
3402 what the correct fix is yet. Maybe the right thing to do is to make
3403 sure that those functions (o-b-remove-result et al.) are neutral
3404 with respect to newlines. Sounds easy, but...
3414 Compare the results of
3415 #+lob: adder(a=5, b=17)
3417 #+resname: python-add(a=5, b=17)
3419 --------------------------------
3427 ---------------------
3428 ** DONE #+srcname arg parsing bug
3429 #+srcname: test-zz(arg=adder(a=1, b=1))
3438 #+srcname: test-zz-nasty(arg=adder(a=adder(a=19,b=adder(a=5,b=2)),b=adder(a=adder(a=1,b=9),b=adder(a=1,b=3))))
3443 #+resname: test-zz-nasty
3448 #+srcname: test-zz-hdr-arg
3449 #+begin_src python :var arg=adder(a=adder(a=19,b=adder(a=5,b=2)),b=adder(a=adder(a=1,b=9),b=adder(a=1,b=3)))
3456 ** DONE Fix nested evaluation and default args
3457 The current parser / evaluator fails with greater levels of nested
3458 function block calls (example below).
3460 *** Initial statement [ded]
3461 If we want to overcome this I think we'd have to redesign some of
3462 the evaluation mechanism. Seeing as we are also facing issues like
3463 dealing with default argument values, and seeing as we now know
3464 how we want the library of babel to behave in addition to the
3465 source blocks, now might be a good time to think about this. It
3466 would be nice to do the full thing at some point, but otoh we may
3467 not consider it a massive priority.
3469 AIui, there are two stages: (i) construct a parse tree, and (ii)
3470 evaluate it and return the value at the root. In the parse tree
3471 each node represents an unevaluated value (either a literal value
3472 or a reference). Node v may have descendent nodes, which represent
3473 values upon which node v's evaluation depends. Once that tree is
3474 constructed, then we evaluate the nodes from the tips towards the
3475 root (a post-order traversal).
3477 [This would also provide a solution for concatenating the STDOUTs
3478 of called blocks, which is a [[*allow%20output%20mode%20to%20return%20stdout%20as%20value][task below]]; we concatenate them in
3479 whatever order the traversal is done in.]
3481 In addition to the variable references (i.e. daughter nodes), each
3482 node would contain the information needed to evaluate that node
3483 (e.g. lang body). Then we would pass a function postorder over the
3484 tree which would call o-b-execute-src-block at each node, finally
3485 returning the value at the root.
3487 Fwiw I made a very tentative small start at stubbing this out in
3488 org-babel-call.el in the 'evaluation' branch. And I've made a start
3489 at sketching a parsing algorithm below.
3490 **** Parse tree algorithm
3491 Seeing as we're just trying to parse a string like
3492 f(a=1,b=g(c=2,d=3)) it shouldn't be too hard. But of course there
3493 are 'proper' parsers written in elisp out there,
3494 e.g. [[http://cedet.sourceforge.net/semantic.shtml][Semantic]]. Perhaps we can find what we need -- our syntax is
3495 pretty much the same as python and R isn't it?
3497 Or, a complete hack, but maybe it would be we easy to transform it
3498 to XML and then parse that with some existing tool?
3500 But if we're doing it ourselves, something very vaguely like this?
3501 (I'm sure there're lots of problems with this)
3503 #+srcname: org-babel-call-parse(call)
3505 ## we are currently reading a reference name: the name of the root function
3506 whereami = "refname"
3507 node = root = Node()
3508 for c in call_string:
3511 whereami = "varname" # now we're reading a variable name
3514 node.daughters = [node.daughters, new]
3517 whereami = "refname"
3519 whereami = "varname"
3526 if whereami = "varname":
3527 node.varnames[varnum] += c
3528 elif whereami = "refname":
3532 *** discussion / investigation
3533 I believe that this issue should be addressed as a bug rather than as
3534 a point for new development. The code in [[file:lisp/org-babel-ref.el][org-babel-ref.el]] already
3535 resolves variable references in a recursive manner which *should* work
3536 in the same manner regardless of the depth of the number of nested
3537 function calls. This recursive evaluation has the effect of
3538 implicitly constructing the parse tree that your are thinking of
3539 constructing explicitly.
3541 Through using some of the commented out debugging statements in
3542 [[file:lisp/org-babel-ref.el][org-babel-ref.el]] I have looked at what may be going wrong in the
3543 current evaluation setup, and it seems that nested variables are being
3544 set using the =:var= header argument, and these variables are being
3545 overridden by the *default* variables which are being entered through
3546 the new functional syntax (see the demonstration header below).
3548 I believe that once this bug is fixed we should be back to fully
3549 resolution of nested arguments. We should capture this functionality
3550 in a test to ensure that we continue to test it as we move forward. I
3551 can take a look at implementing this once I get a chance.
3553 Looks like the problem may be in [[file:lisp/org-babel.el::defun%20org%20babel%20merge%20params%20rest%20plists][org-babel-merge-params]], which seems
3554 to be trampling the provided :vars values.
3556 Nope, now it seems that we are actually looking up the results line,
3557 rather than the actual source-code block, which would make sense given
3558 that the results-line will return the same value regardless of the
3559 arguments supplied. See the output of this [[file:lisp/org-babel-ref.el::message%20type%20S%20type%20debugging][debug-statement]].
3561 We need to be sure that we don't read from a =#+resname:= line when we
3562 have a non-nil set of arguments.
3565 After uncommenting the debugging statements located [[file:lisp/org-babel-ref.el::message%20format%20first%20second%20S%20S%20new%20refere%20new%20referent%20debugging][here]] and more
3566 importantly [[file:lisp/org-babel-ref.el::message%20nested%20args%20S%20args%20debugging][here]], we can see that the current reference code does
3567 evaluate the references correctly, and it uses the =:var= header
3568 argument to set =a=8=, however the default variables specified using
3569 the functional syntax in =adder(a=3, b=2)= is overriding this
3572 ***** doesn't work with functional syntax
3574 #+srcname: adder-func(a=3, b=2)
3579 #+resname: adder-func
3582 #+srcname: after-adder-func(arg=adder-func(a=8))
3587 #+resname: after-adder-func
3590 ***** still does work with =:var= syntax
3592 so it looks like regardless of the syntax used we're not overriding
3593 the default argument values.
3595 #+srcname: adder-header
3596 #+begin_src python :var a=3 :var b=2
3600 #+resname: adder-header
3603 #+srcname: after-adder-header
3604 #+begin_src python :var arg=adder-header(a=8, b=0)
3608 #+resname: after-adder-header
3611 *** Set of test cases
3612 **** Both defaults provided in definition
3613 #+srcname: adder1(a=10,b=20)
3620 ****** DONE Rely on defaults
3628 ******* DONE empty parens () not recognised as lob call
3629 E.g. remove spaces between parens above
3631 updated [[file:lisp/org-babel-lob.el::defvar%20org%20babel%20lob%20one%20liner%20regexp%20lob%20t%20n%20n%20t%20n][org-babel-lob-one-liner-regexp]]
3633 ****** DONE One supplied, one default
3636 #+resname: adder1(a=0)
3643 #+resname: adder1(b=0)
3647 ****** DONE Both supplied
3648 #+lob: adder1(a=1,b=2)
3650 #+resname: adder1(a=1,b=2)
3653 **** One arg lacks default in definition
3654 #+srcname: adder2(a=10,b)
3658 ****** DEFERRED Rely on defaults (one of which is missing)
3663 ## should be error: b has no default
3665 Maybe we should let the programming language handle this case. For
3666 example python spits out an error in the =#+lob= line above. Maybe
3667 rather than catching these errors our-selves we should raise an error
3668 when the source-block returns an error. I'll propose a [[* PROPOSED raise elisp error when source-blocks return errors][task]] for this
3669 idea, I'm not sure how/if it would work...
3671 ****** DEFERRED Default over-ridden
3674 See the above [[* DEFERRED Rely on defaults (one of which is missing)][deferred]] and the new proposed [[* PROPOSED raise elisp error when source-blocks return errors][task]], I think it may be
3675 more flexible to allow the source block language to handle the error.
3678 ## should be error: b has no default
3680 ****** DONE Missing default supplied
3683 #+resname: adder2(b=1)
3690 ****** DONE One over-ridden, one supplied
3691 #+lob: adder2(a=1,b=2)
3693 #+resname: adder2(a=1,b=2)
3698 *** Example that fails
3700 #+srcname: adder(a=0, b=99)
3713 #+srcname: level-one-nesting()
3714 #+begin_src python :var arg=adder(a=one(),b=one())
3718 #+resname: level-one-nesting
3722 #+srcname: level-one-nesting()
3723 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=one()))
3730 *** DONE deeply nested arguments still fails
3732 #+srcname: deeply-nested-args-bug
3733 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=one()))
3740 **** Used to result in this error
3741 : supplied params=nil
3742 : new-refere="adder", new-referent="a=adder(a=one(),b=one()),b=adder(a=one(),b=one())"
3743 : args=((:var . "a=adder(a=one()") (:var . "b=one())") (:var . "b=adder(a=one()") (:var . "b=one())"))
3745 : supplied params=((:var . "a=adder(a=one()") (:var . "b=one())") (:var . "b=adder(a=one()") (:var . "b=one())"))
3746 : new-refere="adder", new-referent="a=one("
3747 : args=((:var . "a=one("))
3749 : supplied params=((:var . "a=one("))
3750 : reference 'one(' not found in this buffer
3752 Need to change the regexp in [[file:lisp/org-babel-ref.el::assign%20any%20arguments%20to%20pass%20to%20source%20block][org-babel-ref-resolve-reference]] so that
3753 it only matches when the parenthesis are balanced. Maybe look at
3754 [[http://www.gnu.org/software/emacs/elisp/html_node/List-Motion.html][this]].
3756 *** DONE Still some problems with deeply nested arguments and defaults
3758 **** DONE Parsing / defaults bug
3759 Try inserting a space between 'a=0,' and 'b=0' and comparing results
3760 #+srcname: parsing-defaults-bug()
3761 #+begin_src python :var arg=adder(a=adder(a=0,b=0))
3765 #+resname: parsing-defaults-bug
3769 #+srcname: deeply-nested-args-bug-orig()
3770 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=adder(a=3, b=4),b=one()))
3774 #+resname: deeply-nested-args-bug-orig
3779 **** DONE Nesting problem II
3780 This generates parsing errors
3782 Fixed: c2bef96b7f644c05be5a38cad6ad1d28723533aa
3784 #+srcname: deeply-nested-args-bug-II-1()
3785 #+begin_src python :var arg=adder(a=adder(a=one(),b=adder(a=2,b=4)))
3789 #+resname: deeply-nested-args-bug-II-1
3793 #+srcname: deeply-nested-args-bug-II-original()
3794 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=adder(a=1,b=4)))
3798 #+resname: deeply-nested-args-bug-II-original
3803 **** DONE Why does this give 8?
3804 It was picking up the wrong definition of adder
3805 #+srcname: deeply-nested-args-bug-2()
3806 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()))
3810 #+resname: deeply-nested-args-bug-2
3814 **** DONE Problem with empty argument list
3815 This gives empty list with () and 'no output' with ( )
3817 I think this is OK now.
3820 #+begin_src python :var arg=adder( )
3827 #+srcname: deeply-nested-args-bug-orig()
3828 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=adder(a=3, b=4),b=one()))
3832 #+resname: deeply-nested-args-bug-orig
3837 **** DONE Nesting problem II
3838 This generates parsing errors
3840 Fixed: c2bef96b7f644c05be5a38cad6ad1d28723533aa
3842 #+srcname: deeply-nested-args-bug-II-1()
3843 #+begin_src python :var arg=adder(a=adder(a=one(),b=adder(a=2,b=4)))
3847 #+resname: deeply-nested-args-bug-II-1
3851 #+srcname: deeply-nested-args-bug-II-original()
3852 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=adder(a=1,b=4)))
3856 #+resname: deeply-nested-args-bug-II-original
3861 **** DONE Why does this give 8?
3862 It was picking up the wrong definition of adder
3863 #+srcname: deeply-nested-args-bug-2()
3864 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()))
3868 #+resname: deeply-nested-args-bug-2
3872 **** DONE Problem with empty argument list
3873 This gives empty list with () and 'no output' with ( )
3875 I think this is OK now.
3878 #+begin_src python :var arg=adder( )
3911 #+srcname: test-zz(arg=adder(a=adder(a=19,b=adder(a=5,b=2)),b=adder(a=adder(a=1,b=9),b=adder(a=1,b=3))))
3917 *** DONE Arg lacking default
3918 This would be good thing to address soon. I'm imagining that
3919 e.g. here, the 'caller' block would return the answer 30. I believe
3920 there's a few issues here: i.e. the naked 'a' without a reference
3921 is not understood; the default arg b=6 is not understood.
3923 #+srcname: adder-with-arg-lacking-default(a, b=6)
3930 #+srcname: caller(var=adder-with-arg-lacking-default(a=24))
3931 #+begin_src python :results silent
3935 ** DONE allow srcname to omit function call parentheses
3936 Someone needs to revisit those regexps. Is there an argument for
3937 moving some of the regexps used to match function calls into
3938 defvars? (i.e. in o-b.el and o-b-ref.el)
3940 This seems to work now. It still might be a good idea to separate
3941 out some of the key regexps into defvars at some point.
3943 #+srcname: omit-parens-test
3944 #+begin_src ruby :results output
3953 ** DONE avoid stripping whitespace from output when :results output
3954 This may be partly solved by using o-b-chomp rather than o-b-trim
3955 in the o-b-LANG-evaluate functions.
3956 ** DEFERRED weird escaped characters in shell prompt break shell evaluation
3957 E.g. this doesn't work. Should the shell sessions set a sane prompt
3958 when they start up? Or is it a question of altering
3959 comint-prompt-regexp? Or altering org-babel regexps?
3962 black=30 ; red=31 ; green=32 ; yellow=33 ; blue=34 ; magenta=35 ; cyan=36 ; white=37
3965 export PS1="\[\033[${prompt_col}m\]\w${prompt_char} \[\033[0m\]"
3968 I just pushed a good amount of changes, could you see if your shell
3969 problems still exist?
3971 The problem's still there. Specifically, aIui, at [[file:lisp/langs/org-babel-sh.el::raw%20org%20babel%20comint%20with%20output%20buffer%20org%20babel%20sh%20eoe%20output%20nil%20insert%20full%20body%20comint%20send%20input%20nil%20t][this line]] of
3972 org-babel-sh.el, raw gets the value
3974 ("" "
\e[0m Sun Jun 14 19:26:24 EDT 2009\n" "
\e[0m org_babel_sh_eoe\n" "
\e[0m ")
3976 and therefore (member org-babel-sh-eoe-output ...) fails
3978 I think that `comint-prompt-regexp' needs to be altered to match
3979 the shell prompt. This shouldn't be too difficult to do by hand,
3980 using the `regexp-builder' command and should probably be part of
3981 the user's regular emacs init. I can't think of a way for us to
3982 set this automatically, and we are SOL without a regexp to match
3984 ** DONE function calls in #+srcname: refs
3986 My srcname references don't seem to be working for function
3987 calls. This needs fixing.
3994 srcname function call doesn't work for calling a source block
3995 #+srcname: caller(var1=called())
4008 They do work for a simple reference
4009 #+srcname: caller2(var1=56)
4018 and they do work for :var header arg
4020 #+begin_src python :var var1=called()
4026 ** DONE LoB: with output to buffer, not working in buffers other than library-of-babel.org
4028 I haven't fixed this yet. org-babel-ref-resolve-reference moves
4029 point around, inside a save-excursion. Somehow when it comes to
4030 inserting the results (after possible further recursive calls to
4031 org-babel-ref-resolve-reference), point hasn't gone back to the
4034 #+tblname: test-data
4039 #+lob: R-plot(data=test-data)
4041 #+lob: python-add(a=2, b=9)
4043 #+resname: python-add(a=2, b=9)
4047 I think this got fixed in the bugfixes before merging results into master.
4049 ** DONE cursor movement when evaluating source blocks
4050 E.g. the pie chart example. Despite the save-window-excursion in
4051 org-babel-execute:R. (I never learned how to do this properly: org-R
4052 jumps all over the place...)
4054 I don't see this now [ded]
4056 ** DONE LoB: calls fail if reference has single character name
4057 commit 21d058869df1ff23f4f8cc26f63045ac9c0190e2
4058 **** This doesn't work
4059 #+lob: R-plot(data=X)
4078 #+lob: R-plot(data=XX)
4080 ** DONE make :results replace the default?
4081 I'm tending to think that appending results to pre-existing results
4082 creates mess, and that the cleaner `replace' option should be the
4083 default. E.g. when a source block creates an image, we would want
4084 that to be updated, rather than have a new one be added.
4088 ** DONE ruby evaluation not working under ubuntu emacs 23
4089 With emacs 23.0.91.1 on ubuntu, for C-h f run-ruby I have the
4090 following, which seems to conflict with [[file:lisp/langs/org-babel-ruby.el::let%20session%20buffer%20save%20window%20excursion%20run%20ruby%20nil%20session%20current%20buffer][this line]] in org-babel-ruby.el.
4093 run-ruby is an interactive compiled Lisp function.
4097 Run an inferior Ruby process, input and output via buffer *ruby*.
4098 If there is a process already running in `*ruby*', switch to that buffer.
4099 With argument, allows you to edit the command line (default is value
4100 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
4101 (after the `comint-mode-hook' is run).
4102 (Type C-h m in the process buffer for a list of commands.)
4105 So, I may have a non-standard inf-ruby.el. Here's my version of
4109 run-ruby is an interactive Lisp function in `inf-ruby.el'.
4111 (run-ruby &optional COMMAND NAME)
4113 Run an inferior Ruby process, input and output via buffer *ruby*.
4114 If there is a process already running in `*ruby*', switch to that buffer.
4115 With argument, allows you to edit the command line (default is value
4116 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
4117 (after the `comint-mode-hook' is run).
4118 (Type C-h m in the process buffer for a list of commands.)
4121 It seems we could either bundle my version of inf-ruby.el (as it's
4122 the newest). Or we could change the use of `run-ruby' so that it
4123 is robust across multiple distributions. I think I'd prefer the
4124 former, unless the older version of inf-ruby is actually bundled
4125 with emacs, in which case maybe we should go out of our way to
4126 support it. Thoughts?
4128 I think for now I'll just include the latest [[file:util/inf-ruby.el][inf-ruby.el]] in the
4129 newly created utility directory. I doubt anyone would have a
4130 problem using the latest version of this file.
4131 ** DONE test failing forcing vector results with =test-forced-vector-results= ruby code block
4132 Note that this only seems to happen the *second* time the test table
4135 #+srcname: bug-trivial-vector
4136 #+begin_src emacs-lisp :results vector silent
4140 #+srcname: bug-forced-vector-results
4141 #+begin_src ruby :var triv=test-trivial-vector :results silent
4145 mysteriously this seems to be fixed...
4146 ** DONE defunct R sessions
4147 Sometimes an old R session will turn defunct, and newly inserted code
4148 will not be evaluated (leading to a hang).
4150 This seems to be fixed by using `inferior-ess-send-input' rather than `comint-send-input'.
4151 ** DONE ruby fails on first call to non-default session
4153 #+srcname: bug-new-session
4154 #+begin_src ruby :session is-new
4158 ** DONE when reading results from =#+resname= line
4160 Errors when trying to read from resname lines.
4162 #+resname: bug-in-resname
4165 #+srcname: bug-in-resname-reader
4166 #+begin_src emacs-lisp :var buggy=bug-in-resname() :results silent
4170 ** DONE R-code broke on "org-babel" rename
4172 #+srcname: bug-R-babels
4177 ** DONE error on trivial R results
4179 So I know it's generally not a good idea to squash error without
4180 handling them, but in this case the error almost always means that
4181 there was no file contents to be read by =org-table-import=, so I
4184 #+srcname: bug-trivial-r1
4185 #+begin_src R :results replace
4186 pie(c(1, 2, 3), labels = c(1, 2, 3))
4189 #+srcname: bug-trivial-r2
4190 #+begin_src R :results replace
4194 #+resname: bug-trivial-r2
4197 #+srcname: bug-trivial-r3
4198 #+begin_src R :results replace
4202 #+resname: bug-trivial-r3
4207 ** DONE ruby new variable creation (multi-line ruby blocks)
4208 Actually it looks like we were dropping all but the last line.
4210 #+srcname: multi-line-ruby-test
4211 #+begin_src ruby :var table=bug-numerical-table :results replace
4213 table.each{|n| total += n}
4220 ** DONE R code execution seems to choke on certain inputs
4221 Currently the R code seems to work on vertical (but not landscape)
4224 #+srcname: little-fake
4225 #+begin_src emacs-lisp
4230 #+begin_src R :var num=little-fake
4237 #+srcname: set-debug-on-error
4238 #+begin_src emacs-lisp :results silent
4239 (setq debug-on-error t)
4242 #+srcname: bug-numerical-table
4243 #+begin_src emacs-lisp :results silent
4250 #+srcname: bug-R-number-evaluation
4251 #+begin_src R :var table=bug-numerical-table
4260 #+tblname: bug-vert-table
4265 #+srcname: bug-R-vertical-table
4266 #+begin_src R :var table=bug-vert-table :results silent
4270 ** DONE org bug/request: prevent certain org behaviour within code blocks
4271 E.g. [[]] gets recognised as a link (when there's text inside the
4272 brackets). This is bad for R code at least, and more generally
4273 could be argued to be inappropriate. Is it difficult to get org to
4274 ignore text in code blocks? [DED]
4276 I believe Carsten addressed this recently on the mailing list with
4277 the comment that it was indeed a difficult issue. I believe this
4278 may be one area where we could wait for an upstream (org-mode) fix.
4280 [Dan] Carsten has fixed this now in the core.
4282 ** DONE with :results replace, non-table output doesn't replace table output
4283 And vice versa. E.g. Try this first with table and then with len(table) [DED]
4284 #+begin_src python :var table=sandbox :results replace
4289 | 4 | "schulte" | 6 |
4292 Yes, this is certainly a problem. I fear that if we begin replacing
4293 anything immediately following a source block (regardless of whether
4294 it matches the type of our current results) we may accidentally delete
4295 hand written portions of the user's org-mode buffer.
4297 I think that the best solution here would be to actually start
4298 labeling results with a line that looks something like...
4302 This would have a couple of benefits...
4303 1) we wouldn't have to worry about possibly deleting non-results
4304 (which is currently an issue)
4305 2) we could reliably replace results even if there are different types
4306 3) we could reference the results of a source-code block in variable
4307 definitions, which would be useful if for example we don't wish to
4308 re-run a source-block every time because it is long-running.
4310 Thoughts? If no-one objects, I believe I will implement the labeling
4313 ** DONE extra quotes for nested string
4314 Well R appears to be reading the tables without issue...
4316 these *should* be quoted
4318 #+begin_src sh :results replace
4323 | "README.markdown" |
4326 | "existing_tools" |
4330 | "test-export.html" |
4331 | "test-export.org" |
4333 #+srcname: test-quotes
4334 #+begin_src ruby :var tab=ls
4340 #+srcname: test-quotes
4341 #+begin_src R :var tab=ls
4347 ** DONE simple ruby arrays not working
4349 As an example eval the following. Adding a line to test
4351 #+tblname: simple-ruby-array
4354 #+srcname: ruby-array-test
4355 #+begin_src ruby :var ar = simple-ruby-array :results silent
4359 ** DONE space trailing language name
4360 fix regexp so it works when there's a space trailing the language name
4362 #+srcname: test-trailing-space
4367 ** DONE Args out of range error
4369 The following block resulted in the error below [DED]. It ran without
4370 error directly in the shell.
4373 for platf in ill aff ; do
4374 for pop in CEU YRI ASI ; do
4375 rm -f $platf/hapmap-genos-$pop-all $platf/hapmap-rs-all
4376 cat $platf/hapmap-genos-$pop-* > $platf/hapmap-genos-$pop-all
4377 cat $platf/hapmap-rs-* > $platf/hapmap-rs-all
4382 executing source block with sh...
4383 finished executing source block
4384 string-equal: Args out of range: "", -1, 0
4386 the error =string-equal: Args out of range: "", -1, 0= looks like what
4387 used to be output when the block returned an empty results string.
4388 This should be fixed in the current version, you should now see the
4389 following message =no result returned by source block=.
4391 ** DONE ruby arrays not recognized as such
4393 Something is wrong in [[file:lisp/org-babel-script.el]] related to the
4394 recognition of ruby arrays as such.
4396 #+begin_src ruby :results replace
4402 #+begin_src python :results replace
4407 ** REJECTED elisp reference fails for literal number
4408 That's a bug in Dan's elisp, not in org-babel.
4409 #+srcname: elisp-test(a=4)
4410 #+begin_src emacs-lisp
4422 Evaluate all the cells in this table for a comprehensive test of the
4423 org-babel functionality.
4425 *Note*: if you have customized =org-babel-default-header-args= then some
4426 of these tests may fail.
4428 #+TBLNAME: org-babel-tests
4429 | functionality | block | arg | expected | results | pass |
4430 |-------------------------+----------------------------+-----+-------------+-------------+------|
4431 | basic evaluation | | | | | pass |
4432 |-------------------------+----------------------------+-----+-------------+-------------+------|
4433 | emacs lisp | basic-elisp | | 5 | 5 | pass |
4434 | shell | basic-shell | | 6 | 6 | pass |
4435 | ruby | basic-ruby | | org-babel | org-babel | pass |
4436 | python | basic-python | | hello world | hello world | pass |
4437 | R | basic-R | | 13 | 13 | pass |
4438 |-------------------------+----------------------------+-----+-------------+-------------+------|
4439 | tables | | | | | pass |
4440 |-------------------------+----------------------------+-----+-------------+-------------+------|
4441 | emacs lisp | table-elisp | | 3 | 3 | pass |
4442 | ruby | table-ruby | | 1-2-3 | 1-2-3 | pass |
4443 | python | table-python | | 5 | 5 | pass |
4444 | R | table-R | | 3.5 | 3.5 | pass |
4445 | R: col names in R | table-R-colnames | | -3 | -3 | pass |
4446 | R: col names in org | table-R-colnames-org | | 169 | 169 | pass |
4447 |-------------------------+----------------------------+-----+-------------+-------------+------|
4448 | source block references | | | | | pass |
4449 |-------------------------+----------------------------+-----+-------------+-------------+------|
4450 | all languages | chained-ref-last | | Array | Array | pass |
4451 |-------------------------+----------------------------+-----+-------------+-------------+------|
4452 | source block functions | | | | | pass |
4453 |-------------------------+----------------------------+-----+-------------+-------------+------|
4454 | emacs lisp | defun-fibb | | fibbd | fibbd | pass |
4455 | run over | Fibonacci | 0 | 1 | 1 | pass |
4456 | a | Fibonacci | 1 | 1 | 1 | pass |
4457 | variety | Fibonacci | 2 | 2 | 2 | pass |
4458 | of | Fibonacci | 3 | 3 | 3 | pass |
4459 | different | Fibonacci | 4 | 5 | 5 | pass |
4460 | arguments | Fibonacci | 5 | 8 | 8 | pass |
4461 |-------------------------+----------------------------+-----+-------------+-------------+------|
4462 | bugs and tasks | | | | | pass |
4463 |-------------------------+----------------------------+-----+-------------+-------------+------|
4464 | simple ruby arrays | ruby-array-test | | 3 | 3 | pass |
4465 | R number evaluation | bug-R-number-evaluation | | 2 | 2 | pass |
4466 | multi-line ruby blocks | multi-line-ruby-test | | 2 | 2 | pass |
4467 | forcing vector results | test-forced-vector-results | | Array | Array | pass |
4468 | deeply nested arguments | deeply-nested-args-bug | | 4 | 4 | pass |
4469 |-------------------------+----------------------------+-----+-------------+-------------+------|
4470 | sessions | | | | | pass |
4471 |-------------------------+----------------------------+-----+-------------+-------------+------|
4472 | set ruby session | set-ruby-session-var | | :set | :set | pass |
4473 | get from ruby session | get-ruby-session-var | | 3 | 3 | pass |
4474 | set python session | set-python-session-var | | set | set | pass |
4475 | get from python session | get-python-session-var | | 4 | 4 | pass |
4476 | set R session | set-R-session-var | | set | set | pass |
4477 | get from R session | get-R-session-var | | 5 | 5 | pass |
4478 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
4479 #+TBLFM: $5=""::$6=""
4482 The second TBLFM line (followed by replacing '[]' with '') can be used
4483 to blank out the table results, in the absence of a better method.
4487 #+srcname: basic-elisp
4488 #+begin_src emacs-lisp :results silent
4493 #+srcname: basic-shell
4494 #+begin_src sh :results silent
4499 #+srcname: date-simple
4500 #+begin_src sh :results silent
4504 #+srcname: basic-ruby
4505 #+begin_src ruby :results silent
4510 #+srcname: basic-python
4511 #+begin_src python :results silent
4517 #+begin_src R :results silent
4525 #+tblname: test-table
4529 #+tblname: test-table-colnames
4530 | var1 | var2 | var3 |
4531 |------+------+------|
4535 #+srcname: table-elisp
4536 #+begin_src emacs-lisp :results silent :var table=test-table
4537 (length (car table))
4541 #+srcname: table-ruby
4542 #+begin_src ruby :results silent :var table=test-table
4543 table.first.join("-")
4547 #+srcname: table-python
4548 #+begin_src python :var table=test-table
4552 #+srcname: table-R(table=test-table)
4557 #+srcname: table-R-colnames(table=test-table-colnames)
4558 #+begin_src R :results silent
4559 sum(table$var2 - table$var3)
4562 #+srcname: R-square(x=default-name-doesnt-exist)
4563 #+begin_src R :colnames t
4567 This should return 169. The fact that R is able to use the column name
4568 to index the data frame (x$var3) proves that a table with column names
4569 (a header row) has been recognised as input for the R-square function
4570 block, and that the R-square block has output an elisp table with
4571 column names, and that the colnames have again been recognised when
4572 creating the R variables in this block.
4573 #+srcname: table-R-colnames-org(x = R-square(x=test-table-colnames))
4583 Lets pass a references through all of our languages...
4585 Lets start by reversing the table from the previous examples
4587 #+srcname: chained-ref-first
4588 #+begin_src python :var table = test-table
4593 #+resname: chained-ref-first
4597 Take the first part of the list
4599 #+srcname: chained-ref-second
4600 #+begin_src R :var table = chained-ref-first
4604 #+resname: chained-ref-second
4608 Turn the numbers into string
4610 #+srcname: chained-ref-third
4611 #+begin_src emacs-lisp :var table = chained-ref-second
4612 (mapcar (lambda (el) (format "%S" el)) table)
4615 #+resname: chained-ref-third
4618 and Check that it is still a list
4620 #+srcname: chained-ref-last
4621 #+begin_src ruby :var table=chained-ref-third
4626 ** source blocks as functions
4628 #+srcname: defun-fibb
4629 #+begin_src emacs-lisp :results silent
4630 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
4634 #+srcname: fibonacci
4635 #+begin_src emacs-lisp :results silent :var n=7
4645 ** sbe tests (these don't seem to be working...)
4646 Testing the insertion of results into org-mode tables.
4648 #+srcname: multi-line-output
4649 #+begin_src ruby :results replace
4650 "the first line ends here
4653 and this is the second one
4659 : the first line ends here\n\n\n and this is the second one\n\neven a third
4661 #+srcname: multi-line-error
4662 #+begin_src ruby :results replace
4663 raise "oh nooooooooooo"
4669 | the first line ends here... | -:5: warning: parenthesize argument(s) for future version... |
4670 #+TBLFM: $1='(sbe "multi-line-output")::$2='(sbe "multi-line-error")
4672 ** forcing results types tests
4674 #+srcname: test-trivial-vector
4675 #+begin_src emacs-lisp :results vector silent
4679 #+srcname: test-forced-vector-results
4680 #+begin_src ruby :var triv=test-trivial-vector :results silent
4686 #+srcname: set-ruby-session-var
4687 #+begin_src ruby :session :results silent
4692 #+srcname: get-ruby-session-var
4693 #+begin_src ruby :session :results silent
4697 #+srcname: set-python-session-var
4698 #+begin_src python :session
4703 #+srcname: get-python-session-var
4704 #+begin_src python :session
4708 #+srcname: set-R-session-var
4709 #+begin_src R :session
4714 #+srcname: get-R-session-var
4715 #+begin_src R :session
4724 To run these examples evaluate [[file:lisp/org-babel-init.el][org-babel-init.el]]
4726 ** org-babel.el beginning functionality
4728 #+begin_src sh :results replace
4733 : Sun Jul 5 18:54:39 EDT 2009
4740 : Sun Jul 05 18:54:35 -0400 2009
4752 #+begin_src R :results replace
4762 hist(rgamma(20,3,3))
4767 ** org-babel plays with tables
4768 Alright, this should demonstrate both the ability of org-babel to read
4769 tables into a lisp source code block, and to then convert the results
4770 of the source code block into an org table. It's using the classic
4771 "lisp is elegant" demonstration transpose function. To try this
4774 1. evaluate [[file:lisp/org-babel-init.el]] to load org-babel and friends
4775 2. evaluate the transpose definition =\C-c\\C-c= on the beginning of
4777 3. evaluate the next source code block, this should read in the table
4778 because of the =:var table=previous=, then transpose the table, and
4779 finally it should insert the transposed table into the buffer
4780 immediately following the block
4784 #+begin_src emacs-lisp :results silent
4785 (defun transpose (table)
4786 (apply #'mapcar* #'list table))
4794 #+begin_src emacs-lisp :var table=sandbox :results replace
4799 #+begin_src emacs-lisp
4804 | 1 | 2 | 3 | 4 | 5 |
4808 #+begin_src ruby :var table=sandbox :results replace
4809 table.first.join(" - ")
4815 #+begin_src python :var table=sandbox
4820 #+begin_src ruby :var table=sandbox :results replace
4825 : [[1, 2, 3], [4, "schulte", 6]]
4829 | 4 | "schulte" | 6 |
4831 #+begin_src python :var table=sandbox :results replace
4837 | "__add__" | "__class__" | "__contains__" | "__delattr__" | "__delitem__" | "__delslice__" | "__doc__" | "__eq__" | "__format__" | "__ge__" | "__getattribute__" | "__getitem__" | "__getslice__" | "__gt__" | "__hash__" | "__iadd__" | "__imul__" | "__init__" | "__iter__" | "__le__" | "__len__" | "__lt__" | "__mul__" | "__ne__" | "__new__" | "__reduce__" | "__reduce_ex__" | "__repr__" | "__reversed__" | "__rmul__" | "__setattr__" | "__setitem__" | "__setslice__" | "__sizeof__" | "__str__" | "__subclasshook__" | "append" | "count" | "extend" | "index" | "insert" | "pop" | "remove" | "reverse" | "sort" |
4839 *** (sandbox table) R
4841 #+TBLNAME: sandbox_r
4845 #+begin_src R :results replace
4846 x <- c(rnorm(10, mean=-3, sd=1), rnorm(10, mean=3, sd=1))
4850 | -3.35473133869346 |
4852 | -3.32819924928633 |
4853 | -2.97310212756194 |
4854 | -2.09640758369576 |
4855 | -5.06054014378736 |
4856 | -2.20713700711221 |
4857 | -1.37618039712037 |
4858 | -1.95839385821742 |
4859 | -3.90407396475502 |
4860 | 2.51168071590226 |
4861 | 3.96753011570494 |
4862 | 3.31793212627865 |
4863 | 1.99829753972341 |
4864 | 4.00403686419829 |
4865 | 4.63723764452927 |
4866 | 3.94636744261313 |
4867 | 3.58355906547775 |
4868 | 3.01563442274226 |
4871 #+begin_src R var tabel=sandbox_r :results replace
4876 | 4 | "schulte" | 6 |
4879 Now shell commands are converted to tables using =org-table-import=
4880 and if these tables are non-trivial (i.e. have multiple elements) then
4881 they are imported as org-mode tables...
4883 #+begin_src sh :results replace
4887 | "total" | 208 | "" | "" | "" | "" | "" | "" |
4888 | "-rw-r--r--" | 1 | "dan" | "dan" | 57 | 2009 | 15 | "block" |
4889 | "-rw-r--r--" | 1 | "dan" | "dan" | 35147 | 2009 | 15 | "COPYING" |
4890 | "-rw-r--r--" | 1 | "dan" | "dan" | 722 | 2009 | 18 | "examples.org" |
4891 | "drwxr-xr-x" | 4 | "dan" | "dan" | 4096 | 2009 | 19 | "existing_tools" |
4892 | "-rw-r--r--" | 1 | "dan" | "dan" | 2207 | 2009 | 14 | "intro.org" |
4893 | "drwxr-xr-x" | 2 | "dan" | "dan" | 4096 | 2009 | 18 | "org-babel" |
4894 | "-rw-r--r--" | 1 | "dan" | "dan" | 277 | 2009 | 20 | "README.markdown" |
4895 | "-rw-r--r--" | 1 | "dan" | "dan" | 11837 | 2009 | 18 | "rorg.html" |
4896 | "-rw-r--r--" | 1 | "dan" | "dan" | 61829 | 2009 | 19 | "#rorg.org#" |
4897 | "-rw-r--r--" | 1 | "dan" | "dan" | 60190 | 2009 | 19 | "rorg.org" |
4898 | "-rw-r--r--" | 1 | "dan" | "dan" | 972 | 2009 | 11 | "test-export.org" |
4901 ** silent evaluation
4909 #+begin_src ruby :results silent
4913 #+begin_src ruby :results replace
4920 ** (sandbox) referencing other source blocks
4921 Doing this in emacs-lisp first because it's trivial to convert
4922 emacs-lisp results to and from emacs-lisp.
4924 *** emacs lisp source reference
4925 This first example performs a calculation in the first source block
4926 named =top=, the results of this calculation are then saved into the
4927 variable =first= by the header argument =:var first=top=, and it is
4928 used in the calculations of the second source block.
4931 #+begin_src emacs-lisp
4935 #+begin_src emacs-lisp :var first=top :results replace
4941 This example is the same as the previous only the variable being
4942 passed through is a table rather than a number.
4944 #+begin_src emacs-lisp :results silent
4945 (defun transpose (table)
4946 (apply #'mapcar* #'list table))
4949 #+TBLNAME: top_table
4953 #+SRCNAME: second_src_example
4954 #+begin_src emacs-lisp :var table=top_table
4958 #+begin_src emacs-lisp :var table=second_src_example :results replace
4963 | 4 | "schulte" | 6 |
4965 Now working for ruby
4972 #+begin_src ruby :var other=start :results replace
4978 #+SRCNAME: start_two
4983 #+begin_src python :var another=start_two :results replace
4988 Since all variables are converted into Emacs Lisp it is no problem to
4989 reference variables specified in another language.
4991 #+SRCNAME: ruby-block
4996 #+SRCNAME: lisp_block
4997 #+begin_src emacs-lisp :var ruby-variable=ruby-block
5001 #+begin_src python :var lisp_var=lisp_block
5010 #+begin_src R :results replace
5017 #+begin_src R :var other=first_r :results replace
5024 ** (sandbox) selective export
5026 For exportation tests and examples see (including exportation of
5027 inline source code blocks) [[file:test-export.org]]
5030 ** (sandbox) source blocks as functions
5033 #+begin_src emacs-lisp :results silent
5038 #+begin_src emacs-lisp :var n=default :results replace
5044 #+begin_src emacs-lisp :var result=triple(n=3, m=98) :results replace
5050 The following just demonstrates the ability to assign variables to
5051 literal values, which was not implemented until recently.
5053 #+begin_src ruby :var num="eric" :results replace
5060 ** (sandbox) inline source blocks
5062 This is an inline source code block src_ruby{1 + 6}. And another
5063 source block with text output src_emacs-lisp{"eric"}.
5065 This is an inline source code block with header
5066 arguments. src_ruby[:var n=fibbd( n = 0 )]{n}
5069 ** (sandbox) integration w/org tables
5071 #+begin_src emacs-lisp :results silent
5072 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
5076 #+begin_src emacs-lisp :var n=4 :results silent
5080 #+begin_src emacs-lisp :results silent
5081 (mapcar #'fibbd '(0 1 2 3 4 5 6 7 8))
5084 Something is not working here. The function `sbe ' works fine when
5085 called from outside of the table (see the source block below), but
5086 produces an error when called from inside the table. I think there
5087 must be some narrowing going on during intra-table emacs-lisp
5090 | original | fibbd |
5091 |----------+-------|
5102 #+TBLFM: $2='(sbe "fibbd" (n $1))
5106 #+begin_src emacs-lisp :results silent
5107 (sbe 'fibbd (n "8"))
5112 LocalWords: DBlocks dblocks org-babel el eric fontification