1 #+TITLE: org-babel --- facilitating communication between programming languages and people
2 #+SEQ_TODO: PROPOSED TODO STARTED | DONE DEFERRED REJECTED
3 #+OPTIONS: H:3 num:nil toc:t
4 #+STARTUP: oddeven hideblocks
6 Through Org-Babel Org-Mode can communicate with programming languages.
7 Code contained in source-code blocks can be evaluated and data can
8 pass seamlessly between different programming languages, Org-Mode
9 constructs (tables, file links, example text) and interactive comint
13 - The [[* Introduction][Introduction]] :: provides a brief overview of the design and use
14 of Org-Babel including tutorials and examples.
15 - In [[* Getting started][Getting Started]] :: find instructions for installing org-babel
16 into your emacs configuration.
17 - The [[* Tasks][Tasks]] :: section contains current and past tasks roughly ordered
18 by TODO state, then importance or date-completed. This would be
19 a good place to suggest ideas for development.
20 - The [[* Bugs][Bugs]] :: section contains bug reports.
21 - The [[* Tests][Tests]] :: section consists of a large table which can be
22 evaluated to run Org-Babel's functional test suite. This
23 provides a good overview of the current functionality with
24 pointers to example source blocks.
25 - The [[* Sandbox][Sandbox]] :: demonstrates much of the early/basic functionality
26 through commented source-code blocks.
28 Also see the [[file:library-of-babel.org][Library of Babel]], an extensible collection of ready-made
29 and easily-shortcut-callable source-code blocks for handling common
34 Org-Babel enables *communication* between programming languages and
38 - communication between programs :: Data passes seamlessly between
39 different programming languages, Org-Mode constructs (tables,
40 file links, example text) and interactive comint buffers.
41 - communication between people :: Data and calculations are embedded
42 in the same document as notes explanations and reports.
44 ** communication between programs
46 Org-Mode supports embedded blocks of source code (in any language)
47 inside of Org documents. Org-Babel allows these blocks of code to be
48 executed from within Org-Mode with natural handling of their inputs
52 with both scalar, file, and table output
54 *** reading information from tables
56 *** reading information from other source blocks (disk usage in your home directory)
58 This will work for Linux and Mac users, not so sure about shell
59 commands for windows users.
61 To run place the cursor on the =#+begin_src= line of the source block
62 labeled directory-pie and press =\C-c\C-c=.
64 #+srcname: directories
65 #+begin_src bash :results replace
66 cd ~ && du -sc * |grep -v total
69 #+resname: directories
71 | 11882808 | "Documents" |
72 | 8210024 | "Downloads" |
73 | 879800 | "Library" |
76 | 5307664 | "Pictures" |
85 #+srcname: directory-pie
86 #+begin_src R :var dirs = directories :session R-pie-example
87 pie(dirs[,1], labels = dirs[,2])
95 *** operations in/on tables
97 #+tblname: grades-table
98 | student | grade | letter |
99 |---------+-------+--------|
106 #+TBLFM: $2='(sbe random-score-generator)::$3='(sbe assign-grade (score $2))
108 #+srcname: assign-grade
109 #+begin_src ruby :var score=99
120 #+srcname: random-score-generator
125 #+srcname: show-distribution
126 #+begin_src R :var grades=grades-table :session *R*
134 ** communication between people
135 Quick overview of Org-Mode's exportation abilities, with links to the
136 online Org-Mode documentation, a focus on source-code blocks, and the
137 exportation options provided by Org-Babel.
139 *** Interactive tutorial
140 This would demonstrate applicability to Reproducible Research, and
141 Literate Programming.
143 *** Tests embedded in documentation
144 org-babels own functional tests are contained in a large org-mode
145 table, allowing the test suite to be run be evaluation of the table
146 and the results to be collected in the same table.
148 *** Emacs initialization files stored in Org-Mode buffers
149 Using `org-babel-tangle' it is possible to embed your Emacs
150 initialization into org-mode files. This allows for folding,
151 note-taking, todo's etc... embedded with the source-code of your Emacs
152 initialization, and through org-mode's publishing features aids in
153 sharing your customizations with others.
155 It may be worthwhile to create a fork of Phil Hagelberg's
156 [[http://github.com/technomancy/emacs-starter-kit/tree/master][emacs-starter-kit]] which uses literate org-mode files for all of the
157 actual elisp customization. These org-mode files could then be
158 exported to html and used to populate the repositories wiki on [[http://github.com/][github]].
163 *** code evaluation (comint buffer sessions and external processes)
164 There are two main ways to evaluate source blocks with org-babel.
166 - external :: By default (if the =:session= header argument is not
167 present) all source code blocks are evaluated in
168 external processes. In these cases an external process
169 is used to evaluate the source-code blocks.
170 - session :: Session based evaluation uses persistent sessions in
171 comint buffers. Sessions can be used across multiple
172 source blocks setting and accessing variables in the
175 Evaluating source blocks in sessions also allows for
176 interaction with the code. To jump to the session of a
177 source block use the `org-babel-pop-to-session' command
178 or press =M-[down]= while inside of a source code block.
179 When called with a prefix argument
180 `org-babel-pop-to-session' will evaluate all header
181 arguments before jumping to the source-code block.
183 *** results (values and outputs)
184 Either the *value* or the *output* of source code blocks can be
185 collected after evaluation.
187 - value :: The default way to collect results from a source-code block
188 is to return the value of the last statement in the block.
189 This can be thought of as the return value of the block.
190 In this case any printed output of the block is ignored.
191 This can be though of a similar to a "functional" value of
193 - output :: Another way of generating results from a source-code block
194 is to collect the output generated by the execution of the
195 block. In this case all printed output is collected
196 throughout the execution of the block. This can be
197 thought of as similar to a "script" style of evaluation.
201 Add the following lines to your .emacs, replacing the path as
202 appropriate. A good place to check that things are up and running
203 would then be [[#sandbox][the sandbox]].
204 #+begin_src emacs-lisp
205 (add-to-list 'load-path "/path/to/org-babel/lisp")
206 (require 'org-babel-init)
211 ** PROPOSED raise elisp error when source-blocks return errors
212 Not sure how/if this would work, but it may be desirable.
214 ** PROPOSED allow `anonymous' function block with function call args?
215 My question here is simply whether we're going to allow
216 #+begin_src python(arg=ref)
220 but with preference given to
221 #+srcname blockname(arg=ref)
222 ** PROPOSED allow :result as synonym for :results?
223 ** PROPOSED allow 'output mode to return stdout as value?
224 Maybe we should allow this. In fact, if block x is called
225 with :results output, and it references blocks y and z, then
226 shouldn't the output of x contain a concatenation of the outputs of
227 y and z, together with x's own output? That would raise the
228 question of what happens if y is defined with :results output and z
229 with :results value. I guess z's (possibly vector/tabular) output
230 would be inside a literal example block containing the whole lot.
231 ** PROPOSED optional timestamp for output
232 Add option to place an (inactive) timestamp at the #+resname, to
233 record when that output was generated.
235 *** source code block timestamps (optional addition)
236 [Eric] If we did this would we then want to place a timestamp on the
237 source-code block, so that we would know if the results are
238 current or out of date? This would have the effect of caching the
239 results of calculations and then only re-running if the
240 source-code has changed. For the caching to work we would need to
241 check not only the timestamp on a source-code block, but also the
242 timestamps of any tables or source-code blocks referenced by the
243 original source-code block.
245 [Dan] I do remember getting frustrated by Sweave always having to
246 re-do everything, so this could be desirable, as long as it's easy
247 to over-ride of course. I'm not sure it should be the default
248 behaviour unless we are very confident that it works well.
250 **** maintaining source-code block timestamps
251 It may make sense to add a hook to `org-edit-special' which could
252 update the source-code blocks timestamp. If the user edits the
253 contents of a source-code block directly I can think of no
254 efficient way of maintaining the timestamp.
256 ** TODO make tangle files read-only?
257 With a file-local variable setting, yea that makes sense. Maybe
258 the header should reference the related org-mode file.
260 ** TODO support for working with =*Org Edit Src Example*= buffers [4/6]
261 *** STARTED Patch against org source.
262 I've worked on several related changes to source code edit buffer
263 behaviour in the org core. My current patch (below) does the
264 following. Detailed explanation / working notes are below.
265 - C-x s offers to save edit buffers
266 - C-x C-c offers to save edit buffers
267 - C-x k warns that you're killing an edit buffer
268 - If you do kill an edit buffer, the overlay in the parent buffer is removed
269 - Edit buffers are named *Org Src <orgbuf>[<lang>]*, where
270 <orgbuf> is the name of the org-mode buffer containing this
271 source code block, and lang is the language major mode. The
272 latter might be unnecessary?
275 diff --git a/lisp/org-src.el b/lisp/org-src.el
276 index 2083c77..2be21e6 100644
277 --- a/lisp/org-src.el
278 +++ b/lisp/org-src.el
279 @@ -113,7 +113,7 @@ but which mess up the display of a snippet in Org exported files.")
281 (defvar org-src-mode-map (make-sparse-keymap))
282 (define-key org-src-mode-map "\C-c'" 'org-edit-src-exit)
283 -(define-key org-src-mode-map "\C-x\C-s" 'org-edit-src-save)
284 +;; (define-key org-src-mode-map "\C-x\C-s" 'org-edit-src-save)
285 (defvar org-edit-src-force-single-line nil)
286 (defvar org-edit-src-from-org-mode nil)
287 (defvar org-edit-src-picture nil)
288 @@ -168,7 +168,8 @@ the edited version."
289 (if (boundp 'org-edit-src-overlay)
290 (org-delete-overlay org-edit-src-overlay)))
291 (kill-buffer buffer))
292 - (setq buffer (generate-new-buffer "*Org Edit Src Example*"))
293 + (setq buffer (generate-new-buffer
294 + (concat "*Org Src " (file-name-nondirectory buffer-file-name) "[" lang "]*")))
295 (setq ovl (org-make-overlay beg end))
296 (org-overlay-put ovl 'face 'secondary-selection)
297 (org-overlay-put ovl 'edit-buffer buffer)
298 @@ -186,8 +187,7 @@ the edited version."
299 '(display nil invisible nil intangible nil))
300 (org-do-remove-indentation)
301 (let ((org-inhibit-startup t))
305 (set (make-local-variable 'org-edit-src-force-single-line) single)
306 (set (make-local-variable 'org-edit-src-from-org-mode) org-mode-p)
308 @@ -201,6 +201,7 @@ the edited version."
309 (org-set-local 'org-edit-src-end-marker end)
310 (org-set-local 'org-edit-src-overlay ovl)
311 (org-set-local 'org-edit-src-nindent nindent)
313 (and org-edit-src-persistent-message
314 (org-set-local 'header-line-format msg)))
316 @@ -400,12 +401,13 @@ the language, a switch telling of the content should be in a single line."
317 (defun org-edit-src-exit ()
318 "Exit special edit and protect problematic lines."
320 - (unless (string-match "\\`*Org Edit " (buffer-name (current-buffer)))
321 - (error "This is not an sub-editing buffer, something is wrong..."))
322 + (unless org-edit-src-from-org-mode
323 + (error "This is not a sub-editing buffer, something is wrong..."))
324 (let ((beg org-edit-src-beg-marker)
325 (end org-edit-src-end-marker)
326 (ovl org-edit-src-overlay)
327 (buffer (current-buffer))
328 + (buffer-file-name nil)
329 (nindent org-edit-src-nindent)
331 (untabify (point-min) (point-max))
332 @@ -464,6 +466,17 @@ the language, a switch telling of the content should be in a single line."
333 (goto-char (min p (point-max)))
334 (message (or msg ""))))
336 +(defun org-src-mode-configure-buffer ()
337 + (setq buffer-offer-save t)
338 + (setq buffer-file-name
339 + (concat (buffer-file-name (marker-buffer org-edit-src-beg-marker))
340 + "[" (buffer-name) "]"))
341 + (setq write-contents-functions '(org-edit-src-save))
342 + (org-add-hook 'kill-buffer-hook
343 + '(lambda () (org-delete-overlay org-edit-src-overlay)) nil 'local))
345 +(org-add-hook 'org-src-mode-hook 'org-src-mode-configure-buffer)
349 ;; arch-tag: 6a1fc84f-dec7-47be-a416-64be56bea5d8
353 **** Detailed working notes to go with that patch
354 ***** Recap of current org-src-mode
356 If you use C-c ' to work on code in a begin_source block, the code
357 buffer is put in minor mode org-src-mode, which features the
358 following two useful key-bindings:
360 | C-x s | org-edit-src-save | save the code in the source code block in the parent org file |
361 | C-c ' | org-edit-src-exit | return to the parent org file with new code |
363 Furthermore, while the edit buffer is alive, the originating code
364 block is subject to a special overlay which links to the edit
365 buffer when you click on it.
367 This is all excellent, and I use it daily, but I think there's
368 still a couple of improvements that we should make.
371 C-x k kills the buffer without questions; the overlay remains, but
372 now links to a deleted buffer.
373 ***** Proposed bug II
374 C-x C-c kills a modified edit buffer silently, without offering to
375 save your work. I have lost work like that a number of times
377 ***** Proposed bug III
378 C-x s does not offer to save a modified edit buffer
379 ***** Notes on solution
380 ****** write-contents-functions
381 A good start seems to be to use org-src-mode-hook to add
382 org-edit-src-save to the write-contents-functions list. This
383 means that when it comes to saving, org-edit-src-save will be
384 called and no subsequent attempt will be made to save the buffer
385 in the normal way. (This should obviate the remapping of C-x C-s
386 to org-edit-src-save in org-src.el)
387 ****** buffer-offer-save
388 We also want to set this to t.
390 ****** Where does this get us?
392 - C-x s still does *not* offer to save the edit buffer. That's
393 because buffer-file-name is nil.
395 - C-x C-c does ask us whether we want to save the
396 edit buffer. However, since buffer-file-name is nil it asks us
397 for a file name. The check in org-edit-src-exit throws an error
398 unless the buffer is named '* Org Edit '...
400 - C-x k kills the buffer silently, leaving a broken overlay
401 link. If buffer-file-name were set, it would have warned that
402 the buffer was modified.
404 ****** buffer-file-name
405 So, that all suggests that we need to set buffer-file-name, even
406 though we don't really want to associate this buffer with a file
407 in the normal way. As for the file name, my current suggestion
408 is parent-org-filename[edit-buffer-name].
410 [I had to move the (org-src-mode) call to the end of
411 org-edit-src-code to make sure that the required variables were
412 defined when the hook was called.]
414 ****** And so where are we now?
415 - C-x s *does* offer to save the edit buffer, but in saving
416 produces a warning that the edit buffer is modified.
417 - C-x k now gives a warning that the edit buffer is modified
419 - C-x C-c is working as desired, except that again we get
420 warnings that the edit buffer is modified, once when we save,
421 and again just before exiting emacs.
422 - And C-c ' now issues a warning that the edit buffer is
423 modified when we leave it, which we don't want.
424 ****** So, we need to get rid of the buffer modification warnings.
425 I've made buffer-file-name nil inside the let binding in
428 - C-x s behaves as desired, except that as was already the case,
429 the edit buffer is always considered modified, and so repeated
430 invocations keep saving it.
431 - As was already the case, C-x k always gives a warning that the
432 edit buffer has been modified.
433 - C-x C-c is as desired (offers to save the edit buffer) except
434 that it warns of the modified buffer just before exiting.
435 - C-c ' is as it should be (silent)
437 We've got the desired behaviour, at the cost of being forced to
438 assign a buffer-file-name to the edit buffer. The consequence is
439 that the edit buffer is considered to always be modified, since
440 a file of that name is never actually written to (doesn't even
441 exist). I couldn't see a way to trick emacs into believing that
442 the buffer was unmodified since last save. But in any case, I
443 think there's an argument that these modifications warnings are
444 a good thing, because one should not leave active edit buffers
445 around: you should always have exited with C-c ' first.
447 *** DONE name edit buffer according to #+srcname (and language?)
448 See above patch agains org.
449 *** DONE optionally evaluate header references when we switch to =*Org Edit Src*= buffer
450 That seems to imply that the header references need to be evaluated
451 and transformed into the target language object when we hit C-c ' to
452 enter the *Org Edit Src* buffer [DED]
454 Good point, I heartily agree that this should be supported [Eric]
456 (or at least before the first time we attempt to evaluate code in that
457 buffer -- I suppose there might be an argument for lazy evaluation, in
458 case someone hits C-c ' but is "just looking" and not actually
459 evaluating anything.) Of course if evaluating the reference is
460 computationally intensive then the user might have to wait before they
461 get the *Org Edit Src* buffer. [DED]
463 I fear that it may be hard to anticipate when the references will be
464 needed, some major-modes do on-the-fly evaluation while the buffer is
465 being edited. I think that we should either do this before the buffer
466 is opened or not at all, specifically I think we should resolve
467 references if the user calls C-c ' with a prefix argument. Does that
468 sound reasonable? [Eric]
472 [Dan] So now that we have org-src-mode and org-src-mode-hook, I guess
473 org-babel should do this by using the hook to make sure that, when C-c
474 C-' is issued on a source block, any references are resolved and
475 assignments are made in the appropriate session.
477 #+tblname: my-little-table
481 #+srcname: resolve-vars-on-edit
482 #+begin_src ruby :var table=my-little-table :results silent :session test
483 table.size.times.do |n|
489 *** TODO set buffer-local-process variables appropriately [DED]
490 I think something like this would be great. You've probably
491 already thought of this, but just to note it down: it would be really
492 nice if org-babel's notion of a buffer's 'session/process' played
493 nicely with ESS's notion of the buffer's session/process. ESS keeps
494 the current process name for a buffer in a buffer-local variable
495 ess-local-process-name. So one thing we will probably want to do is
496 make sure that the *Org Edit Src Example* buffer sets that variable
499 I had not thought of that, but I agree whole heartedly. [Eric]
501 Once this is done every variable should be able to dump regions into
502 their inferior-process buffer using major-mode functions.
503 *** REJECTED send code to inferior process
504 Another thought on this topic: I think we will want users to send
505 chunks of code to the interpreter from within the *Org Edit Src*
506 buffer, and I think that's what you have in mind already. In ESS that
507 is done using the ess-eval-* functions. [DED]
509 I think we can leave this up to the major-mode in the source code
510 buffer, as almost every source-code major mode will have functions for
511 doing things like sending regions to the inferior process. If
512 anything we might need to set the value of the buffer local inferior
513 process variable. [Eric]
515 *** DONE some possible requests/proposed changes for Carsten [4/4]
516 While I remember, some possible requests/proposed changes for Carsten
517 come to mind in that regard:
519 **** DONE Remap C-x C-s to save the source to the org buffer?
520 I've done this personally and I find it essential. I'm using
521 #+begin_src emacs-lisp
522 (defun org-edit-src-save ()
523 "Update the parent org buffer with the edited source code, save
524 the parent org-buffer, and return to the source code edit
533 (define-key org-exit-edit-mode-map "\C-x\C-s" 'org-edit-src-save)
537 I think this is great, but I think it should be implemented in the
540 **** DEFERRED Rename buffer and minor mode?
541 Something shorter than *Org Edit Src Example* for the buffer
542 name. org-babel is bringing org's source code interaction to a
543 level of maturity where the 'example' is no longer
544 appropriate. And if further keybindings are going to be added to
545 the minor mode then maybe org-edit-src-mode is a better name than
548 Maybe we should name the buffer with a combination of the source
549 code and the session. I think that makes sense.
551 [ES] Are you also suggesting a new org-edit-src minor mode?
552 [DED] org-exit-edit-mode is a minor mode that already exists:
554 Minor mode installing a single key binding, "C-c '" to exit special edit.
556 org-edit-src-save now has a binding in that mode, so I guess all
557 I'm saying at this stage is that it's a bit of a misnomer. But
558 perhaps we will also have more functionality to add to that minor
559 mode, making it even more of a misnomer. Perhaps something like
560 org-src-mode would be better.
561 **** DONE Changed minor mode name and added hooks
563 **** DONE a hook called when the src edit buffer is created
564 This should be implemented in the org-mode core
566 ** TODO resolve references to other org buffers/files
567 This would allow source blocks to call upon tables, source-blocks,
568 and results in other org buffers/files.
571 - [[file:lisp/org-babel-ref.el::TODO%20allow%20searching%20for%20names%20in%20other%20buffers][org-babel-ref.el:searching-in-other-buffers]]
572 - [[file:lisp/org-babel.el::defun%20org-babel%20find%20named%20result%20name][org-babel.el#org-babel-find-named-result]]
573 ** TODO resolve references to other non-org files
574 - tabular data in .csv, .tsv etc format
575 - files of interpreted code: anything stopping us giving such files
576 similar status to a source code block?
577 - Would be nice to allow org and non-org files to be remote
578 ** TODO figure out how to handle errors during evaluation
579 I expect it will be hard to do this properly, but ultimately it
580 would be nice to be able to specify somewhere to receive STDERR,
581 and to be warned if it is non-empty.
583 Probably simpler in non-session evaluation than session? At least
584 the mechanism will be different I guess.
586 R has a try function, with error handling, along the lines of
587 python. I bet ruby does too. Maybe more of an issue for functional
588 style; in my proposed scripting style the error just gets dumped to
589 the org buffer and the user is thus alerted.
590 ** STARTED figure out how to handle graphic output
592 This is listed under [[* graphical output][graphical output]] in out objectives.
594 This should take advantage of the =:results file= option, and
595 languages which almost always produce graphical output should set
596 =:results file= to true by default (this is currently done for the
597 gnuplot and ditaa languages). That would handle placing these results
598 in the buffer. Then if there is a combination of =silent= and =file=
599 =:results= headers we could drop the results to a temp buffer and pop
602 Display of file results is addressed in the [[* =\C-c \C-o= to open results of source block][open-results-task]].
604 *** TODO R graphics to screen means session evaluation
605 If R graphical output is going to screen then evaluation must be
606 in a session, otherwise the graphics will disappear as soon as the
609 *** Adding to a discussion started in email
610 I'm not deeply wedded to these ideas, just noting them down. I'm
611 probably just thinking of R and haven't really thought about how
612 this fits with the other graphics-generating languages.
614 > I used the approach below to get graphical file output
615 > today, which is one idea at least. Maybe it could be linked up with
616 > your :results file variable. (Or do we need a :results image for R?)
619 I don't think we need a special image results variable, but I may be
620 missing what the code below accomplishes. Would the task I added about
621 adding org-open-at-point functionality to source code blocks take care
624 Dan: I'm not sure. I think the ability for a script to generate both
625 text and graphical output might be a natural expectation, at least for
631 > #+srcname: cohort-scatter-plots-2d(org_babel_graphical_output_file="cohort-scatter-plots-2d.png")
633 > if(exists("org_babel_output_file"))
634 > png(filename=org_babel_graphical_output_file, width=1000, height=1000)
635 > ## plotting code in here
636 > if(exists("org_babel_graphical_output_file")) dev.off()
639 Dan: Yes, the results :file option is nice for dealing with graphical
640 output, and that could well be enough. Something based on the scheme
641 above would have a couple of points in its favour:
642 1. It's easy to switch between output going to on-screen graphics and
643 output going to file: Output will go to screen unless a string variable
644 with a standard name (e.g. ""org_babel_graphical_output_file"")
645 exists in which case it will go to the file indicated by the value
647 2. The block can return a result / script output, as well as produce
650 In interactive use we might want to allow the user to choose between
651 screen and file output. In non-interactive use such as export, it
652 would be file output (subject to the :exports directives).
654 ** TODO Finalise behaviour regarding vector/scalar output
655 *** DONE Stop spaces causing vector output
656 This simple example of multilingual chaining produces vector output if
657 there are spaces in the message and scalar otherwise.
661 #+srcname: msg-from-R(msg=msg-from-python)
663 paste(msg, "und R", sep=" ")
667 : org-babel speaks elisp y python und R
669 #+srcname: msg-from-python(msg=msg-from-elisp)
674 #+srcname: msg-from-elisp(msg="org-babel speaks")
675 #+begin_src emacs-lisp
676 (concat msg " elisp")
678 ** STARTED share org-babel [1/4]
679 how should we share org-babel?
681 *** DONE post to org-mode
682 *** TODO post to ess mailing list
683 *** TODO create a org-babel page on worg
684 *** TODO create a short screencast demonstrating org-babel in action
687 we need to think up some good examples
689 **** interactive tutorials
690 This could be a place to use [[* org-babel assertions][org-babel assertions]].
692 for example the first step of a tutorial could assert that the version
693 of the software-package (or whatever) is equal to some value, then
694 source-code blocks could be used with confidence (and executed
695 directly from) the rest of the tutorial.
697 **** answering a text-book question w/code example
698 org-babel is an ideal environment enabling both the development and
699 demonstrationg of the code snippets required as answers to many
702 **** something using tables
703 maybe something along the lines of calculations from collected grades
706 Maybe something like the following which outputs sizes of directories
707 under the home directory, and then instead of the trivial =emacs-lisp=
708 block we could use an R block to create a nice pie chart of the
712 #+begin_src bash :results replace
716 #+begin_src emacs-lisp :var sizes=sizes :results replace
720 ** TODO command line execution
721 Allow source code blocks to be called form the command line. This
722 will be easy using the =sbe= function in [[file:lisp/org-babel-table.el][org-babel-table.el]].
724 This will rely upon [[* resolve references to other buffers][resolve references to other buffers]].
726 ** TODO inline source code blocks [3/5]
727 Like the =\R{ code }= blocks
729 not sure what the format should be, maybe just something simple
730 like =src_lang[]{}= where lang is the name of the source code
731 language to be evaluated, =[]= is optional and contains any header
732 arguments and ={}= contains the code.
734 (see [[* (sandbox) inline source blocks][the-sandbox]])
736 *** DONE evaluation with \C-c\C-c
737 Putting aside the header argument issue for now we can just run these
738 with the following default header arguments
739 - =:results= :: silent
740 - =:exports= :: results
742 *** DONE inline exportation
743 Need to add an interblock hook (or some such) through org-exp-blocks
744 *** DONE header arguments
745 We should make it possible to use header arguments.
747 *** TODO fontification
748 we should color these blocks differently
750 *** TODO refine html exportation
751 should use a span class, and should show original source in tool-tip
753 ** TODO LoB: re-implement plotting and analysis functions from org-R
754 I'll do this soon, now that we things are a bit more settled and we
755 have column names in R.
756 ** PROPOSED Creating presentations
757 The [[mairix:t:@@9854.1246500519@gamaville.dokosmarshall.org][recent thread]] containing posts by Nick Dokos and Sebastian
758 Vaubán on exporting to beamer looked very interesting, but I
759 haven't had time to try it out yet. I would really like it if,
760 eventually, we can generate a presentation (with graphics generated
761 by code blocks) from the same org file that contains all the notes
762 and code etc. I just wanted that to be on record in this document;
763 I don't have anything more profound to say about it at the moment,
764 and I'm not sure to what extent it is an org-babel issue.
765 ** PROPOSED conversion between org-babel and noweb (e.g. .Rnw) format
766 I haven't thought about this properly. Just noting it down. What
767 Sweave uses is called "R noweb" (.Rnw).
769 I found a good description of noweb in the following article (see
770 the [[http://www.cs.tufts.edu/~nr/pubs/lpsimp.pdf][pdf]]).
772 I think there are two parts to noweb, the construction of
773 documentation and the extraction of source-code (with notangle).
775 *documentation*: org-mode handles all of our documentation needs in
776 a manner that I believe is superior to noweb.
778 *source extraction* At this point I don't see anyone writing large
779 applications with 100% of the source code contained in org-babel
780 files, rather I see org-babel files containing things like
781 - notes with active code chunks
782 - interactive tutorials
783 - requirements documents with code running test suites
784 - and of course experimental reports with the code to run the
785 experiment, and perform analysis
787 Basically I think the scope of the programs written in org-babel
788 (at least initially) will be small enough that it wont require the
789 addition of a tangle type program to extract all of the source code
790 into a running application.
792 On the other hand, since we already have named blocks of source
793 code which reference other blocks on which they rely, this
794 shouldn't be too hard to implement either on our own, or possibly
795 relying on something like noweb/notangle.
797 ** PROPOSED support for passing paths to files between source blocks
798 Maybe this should be it's own result type (in addition to scalars and
799 vectors). The reason being that some source-code blocks (for example
800 ditaa or anything that results in the creation of a file) may want to
801 pass a file path back to org-mode which could then be inserted into
802 the org-mode buffer as a link to the file...
804 This would allow for display of images upon export providing
805 functionality similar to =org-exp-blocks= only in a more general
807 ** DEFERRED Support rownames and other org babel table features?
809 The full org table features are detailed in the manual [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]].
812 Perhaps add a :rownames header arg. This would be an integer
813 (usually 1) which would have the effect of post-processing all the
814 variables created in the R session in the following way: if the
815 integer is j, set the row names to the contents of column j and
816 delete column j. Perhaps it is artificial to allow this integer to
817 take any value other than 1. The default would be nil which would
818 mean no such behaviour.
820 Actually I don't know about that. If multiple variables are passed
821 in, it's not appropriate to alter them all in the same way. The
822 rownames specification would normally refer to just one of the
823 variables. For now maybe just say this has to be done in R. E.g.
825 #+TBLNAME: sample-sizes
826 | collection | size | exclude | include | exclude2 | include2 |
827 |-----------------+------+---------+---------+----------+----------|
828 | 58C | 2936 | 8 | 2928 | 256 | 2680 |
829 | MS | 5852 | 771 | 5081 | 771 | 5081 |
830 | NBS | 2929 | 64 | 2865 | 402 | 2527 |
831 | POBI | 2717 | 1 | 2716 | 1 | 2716 |
832 | 58C+MS+NBS+POBI | | | 13590 | | 13004 |
833 #+TBLFM: @2$4=@2$2 - @2$3::@2$6=@2$2 - @2$5::@3$4=@3$2-@3$3::@3$6=@3$2 - @3$5::@4$4=@4$2 - @4$3::@4$6=@4$2 - @4$5::@5$4=@5$2-@5$3::@5$6=@5$2 - @5$5::@6$4=vsum(@2$4..@5$4)::@6$6=vsum(@2$6..@5$6)
835 #+srcname: make-size-table(size=sample-sizes)
837 rownames(size) <- size[,1]
843 [I don't think it's as problematic as this makes out]
844 This is non-trivial, but may be worth doing, in particular to
845 develop a nice framework for sending data to/from R.
847 In R, indexing vector elements, and rows and columns, using
848 strings rather than integers is an important part of the
850 - elements of a vector may have names
851 - matrices and data.frames may have "column names" and "row names"
852 which can be used for indexing
853 - In a data frame, row names *must* be unique
861 > mat <- matrix(1:4, nrow=2, ncol=2, dimnames=list(c("r1","r2"), c("c1","c2")))
866 > # The names are separate from the data: they do not interfere with operations on the data
873 > df <- data.frame(var1=1:26, var2=26:1, row.names=letters)
875 [1] 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
881 So it's tempting to try to provide support for this in org-babel. For example
882 - allow R to refer to columns of a :var reference by their names
883 - When appropriate, results from R appear in the org buffer with "named
886 However none (?) of the other languages we are currently supporting
887 really have a native matrix type, let alone "column names" or "row
888 names". Names are used in e.g. python and perl to refer to entries
891 It currently seems to me that support for this in org-babel would
892 require setting rules about when org tables are considered to have
893 named columns/fields, and ensuring that (a) languages with a notion
894 of named columns/fields use them appropriately and (b) languages
895 with no such notion do not treat then as data.
897 - Org allows something that *looks* like column names to be separated
899 - Org also allows a row to *function* as column names when special
900 markers are placed in the first column. An hline is unnecessary
901 (indeed hlines are purely cosmetic in org [correct?]
902 - Org does not have a notion of "row names" [correct?]
904 The full org table functionality exeplified [[http://orgmode.org/manual/Advanced-features.html#Advanced-features][here]] has features that
905 we would not support in e.g. R (like names for the row below).
907 **** Initial statement: allow tables with hline to be passed as args into R
908 This doesn't seem to work at the moment (example below). It would
909 also be nice to have a natural way for the column names of the org
910 table to become the column names of the R data frame, and to have
911 the option to specify that the first column is to be used as row
912 names in R (these must be unique). But this might require a bit of
917 | col1 | col2 | col3 |
918 |------+---------+------|
926 #+begin_src R :var tabel=egtable :colnames t
931 | "col1" | "col2" | "col3" |
932 |--------+-----------+--------|
934 | 4 | "schulte" | 6 |
937 Another example is in the [[*operations%20in%20on%20tables][grades example]].
939 ** DEFERRED use textConnection to pass tsv to R?
940 When passing args from the org buffer to R, the following route is
941 used: arg in buffer -> elisp -> tsv on file -> data frame in R. I
942 think it would be possible to avoid having to write to file by
943 constructing an R expression in org-babel-R-assign-elisp, something
946 #+begin_src emacs-lisp
947 (org-babel-R-input-command
948 (format "%s <- read.table(textConnection(\"%s\"), sep=\"\\t\", as.is=TRUE)"
949 name (orgtbl-to-tsv value '(:sep "\t" :fmt org-babel-R-quote-tsv-field))))
952 I haven't tried to implement this yet as it's basically just
953 fiddling with something that works. The only reason for it I can
954 think of would be efficiency and I haven't tested that.
956 This Didn't work after an initial test. I still think this is a
957 good idea (I also think we should try to do something similar when
958 writing out results frmo R to elisp) however as it wouldn't result
959 in any functional changes I'm bumping it down to deferred for
964 #+tblname: quick-test
967 #+srcname: quick-test-src-blk
968 #+begin_src R :var vec=quick-test
978 ** DEFERRED Rework Interaction with Running Processes [2/5]
979 *** DONE robust to errors interrupting execution
981 #+srcname: long-runner-ruby
982 #+begin_src ruby :results silent
987 *** DEFERRED use =C-g= keyboard-quit to push processing into the background
988 This may be possible using the `run-with-timer' command.
990 I have no idea how this could work...
992 #+srcname: long-runner-ruby
993 #+begin_src ruby :results silent
998 *** TODO ability to select which of multiple sessions is being used
999 Increasingly it is looking like we're going to want to run all
1000 source code blocks in comint buffer (sessions). Which will have
1002 1) allowing background execution
1003 2) maintaining state between source-blocks
1004 - allowing inline blocks w/o header arguments
1007 (like ess-switch-process in .R buffers)
1009 Maybe this could be packaged into a header argument, something
1010 like =:R_session= which could accept either the name of the
1011 session to use, or the string =prompt=, in which case we could use
1012 the =ess-switch-process= command to select a new process.
1014 *** TODO evaluation of shell code as background process?
1015 After C-c C-c on an R code block, the process may appear to
1016 block, but C-g can be used to reclaim control of the .org buffer,
1017 without interrupting the R evalution. However I believe this is not
1018 true of bash/sh evaluation. [Haven't tried other languages] Perhaps
1019 a solution is just to background the individual shell commands.
1021 The other languages (aside from emacs lisp) are run through the
1022 shell, so if we find a shell solution it should work for them as
1025 Adding an ampersand seems to be a supported way to run commands in
1026 the background (see [[http://www.emacswiki.org/emacs/ExecuteExternalCommand#toc4][external-commands]]). Although a more extensible
1027 solution may involve the use of the [[elisp:(progn (describe-function 'call-process-region) nil)][call-process-region]] function.
1029 Going to try this out in a new file [[file:lisp/org-babel-proc.el][org-babel-proc.el]]. This should
1030 contain functions for asynchronously running generic shell commands
1031 in the background, and then returning their input.
1033 **** partial update of org-mode buffer
1034 The sleekest solution to this may be using a comint buffer, and
1035 then defining a filter function which would incrementally interpret
1036 the results as they are returned, including insertion into the
1037 org-mode buffer. This may actually cause more problems than it is
1038 worth, what with the complexities of identifying the types of
1039 incrementally returned results, and the need for maintenance of a
1040 process marker in the org buffer.
1042 **** 'working' spinner
1043 It may be nice and not too difficult to place a spinner on/near the
1044 evaluating source code block
1046 *** TODO conversion of output from interactive shell, R (and python) sessions to org-babel buffers
1047 [DED] This would be a nice feature I think. Although an org-babel
1048 purist would say that it's working the wrong way round... After
1049 some interactive work in a *R* buffer, you save the buffer, maybe
1050 edit out some lines, and then convert it to org-babel format for
1051 posterity. Same for a shell session either in a *shell* buffer, or
1052 pasted from another terminal emulator. And python of course.
1054 ** DEFERRED improve the source-block snippet
1055 any real improvement seems somewhat beyond the ability of yasnippet
1058 [[file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name%20Chapter%20title][file:~/src/emacs-starter-kit/src/snippets/text-mode/rst-mode/chap::name Chapter title]]
1060 ,#name : Chapter title
1063 ${1:$(make-string (string-width text) ?\=)}
1068 [[file:snippets/org-mode/sb][sb -- snippet]]
1070 waiting for guidance from those more familiar with yasnippets
1072 ** REJECTED re-implement R evaluation using ess-command or ess-execute
1073 I don't have any complaints with the current R evaluation code or
1074 behaviour, but I think it would be good to use the ESS functions
1075 from a political point of view. Plus of course it has the normal
1076 benefits of an API (insulates us from any underlying changes etc). [DED]
1078 I'll look into this. I believe that I looked at and rejected these
1079 functions initially but now I can't remember why. I agree with
1080 your overall point about using API's where available. I will take
1081 a look back at these and either switch to using the ess commands,
1082 or at least articulate under this TODO the reasons for using our
1083 custom R-interaction commands. [Eric]
1087 Lets just replace =org-babel-R-input-command= with =ess-execute=.
1089 I tried this, and although it works in some situations, I find that
1090 =ess-command= will often just hang indefinitely without returning
1091 results. Also =ess-execute= will occasionally hang, and pops up
1092 the buffer containing the results of the command's execution, which
1093 is undesirable. For now these functions can not be used. Maybe
1094 someone more familiar with the ESS code can recommend proper usage
1095 of =ess-command= or some other lower-level function which could be
1096 used in place of [[file:lisp/org-babel-R.el::defun%20org-babel%20R%20input%20command%20command][org-babel-R-input-command]].
1100 #+begin_quote ess-command
1101 (ess-command COM &optional BUF SLEEP NO-PROMPT-CHECK)
1103 Send the ESS process command COM and delete the output
1104 from the ESS process buffer. If an optional second argument BUF exists
1105 save the output in that buffer. BUF is erased before use.
1106 COM should have a terminating newline.
1107 Guarantees that the value of .Last.value will be preserved.
1108 When optional third arg SLEEP is non-nil, `(sleep-for (* a SLEEP))'
1109 will be used in a few places where `a' is proportional to `ess-cmd-delay'.
1112 #+begin_quote ess-execute
1113 (ess-execute COMMAND &optional INVERT BUFF MESSAGE)
1115 Send a command to the ESS process.
1116 A newline is automatically added to COMMAND. Prefix arg (or second arg
1117 INVERT) means invert the meaning of
1118 `ess-execute-in-process-buffer'. If INVERT is 'buffer, output is
1119 forced to go to the process buffer. If the output is going to a
1120 buffer, name it *BUFF*. This buffer is erased before use. Optional
1121 fourth arg MESSAGE is text to print at the top of the buffer (defaults
1122 to the command if BUFF is not given.)
1125 *** out current setup
1127 1) The body of the R source code block is wrapped in a function
1128 2) The function is called inside of a =write.table= function call
1129 writing the results to a table
1130 3) The table is read using =org-table-import=
1132 ** DONE take default values for header args from properties
1133 Use file-wide and subtree wide properties to set default values for
1136 [DED] One thing I'm finding when working with R is that an org file
1137 may contain many source blocks, but that I just want to evaluate a
1138 subset of them. Typically this is in order to take up where I left
1139 off: I need to recreate a bunch of variables in the session
1140 environment. I'm thinking maybe we want to use a tag-based
1141 mechanism similar to :export: and :noexport: to control evaluation
1142 on a per-subtree basis.
1144 *** test-header with properties
1150 Ahh... as is so often the case, just had to wrap
1151 `org-babel-params-from-properties' in a `save-match-data' form.
1153 #+tblname: why-def-props-cause-probs
1156 #+srcname: default-props-implementation
1157 #+begin_src emacs-lisp :tangle no :var my-lis=why-def-props-cause-probs :results silent
1158 (+ (length my-lis) def)
1161 ** DONE new reference syntax *inside* source code blocks
1162 This is from an email discussion on the org-mode mailing list with
1163 Sébastien. The goal here is to mimic the source-block reference style
1164 of Noweb. Upon export and/or tangle these references could be
1165 replaced with the actual body of the referenced source-code block.
1167 See the following for an example.
1169 #+srcname: ems-ruby-print-header
1171 puts "---------------------------header---------------------------"
1174 #+srcname: emacs-ruby-print-footer
1176 puts "---------------------------footer---------------------------"
1179 #+srcname: ems-ruby-print-message
1180 #+begin_src ruby :file ruby-noweb.rb
1181 # <<ems-ruby-print-header>>
1183 # <<ems-ruby-print-footer>>
1186 Upon export the previous source-code block would result in a file
1187 being generated at =ruby-noweb.rb= with the following contents
1189 : puts "---------------------------header---------------------------"
1191 : puts "---------------------------footer---------------------------"
1193 the body of a source-code block with all =<<src-name>>= references
1194 expanded can now be returned by `org-babel-expand-noweb-references'.
1195 This function is now called by default on all source-code blocks on
1198 ** DONE re-work tangling system
1199 Sometimes when tangling a file (e.g. when extracting elisp from a
1200 org-mode file) we want to get nearly every source-code block.
1202 Sometimes we want to only extract those source-code blocks which
1203 reference a indicate that they should be extracted (e.g. traditional
1204 literate programming along the Noweb model)
1206 I'm not sure how we can devise a single simple tangling system that
1207 naturally fits both of these use cases.
1210 the =tangle= header argument will default to =no= meaning source-code
1211 blocks will *not* be exported by default. In order for a source-code
1212 block to be tangled it needs to have an output file specified. This
1213 can happen in two ways...
1215 1) a file-wide default output file can be passed to `org-babel-tangle'
1216 which will then be used for all blocks
1217 2) if the value of the =tangle= header argument is anything other than
1218 =no= or =yes= then it is used as the file name
1220 #+srcname: test-new-tangling
1221 #+begin_src emacs-lisp
1222 (org-babel-load-file "test-tangle.org")
1223 (if (string= test-tangle-advert "use org-babel-tangle for all your emacs initialization files!!")
1231 ** DONE =\C-c \C-o= to open results of source block
1232 by adding a =defadvice= to =org-open-at-point= we can use the common
1233 =\C-c \C-o= keybinding to open the results of a source-code block.
1234 This would be especially useful for source-code blocks which generate
1235 graphical results and insert a file link as the results in the
1236 org-mode buffer. (see [[* figure out how to handle graphic output][TODO figure out how to handle graphic output]]).
1237 This could also act reasonably with other results types...
1239 - file :: use org-open-at-point to open the file
1240 - scalar :: open results unquoted in a new buffer
1241 - tabular :: export the table to a new buffer and open that buffer
1243 when called with a prefix argument the block is re-run
1245 #+srcname: task-opening-results-of-blocks
1246 #+begin_src ditaa :results replace :file blue.png :cmdline -r
1257 [[file:blue.png][blue.png]]
1259 #+srcname: task-open-vector
1260 #+begin_src emacs-lisp
1268 #+srcname: task-open-scalar
1269 #+begin_src ruby :results output
1285 ** DONE Stop spaces causing vector output
1286 This simple example of multilingual chaining produces vector output if
1287 there are spaces in the message and scalar otherwise.
1291 #+srcname: msg-from-R(msg=msg-from-python)
1293 paste(msg, "und R", sep=" ")
1297 : org-babel speaks elisp y python und R
1299 #+srcname: msg-from-python(msg=msg-from-elisp)
1304 #+srcname: msg-from-elisp(msg="org-babel speaks")
1305 #+begin_src emacs-lisp
1306 (concat msg " elisp")
1309 ** DONE add =:tangle= family of header arguments
1311 - no :: don't include source-code block when tangling
1312 - yes :: do include source-code block when tangling
1314 this is tested in [[file:test-tangle.org::*Emacs%20Lisp%20initialization%20stuff][test-tangle.org]]
1316 ** DONE extensible library of callable source blocks
1318 This is covered by the [[file:library-of-babel.org][Library of Babel]], which will contain
1319 ready-made source blocks designed to carry out useful common tasks.
1320 *** Initial statement [Eric]
1321 Much of the power of org-R seems to be in it's helper functions for
1322 the quick graphing of tables. Should we try to re-implement these
1323 functions on top of org-babel?
1325 I'm thinking this may be useful both to add features to org-babel-R and
1326 also to potentially suggest extensions of the framework. For example
1327 one that comes to mind is the ability to treat a source-code block
1328 like a function which accepts arguments and returns results. Actually
1329 this can be it's own TODO (see [[* source blocks as functions][source blocks as functions]]).
1330 *** Objectives [Dan]
1331 - We want to provide convenient off-the-shelf actions
1332 (e.g. plotting data) that make use of our new code evaluation
1333 environment but do not require any actual coding.
1334 *** Initial Design proposal [Dan]
1335 - *Input data* will be specified using the same mechanism as :var
1336 references, thus the input data may come from a table, or
1337 another source block, and it is initially available as an elisp
1339 - We introduce a new #+ line, e.g. #+BABELDO. C-c C-c on that
1340 line will apply an *action* to the referenced data.
1341 - *Actions correspond to source blocks*: our library of available
1342 actions will be a library of org-babel source blocks. Thus the
1343 code for executing an action, and the code for dealing with the
1344 output of the action will be the same code as for executing
1345 source blocks in general
1346 - Optionally, the user can have the relevant source block inserted
1347 into the org buffer after the (say) #+BABELDO line. This will
1348 allow the user to fine tune the action by modifying the code
1349 (especially useful for plots).
1350 - So maybe a #+BABELDO line will have header args
1351 - :data (a reference to a table or source code block)
1352 - :action (or should that be :srcname?) which will be something
1353 like :action pie-chart, referring to a source block which will
1354 be executed with the :data referent passed in using a :var arg.
1355 - :showcode or something controlling whether to show the code
1357 *** Modification to design
1358 I'm implementing this, at least initially, as a new interpreter
1359 named 'babel', which has an empty body. 'babel' blocks take
1360 a :srcname header arg, and look for the source-code block with
1361 that name. They then execute the referenced block, after first
1362 appending their own header args on to the target block's header
1365 If the target block is in the library of babel (a.o.t. e.g. the
1366 current buffer), then the code in the block will refer to the
1367 input data with a name dictated by convention (e.g. __data__
1368 (something which is syntactically legal in all languages...). Thus
1369 the babel block will use a :var __data__ = whatever header arg to
1370 reference the data to be plotted.
1372 ** DONE Column names in R input/output
1373 This has been implemented: Automatic on input to R; optional in
1374 output. Note that this equates column names with the header row in
1375 an org table; whereas org actually has a mechanism whereby a row
1376 with a '!' in the first field defines column names. I have not
1377 attempted to support these org table mechanisms yet. See [[*Support%20rownames%20and%20other%20org%20babel%20table%20features][this
1378 DEFERRED todo item]].
1379 ** DONE use example block for large amounts of stdout output?
1380 We're currently `examplizing' with : at the beginning of the line,
1381 but should larger amounts of output be in a
1382 \#+begin_example...\#+end_example block? What's the cutoff? > 1
1383 line? This would be nice as it would allow folding of lengthy
1384 output. Sometimes one will want to see stdout just to check
1385 everything looks OK, and then fold it away.
1387 I'm addressing this in branch 'examplizing-output'.
1388 Yea, that makes sense. (either that or allow folding of large
1389 blocks escaped with =:=).
1391 Proposed cutoff of 10 lines, we can save this value in a user
1392 customizable variable.
1393 *** DONE add ability to remove such results
1394 ** DONE exclusive =exports= params
1396 #+srcname: implement-export-exclusivity
1403 ** DONE LoB: allow output in buffer
1404 ** DONE allow default header arguments by language
1405 org-babel-default-header-args:lang-name
1407 An example of when this is useful is for languages which always return
1408 files as their results (e.g. [[*** ditaa][ditaa]], and [[*** gnuplot][gnuplot]]).
1409 ** DONE singe-function tangling and loading elisp from literate org-mode file [3/3]
1411 This function should tangle the org-mode file for elisp, and then call
1412 `load-file' on the resulting tangled file.
1414 #+srcname: test-loading-embedded-emacs-lisp
1415 #+begin_src emacs-lisp :results replace
1416 (setq test-tangle-advert nil)
1417 (setq test-tangle-loading nil)
1418 (setq results (list :before test-tangle-loading test-tangle-advert))
1419 (org-babel-load-file "test-tangle.org")
1420 (setq results (list (list :after test-tangle-loading test-tangle-advert) results))
1421 (delete-file "test-tangle.el")
1425 #+resname: test-loading-embedded-emacs-lisp
1426 | :before | nil | nil |
1427 | :after | "org-babel tangles" | "use org-babel-tangle for all your emacs initialization files!!" |
1429 *** DONE add optional language limiter to org-babel-tangle
1430 This should check to see if there is any need to re-export
1432 *** DONE ensure that org-babel-tangle returns the path to the tangled file(s)
1434 #+srcname: test-return-value-of-org-babel-tangle
1435 #+begin_src emacs-lisp :results replace
1436 (mapcar #'file-name-nondirectory (org-babel-tangle-file "test-tangle.org" "emacs-lisp"))
1440 | "test-tangle.el" |
1442 *** DONE only tangle the file if it's actually necessary
1443 ** DONE add a function to jump to a source-block by name
1444 I've had an initial stab at that in org-babel-find-named-block
1445 (library-of-babel branch).
1447 At the same time I introduced org-babel-named-src-block-regexp, to
1448 match src-blocks with srcname.
1450 This is now working with the command
1451 `org-babel-goto-named-source-block', all we need is a good key
1454 ** DONE add =:none= session argument (for purely functional execution) [4/4]
1455 This would allow source blocks to be run in their own new process
1457 - These blocks could then also be run in the background (since we can
1458 detach and just wait for the process to signal that it has terminated)
1459 - We wouldn't be drowning in session buffers after running the tests
1460 - we can re-use much of the session code to run in a more /functional/
1463 While session provide a lot of cool features, like persistent
1464 environments, [[* DONE function to bring up inferior-process buffer][pop-to-session]], and hints at exportation for
1465 org-babel-tangle, they also have some down sides and I'm thinking that
1466 session-based execution maybe shouldn't be the default behavior.
1468 Down-sides to sessions
1469 - *much* more complicated than functional evaluation
1470 - maintaining the state of the session has weird issues
1471 - waiting for evaluation to finish
1472 - prompt issues like [[* TODO weird escaped characters in shell prompt break shell evaluation][shell-prompt-escapes-bug]]
1473 - can't run in background
1474 - litter emacs with session buffers
1478 #+srcname: ruby-task-no-session
1479 #+begin_src ruby :results replace output
1485 #+resname: ruby-task-no-session
1490 #+srcname: task-python-none-session
1491 #+begin_src python :session none :results replace value
1497 #+resname: task-python-none-session
1502 #+srcname: task-session-none-sh
1503 #+begin_src sh :results replace
1508 #+resname: task-session-none-sh
1514 #+srcname: task-no-session-R
1515 #+begin_src R :results replace output
1522 #+resname: task-no-session-R
1526 ** DONE fully purge org-babel-R of direct comint interaction
1527 try to remove all code under the [[file:lisp/org-babel-R.el::functions%20for%20evaluation%20of%20R%20code][;; functions for evaluation of R code]] line
1529 ** DONE Create objects in top level (global) environment [5/5]
1532 *** initial requirement statement [DED]
1533 At the moment, objects created by computations performed in the
1534 code block are evaluated in the scope of the
1535 code-block-function-body and therefore disappear when the code
1536 block is evaluated {unless you employ some extra trickery like
1537 assign('name', object, env=globalenv()) }. I think it will be
1538 desirable to also allow for a style wherein objects that are
1539 created in one code block persist in the R global environment and
1540 can be re-used in a separate block.
1542 This is what Sweave does, and while I'm not saying we have to be
1543 the same as Sweave, it wouldn't be hard for us to provide the same
1544 behaviour in this case; if we don't, we risk undeservedly being
1545 written off as an oddity by some.
1547 IOW one aspect of org-babel is that of a sort of functional
1548 meta-programming language. This is crazy, in a very good
1549 way. Nevertheless, wrt R I think there's going to be a lot of value
1550 in providing for a working style in which the objects are stored in
1551 the R session, rather than elisp/org buffer. This will be a very
1552 familiar working style to lots of people.
1554 There are no doubt a number of different ways of accomplishing
1555 this, the simplest being a hack like adding
1558 for(objname in ls())
1559 assign(objname, get(objname), envir=globalenv())
1562 to the source code block function body. (Maybe wrap it in an on.exit() call).
1564 However this may deserve to be thought about more carefully, perhaps
1565 with a view to having a uniform approach across languages. E.g. shell
1566 code blocks have the same semantics at the moment (no persistence of
1567 variables across code blocks), because the body is evaluated in a new
1568 bash shell process rather than a running shell. And I guess the same
1569 is true for python. However, in both these cases, you could imagine
1570 implementing the alternative in which the body is evaluated in a
1571 persistent interactive session. It's just that it's particularly
1572 natural for R, seeing as both ESS and org-babel evaluate commands in a
1573 single persistent R session.
1577 Thanks for bringing this up. I think you are absolutely correct that we
1578 should provide support for a persistent environment (maybe called a
1579 *session*) in which to evaluate code blocks. I think the current setup
1580 demonstrates my personal bias for a functional style of programming
1581 which is certainly not ideal in all contexts.
1583 While the R function you mention does look like an elegant solution, I
1584 think we should choose an implementation that would be the same across
1585 all source code types. Specifically I think we should allow the user to
1586 specify an optional *session* as a header variable (when not present we
1587 assume a default session for each language). The session name could be
1588 used to name a comint buffer (like the *R* buffer) in which all
1589 evaluation would take place (within which variables would retain their
1590 values --at least once I remove some of the functional method wrappings
1591 currently in place-- ).
1593 This would allow multiple environments to be used in the same buffer,
1594 and once this setup was implemented we should be able to fairly easily
1595 implement commands for jumping between source code blocks and the
1596 related session buffers, as well as for dumping the last N commands from
1597 a session into a new or existing source code block.
1599 Please let me know if you foresee any problems with this proposed setup,
1600 or if you think any parts might be confusing for people coming from
1601 Sweave. I'll hopefully find some time to work on this later in the
1604 *** can functional and interpreted/interactive models coexist?
1606 Even though both of these use the same =*R*= buffer the value of =a=
1607 is not preserved because it is assigned inside of a functional
1610 #+srcname: task-R-sessions
1617 #+srcname: task-R-same-session
1622 This functional wrapper was implemented in order to efficiently return
1623 the results of the execution of the entire source code block. However
1624 it inhibits the evaluation of source code blocks in the top level,
1625 which would allow for persistence of variable assignment across
1626 evaluations. How can we allow *both* evaluation in the top level, and
1627 efficient capture of the return value of an entire source code block
1628 in a language independent manner?
1630 Possible solutions...
1631 1) we can't so we will have to implement two types of evaluation
1632 depending on which is appropriate (functional or imperative)
1633 2) we remove the functional wrapper and parse the source code block
1634 into it's top level statements (most often but not always on line
1635 breaks) so that we can isolate the final segment which is our
1637 3) we add some sort of "#+return" line to the code block
1638 4) we take advantage of each languages support for meta-programming
1639 through =eval= type functions, and use said to evaluate the entire
1640 blocks in such a way that their environment can be combined with the
1641 global environment, and their results are still captured.
1642 5) I believe that most modern languages which support interactive
1643 sessions have support for a =last_result= type function, which
1644 returns the result of the last input without re-calculation. If
1645 widely enough present this would be the ideal solution to a
1646 combination of functional and imperative styles.
1648 None of these solutions seem very desirable, but for now I don't see
1649 what else would be possible.
1651 Of these options I was leaning towards (1) and (4) but now believe
1652 that if it is possible option (5) will be ideal.
1654 **** (1) both functional and imperative evaluation
1656 - can take advantage of built in functions for sending regions to the
1658 - retains the proven tested and working functional wrappers
1661 - introduces the complication of keeping track of which type of
1662 evaluation is best suited to a particular context
1663 - the current functional wrappers may require some changes in order to
1664 include the existing global context
1666 **** (4) exploit language meta-programming constructs to explicitly evaluate code
1668 - only one type of evaluation
1671 - some languages may not have sufficient meta-programming constructs
1673 **** (5) exploit some =last_value= functionality if present
1675 Need to ensure that most languages have such a function, those without
1676 will simply have to implement their own similar solution...
1678 | language | =last_value= function |
1679 |------------+-----------------------------|
1683 | shell | see [[* last command for shells][last command for shells]] |
1684 | emacs-lisp | see [[* emacs-lisp will be a special case][special-case]] |
1686 #+srcname: task-last-value
1691 ***** last command for shells
1692 Do this using the =tee= shell command, and continually pipe the output
1695 Got this idea from the following [[http://linux.derkeiler.com/Mailing-Lists/Fedora/2004-01/0898.html][email-thread]].
1697 suggested from mailing list
1699 #+srcname: bash-save-last-output-to-file
1703 bash -c "$line" | tee /tmp/last.out1
1704 mv /tmp/last.out1 /tmp/last.out
1708 another proposed solution from the above thread
1710 #+srcname: bash-save-in-variable
1713 # so - Save Output. Saves output of command in OUT shell variable.
1723 "^M": " | tee /tmp/h_lastcmd.out ^[k"
1726 export __=/tmp/h_lastcmd.out
1728 If you try it, Alt-k will stand for the old Enter; use "command $__" to
1729 access the last output.
1735 Herculano de Lima Einloft Neto
1738 ***** emacs-lisp will be a special case
1739 While it is possible for emacs-lisp to be run in a console type
1740 environment (see the =elim= function) it is *not* possible to run
1741 emacs-lisp in a different *session*. Meaning any variable set top
1742 level of the console environment will be set *everywhere* inside
1743 emacs. For this reason I think that it doesn't make any sense to
1744 worry about session support for emacs-lisp.
1746 *** Further thoughts on 'scripting' vs. functional approaches
1748 These are just thoughts, I don't know how sure I am about this.
1749 And again, perhaps I'm not saying anything very radical, just that
1750 it would be nice to have some options supporting things like
1751 receiving text output in the org buffer.
1753 I can see that you've already gone some way down the road towards
1754 the 'last value' approach, so sorry if my comments come rather
1755 late. I am concerned that we are not giving sufficient attention
1756 to stdout / the text that is returned by the interpreters. In
1757 contrast, many of our potential users will be accustomed to a
1758 'scripting' approach, where they are outputting text at various
1759 points in the code block, not just at the end. I am leaning
1760 towards thinking that we should have 2 modes of evaluation:
1761 'script' mode, and 'functional' mode.
1763 In script mode, evaluation of a code block would result in *all*
1764 text output from that code block appearing as output in the org
1765 buffer, presumably as an #+begin_example...#+end_example. There
1766 could be an :echo option controlling whether the input commands
1767 also appear in the output. [This is like Sweave].
1769 In functional mode, the *result* of the code block is available as
1770 an elisp object, and may appear in the org buffer as an org
1771 table/string, via the mechanisms you have developed already.
1773 One thing I'm wondering about is whether, in script mode, there
1774 simply should not be a return value. Perhaps this is not so
1775 different from what exists: script mode would be new, and what
1776 exists currently would be functional mode.
1778 I think it's likely that, while code evaluation will be exciting
1779 to people, a large majority of our users in a large majority of
1780 their usage will not attempt to actually use the return value from
1781 a source code block in any meaningful way. In that case, it seems
1782 rather restrictive to only allow them to see output from the end
1785 Instead I think the most accessible way to introduce org-babel to
1786 people, at least while they are learning it, is as an immensely
1787 powerful environment in which to embed their 'scripts', which now
1788 also allows them to 'run' their 'scripts'. Especially as such
1789 people are likely to be the least capable of the user-base, a
1790 possible design-rule would be to make the scripting style of usage
1791 easy (default?), perhaps requiring a special option to enable a
1792 functional style. Those who will use the functional style won't
1793 have a problem understanding what's going on, whereas the 'skript
1794 kiddies' might not even know the syntax for defining a function in
1795 their language of choice. And of course we can allow the user to
1796 set a variable in their .emacs controlling the preference, so that
1797 functional users are not inconveniennced by having to provide
1798 header args the whole time.
1800 Please don't get the impression that I am down-valuing the
1801 functional style of org-babel. I am constantly horrified at the
1802 messy 'scripts' that my colleagues produce in perl or R or
1803 whatever! Nevertheless that seems to be how a lot of people work.
1805 I think you were leaning towards the last-value approach because
1806 it offered the possibility of unified code supporting both the
1807 single evaluation environment and the functional style. If you
1808 agree with any of the above then perhaps it will impact upon this
1809 and mean that the code in the two branches has to differ a bit. In
1810 that case, functional mode could perhaps after all evaluate each
1811 code block in its own environment, thus (re)approaching 'true'
1812 functional programming (side-effects are hard to achieve).
1816 echo "There are `wc -l files` files in this directory"
1820 *** even more thoughts on evaluation, results, models and options
1822 Thanks Dan, These comments are invaluable.
1824 What do you think about this as a new list of priorities/requirements
1825 for the execution of source-code blocks.
1828 1) we want the evaluation of the source code block to take place in a
1829 session which can persist state (variables, current directory,
1831 2) source code blocks can specify their session with a header argument
1832 3) each session should correspond to an Emacs comint buffer so that the
1833 user can drop into the session and experiment with live code
1836 1) each source-code block generates some form of results which (as
1837 we have already implemented) is transfered into emacs-lisp
1838 after which it can be inserted into the org-mode buffer, or
1839 used by other source-code blocks
1840 2) when the results are translated into emacs-lisp, forced to be
1841 interpreted as a scalar (dumping their raw values into the
1842 org-mode buffer), as a vector (which is often desirable with R
1843 code blocks), or interpreted on the fly (the default option).
1844 Note that this is very nearly currently implemented through the
1845 [[* DONE results-type header (vector/file)][results-type-header]].
1846 3) there should be *two* means of collecting results from the
1847 execution of a source code block. *Either* the value of the
1848 last statement of the source code block, or the collection of
1849 all that has been passed to STDOUT during the evaluation.
1851 **** header argument or return line (*header argument*)
1853 Rather than using a header argument to specify how the return value
1854 should be passed back, I'm leaning towards the use of a =#+RETURN=
1855 line inside the block. If such a line *is not present* then we
1856 default to using STDOUT to collect results, but if such a line *is
1857 present* then we use it's value as the results of the block. I
1858 think this will allow for the most elegant specification between
1859 functional and script execution. This also cleans up some issues
1860 of implementation and finding which statement is the last
1863 Having given this more thought, I think a header argument is
1864 preferable. The =#+return:= line adds new complicating syntax for
1865 something that does little more than we would accomplish through
1866 the addition of a header argument. The only benefit being that we
1867 know where the final statement starts, which is not an issue in
1868 those languages which contain 'last value' operators.
1870 new header =:results= arguments
1871 - script :: explicitly states that we want to use STDOUT to
1872 initialize our results
1873 - return_last :: stdout is ignored instead the *value* of the final
1874 statement in the block is returned
1875 - echo :: means echo the contents of the source-code block along
1876 with the results (this implies the *script* =:results=
1879 *** DONE rework evaluation lang-by-lang [4/4]
1881 This should include...
1882 - functional results working with the comint buffer
1884 - script :: return the output of STDOUT
1885 - write a macro which runs the first redirection, executes the
1886 body, then runs the second redirection
1887 - last :: return the value of the last statement
1890 - sessions in comint buffers
1892 **** DONE Ruby [4/4]
1893 - [X] functional results working with comint
1894 - [X] script results
1895 - [X] ensure scalar/vector results args are taken into consideration
1896 - [X] ensure callable by other source block
1898 #+srcname: ruby-use-last-output
1899 #+begin_src ruby :results replace
1906 #+resname: ruby-use-last-output
1909 #+srcname: task-call-use-last-output
1910 #+begin_src ruby :var last=ruby-use-last-output :results replace
1911 last.flatten.size + 1
1914 #+resname: task-call-use-last-output
1919 #+srcname: first-ruby-session-task
1920 #+begin_src ruby :session schulte :results silent
1924 #+srcname: second-ruby-session-task
1925 #+begin_src ruby :session schulte :results silent
1929 #+srcname: without-the-right-session
1930 #+begin_src ruby :results silent
1936 - [X] functional results working with comint
1937 - [X] script results
1938 - [X] ensure scalar/vector results args are taken into consideration
1939 - [X] ensure callable by other source block
1941 To redirect output to a file, you can use the =sink()= command.
1944 #+begin_src R :results value vector silent
1951 #+srcname: task-R-use-other-output
1952 #+begin_src R :var twoentyseven=task_R_B() :results replace value
1957 #+resname: task-R-use-other-output
1960 **** DONE Python [4/4]
1961 - [X] functional results working with comint
1962 - [X] script results
1963 - [X] ensure scalar/vector results args are taken into consideration
1964 - [X] ensure callable by other source block
1966 #+srcname: task-new-eval-for-python
1967 #+begin_src python :results silent output scalar
1973 #+srcname: task-use-new-eval
1974 #+begin_src python :var tasking=task-new-eval-for-python() :results replace
1978 #+resname: task-use-new-eval
1981 **** DONE Shells [4/4]
1982 - [X] functional results working with comint
1983 - [X] script results
1984 - [X] ensure scalar/vector results args are taken into consideration
1985 - [X] ensure callable by other source block
1987 #+srcname: task-shell-new-evaluation
1988 #+begin_src sh :results silent value scalar
1993 #+srcname: task-call-other-shell
1994 #+begin_src sh :var other=task-shell-new-evaluation() :results replace scalar
1995 echo $other ' is the old date'
1998 #+resname: task-call-other-shell
1999 : $ Fri Jun 12 13:08:37 PDT 2009 is the old date
2001 *** DONE implement a *session* header argument [4/4]
2002 =:session= header argument to override the default *session* buffer
2006 #+srcname: task-ruby-named-session
2007 #+begin_src ruby :session schulte :results replace
2008 schulte = :in_schulte
2011 #+resname: task-ruby-named-session
2014 #+srcname: another-in-schulte
2015 #+begin_src ruby :session schulte
2019 #+resname: another-in-schulte
2026 #+srcname: python-session-task
2027 #+begin_src python :session what :results silent
2031 #+srcname: python-get-from-session
2032 #+begin_src python :session what :results replace
2036 #+resname: python-get-from-session
2041 #+srcname: task-shell-sessions
2042 #+begin_src sh :session what
2046 #+srcname: task-shell-sessions-what
2047 #+begin_src sh :session what :results replace
2051 #+resname: task-shell-sessions-what
2056 #+srcname: task-R-session
2057 #+begin_src R :session what :results replace
2063 #+resname: task-R-session
2066 #+srcname: another-task-R-session
2067 #+begin_src R :session what :results replace
2071 *** DONE function to bring up inferior-process buffer [4/4]
2073 This should be callable from inside of a source-code block in an
2074 org-mode buffer. It should evaluate the header arguments, then bring
2075 up the inf-proc buffer using =pop-to-buffer=.
2077 For lack of a better place, lets add this to the `org-metadown-hook'
2080 To give this a try, place the cursor on a source block with variables,
2081 (optionally git a prefix argument) then hold meta and press down.
2085 #+srcname: task-ruby-pop-to-session
2086 #+begin_src ruby :var num=9 :var another="something else"
2087 num.times{|n| puts another}
2092 #+srcname: task-python-pop-to-session
2093 #+begin_src python :var num=9 :var another="something else"
2098 #+srcname: task-R-pop-to-session
2099 #+begin_src R :var a=9 :var b=8
2105 #+srcname: task-shell-pop-sessions
2106 #+begin_src sh :var NAME="eric"
2110 *** DEFERRED function to dump last N lines from inf-proc buffer into the current source block
2112 Callable with a prefix argument to specify how many lines should be
2113 dumped into the source-code buffer.
2115 *** REJECTED comint notes
2117 Implementing comint integration in [[file:lisp/org-babel-comint.el][org-babel-comint.el]].
2120 - handling of outputs
2121 - split raw output from process by prompts
2122 - a ring of the outputs, buffer-local, `org-babel-comint-output-ring'
2123 - a switch for dumping all outputs to a buffer
2124 - inputting commands
2126 Lets drop all this language specific stuff, and just use
2127 org-babel-comint to split up our outputs, and return either the last
2128 value of an execution or the combination of values from the
2131 **** comint filter functions
2132 : ;; comint-input-filter-functions hook process-in-a-buffer
2133 : ;; comint-output-filter-functions hook function modes.
2134 : ;; comint-preoutput-filter-functions hook
2135 : ;; comint-input-filter function ...
2137 #+srcname: obc-filter-ruby
2138 #+begin_src ruby :results last
2146 ** DONE Remove protective commas from # comments before evaluating
2147 org inserts protective commas in front of ## comments in language
2148 modes that use them. We need to remove them prior to sending code
2151 #+srcname: testing-removal-of-protective-comas
2153 ,# this one might break it??
2157 ** DONE pass multiple reference arguments into R
2158 Can we do this? I wasn't sure how to supply multiple 'var' header
2159 args. Just delete this if I'm being dense.
2161 This should be working, see the following example...
2163 #+srcname: two-arg-example
2164 #+begin_src R :var n=2 :var m=8
2168 #+resname: two-arg-example
2171 ** DONE ensure that table ranges work
2172 when a table range is passed to org-babel as an argument, it should be
2173 interpreted as a vector.
2176 | 2 | 3 | Fixnum:1 |
2177 | 3 | 4 | Array:123456 |
2181 #+TBLFM: @1$3='(sbe simple-sbe-example (n 4))::@2$3='(sbe task-table-range (n @1$1..@6$1))::@3$3='(sbe task-table-range (n (@1$1..@6$1)))
2183 #+srcname: simple-sbe-example
2184 #+begin_src emacs-lisp
2188 #+srcname: task-table-range
2189 #+begin_src ruby :var n=simple-sbe-example
2193 #+srcname: simple-results
2194 #+begin_src emacs-lisp :var n=task-table-range(n=(1 2 3))
2198 #+resname: simple-results
2201 #+srcname: task-arr-referent
2202 #+begin_src ruby :var ar=(1 2 3)
2206 #+resname: task-arr-referent
2209 ** DONE global variable indicating default to vector output
2210 how about an alist... =org-babel-default-header-args= this may already
2211 exist... just execute the following and all source blocks will default
2214 #+begin_src emacs-lisp
2215 (setq org-babel-default-header-args '((:results . "vector")))
2218 ** DONE name named results if source block is named
2219 currently this isn't happening although it should be
2221 #+srcname: test-naming-named-source-blocks
2222 #+begin_src emacs-lisp
2226 #+resname: test-naming-named-source-blocks
2228 ** DONE (simple caching) check for named results before source blocks
2229 see the TODO comment in [[file:lisp/org-babel-ref.el::TODO%20This%20should%20explicitly%20look%20for%20resname%20lines%20before][org-babel-ref.el#org-babel-ref-resolve-reference]]
2230 ** DONE set =:results silent= when eval with prefix argument
2232 #+begin_src emacs-lisp
2235 ** DONE results-type header (vector/file) [3/3]
2236 In response to a point in Dan's email. We should allow the user to
2237 force scalar or vector results. This could be done with a header
2238 argument, and the default behavior could be controlled through a
2239 configuration variable.
2241 #+srcname: task-trivial-vector
2242 #+begin_src ruby :results replace vector
2249 since it doesn't make sense to turn a vector into a scalar, lets
2250 just add a two values...
2252 - vector :: forces the results to be a vector (potentially 1 dimensional)
2253 - file :: this throws an error if the result isn't a string, and
2254 tries to treat it as a path to a file.
2256 I'm just going to cram all of these into the =:results= header
2257 argument. Then if we allow multiple header arguments it should
2258 work out, for example one possible header argument string could be
2259 =:results replace vector file=, which would *replace* any existing
2260 results forcing the results into an org-mode table, and
2261 interpreting any strings as file paths.
2263 *** DONE multiple =:results= headers
2265 #+srcname: multiple-result-headers
2266 #+begin_src ruby :results replace silent
2272 *** DONE file result types
2273 When inserting into an org-mode buffer create a link with the path
2274 being the value, and optionally the display being the
2275 =file-name-nondirectory= if it exists.
2277 #+srcname: task-file-result
2278 #+begin_src python :results replace file
2283 [[something][something]]
2286 This will be useful because blocks like =ditaa= and =dot= can return
2287 the string path of their files, and can add =file= to their results
2290 *** DONE vector result types
2292 #+srcname: task-force-results
2293 #+begin_src emacs-lisp :results vector
2300 ** DONE results name
2301 In order to do this we will need to start naming our results.
2302 Since the source blocks are named with =#+srcname:= lines we can
2303 name results with =#+resname:= lines (if the source block has no
2304 name then no name is given to the =#+resname:= line on creation,
2305 otherwise the name of the source block is used).
2307 This will have the additional benefit of allowing results and
2308 source blocks to be located in different places in a buffer (and
2309 eventually in different buffers entirely).
2311 #+srcname: developing-resnames
2312 #+begin_src emacs-lisp :results silent
2316 Once source blocks are able to find their own =#+resname:= lines
2319 #+srcname: sbe-w-new-results
2320 #+begin_src emacs-lisp :results replace
2321 (sbe "developing-resnames")
2327 *** TODO change the results insertion functions to use these lines
2329 *** TODO teach references to resolve =#+resname= lines.
2331 ** DONE org-babel tests org-babel [1/1]
2332 since we are accumulating this nice collection of source-code blocks
2333 in the sandbox section we should make use of them as unit tests.
2334 What's more, we should be able to actually use org-babel to run these
2337 We would just need to cycle over every source code block under the
2338 sandbox, run it, and assert that the return value is equal to what we
2341 I have the feeling that this should be possible using only org-babel
2342 functions with minimal or no additional elisp. It would be very cool
2343 for org-babel to be able to test itself.
2345 This is now done, see [[* Tests]].
2347 *** DEFERRED org-babel assertions (may not be necessary)
2348 These could be used to make assertions about the results of a
2349 source-code block. If the assertion fails then the point could be
2350 moved to the block, and error messages and highlighting etc... could
2353 ** DONE make C-c C-c work anywhere within source code block?
2354 This seems like it would be nice to me, but perhaps it would be
2355 inefficient or ugly in implementation? I suppose you could search
2356 forward, and if you find #+end_src before you find #+begin_src,
2357 then you're inside one. [DED]
2359 Agreed, I think inside of the =#+srcname: line= would be useful as
2362 #+srcname: testing-out-cc
2363 #+begin_src emacs-lisp
2367 ** DONE integration with org tables
2368 We should make it easy to call org-babel source blocks from org-mode
2369 table formulas. This is practical now that it is possible to pass
2370 arguments to org-babel source blocks.
2372 See the related [[* (sandbox) integration w/org tables][sandbox]] header for tests/examples.
2374 *** digging in org-table.el
2375 In the past [[file:~/src/org/lisp/org-table.el::org%20table%20el%20The%20table%20editor%20for%20Org%20mode][org-table.el]] has proven difficult to work with.
2377 Should be a hook in [[file:~/src/org/lisp/org-table.el::defun%20org%20table%20eval%20formula%20optional%20arg%20equation][org-table-eval-formula]].
2379 Looks like I need to change this [[file:~/src/org/lisp/org-table.el::if%20lispp][if statement]] (line 2239) into a cond
2382 ** DONE source blocks as functions
2384 Allow source code blocks to be called like functions, with arguments
2385 specified. We are already able to call a source-code block and assign
2386 it's return result to a variable. This would just add the ability to
2387 specify the values of the arguments to the source code block assuming
2388 any exist. For an example see
2390 When a variable appears in a header argument, how do we differentiate
2391 between it's value being a reference or a literal value? I guess this
2392 could work just like a programming language. If it's escaped or in
2393 quotes, then we count it as a literal, otherwise we try to look it up
2396 ** DONE folding of code blocks? [2/2]
2397 [DED] In similar way to using outline-minor-mode for folding function
2398 bodies, can we fold code blocks? #+begin whatever statements are
2399 pretty ugly, and in any case when you're thinking about the overall
2400 game plan you don't necessarily want to see the code for each Step.
2402 *** DONE folding of source code block
2403 Sounds good, and wasn't too hard to implement. Code blocks should
2404 now be fold-able in the same manner as headlines (by pressing TAB
2407 *** REJECTED folding of results
2408 So, lets do a three-stage tab cycle... First fold the src block,
2409 then fold the results, then unfold.
2411 There's no way to tell if the results are a table or not w/o
2412 actually executing the block which would be too expensive of an
2415 ** DONE selective export of text, code, figures
2416 [DED] The org-babel buffer contains everything (code, headings and
2417 notes/prose describing what you're up to, textual/numeric/graphical
2418 code output, etc). However on export to html / LaTeX one might want
2419 to include only a subset of that content. For example you might
2420 want to create a presentation of what you've done which omits the
2423 [EMS] So I think this should be implemented as a property which can
2424 be set globally or on the outline header level (I need to review
2425 the mechanics of org-mode properties). And then as a source block
2426 header argument which will apply only to a specific source code
2427 block. A header argument of =:export= with values of
2429 - =code= :: just show the code in the source code block
2430 - =none= :: don't show the code or the results of the evaluation
2431 - =results= :: just show the results of the code evaluation (don't
2432 show the actual code)
2433 - =both= :: show both the source code, and the results
2435 this will be done in [[* (sandbox) selective export][(sandbox) selective export]].
2437 ** DONE a header argument specifying silent evaluation (no output)
2438 This would be useful across all types of source block. Currently
2439 there is a =:replace t= option to control output, this could be
2440 generalized to an =:output= option which could take the following
2441 options (maybe more)
2443 - =t= :: this would be the default, and would simply insert the
2444 results after the source block
2445 - =replace= :: to replace any results which may already be there
2446 - =silent= :: this would inhibit any insertion of the results
2448 This is now implemented see the example in the [[* silent evaluation][sandbox]]
2450 ** DONE assign variables from tables in R
2451 This is now working (see [[* (sandbox table) R][(sandbox-table)-R]]). Although it's not that
2452 impressive until we are able to print table results from R.
2454 ** DONE insert 2-D R results as tables
2455 everything is working but R and shell
2461 This has already been tackled by Dan in [[file:existing_tools/org-R.el::defconst%20org%20R%20write%20org%20table%20def][org-R:check-dimensions]]. The
2462 functions there should be useful in combination with [[http://cran.r-project.org/doc/manuals/R-data.html#Export-to-text-files][R-export-to-csv]]
2463 as a means of converting multidimensional R objects to emacs lisp.
2465 It may be as simple as first checking if the data is multidimensional,
2466 and then, if so using =write= to write the data out to a temporary
2467 file from which emacs can read the data in using =org-table-import=.
2469 Looking into this further, is seems that there is no such thing as a
2470 scalar in R [[http://tolstoy.newcastle.edu.au/R/help/03a/3733.html][R-scalar-vs-vector]] In that light I am not sure how to
2471 deal with trivial vectors (scalars) in R. I'm tempted to just treat
2472 them as vectors, but then that would lead to a proliferation of
2473 trivial 1-cell tables...
2475 ** DONE allow variable initialization from source blocks
2476 Currently it is possible to initialize a variable from an org-mode
2477 table with a block argument like =table=sandbox= (note that the
2478 variable doesn't have to named =table=) as in the following example
2484 #+begin_src emacs-lisp :var table=sandbox :results replace
2485 (message (format "table = %S" table))
2488 : "table = ((1 2 3) (4 \"schulte\" 6))"
2490 It would be good to allow initialization of variables from the results
2491 of other source blocks in the same manner. This would probably
2492 require the addition of =#+SRCNAME: example= lines for the naming of
2493 source blocks, also the =table=sandbox= syntax may have to be expanded
2494 to specify whether the target is a source code block or a table
2495 (alternately we could just match the first one with the given name
2496 whether it's a table or a source code block).
2498 At least initially I'll try to implement this so that there is no need
2499 to specify whether the reference is to a table or a source-code block.
2500 That seems to be simpler both in terms of use and implementation.
2502 This is now working for emacs-lisp, ruby and python (and mixtures of
2503 the three) source blocks. See the examples in the [[* (sandbox) referencing other source blocks][sandbox]].
2505 This is currently working only with emacs lisp as in the following
2506 example in the [[* emacs lisp source reference][emacs lisp source reference]].
2509 ** TODO Add languages [10/13]
2510 I'm sure there are many more that aren't listed here. Please add
2511 them, and bubble any that you particularly care about up to the top.
2513 Any new language should be implemented in a org-babel-lang.el file.
2514 Follow the pattern set by [[file:lisp/org-babel-script.el][org-babel-script.el]], [[file:lisp/org-babel-shell.el][org-babel-shell.el]] and
2515 [[file:lisp/org-babel-R.el][org-babel-R.el]].
2518 This could probably be added to [[file:lisp/org-babel-script.el][org-babel-script.el]]
2522 - support for sessions
2523 - add more useful header arguments (user, passwd, database, etc...)
2524 - support for more engines (currently only supports mysql)
2525 - what's a reasonable way to drop table data into SQL?
2527 #+srcname: sql-example
2528 #+begin_src sql :engine mysql
2534 | "information_schema" |
2538 trivial [[file:lisp/langs/org-babel-css.el][org-babel-css.el]]
2541 (see [[* file result types][file result types]])
2543 #+srcname: implementing-ditaa
2544 #+begin_src ditaa :results replace :file blue.png :cmdline -r
2554 #+resname: implementing-ditaa
2555 [[file:blue.png][blue.png]]
2557 *** DONE gnuplot [7/7]
2558 (see [[* file result types][file result types]])
2560 #+PLOT: title:"Citas" ind:1 deps:(3) type:2d with:histograms set:"yrange [0:]"
2561 #+TBLNAME: gnuplot-data
2562 | independent var | first dependent var | second dependent var |
2563 |-----------------+---------------------+----------------------|
2564 | 0.1 | 0.425 | 0.375 |
2565 | 0.2 | 0.3125 | 0.3375 |
2566 | 0.3 | 0.24999993 | 0.28333338 |
2567 | 0.4 | 0.275 | 0.28125 |
2568 | 0.5 | 0.26 | 0.27 |
2569 | 0.6 | 0.25833338 | 0.24999993 |
2570 | 0.7 | 0.24642845 | 0.23928553 |
2571 | 0.8 | 0.23125 | 0.2375 |
2572 | 0.9 | 0.23333323 | 0.2333332 |
2573 | 1 | 0.2225 | 0.22 |
2574 | 1.1 | 0.20909075 | 0.22272708 |
2575 | 1.2 | 0.19999998 | 0.21458333 |
2576 | 1.3 | 0.19615368 | 0.21730748 |
2578 #+srcname: implementing-gnuplot
2579 #+begin_src gnuplot :var data=gnuplot-data :results silent
2580 set title "Implementing Gnuplot"
2581 plot data using 1:2 with lines
2584 **** DONE add variables
2585 gnuplot 4.2 and up support user defined variables. This is how
2586 we will handle variables with org-babel (meaning we will need to
2587 require gnuplot 4.2 and up for variable support, which can be
2588 install using [[http://www.macports.org/install.php][macports]] on Mac OSX).
2590 - scalar variables should be replaced in the body of the gnuplot code
2591 - vector variables should be exported to tab-separated files, and
2592 the variable names should be replaced with the path to the files
2594 **** DONE direct plotting w/o session
2595 **** DEFERRED gnuplot support for column/row names
2596 This should be implemented along the lines of the [[* STARTED Column (and row) names of tables in R input/output][R-colname-support]].
2598 We can do something similar to the :labels param in org-plot, we just
2599 have to be careful to ensure that each label is aligned with the
2602 This may be walking too close to an entirely prebuilt plotting tool
2603 rather than straight gnuplot code evaluation. For now I think this
2606 **** DONE a =file= header argument
2607 to specify a file holding the results
2609 #+srcname: gnuplot-to-file-implementation
2610 #+begin_src gnuplot :file plot.png :var data=gnuplot-data
2611 plot data using 1:2, data using 1:3 with lines
2615 [[file:plot.png][plot.png]]
2617 **** DONE helpers from org-plot.el
2618 There are a variety of helpers in org-plot which can be fit nicely
2619 into custom gnuplot header arguments.
2621 These should all be in place by now.
2623 **** DEFERRED header argument specifying 3D data
2626 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2627 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2628 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2629 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 |
2630 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2631 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2632 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 1 | 0 |
2633 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
2634 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
2635 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2636 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
2637 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 0 |
2638 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2639 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2641 #+srcname: implementing-gnuplot-grid-plots
2642 #+begin_src gnuplot :vars data=org-grid
2646 **** DONE gnuplot sessions
2647 Working on this, we won't support multiple sessions as `gnuplot-mode'
2648 isn't setup for such things.
2650 Also we can't display results with the default :none session, so for
2651 gnuplot we really want the default behavior to be :default, and to
2652 only run a :none session when explicitly specified.
2654 #+srcname: implementing-gnuplot-sessions
2655 #+begin_src gnuplot :var data=gnuplot-data :session none :file session.png
2656 set title "Implementing Gnuplot Sessions"
2657 plot data using 1:2 with lines
2661 [[file:session.png][session.png]]
2664 (see [[* file result types][file result types]])
2666 #+srcname: implementing-dot-support
2667 #+begin_src dot :file test-dot.png :cmdline -Tpng
2668 digraph data_relationships {
2669 "data_requirement" [shape=Mrecord, label="{DataRequirement|description\lformat\l}"]
2670 "data_product" [shape=Mrecord, label="{DataProduct|name\lversion\lpoc\lformat\l}"]
2671 "data_requirement" -> "data_product"
2676 [[file:test-dot.png][test-dot.png]]
2679 (see [[* file result types][file result types]])
2681 for information on asymptote see http://asymptote.sourceforge.net
2683 #+begin_src asymptote :file asymptote-test.png
2688 real f(real t) {return 1+cos(t);}
2690 path g=polargraph(f,0,2pi,operator ..)--cycle;
2693 xaxis("$x$",above=true);
2694 yaxis("$y$",above=true);
2696 dot("$(a,0)$",(1,0),N);
2697 dot("$(2a,0)$",(2,0),N+E);
2701 [[file:asymptote-test.png][asymptote-test.png]]
2711 ** TODO export problems when support for a language is missing
2712 we should come up with a way to gracefully degrade when support for a
2713 specific language is missing
2715 > To demonstrate creation of documents, open the "test-export.org" file in
2716 > the base of the org-babel directory, and export it as you would any
2717 > other org-mode file. The "exports" header argument controls how
2718 > source-code blocks are exported, with the following options
2720 > - none :: no part of the source-code block is exported in the document
2721 > - results :: only the output of the evaluated block is exported
2722 > - code :: the code itself is exported
2723 > - both :: both the code and results are exported
2725 I have this error showing up:
2727 executing Ruby source code block
2728 apply: Searching for program: no such file or directory, irb
2730 ** TODO problem with newlines in output when :results value
2731 #+begin_src python :results value
2732 '\n'.join(map(str, range(4)))
2738 Whereas I was hoping for
2745 This is some sort of non-printing char / quoting issue I think. Note
2748 #+begin_src python :results value
2749 '\\n'.join(map(str, range(4)))
2756 #+begin_src python :results output
2757 print('\n'.join(map(str, range(4))))
2766 *** collapsing consecutive newlines in string output
2768 This is an example of the same bug
2770 #+srcname: multi-line-string-output
2771 #+begin_src ruby :results output
2772 "the first line ends here
2775 and this is the second one
2780 This doesn't produce anything at all now. I believe that's because
2781 I've changed things so that :results output really does *not* get the
2782 value of the block, only the STDOUT. So if we add a print statement
2785 #+srcname: multi-line-string-output
2786 #+begin_src ruby :results output
2787 print "the first line ends here
2790 and this is the second one
2796 : the first line ends here
2799 : and this is the second one
2803 However, the behaviour with :results value is wrong
2805 #+srcname: multi-line-string-value
2807 "the first line ends here
2810 and this is the second one
2818 ** TODO prompt characters appearing in output with R
2819 #+begin_src R :session *R* :results output
2828 ** TODO o-b-execute-subtree overwrites heading when subtree is folded
2830 Try M-x org-babel-execute-subtree with the subtree folded and
2831 point at the beginning of the heading line.
2836 ** TODO Allow source blocks to be recognised when #+ are not first characters on the line
2837 I think Carsten has recently altered the core so that #+ can have
2838 preceding whitespace, at least for literal/code examples. org-babel
2839 should support this.
2841 ** TODO non-orgtbl formatted lists
2844 #+srcname: this-doesn't-match-orgtbl
2845 #+begin_src emacs-lisp :results replace
2846 '((:results . "replace"))
2849 #+resname: this-doesn't-match-orgtbl
2851 ** PROPOSED allow un-named arguments
2860 ## produces no output
2862 It's not essential but would be nice for this to work. To do it
2863 properly, would mean that we'd have to specify rules for how a string
2864 of supplied arguments (some possibly named) interact with the
2865 arguments in the definition (some possibly with defaults) to give
2866 values to the variables in the funbction body.
2867 ** PROPOSED external shell execution can't isolate return values
2868 I have no idea how to do this as of yet. The result is that when
2869 shell functions are run w/o a session there is no difference between
2870 the =output= and =value= result arguments.
2872 Yea, I don't know how to do this either. I searched extensively on
2873 how to isolate the *last* output of a series of shell commands (see
2874 [[* last command for
2875 shells][last command for shells]]). The results of the search were basically
2876 that it was not possible (or at least not accomplish-able with a
2877 reasonable amount of effort).
2879 That fact combined with the tenancy to all ways use standard out in
2880 shell scripts led me to treat these two options (=output= and =value=)
2881 as identical in shell evaluation. Not ideal but maybe good enough for
2884 In the `results' branch I've changed this so that they're not quite
2885 identical: output results in raw stdout contents, whereas value
2886 converts it to elisp, perhaps to a table if it looks tabular. This is
2887 the same for the other languages. [Dan]
2889 ** TODO adding blank line when source-block produces no output
2891 #+srcname: show-org-babel-trim
2893 find . \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o -type f \( -iname \*.el \) -exec grep -i -nH -e org-babel-trim {} \;
2896 ** DONE are the org-babel-trim s necessary?
2897 at the end of e.g. org-babel-R-evaluate, org-babel-python-evaluate, but
2898 not org-babel-ruby-evaluate
2900 I think it depends on the language, if we find that extra blank
2901 lines are being inserted in a particular language that is a good
2902 indication that the trim or chomp functions may be appropriate.
2904 org-babel-trim and the related org-babel-chomp are use throughout
2907 #+srcname: show-org-babel-trim-usage
2908 #+begin_src sh :results output
2909 find lisp/ \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o -type f \( -iname \*.el \) -exec grep -i -nH org-babel-trim {} \;
2914 lisp//langs/org-babel-python.el:49: vars "\n") "\n" (org-babel-trim body) "\n")) ;; then the source block body
2915 lisp//langs/org-babel-python.el:143: (org-remove-indentation (org-babel-trim body)) "[\r\n]")))
2916 lisp//langs/org-babel-python.el:158: #'org-babel-trim
2917 lisp//langs/org-babel-python.el:166: (reverse (mapcar #'org-babel-trim raw)))))))
2918 lisp//langs/org-babel-python.el:169: (output (org-babel-trim (mapconcat #'identity (reverse (cdr results)) "\n")))
2919 lisp//langs/org-babel-python.el:170: (value (org-babel-python-table-or-string (org-babel-trim (car results)))))))))
2920 lisp//langs/org-babel-ruby.el:149: (mapcar #'org-babel-trim raw)))))))
2921 lisp//langs/org-babel-sh.el:148: (mapcar #'org-babel-trim raw)))))))
2922 lisp//langs/org-babel-sh.el:151: (output (org-babel-trim (mapconcat #'org-babel-trim (reverse results) "\n")))
2923 lisp//org-babel-ref.el:161: (mapcar #'org-babel-trim (reverse (cons buffer return)))))
2924 lisp//org-babel.el:198: (with-temp-buffer (insert (org-babel-trim body)) (copy-region-as-kill (point-min) (point-max)))
2925 lisp//org-babel.el:465: (org-babel-trim
2926 lisp//org-babel.el:706:(defun org-babel-trim (string &optional regexp)
2929 #+srcname: show-org-babel-chomp-usage
2930 #+begin_src sh :results output
2931 find lisp/ \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o -type f \( -iname \*.el \) -exec grep -i -nH org-babel-chomp {} \;
2936 lisp//langs/org-babel-R.el:122: (full-body (mapconcat #'org-babel-chomp
2937 lisp//langs/org-babel-R.el:143: (delete nil (mapcar #'extractor (mapcar #'org-babel-chomp raw))) "\n"))))))))
2938 lisp//langs/org-babel-ruby.el:143: #'org-babel-chomp
2939 lisp//langs/org-babel-sh.el:142: (full-body (mapconcat #'org-babel-chomp
2940 lisp//org-babel-tangle.el:163: (insert (format "\n%s\n" (org-babel-chomp body)))
2941 lisp//org-babel.el:362: (org-babel-chomp (match-string 2 arg)))
2942 lisp//org-babel.el:698:(defun org-babel-chomp (string &optional regexp)
2943 lisp//org-babel.el:707: "Like `org-babel-chomp' only it runs on both the front and back of the string"
2944 lisp//org-babel.el:708: (org-babel-chomp (org-babel-reverse-string
2945 lisp//org-babel.el:709: (org-babel-chomp (org-babel-reverse-string string) regexp)) regexp))
2948 ** DONE LoB is not populated on startup
2949 org-babel-library-of-babel is nil for me on startup. I have to
2950 evaluate the [[file:lisp/org-babel-lob.el::][org-babel-lob-ingest]] line manually.
2952 #+tblname: R-plot-example-data
2959 #+lob: R-plot(data=R-plot-example-data)
2963 I've added a section to [[file:lisp/org-babel-init.el][org-babel-init.el]] which will load the
2964 library of babel on startup.
2966 Note that this needs to be done in [[file:lisp/org-babel-init.el][org-babel-init.el]] rather than in
2967 [[file:lisp/org-babel-lob.el][org-babel-lob.el]], not entirely sure why, something about it being
2970 Also, I'm now having the file closed if it wasn't being visited by
2971 a buffer before being loaded.
2973 ** DONE use new merge function [[file:lisp/org-babel-ref.el::t%20nil%20org%20combine%20plists%20args%20nil][here]]?
2974 And at other occurrences of org-combine-plists?
2975 ** DONE creeping blank lines
2976 There's still inappropriate addition of blank lines in some circumstances.
2978 Hmm, it's a bit confusing. It's to do with o-b-remove-result. LoB
2979 removes the entire (#+resname and result) and starts from scratch,
2980 whereas #+begin_src only removes the result. I haven't worked out
2981 what the correct fix is yet. Maybe the right thing to do is to make
2982 sure that those functions (o-b-remove-result et al.) are neutral
2983 with respect to newlines. Sounds easy, but...
2993 Compare the results of
2994 #+lob: adder(a=5, b=17)
2996 #+resname: python-add(a=5, b=17)
2998 --------------------------------
3006 ---------------------
3007 ** DONE #+srcname arg parsing bug
3008 #+srcname: test-zz(arg=adder(a=1, b=1))
3017 #+srcname: test-zz-nasty(arg=adder(a=adder(a=19,b=adder(a=5,b=2)),b=adder(a=adder(a=1,b=9),b=adder(a=1,b=3))))
3022 #+resname: test-zz-nasty
3027 #+srcname: test-zz-hdr-arg
3028 #+begin_src python :var arg=adder(a=adder(a=19,b=adder(a=5,b=2)),b=adder(a=adder(a=1,b=9),b=adder(a=1,b=3)))
3035 ** DONE Fix nested evaluation and default args
3036 The current parser / evaluator fails with greater levels of nested
3037 function block calls (example below).
3039 *** Initial statement [ded]
3040 If we want to overcome this I think we'd have to redesign some of
3041 the evaluation mechanism. Seeing as we are also facing issues like
3042 dealing with default argument values, and seeing as we now know
3043 how we want the library of babel to behave in addition to the
3044 source blocks, now might be a good time to think about this. It
3045 would be nice to do the full thing at some point, but otoh we may
3046 not consider it a massive priority.
3048 AIui, there are two stages: (i) construct a parse tree, and (ii)
3049 evaluate it and return the value at the root. In the parse tree
3050 each node represents an unevaluated value (either a literal value
3051 or a reference). Node v may have descendent nodes, which represent
3052 values upon which node v's evaluation depends. Once that tree is
3053 constructed, then we evaluate the nodes from the tips towards the
3054 root (a post-order traversal).
3056 [This would also provide a solution for concatenating the STDOUTs
3057 of called blocks, which is a [[*allow%20output%20mode%20to%20return%20stdout%20as%20value][task below]]; we concatenate them in
3058 whatever order the traversal is done in.]
3060 In addition to the variable references (i.e. daughter nodes), each
3061 node would contain the information needed to evaluate that node
3062 (e.g. lang body). Then we would pass a function postorder over the
3063 tree which would call o-b-execute-src-block at each node, finally
3064 returning the value at the root.
3066 Fwiw I made a very tentative small start at stubbing this out in
3067 org-babel-call.el in the 'evaluation' branch. And I've made a start
3068 at sketching a parsing algorithm below.
3069 **** Parse tree algorithm
3070 Seeing as we're just trying to parse a string like
3071 f(a=1,b=g(c=2,d=3)) it shouldn't be too hard. But of course there
3072 are 'proper' parsers written in elisp out there,
3073 e.g. [[http://cedet.sourceforge.net/semantic.shtml][Semantic]]. Perhaps we can find what we need -- our syntax is
3074 pretty much the same as python and R isn't it?
3076 Or, a complete hack, but maybe it would be we easy to transform it
3077 to XML and then parse that with some existing tool?
3079 But if we're doing it ourselves, something very vaguely like this?
3080 (I'm sure there're lots of problems with this)
3082 #+srcname: org-babel-call-parse(call)
3084 ## we are currently reading a reference name: the name of the root function
3085 whereami = "refname"
3086 node = root = Node()
3087 for c in call_string:
3090 whereami = "varname" # now we're reading a variable name
3093 node.daughters = [node.daughters, new]
3096 whereami = "refname"
3098 whereami = "varname"
3105 if whereami = "varname":
3106 node.varnames[varnum] += c
3107 elif whereami = "refname":
3111 *** discussion / investigation
3112 I believe that this issue should be addressed as a bug rather than as
3113 a point for new development. The code in [[file:lisp/org-babel-ref.el][org-babel-ref.el]] already
3114 resolves variable references in a recursive manner which *should* work
3115 in the same manner regardless of the depth of the number of nested
3116 function calls. This recursive evaluation has the effect of
3117 implicitly constructing the parse tree that your are thinking of
3118 constructing explicitly.
3120 Through using some of the commented out debugging statements in
3121 [[file:lisp/org-babel-ref.el][org-babel-ref.el]] I have looked at what may be going wrong in the
3122 current evaluation setup, and it seems that nested variables are being
3123 set using the =:var= header argument, and these variables are being
3124 overridden by the *default* variables which are being entered through
3125 the new functional syntax (see the demonstration header below).
3127 I believe that once this bug is fixed we should be back to fully
3128 resolution of nested arguments. We should capture this functionality
3129 in a test to ensure that we continue to test it as we move forward. I
3130 can take a look at implementing this once I get a chance.
3132 Looks like the problem may be in [[file:lisp/org-babel.el::defun%20org%20babel%20merge%20params%20rest%20plists][org-babel-merge-params]], which seems
3133 to be trampling the provided :vars values.
3135 Nope, now it seems that we are actually looking up the results line,
3136 rather than the actual source-code block, which would make sense given
3137 that the results-line will return the same value regardless of the
3138 arguments supplied. See the output of this [[file:lisp/org-babel-ref.el::message%20type%20S%20type%20debugging][debug-statement]].
3140 We need to be sure that we don't read from a =#+resname:= line when we
3141 have a non-nil set of arguments.
3144 After uncommenting the debugging statements located [[file:lisp/org-babel-ref.el::message%20format%20first%20second%20S%20S%20new%20refere%20new%20referent%20debugging][here]] and more
3145 importantly [[file:lisp/org-babel-ref.el::message%20nested%20args%20S%20args%20debugging][here]], we can see that the current reference code does
3146 evaluate the references correctly, and it uses the =:var= header
3147 argument to set =a=8=, however the default variables specified using
3148 the functional syntax in =adder(a=3, b=2)= is overriding this
3151 ***** doesn't work with functional syntax
3153 #+srcname: adder-func(a=3, b=2)
3158 #+resname: adder-func
3161 #+srcname: after-adder-func(arg=adder-func(a=8))
3166 #+resname: after-adder-func
3169 ***** still does work with =:var= syntax
3171 so it looks like regardless of the syntax used we're not overriding
3172 the default argument values.
3174 #+srcname: adder-header
3175 #+begin_src python :var a=3 :var b=2
3179 #+resname: adder-header
3182 #+srcname: after-adder-header
3183 #+begin_src python :var arg=adder-header(a=8, b=0)
3187 #+resname: after-adder-header
3190 *** Set of test cases
3191 **** Both defaults provided in definition
3192 #+srcname: adder1(a=10,b=20)
3199 ****** DONE Rely on defaults
3207 ******* DONE empty parens () not recognised as lob call
3208 E.g. remove spaces between parens above
3210 updated [[file:lisp/org-babel-lob.el::defvar%20org%20babel%20lob%20one%20liner%20regexp%20lob%20t%20n%20n%20t%20n][org-babel-lob-one-liner-regexp]]
3212 ****** DONE One supplied, one default
3215 #+resname: adder1(a=0)
3222 #+resname: adder1(b=0)
3226 ****** DONE Both supplied
3227 #+lob: adder1(a=1,b=2)
3229 #+resname: adder1(a=1,b=2)
3232 **** One arg lacks default in definition
3233 #+srcname: adder2(a=10,b)
3237 ****** DEFERRED Rely on defaults (one of which is missing)
3242 ## should be error: b has no default
3244 Maybe we should let the programming language handle this case. For
3245 example python spits out an error in the =#+lob= line above. Maybe
3246 rather than catching these errors our-selves we should raise an error
3247 when the source-block returns an error. I'll propose a [[* PROPOSED raise elisp error when source-blocks return errors][task]] for this
3248 idea, I'm not sure how/if it would work...
3250 ****** DEFERRED Default over-ridden
3253 See the above [[* DEFERRED Rely on defaults (one of which is missing)][deferred]] and the new proposed [[* PROPOSED raise elisp error when source-blocks return errors][task]], I think it may be
3254 more flexible to allow the source block language to handle the error.
3257 ## should be error: b has no default
3259 ****** DONE Missing default supplied
3262 #+resname: adder2(b=1)
3269 ****** DONE One over-ridden, one supplied
3270 #+lob: adder2(a=1,b=2)
3272 #+resname: adder2(a=1,b=2)
3277 *** Example that fails
3279 #+srcname: adder(a=0, b=99)
3292 #+srcname: level-one-nesting()
3293 #+begin_src python :var arg=adder(a=one(),b=one())
3297 #+resname: level-one-nesting
3301 #+srcname: level-one-nesting()
3302 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=one()))
3309 *** DONE deeply nested arguments still fails
3311 #+srcname: deeply-nested-args-bug
3312 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=one()))
3319 **** Used to result in this error
3320 : supplied params=nil
3321 : new-refere="adder", new-referent="a=adder(a=one(),b=one()),b=adder(a=one(),b=one())"
3322 : args=((:var . "a=adder(a=one()") (:var . "b=one())") (:var . "b=adder(a=one()") (:var . "b=one())"))
3324 : supplied params=((:var . "a=adder(a=one()") (:var . "b=one())") (:var . "b=adder(a=one()") (:var . "b=one())"))
3325 : new-refere="adder", new-referent="a=one("
3326 : args=((:var . "a=one("))
3328 : supplied params=((:var . "a=one("))
3329 : reference 'one(' not found in this buffer
3331 Need to change the regexp in [[file:lisp/org-babel-ref.el::assign%20any%20arguments%20to%20pass%20to%20source%20block][org-babel-ref-resolve-reference]] so that
3332 it only matches when the parenthesis are balanced. Maybe look at
3333 [[http://www.gnu.org/software/emacs/elisp/html_node/List-Motion.html][this]].
3335 *** DONE Still some problems with deeply nested arguments and defaults
3337 **** DONE Parsing / defaults bug
3338 Try inserting a space between 'a=0,' and 'b=0' and comparing results
3339 #+srcname: parsing-defaults-bug()
3340 #+begin_src python :var arg=adder(a=adder(a=0,b=0))
3344 #+resname: parsing-defaults-bug
3348 #+srcname: deeply-nested-args-bug-orig()
3349 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=adder(a=3, b=4),b=one()))
3353 #+resname: deeply-nested-args-bug-orig
3358 **** DONE Nesting problem II
3359 This generates parsing errors
3361 Fixed: c2bef96b7f644c05be5a38cad6ad1d28723533aa
3363 #+srcname: deeply-nested-args-bug-II-1()
3364 #+begin_src python :var arg=adder(a=adder(a=one(),b=adder(a=2,b=4)))
3368 #+resname: deeply-nested-args-bug-II-1
3372 #+srcname: deeply-nested-args-bug-II-original()
3373 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=adder(a=1,b=4)))
3377 #+resname: deeply-nested-args-bug-II-original
3382 **** DONE Why does this give 8?
3383 It was picking up the wrong definition of adder
3384 #+srcname: deeply-nested-args-bug-2()
3385 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()))
3389 #+resname: deeply-nested-args-bug-2
3393 **** DONE Problem with empty argument list
3394 This gives empty list with () and 'no output' with ( )
3396 I think this is OK now.
3399 #+begin_src python :var arg=adder( )
3406 #+srcname: deeply-nested-args-bug-orig()
3407 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=adder(a=3, b=4),b=one()))
3411 #+resname: deeply-nested-args-bug-orig
3416 **** DONE Nesting problem II
3417 This generates parsing errors
3419 Fixed: c2bef96b7f644c05be5a38cad6ad1d28723533aa
3421 #+srcname: deeply-nested-args-bug-II-1()
3422 #+begin_src python :var arg=adder(a=adder(a=one(),b=adder(a=2,b=4)))
3426 #+resname: deeply-nested-args-bug-II-1
3430 #+srcname: deeply-nested-args-bug-II-original()
3431 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()),b=adder(a=one(),b=adder(a=1,b=4)))
3435 #+resname: deeply-nested-args-bug-II-original
3440 **** DONE Why does this give 8?
3441 It was picking up the wrong definition of adder
3442 #+srcname: deeply-nested-args-bug-2()
3443 #+begin_src python :var arg=adder(a=adder(a=one(),b=one()))
3447 #+resname: deeply-nested-args-bug-2
3451 **** DONE Problem with empty argument list
3452 This gives empty list with () and 'no output' with ( )
3454 I think this is OK now.
3457 #+begin_src python :var arg=adder( )
3490 #+srcname: test-zz(arg=adder(a=adder(a=19,b=adder(a=5,b=2)),b=adder(a=adder(a=1,b=9),b=adder(a=1,b=3))))
3496 *** DONE Arg lacking default
3497 This would be good thing to address soon. I'm imagining that
3498 e.g. here, the 'caller' block would return the answer 30. I believe
3499 there's a few issues here: i.e. the naked 'a' without a reference
3500 is not understood; the default arg b=6 is not understood.
3502 #+srcname: adder-with-arg-lacking-default(a, b=6)
3509 #+srcname: caller(var=adder-with-arg-lacking-default(a=24))
3510 #+begin_src python :results silent
3514 ** DONE allow srcname to omit function call parentheses
3515 Someone needs to revisit those regexps. Is there an argument for
3516 moving some of the regexps used to match function calls into
3517 defvars? (i.e. in o-b.el and o-b-ref.el)
3519 This seems to work now. It still might be a good idea to separate
3520 out some of the key regexps into defvars at some point.
3522 #+srcname: omit-parens-test
3523 #+begin_src ruby :results output
3532 ** DONE avoid stripping whitespace from output when :results output
3533 This may be partly solved by using o-b-chomp rather than o-b-trim
3534 in the o-b-LANG-evaluate functions.
3535 ** DEFERRED weird escaped characters in shell prompt break shell evaluation
3536 E.g. this doesn't work. Should the shell sessions set a sane prompt
3537 when they start up? Or is it a question of altering
3538 comint-prompt-regexp? Or altering org-babel regexps?
3541 black=30 ; red=31 ; green=32 ; yellow=33 ; blue=34 ; magenta=35 ; cyan=36 ; white=37
3544 export PS1="\[\033[${prompt_col}m\]\w${prompt_char} \[\033[0m\]"
3547 I just pushed a good amount of changes, could you see if your shell
3548 problems still exist?
3550 The problem's still there. Specifically, aIui, at [[file:lisp/langs/org-babel-sh.el::raw%20org%20babel%20comint%20with%20output%20buffer%20org%20babel%20sh%20eoe%20output%20nil%20insert%20full%20body%20comint%20send%20input%20nil%20t][this line]] of
3551 org-babel-sh.el, raw gets the value
3553 ("" "
\e[0m Sun Jun 14 19:26:24 EDT 2009\n" "
\e[0m org_babel_sh_eoe\n" "
\e[0m ")
3555 and therefore (member org-babel-sh-eoe-output ...) fails
3557 I think that `comint-prompt-regexp' needs to be altered to match
3558 the shell prompt. This shouldn't be too difficult to do by hand,
3559 using the `regexp-builder' command and should probably be part of
3560 the user's regular emacs init. I can't think of a way for us to
3561 set this automatically, and we are SOL without a regexp to match
3563 ** DONE function calls in #+srcname: refs
3565 My srcname references don't seem to be working for function
3566 calls. This needs fixing.
3573 srcname function call doesn't work for calling a source block
3574 #+srcname: caller(var1=called())
3587 They do work for a simple reference
3588 #+srcname: caller2(var1=56)
3597 and they do work for :var header arg
3599 #+begin_src python :var var1=called()
3605 ** DONE LoB: with output to buffer, not working in buffers other than library-of-babel.org
3607 I haven't fixed this yet. org-babel-ref-resolve-reference moves
3608 point around, inside a save-excursion. Somehow when it comes to
3609 inserting the results (after possible further recursive calls to
3610 org-babel-ref-resolve-reference), point hasn't gone back to the
3613 #+tblname: test-data
3618 #+lob: R-plot(data=test-data)
3620 #+lob: python-add(a=2, b=9)
3622 #+resname: python-add(a=2, b=9)
3626 I think this got fixed in the bugfixes before merging results into master.
3628 ** DONE cursor movement when evaluating source blocks
3629 E.g. the pie chart example. Despite the save-window-excursion in
3630 org-babel-execute:R. (I never learned how to do this properly: org-R
3631 jumps all over the place...)
3633 I don't see this now [ded]
3635 ** DONE LoB: calls fail if reference has single character name
3636 commit 21d058869df1ff23f4f8cc26f63045ac9c0190e2
3637 **** This doesn't work
3638 #+lob: R-plot(data=X)
3657 #+lob: R-plot(data=XX)
3659 ** DONE make :results replace the default?
3660 I'm tending to think that appending results to pre-existing results
3661 creates mess, and that the cleaner `replace' option should be the
3662 default. E.g. when a source block creates an image, we would want
3663 that to be updated, rather than have a new one be added.
3667 ** DONE ruby evaluation not working under ubuntu emacs 23
3668 With emacs 23.0.91.1 on ubuntu, for C-h f run-ruby I have the
3669 following, which seems to conflict with [[file:lisp/langs/org-babel-ruby.el::let%20session%20buffer%20save%20window%20excursion%20run%20ruby%20nil%20session%20current%20buffer][this line]] in org-babel-ruby.el.
3672 run-ruby is an interactive compiled Lisp function.
3676 Run an inferior Ruby process, input and output via buffer *ruby*.
3677 If there is a process already running in `*ruby*', switch to that buffer.
3678 With argument, allows you to edit the command line (default is value
3679 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
3680 (after the `comint-mode-hook' is run).
3681 (Type C-h m in the process buffer for a list of commands.)
3684 So, I may have a non-standard inf-ruby.el. Here's my version of
3688 run-ruby is an interactive Lisp function in `inf-ruby.el'.
3690 (run-ruby &optional COMMAND NAME)
3692 Run an inferior Ruby process, input and output via buffer *ruby*.
3693 If there is a process already running in `*ruby*', switch to that buffer.
3694 With argument, allows you to edit the command line (default is value
3695 of `ruby-program-name'). Runs the hooks `inferior-ruby-mode-hook'
3696 (after the `comint-mode-hook' is run).
3697 (Type C-h m in the process buffer for a list of commands.)
3700 It seems we could either bundle my version of inf-ruby.el (as it's
3701 the newest). Or we could change the use of `run-ruby' so that it
3702 is robust across multiple distributions. I think I'd prefer the
3703 former, unless the older version of inf-ruby is actually bundled
3704 with emacs, in which case maybe we should go out of our way to
3705 support it. Thoughts?
3707 I think for now I'll just include the latest [[file:util/inf-ruby.el][inf-ruby.el]] in the
3708 newly created utility directory. I doubt anyone would have a
3709 problem using the latest version of this file.
3710 ** DONE test failing forcing vector results with =test-forced-vector-results= ruby code block
3711 Note that this only seems to happen the *second* time the test table
3714 #+srcname: bug-trivial-vector
3715 #+begin_src emacs-lisp :results vector silent
3719 #+srcname: bug-forced-vector-results
3720 #+begin_src ruby :var triv=test-trivial-vector :results silent
3724 mysteriously this seems to be fixed...
3725 ** DONE defunct R sessions
3726 Sometimes an old R session will turn defunct, and newly inserted code
3727 will not be evaluated (leading to a hang).
3729 This seems to be fixed by using `inferior-ess-send-input' rather than `comint-send-input'.
3730 ** DONE ruby fails on first call to non-default session
3732 #+srcname: bug-new-session
3733 #+begin_src ruby :session is-new
3737 ** DONE when reading results from =#+resname= line
3739 Errors when trying to read from resname lines.
3741 #+resname: bug-in-resname
3744 #+srcname: bug-in-resname-reader
3745 #+begin_src emacs-lisp :var buggy=bug-in-resname() :results silent
3749 ** DONE R-code broke on "org-babel" rename
3751 #+srcname: bug-R-babels
3756 ** DONE error on trivial R results
3758 So I know it's generally not a good idea to squash error without
3759 handling them, but in this case the error almost always means that
3760 there was no file contents to be read by =org-table-import=, so I
3763 #+srcname: bug-trivial-r1
3764 #+begin_src R :results replace
3765 pie(c(1, 2, 3), labels = c(1, 2, 3))
3768 #+srcname: bug-trivial-r2
3769 #+begin_src R :results replace
3773 #+resname: bug-trivial-r2
3776 #+srcname: bug-trivial-r3
3777 #+begin_src R :results replace
3781 #+resname: bug-trivial-r3
3786 ** DONE ruby new variable creation (multi-line ruby blocks)
3787 Actually it looks like we were dropping all but the last line.
3789 #+srcname: multi-line-ruby-test
3790 #+begin_src ruby :var table=bug-numerical-table :results replace
3792 table.each{|n| total += n}
3799 ** DONE R code execution seems to choke on certain inputs
3800 Currently the R code seems to work on vertical (but not landscape)
3803 #+srcname: little-fake
3804 #+begin_src emacs-lisp
3809 #+begin_src R :var num=little-fake
3816 #+srcname: set-debug-on-error
3817 #+begin_src emacs-lisp :results silent
3818 (setq debug-on-error t)
3821 #+srcname: bug-numerical-table
3822 #+begin_src emacs-lisp :results silent
3829 #+srcname: bug-R-number-evaluation
3830 #+begin_src R :var table=bug-numerical-table
3839 #+tblname: bug-vert-table
3844 #+srcname: bug-R-vertical-table
3845 #+begin_src R :var table=bug-vert-table :results silent
3849 ** DONE org bug/request: prevent certain org behaviour within code blocks
3850 E.g. [[]] gets recognised as a link (when there's text inside the
3851 brackets). This is bad for R code at least, and more generally
3852 could be argued to be inappropriate. Is it difficult to get org to
3853 ignore text in code blocks? [DED]
3855 I believe Carsten addressed this recently on the mailing list with
3856 the comment that it was indeed a difficult issue. I believe this
3857 may be one area where we could wait for an upstream (org-mode) fix.
3859 [Dan] Carsten has fixed this now in the core.
3861 ** DONE with :results replace, non-table output doesn't replace table output
3862 And vice versa. E.g. Try this first with table and then with len(table) [DED]
3863 #+begin_src python :var table=sandbox :results replace
3868 | 4 | "schulte" | 6 |
3871 Yes, this is certainly a problem. I fear that if we begin replacing
3872 anything immediately following a source block (regardless of whether
3873 it matches the type of our current results) we may accidentally delete
3874 hand written portions of the user's org-mode buffer.
3876 I think that the best solution here would be to actually start
3877 labeling results with a line that looks something like...
3881 This would have a couple of benefits...
3882 1) we wouldn't have to worry about possibly deleting non-results
3883 (which is currently an issue)
3884 2) we could reliably replace results even if there are different types
3885 3) we could reference the results of a source-code block in variable
3886 definitions, which would be useful if for example we don't wish to
3887 re-run a source-block every time because it is long-running.
3889 Thoughts? If no-one objects, I believe I will implement the labeling
3892 ** DONE extra quotes for nested string
3893 Well R appears to be reading the tables without issue...
3895 these *should* be quoted
3897 #+begin_src sh :results replace
3902 | "README.markdown" |
3905 | "existing_tools" |
3909 | "test-export.html" |
3910 | "test-export.org" |
3912 #+srcname: test-quotes
3913 #+begin_src ruby :var tab=ls
3919 #+srcname: test-quotes
3920 #+begin_src R :var tab=ls
3926 ** DONE simple ruby arrays not working
3928 As an example eval the following. Adding a line to test
3930 #+tblname: simple-ruby-array
3933 #+srcname: ruby-array-test
3934 #+begin_src ruby :var ar = simple-ruby-array :results silent
3938 ** DONE space trailing language name
3939 fix regexp so it works when there's a space trailing the language name
3941 #+srcname: test-trailing-space
3946 ** DONE Args out of range error
3948 The following block resulted in the error below [DED]. It ran without
3949 error directly in the shell.
3952 for platf in ill aff ; do
3953 for pop in CEU YRI ASI ; do
3954 rm -f $platf/hapmap-genos-$pop-all $platf/hapmap-rs-all
3955 cat $platf/hapmap-genos-$pop-* > $platf/hapmap-genos-$pop-all
3956 cat $platf/hapmap-rs-* > $platf/hapmap-rs-all
3961 executing source block with sh...
3962 finished executing source block
3963 string-equal: Args out of range: "", -1, 0
3965 the error =string-equal: Args out of range: "", -1, 0= looks like what
3966 used to be output when the block returned an empty results string.
3967 This should be fixed in the current version, you should now see the
3968 following message =no result returned by source block=.
3970 ** DONE ruby arrays not recognized as such
3972 Something is wrong in [[file:lisp/org-babel-script.el]] related to the
3973 recognition of ruby arrays as such.
3975 #+begin_src ruby :results replace
3981 #+begin_src python :results replace
3986 ** REJECTED elisp reference fails for literal number
3987 That's a bug in Dan's elisp, not in org-babel.
3988 #+srcname: elisp-test(a=4)
3989 #+begin_src emacs-lisp
4001 Evaluate all the cells in this table for a comprehensive test of the
4002 org-babel functionality.
4004 *Note*: if you have customized =org-babel-default-header-args= then some
4005 of these tests may fail.
4007 #+TBLNAME: org-babel-tests
4008 | functionality | block | arg | expected | results | pass |
4009 |-------------------------+----------------------------+-----+-------------+-------------+------|
4010 | basic evaluation | | | | | pass |
4011 |-------------------------+----------------------------+-----+-------------+-------------+------|
4012 | emacs lisp | basic-elisp | | 5 | 5 | pass |
4013 | shell | basic-shell | | 6 | 6 | pass |
4014 | ruby | basic-ruby | | org-babel | org-babel | pass |
4015 | python | basic-python | | hello world | hello world | pass |
4016 | R | basic-R | | 13 | 13 | pass |
4017 |-------------------------+----------------------------+-----+-------------+-------------+------|
4018 | tables | | | | | pass |
4019 |-------------------------+----------------------------+-----+-------------+-------------+------|
4020 | emacs lisp | table-elisp | | 3 | 3 | pass |
4021 | ruby | table-ruby | | 1-2-3 | 1-2-3 | pass |
4022 | python | table-python | | 5 | 5 | pass |
4023 | R | table-R | | 3.5 | 3.5 | pass |
4024 | R: col names in R | table-R-colnames | | -3 | -3 | pass |
4025 | R: col names in org | table-R-colnames-org | | 169 | 169 | pass |
4026 |-------------------------+----------------------------+-----+-------------+-------------+------|
4027 | source block references | | | | | pass |
4028 |-------------------------+----------------------------+-----+-------------+-------------+------|
4029 | all languages | chained-ref-last | | Array | Array | pass |
4030 |-------------------------+----------------------------+-----+-------------+-------------+------|
4031 | source block functions | | | | | pass |
4032 |-------------------------+----------------------------+-----+-------------+-------------+------|
4033 | emacs lisp | defun-fibb | | fibbd | fibbd | pass |
4034 | run over | Fibonacci | 0 | 1 | 1 | pass |
4035 | a | Fibonacci | 1 | 1 | 1 | pass |
4036 | variety | Fibonacci | 2 | 2 | 2 | pass |
4037 | of | Fibonacci | 3 | 3 | 3 | pass |
4038 | different | Fibonacci | 4 | 5 | 5 | pass |
4039 | arguments | Fibonacci | 5 | 8 | 8 | pass |
4040 |-------------------------+----------------------------+-----+-------------+-------------+------|
4041 | bugs and tasks | | | | | pass |
4042 |-------------------------+----------------------------+-----+-------------+-------------+------|
4043 | simple ruby arrays | ruby-array-test | | 3 | 3 | pass |
4044 | R number evaluation | bug-R-number-evaluation | | 2 | 2 | pass |
4045 | multi-line ruby blocks | multi-line-ruby-test | | 2 | 2 | pass |
4046 | forcing vector results | test-forced-vector-results | | Array | Array | pass |
4047 | deeply nested arguments | deeply-nested-args-bug | | 4 | 4 | pass |
4048 |-------------------------+----------------------------+-----+-------------+-------------+------|
4049 | sessions | | | | | pass |
4050 |-------------------------+----------------------------+-----+-------------+-------------+------|
4051 | set ruby session | set-ruby-session-var | | :set | :set | pass |
4052 | get from ruby session | get-ruby-session-var | | 3 | 3 | pass |
4053 | set python session | set-python-session-var | | set | set | pass |
4054 | get from python session | get-python-session-var | | 4 | 4 | pass |
4055 | set R session | set-R-session-var | | set | set | pass |
4056 | get from R session | get-R-session-var | | 5 | 5 | pass |
4057 #+TBLFM: $5='(if (= (length $3) 1) (progn (message (format "running %S" '(sbe $2 (n $3)))) (sbe $2 (n $3))) (sbe $2))::$6='(if (string= $4 $5) "pass" (format "expected %S but was %S" $4 $5))
4058 #+TBLFM: $5=""::$6=""
4061 The second TBLFM line (followed by replacing '[]' with '') can be used
4062 to blank out the table results, in the absence of a better method.
4066 #+srcname: basic-elisp
4067 #+begin_src emacs-lisp :results silent
4072 #+srcname: basic-shell
4073 #+begin_src sh :results silent
4078 #+srcname: date-simple
4079 #+begin_src sh :results silent
4083 #+srcname: basic-ruby
4084 #+begin_src ruby :results silent
4089 #+srcname: basic-python
4090 #+begin_src python :results silent
4096 #+begin_src R :results silent
4104 #+tblname: test-table
4108 #+tblname: test-table-colnames
4109 | var1 | var2 | var3 |
4110 |------+------+------|
4114 #+srcname: table-elisp
4115 #+begin_src emacs-lisp :results silent :var table=test-table
4116 (length (car table))
4120 #+srcname: table-ruby
4121 #+begin_src ruby :results silent :var table=test-table
4122 table.first.join("-")
4126 #+srcname: table-python
4127 #+begin_src python :var table=test-table
4131 #+srcname: table-R(table=test-table)
4136 #+srcname: table-R-colnames(table=test-table-colnames)
4137 #+begin_src R :results silent
4138 sum(table$var2 - table$var3)
4141 #+srcname: R-square(x=default-name-doesnt-exist)
4142 #+begin_src R :colnames t
4146 This should return 169. The fact that R is able to use the column name
4147 to index the data frame (x$var3) proves that a table with column names
4148 (a header row) has been recognised as input for the R-square function
4149 block, and that the R-square block has output an elisp table with
4150 column names, and that the colnames have again been recognised when
4151 creating the R variables in this block.
4152 #+srcname: table-R-colnames-org(x = R-square(x=test-table-colnames))
4162 Lets pass a references through all of our languages...
4164 Lets start by reversing the table from the previous examples
4166 #+srcname: chained-ref-first
4167 #+begin_src python :var table = test-table
4172 #+resname: chained-ref-first
4176 Take the first part of the list
4178 #+srcname: chained-ref-second
4179 #+begin_src R :var table = chained-ref-first
4183 #+resname: chained-ref-second
4187 Turn the numbers into string
4189 #+srcname: chained-ref-third
4190 #+begin_src emacs-lisp :var table = chained-ref-second
4191 (mapcar (lambda (el) (format "%S" el)) table)
4194 #+resname: chained-ref-third
4197 and Check that it is still a list
4199 #+srcname: chained-ref-last
4200 #+begin_src ruby :var table=chained-ref-third
4205 ** source blocks as functions
4207 #+srcname: defun-fibb
4208 #+begin_src emacs-lisp :results silent
4209 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
4213 #+srcname: fibonacci
4214 #+begin_src emacs-lisp :results silent :var n=7
4224 ** sbe tests (these don't seem to be working...)
4225 Testing the insertion of results into org-mode tables.
4227 #+srcname: multi-line-output
4228 #+begin_src ruby :results replace
4229 "the first line ends here
4232 and this is the second one
4238 : the first line ends here\n\n\n and this is the second one\n\neven a third
4240 #+srcname: multi-line-error
4241 #+begin_src ruby :results replace
4242 raise "oh nooooooooooo"
4248 | the first line ends here... | -:5: warning: parenthesize argument(s) for future version... |
4249 #+TBLFM: $1='(sbe "multi-line-output")::$2='(sbe "multi-line-error")
4251 ** forcing results types tests
4253 #+srcname: test-trivial-vector
4254 #+begin_src emacs-lisp :results vector silent
4258 #+srcname: test-forced-vector-results
4259 #+begin_src ruby :var triv=test-trivial-vector :results silent
4265 #+srcname: set-ruby-session-var
4266 #+begin_src ruby :session :results silent
4271 #+srcname: get-ruby-session-var
4272 #+begin_src ruby :session :results silent
4276 #+srcname: set-python-session-var
4277 #+begin_src python :session
4282 #+srcname: get-python-session-var
4283 #+begin_src python :session
4287 #+srcname: set-R-session-var
4288 #+begin_src R :session
4293 #+srcname: get-R-session-var
4294 #+begin_src R :session
4303 To run these examples evaluate [[file:lisp/org-babel-init.el][org-babel-init.el]]
4305 ** org-babel.el beginning functionality
4307 #+begin_src sh :results replace
4312 : Sun Jul 5 18:54:39 EDT 2009
4319 : Sun Jul 05 18:54:35 -0400 2009
4331 #+begin_src R :results replace
4341 hist(rgamma(20,3,3))
4346 ** org-babel plays with tables
4347 Alright, this should demonstrate both the ability of org-babel to read
4348 tables into a lisp source code block, and to then convert the results
4349 of the source code block into an org table. It's using the classic
4350 "lisp is elegant" demonstration transpose function. To try this
4353 1. evaluate [[file:lisp/org-babel-init.el]] to load org-babel and friends
4354 2. evaluate the transpose definition =\C-c\\C-c= on the beginning of
4356 3. evaluate the next source code block, this should read in the table
4357 because of the =:var table=previous=, then transpose the table, and
4358 finally it should insert the transposed table into the buffer
4359 immediately following the block
4363 #+begin_src emacs-lisp :results silent
4364 (defun transpose (table)
4365 (apply #'mapcar* #'list table))
4373 #+begin_src emacs-lisp :var table=sandbox :results replace
4378 #+begin_src emacs-lisp
4383 | 1 | 2 | 3 | 4 | 5 |
4387 #+begin_src ruby :var table=sandbox :results replace
4388 table.first.join(" - ")
4394 #+begin_src python :var table=sandbox
4399 #+begin_src ruby :var table=sandbox :results replace
4404 : [[1, 2, 3], [4, "schulte", 6]]
4408 | 4 | "schulte" | 6 |
4410 #+begin_src python :var table=sandbox :results replace
4416 | "__add__" | "__class__" | "__contains__" | "__delattr__" | "__delitem__" | "__delslice__" | "__doc__" | "__eq__" | "__format__" | "__ge__" | "__getattribute__" | "__getitem__" | "__getslice__" | "__gt__" | "__hash__" | "__iadd__" | "__imul__" | "__init__" | "__iter__" | "__le__" | "__len__" | "__lt__" | "__mul__" | "__ne__" | "__new__" | "__reduce__" | "__reduce_ex__" | "__repr__" | "__reversed__" | "__rmul__" | "__setattr__" | "__setitem__" | "__setslice__" | "__sizeof__" | "__str__" | "__subclasshook__" | "append" | "count" | "extend" | "index" | "insert" | "pop" | "remove" | "reverse" | "sort" |
4418 *** (sandbox table) R
4420 #+TBLNAME: sandbox_r
4424 #+begin_src R :results replace
4425 x <- c(rnorm(10, mean=-3, sd=1), rnorm(10, mean=3, sd=1))
4429 | -3.35473133869346 |
4431 | -3.32819924928633 |
4432 | -2.97310212756194 |
4433 | -2.09640758369576 |
4434 | -5.06054014378736 |
4435 | -2.20713700711221 |
4436 | -1.37618039712037 |
4437 | -1.95839385821742 |
4438 | -3.90407396475502 |
4439 | 2.51168071590226 |
4440 | 3.96753011570494 |
4441 | 3.31793212627865 |
4442 | 1.99829753972341 |
4443 | 4.00403686419829 |
4444 | 4.63723764452927 |
4445 | 3.94636744261313 |
4446 | 3.58355906547775 |
4447 | 3.01563442274226 |
4450 #+begin_src R var tabel=sandbox_r :results replace
4455 | 4 | "schulte" | 6 |
4458 Now shell commands are converted to tables using =org-table-import=
4459 and if these tables are non-trivial (i.e. have multiple elements) then
4460 they are imported as org-mode tables...
4462 #+begin_src sh :results replace
4466 | "total" | 208 | "" | "" | "" | "" | "" | "" |
4467 | "-rw-r--r--" | 1 | "dan" | "dan" | 57 | 2009 | 15 | "block" |
4468 | "-rw-r--r--" | 1 | "dan" | "dan" | 35147 | 2009 | 15 | "COPYING" |
4469 | "-rw-r--r--" | 1 | "dan" | "dan" | 722 | 2009 | 18 | "examples.org" |
4470 | "drwxr-xr-x" | 4 | "dan" | "dan" | 4096 | 2009 | 19 | "existing_tools" |
4471 | "-rw-r--r--" | 1 | "dan" | "dan" | 2207 | 2009 | 14 | "intro.org" |
4472 | "drwxr-xr-x" | 2 | "dan" | "dan" | 4096 | 2009 | 18 | "org-babel" |
4473 | "-rw-r--r--" | 1 | "dan" | "dan" | 277 | 2009 | 20 | "README.markdown" |
4474 | "-rw-r--r--" | 1 | "dan" | "dan" | 11837 | 2009 | 18 | "rorg.html" |
4475 | "-rw-r--r--" | 1 | "dan" | "dan" | 61829 | 2009 | 19 | "#rorg.org#" |
4476 | "-rw-r--r--" | 1 | "dan" | "dan" | 60190 | 2009 | 19 | "rorg.org" |
4477 | "-rw-r--r--" | 1 | "dan" | "dan" | 972 | 2009 | 11 | "test-export.org" |
4480 ** silent evaluation
4488 #+begin_src ruby :results silent
4492 #+begin_src ruby :results replace
4499 ** (sandbox) referencing other source blocks
4500 Doing this in emacs-lisp first because it's trivial to convert
4501 emacs-lisp results to and from emacs-lisp.
4503 *** emacs lisp source reference
4504 This first example performs a calculation in the first source block
4505 named =top=, the results of this calculation are then saved into the
4506 variable =first= by the header argument =:var first=top=, and it is
4507 used in the calculations of the second source block.
4510 #+begin_src emacs-lisp
4514 #+begin_src emacs-lisp :var first=top :results replace
4520 This example is the same as the previous only the variable being
4521 passed through is a table rather than a number.
4523 #+begin_src emacs-lisp :results silent
4524 (defun transpose (table)
4525 (apply #'mapcar* #'list table))
4528 #+TBLNAME: top_table
4532 #+SRCNAME: second_src_example
4533 #+begin_src emacs-lisp :var table=top_table
4537 #+begin_src emacs-lisp :var table=second_src_example :results replace
4542 | 4 | "schulte" | 6 |
4544 Now working for ruby
4551 #+begin_src ruby :var other=start :results replace
4557 #+SRCNAME: start_two
4562 #+begin_src python :var another=start_two :results replace
4567 Since all variables are converted into Emacs Lisp it is no problem to
4568 reference variables specified in another language.
4570 #+SRCNAME: ruby-block
4575 #+SRCNAME: lisp_block
4576 #+begin_src emacs-lisp :var ruby-variable=ruby-block
4580 #+begin_src python :var lisp_var=lisp_block
4589 #+begin_src R :results replace
4596 #+begin_src R :var other=first_r :results replace
4603 ** (sandbox) selective export
4605 For exportation tests and examples see (including exportation of
4606 inline source code blocks) [[file:test-export.org]]
4609 ** (sandbox) source blocks as functions
4612 #+begin_src emacs-lisp :results silent
4617 #+begin_src emacs-lisp :var n=default :results replace
4623 #+begin_src emacs-lisp :var result=triple(n=3, m=98) :results replace
4629 The following just demonstrates the ability to assign variables to
4630 literal values, which was not implemented until recently.
4632 #+begin_src ruby :var num="eric" :results replace
4639 ** (sandbox) inline source blocks
4641 This is an inline source code block src_ruby{1 + 6}. And another
4642 source block with text output src_emacs-lisp{"eric"}.
4644 This is an inline source code block with header
4645 arguments. src_ruby[:var n=fibbd( n = 0 )]{n}
4648 ** (sandbox) integration w/org tables
4650 #+begin_src emacs-lisp :results silent
4651 (defun fibbd (n) (if (< n 2) 1 (+ (fibbd (- n 1)) (fibbd (- n 2)))))
4655 #+begin_src emacs-lisp :var n=4 :results silent
4659 #+begin_src emacs-lisp :results silent
4660 (mapcar #'fibbd '(0 1 2 3 4 5 6 7 8))
4663 Something is not working here. The function `sbe ' works fine when
4664 called from outside of the table (see the source block below), but
4665 produces an error when called from inside the table. I think there
4666 must be some narrowing going on during intra-table emacs-lisp
4669 | original | fibbd |
4670 |----------+-------|
4681 #+TBLFM: $2='(sbe "fibbd" (n $1))
4685 #+begin_src emacs-lisp :results silent
4686 (sbe 'fibbd (n "8"))
4691 LocalWords: DBlocks dblocks org-babel el eric fontification