1 #+OPTIONS: H:3 num:nil toc:t \n:nil @:t ::t |:t ^:t -:t f:t *:t TeX:t LaTeX:t skip:nil d:(HIDE) tags:not-in-toc
2 #+TITLE: rorg --- Code evaluation in org-mode, with an emphasis on R
3 #+SEQ_TODO: TODO PROPOSED | DONE DROPPED MAYBE
6 * Overview (Dan [2009-02-08 Sun])
8 This project is basically about putting source code into org
9 files. This isn't just code to look pretty as a source code example,
10 but code to be evaluated. Org files have 3 main export targets: org,
11 html and latex. Thus the aim of this project is to produce files in
12 those formats that have benefitted in some way from the evaluation of
13 source code that is present in the source org file. We have a current
14 focus on R code, but we are regarding that more as a working example
15 than as a defining feature of the project.
17 Code evaluation can have three relevant consequences. Our aim is to
18 deal with these consequences as follows:
20 *** It produces text/numeric output
21 We (optionally) incorporate the text output as text in the target
23 *** It produces graphical output
24 We either link to the graphics or (html/latex) include them inline.
25 *** It creates some non-graphics files
26 ? We link to other file output
27 *** It alters the environment by side effect in some other way
30 ** Implementation questions
31 These objectives raise three questions:
33 1. How is the code placed in the org file?
34 2. When is the code evaluated?
35 3. What is the result of code evaluation?
37 *** How is the code placed in the org file?
38 Using some version of the code block ideas that Eric and Austin
39 have worked on. (In addition, an aim of org-R was to allow Org
40 users who are not R users to specify R code implicitly, using
41 native org syntax. I'd like to maintain that, but it's not central
44 *** When is the code evaluated?
45 Let's use an asterisk to indicate content which includes the
46 *result* of code evaluation, rather than the code itself. Clearly
47 we have a requirement for the following transformation:
51 Let's say this transformation is effected by a function
52 `org-eval-buffer'. This transformation is necessary when the
53 target format is org (say you want to update the values in an org
54 table, or generate a plot and create an org link to it), and it
55 can also be used as the first step by which to reach html and
60 org \to org* \to latex
62 Thus in principle we can reach our 3 target formats with
63 `org-eval-buffer', `org-export-as-latex' and `org-export-as-html'.
65 An extra transformation that we might want is
69 I.e. export to latex without evaluation of code, in such a way that R
70 code can subsequently be evaluated using
71 =Sweave(driver=RweaveLatex)=, which is what the R community is
72 used to. This would provide a `bail out' avenue where users can
73 escape org mode and enter a workflow in which the latex/noweb file
76 **** How do we implement `org-eval-buffer'?
78 AIUI The following can all be viewed as implementations of
79 org-eval-buffer for R code:
82 This is the beginnings of a general evaluation mechanism, that
83 could evaluate python, ruby, shell, perl, in addition to R.
84 The header says it's based on org-eval, what is org-eval??
87 This accomplishes org \to org* in elisp by visiting code blocks
88 and evaluating code using ESS.
91 This accomplishes org \to org* using R via
93 : Sweave("file-with-unevaluated-code.org", driver=RweaveOrg, syntax=SweaveSyntaxOrg)
95 ***** org-exp-blocks.el
96 Like org-R, this achieves org \to org* in elisp by visiting code
97 blocks and using ESS to evaluate R code.
100 *** What is the result of code evaluation?
101 Here we have to consider text/numeric output, and graphical
102 output. And also the stage at which evaluation occurs
104 ****** Text / numerical output
105 In the case of org \to org*, I would argue that, where
106 appropriate, it should be stored in org tables. Thus an advantage
107 our project would have over Sweave is that tabular output is
108 automatically conveqrted to native tables on export to HTML and
110 ****** Graphical output
111 We place an org link to the file. This is done already by
112 org-R-apply, and by RweaveOrg.
113 ***** latex \to latex*
114 This is done by Sweave(driver=RweaveLatex) and so is out of our hands
118 ** Eric <2009-02-06 Fri 15:41>
119 I think we're getting close to a comprehensive set of objectives
120 (although since you two are the real R user's I leave that decision up
121 to you). Once we've agreed on a set of objectives and agreed on at
122 least to broad strokes of implementation, I think we should start
123 listing out and assigning tasks.
127 ** Send data to R from org
128 Org-mode includes orgtbl-mode, an extremely convenient way of using
129 tabular data in a plain text file. Currently, spreadsheet
130 functionality is available in org tables using the emacs package
131 calc. It would be a boon both to org users and R users to allow
132 org tables to be manipulated with the R programming language. Org
133 tables give R users an easy way to enter and display data; R gives
134 org users a powerful way to perform vector operations, statistical
135 tests, and visualization on their tables.
139 Naive implementation would be to use =(org-export-table "tmp.csv")=
140 and =(ess-execute "read.csv('tmp.csv')")=.
142 org-R passes data to R from two sources: org tables, or csv
143 files. Org tables are first exported to a temporary csv file
144 using [[file:existing_tools/org-R.el::defun%20org%20R%20export%20to%20csv%20csv%20file%20options][org-R-export-to-csv]].
146 org-exp-blocks uses [[org-interblock-R-command-to-string]] to send
147 commands to an R process running in a comint buffer through ESS.
148 org-exp-blocks has no support for dumping table data to R process, or
155 ** Evaluate R code from org and deal with output appropriately
157 When R code evaluation generates vectors and 2-dimensional arrays,
158 this should be formatted appropriately in org buffers (orgtbl-mode) as well
159 as in export targets (html, latex)
161 Agreed, if we can convert the vector data to lists then we can use
162 the many orgtbl-to-* functions to convert the list to whatever
163 output format we desire. See `orgtbl-to-orgtbl, `orgtbl-to-latex',
164 `orgtbl-to-html', `orgtbl-to-csv', etc...
168 org-R converts R output (vectors, or matrices / 2d-arrays) to an
169 org table and stores it in the org buffer, or in a separate org
170 file (csv output would also be perfectly possible).
174 R can generate graphical output on a screen graphics device
175 (e.g. X11, quartz), and in various standard image file formats
176 (png, jpg, ps, pdf, etc). When graphical output is generated by
177 evaluation of R code in Org, at least the following two things are desirable:
178 1. output to screen for immediate viewing is possible
179 2. graphical output to file is linked to appropriately from the
180 org file This should have the automatic consequence that it is
181 included appropriately in subsequent export targets (html,
186 org-R does (1) if no output file is specified and (2) otherwise
188 org-exp-blocks tries to do 2, but I don't think that part was
194 ** Evaluate R code on export
195 At first I was leaning towards leaving the exporting to Sweave, but it
196 seems that once we have evaluation or R working, it will not be
197 difficult to implement full evaluation of R blocks, one-liners, and
198 creation of R graphics on export directly in elisp.
200 I think that this would be worth the effort since it would eliminate
201 the need for using Sweave, and would allow for exportation to any
202 target format supported by org-mode.
206 ** Special editing and evaluation of source code in R blocks
207 Unfortunately org-mode how two different block types, both useful.
208 In developing RweaveOrg, a third was introduced.
210 Eric is leaning towards using the =#+begin_src= blocks, as that is
211 really what these blocks contain: source code. Austin believes
212 that specifying export options at the beginning of a block is
213 useful functionality, to be preserved if possible.
215 Note that upper and lower case are not relevant in block headings.
217 *** PROPOSED R-block proposal
218 I (Eric) propose that we use the syntax of source code blocks as they
219 currently exist in org-mode with the addition of *evaluation*,
220 *header-arguments*, *exportation*, *single-line-blocks*, and
221 *references-to-table-data*.
223 1) *evaluation*: These blocks can be evaluated through =\C-c\C-c= with
224 a slight addition to the code already present and working in
225 [[file:existing_tools/org-eval-light.el][org-eval-light.el]]. All we should need to add for R support would
226 be an appropriate entry in [[org-eval-light-interpreters]] with a
227 corresponding evaluation function. For an example usinga
228 org-eval-light see [[* src block evaluation w/org-eval-light]].
230 2) *header-arguments*: These can be implemented along the lines of
231 Austin's header arguments in [[file:existing_tools/RweaveOrg/org-sweave.el][org-sweave.el]].
233 3) *exportation*: Should be as similar as possible to that done by
234 Sweave, and hopefully can re-use some of the code currently present
235 in [[file:existing_tools/exp-blocks/org-exp-blocks.el ][org-exp-blocks.el]].
237 4) *single-line-blocks*: It seems that it is useful to be able to
238 place a single line of R code on a line by itself. Should we add
239 syntax for this similar to Dan's =#+R:= lines? I would lean
240 towards something here that can be re-used for any type of source
241 code in the same manner as the =#+begin_src R= blocks, maybe
244 5) *references-to-table-data*: I get this impression that this is
245 vital to the efficient use of R code in an org file, so we should
246 come up with a way to reference table data from a single-line-block
247 or from an R source-code block. It looks like Dan has already done
248 this in [[file:existing_tools/org-R.el][org-R.el]].
250 What do you think? Does this accomplish everything we want to be able
251 to do with embedded R source code blocks?
253 **** src block evaluation w/org-eval-light
254 here's an example using org-eval-light.el
256 first load the org-eval-light.el file
258 [[elisp:(load (expand-file-name "org-eval-light.el" (expand-file-name "existing_tools" (file-name-directory buffer-file-name))))]]
260 then press =\C-c\C-c= inside of the following src code snippet. The
261 results should appear in a comment immediately following the source
262 code block. It shouldn't be too hard to add R support to this
263 function through the `org-eval-light-interpreters' variable.
265 (Dan: The following causes error on export to HTML hence spaces inserted at bol)
271 *** Source code blocks
272 Org has an extremely useful method of editing source code and
273 examples in their native modes. In the case of R code, we want to
274 be able to use the full functionality of ESS mode, including
275 interactive evaluation of code.
277 Source code blocks look like the following and allow for the
278 special editing of code inside of the block through
283 ,## hit C-c ' within this block to enter a temporary buffer in r-mode.
285 ,## while in the temporary buffer, hit C-c C-c on this comment to
286 ,## evaluate this block
290 ,## hit C-c ' to exit the temporary buffer
294 dblocks are useful because org-mode will automatically call
295 `org-dblock-write:dblock-type' where dblock-type is the string
296 following the =#+BEGIN:= portion of the line.
298 dblocks look like the following and allow for evaluation of the
299 code inside of the block by calling =\C-c\C-c= on the header of
306 In developing RweaveOrg, Austin created [[file:existing_tools/RweaveOrg/org-sweave.el][org-sweave.el]]. This
307 allows for the kind of blocks shown in [[file:existing_tools/RweaveOrg/testing.Rorg][testing.Rorg]]. These blocks
308 have the advantage of accepting options to the Sweave preprocessor
309 following the #+BEGIN_R declaration.
311 ** Interaction with the R process
313 We should take care to implement this in such a way that all of the
314 different components which have to interactive with R including:
315 - evaluation of source code blocks
316 - automatic evaluation on export
317 - evaluation of \R{} snippets
318 - evaluation of single source code lines
319 - sending/receiving vector data
321 I think we currently have two implementations of interaction with R
322 processes; [[file:existing_tools/org-R.el][org-R.el]] and [[file:existing_tools/exp-blocks/org-exp-blocks.el ][org-exp-blocks.el]]. We should be sure to take
323 the best of each of these approaches.
330 LocalWords: DBlocks dblocks