2 Time-stamp: <2012-10-04 02:31:38 tony>
3 Creation: <2008-09-08 08:06:30 tony>
8 Author: AJ Rossini <blindglobe@gmail.com>
9 Copyright: (c) 2007-2010, AJ Rossini <blindglobe@gmail.com>. BSD.
10 Purpose: Stuff that needs to be made working sits inside the
13 This file contains the current challenges to solve,
14 including a description of the setup and the work to
15 solve. Solutions welcome.
17 What is this talk of 'release'? Klingons do not make software
18 'releases'. Our software 'escapes', leaving a bloody trail of
19 designers and quality assurance people in its wake.
21 * Design Documentation
23 ** (Internal) Package and (External) System Hierarchy
24 *** Singletons (primary building blocks)
26 These are packages as well as
28 | asdf | common system loader |
29 | xarray | common access structure to array-like |
30 | | (matrix, vector) structures. |
31 | cls-config | initialization of Lisp state, variables, etc, |
32 | | localization to the particular lisp. |
33 | lift | unit-testing |
34 | cffi | foriegn function library |
40 as of now, all are in QL heirarchy
42 *** Dependency structure
44 | lisp-matrix | general purpose matrix package, linking to lapack | |
45 | | for numerics. Depends on: | |
48 | | cl-blapack | cffi |
50 | cls-dataframe | in the same spirit as lisp-matrix, a means to | |
51 | | create tables. Perhaps better called datatables? | |
52 | cls-probability | depends on gsll, cl-variates, cl-? initially, | |
68 Usually, we need to load it before going on.
75 though sometimes we might want to recompile fully. (Can this be
76 done via QL? Need to check)
78 #+name: compile-it-all
80 (asdf:oos 'asdf:compile-op :cls :force T)
85 - State "DONE" from "CURR" [2010-10-12 Tue 13:48] \\
86 setup is mostly complete
87 - State "CURR" from "TODO" [2010-10-12 Tue 13:47]
88 - State "TODO" from "" [2010-10-12 Tue 13:47]
89 This is an example of a custom setup, not really interesting at
90 this point except to remind Tony how to program.
95 (defun init-CLS (&key (compile 'nil))
96 (let ((packagesToLoad (list ;; core system
97 :lift :lisp-matrix :cls
99 ;; :cl-cairo2-x11 :iterate
102 :cl-pdf :cl-typesetting
104 :asdf-system-connections :xarray
106 :metatilities-base :anaphora :tinaa
107 :cl-ppcre :cl-markdown :docudown
108 ;; version and validate CLOS objects
109 ;; :versioned-objects :validations
112 ;; :cl-glu :cl-glut :cl-glut-examples
117 (mapcar #'(lambda (x)
119 (asdf:oos 'asdf:compile-op x :force T)
120 (asdf:oos 'asdf:load-op x)))
123 (init-CLS)) ;; vs (init-CLS :compile T)
127 | | #<PACKAGE "COMMON-LISP-USER"> |
129 ** TODO [#A] Integrate with quicklist support.
130 - State "TODO" from "" [2010-11-30 Tue 18:00]
132 important to merge with quicklisp system loader support.
133 ** CURR [#A] Testing: unit, regression, examples. [0/3]
134 - State "CURR" from "TODO" [2010-10-12 Tue 13:51]
135 - State "TODO" from "" [2010-10-12 Tue 13:51]
136 Testing consists of unit tests, which internally verify subsets of
137 code, regression tests, and functional tests (in increasing order
139 *** CURR [#B] Unit tests
140 - State "CURR" from "TODO" [2010-11-04 Thu 18:33]
141 - State "CURR" from "TODO" [2010-10-12 Tue 13:48]
142 - State "TODO" from "" [2010-10-12 Tue 13:48]
143 Unit tests have been started using LIFT. Need to consider some of
144 the other systems that provide testing, when people add them to the
145 mix of libraries that we need, along with examples of how to use.
147 #+srcname: ex-cls-unittest
149 (in-package :lisp-stat-unittests)
150 (run-tests :suite 'lisp-stat-ut)
154 : #<Results for LISP-STAT-UT 78 Tests, 7 Failures, 20 Errors>
156 ;; => tests = 78, failures = 7, errors = 20
160 The following needs to be solved in order to have a decent
161 installation qualification (IQ) and performance qualification (PQ)
163 #+srcname: cls-unittest
165 (in-package :lisp-stat-unittests)
166 (asdf:oos 'asdf:test-op 'cls)
167 ;; which runs (describe (run-tests :suite 'lisp-stat-ut))
172 and check documentation to see if it is useful.
175 (in-package :lisp-stat-unittests)
177 (describe 'lisp-stat-ut)
178 (documentation 'lisp-stat-ut 'type)
180 ;; FIXME: Example: currently not relevant, yet
181 ;; (describe (lift::run-test :test-case 'lisp-stat-unittests::create-proto
182 ;; :suite 'lisp-stat-unittests::lisp-stat-ut-proto))
184 (describe (lift::run-tests :suite 'lisp-stat-ut-dataframe))
185 (lift::run-tests :suite 'lisp-stat-ut-dataframe)
187 (describe (lift::run-test
188 :test-case 'lisp-stat-unittests::create-proto
189 :suite 'lisp-stat-unittests::lisp-stat-ut-proto))
192 *** TODO [#B] Regression Tests
193 - State "TODO" from "" [2010-10-12 Tue 13:54]
195 *** TODO [#B] Functional Tests
196 - State "TODO" from "" [2010-10-12 Tue 13:54]
198 ** CURR [#B] Functional Examples that need to work [1/2]
199 - State "CURR" from "TODO" [2010-11-30 Tue 17:57]
200 - State "TODO" from "" [2010-10-12 Tue 13:55]
202 These examples should be functional forms within CLS, describing
203 working functionality which is needed for work.
205 *** TODO [#B] Scoping with datasets
206 - State "TODO" from "" [2010-11-04 Thu 18:46]
208 The following needs to work, and a related syntax for resampling
209 and similar synthetic data approaches (bootstrapping, imputation)
210 ought to use similar syntax as well.
211 #+srcname: DataSetNameScoping
213 (in-package :ls-user)
215 ;; Syntax examples using lexical scope, closures, and bindings to
216 ;; ensure a clean communication of results
217 ;; This is actually a bit tricky, since we need to clarify whether
218 ;; it is line-at-a-time that we are considering or if there is
219 ;; another mapping strategy. In particular, one could imagine a
220 ;; looping-over-observations function, or a
221 ;; looping-over-independent-observations function which leverages a
222 ;; grouping variable which provides guidance for what is considered
223 ;; independent from the sampling frame being considered. The frame
224 ;; itself (definable via some form of metadata to clarify scope?)
225 ;; could clearly provide a bit of relativity for clarifying what
226 ;; statistical independence means.
228 (with-data dataset ((dsvarname1 [usevarname1])
229 (dsvarname2 [usevarname2]))
232 ;; SAS-centric approach to spec'ing work
233 (looping-over-observations
234 dataset ((dsvarname1 [usevarname1])
235 (dsvarname2 [usevarname2]))
238 ;; SAS plus "statistical sensibility"... for example, if an
239 ;; independent observation actually consists of many observations so
240 ;; that a dataframe of independence results -- for example,
241 ;; longitudinal data or spatial data or local-truncated network data
242 ;; are clean examples of such happening -- then we get the data
243 ;; frame or row representing the independent result.
244 (looping-over-independent-observations
245 dataset independence-defining-variable
246 ((dsvarname1 [usevarname1])
247 (dsvarname2 [usevarname2]))
252 *** DONE [#B] Dataframe variable typing
253 - State "DONE" from "CURR" [2010-11-30 Tue 17:56] \\
254 check-type approach works, we would just have to throw a catchable
255 error if we want to use it in a reliable fashion.
256 - State "CURR" from "TODO" [2010-11-30 Tue 17:56]
257 - State "TODO" from "" [2010-11-04 Thu 18:48]
259 Seems to generally work, need to ensure that we use this for
262 #+srcname: DFvarTyping
264 (in-package :ls-user)
265 (defparameter *df-test*
266 (make-instance 'dataframe-array
267 :storage #2A (('a "test0" 0 0d0)
273 :case-labels (list "0" "1" 2 "3" "4")
274 :var-labels (list "symbol" "string" "integer" "double-float")
275 :var-types (list 'symbol 'string 'integer 'double-float)))
277 ;; with SBCL, ints become floats? Need to adjust output
278 ;; representation appropriately..
281 (defun check-var (df colnum)
282 (let ((nobs (xdim (dataset df) 0)))
284 (check-type (xref df i colnum) (elt (var-types df) i)))))
286 (xdim (dataset *df-test*) 1)
287 (xdim (dataset *df-test*) 0)
289 (check-var *df-test* 0)
292 (xref *df-test* 1 1))
294 (check-type (xref *df-test* 1 1)
295 string) ;; => nil, so good.
296 (check-type (xref *df-test* 1 1)
297 vector) ;; => nil, so good.
298 (check-type (xref *df-test* 1 1)
299 real) ;; => simple-error type thrown, so good.
301 ;; How to nest errors within errors?
302 (check-type (check-type (xref *df-test* 1 1) real) ;; => error thrown, so good.
306 (check-type *df-test*
307 dataframe-array) ; nil is good.
309 (integerp (xref *df-test* 1 2))
310 (floatp (xref *df-test* 1 2))
311 (integerp (xref *df-test* 1 3))
312 (type-of (xref *df-test* 1 3))
313 (floatp (xref *df-test* 1 3))
315 (type-of (vector 1 1d0))
321 (xref *df-test* 1 '*)
324 ** CURR [#A] Random Numbers [2/6]
325 - State "CURR" from "TODO" [2010-11-05 Fri 15:41]
326 - State "TODO" from "" [2010-10-14 Thu 00:12]
328 Need to select and choose a probability system (probability
329 functions, random numbers). Goal is to have a general framework
330 for representing probability functions, functionals on
331 probabilities, and reproducible random streams based on such
333 *** CURR [#B] CL-VARIATES system evaluation [2/3]
334 - State "CURR" from "TODO" [2010-11-05 Fri 15:40]
335 - State "TODO" from "" [2010-10-12 Tue 14:16]
337 CL-VARIATES is a system developed by Gary W King. It uses streams
338 with seeds, and is hence reproducible. (Random comment: why do CL
339 programmers as a class ignore computational reproducibility?)
340 **** DONE [#B] load and verify
341 - State "DONE" from "CURR" [2010-11-04 Thu 18:59] \\
342 load, init, and verify performance.
343 - State "CURR" from "TODO" [2010-11-04 Thu 18:58]
344 - State "TODO" from "" [2010-11-04 Thu 18:58]
346 <2010-11-30 Tue> : just modified cls.asd to ensure that we load
347 as appropriate the correct random variate package.
349 #+srcname: Loading-CL-VARIATES
351 (in-package :cl-user)
352 (asdf:oos 'asdf:load-op 'cl-variates)
353 (asdf:oos 'asdf:load-op 'cl-variates-test)
357 : #<ASDF:LOAD-OP NIL {C2C30E1}>
361 #+srcname: CL-VARIATES-UNITTESTS
364 (in-package :cl-variates-test)
366 (run-tests :suite 'cl-variates-test)
367 (describe (run-tests :suite 'cl-variates-test))
371 **** DONE [#B] Examples of use
372 - State "DONE" from "CURR" [2010-11-05 Fri 15:39] \\
373 basic example of reproducible draws from the uniform and normal random
375 - State "CURR" from "TODO" [2010-11-05 Fri 15:39]
376 - State "TODO" from "" [2010-11-04 Thu 19:01]
378 #+srcname: CL-VARIATES-REPRO
381 (in-package :cl-variates-user)
383 (defparameter state (make-random-number-generator))
384 (setf (random-seed state) 44)
387 (loop for i from 1 to 10 collect
388 (random-range state 0 10))
389 ;; => (1 5 1 0 7 1 2 2 8 10)
390 (setf (random-seed state) 44)
391 (loop for i from 1 to 10 collect
392 (random-range state 0 10))
393 ;; => (1 5 1 0 7 1 2 2 8 10)
395 (setf (random-seed state) 44)
397 (loop for i from 1 to 10 collect
398 (normal-random state 0 1))
400 ;; (-1.2968656102820426 0.40746363934173213 -0.8594712469518473 0.8795681301148328
401 ;; 1.0731526250004264 -0.8161629082481728 0.7001813608754809 0.1078045427044097
402 ;; 0.20750134211656893 -0.14501914108452274)
404 (setf (random-seed state) 44)
405 (loop for i from 1 to 10 collect
406 (normal-random state 0 1))
408 ;; (-1.2968656102820426 0.40746363934173213 -0.8594712469518473 0.8795681301148328
409 ;; 1.0731526250004264 -0.8161629082481728 0.7001813608754809 0.1078045427044097
410 ;; 0.20750134211656893 -0.14501914108452274)
414 **** CURR [#B] Full example of general usage
415 - State "CURR" from "TODO" [2010-11-05 Fri 15:40]
416 - State "TODO" from "" [2010-11-05 Fri 15:40]
418 What we want to do here is describe the basic available API that
419 is present. So while the previous work describes what the basic
420 reproducibility approach would be in terms of generating lists of
421 reproducible pRNG streams, we need the full range of possible
422 probability laws that are present.
424 One of the good things about cl-variates is that it provides for
425 reproducibility. One of the bad things is that it has a mixed
428 *** TODO [#B] CL-RANDOM system evaluation
429 - State "TODO" from "" [2010-11-05 Fri 15:40]
432 1. no seed setting for random numbers
433 2. contamination of a probability support with optimization and
438 2. nice design for generics.
440 *** TODO [#B] Native CLS (from XLS)
441 - State "TODO" from "" [2010-11-05 Fri 15:40]
443 ** TODO [#B] Numerical Linear Algebra
444 - State "TODO" from "" [2010-10-14 Thu 00:12]
446 *** TODO [#B] LLA evaluation
447 - State "TODO" from "" [2010-10-12 Tue 14:13]
448 ;;; experiments with LLA
449 (in-package :cl-user)
450 (asdf:oos 'asdf:load-op 'lla)
451 (in-package :lla-user)
453 *** CURR [#B] Lisp-Matrix system evaluation
454 - State "CURR" from "TODO" [2010-10-12 Tue 14:13]
455 - State "TODO" from "" [2010-10-12 Tue 14:13]
457 *** TODO [#B] LispLab system evaluation
458 - State "TODO" from "" [2010-10-12 Tue 14:13]
460 ** TODO [#B] Statistical Procedures to implement
461 - State "TODO" from "" [2010-10-14 Thu 00:12]
464 (in-package :cls-user)
469 ;; population design eval and opt
477 number of samples/cost of lab analysis and collection
481 (defun pfim (&key model ( constraints ( summary-function )
483 (list num-subjects num-times list-times))))
487 Each individal has a deisgn psi_i
488 nubmer of samples n_i and sampling times t_{i{1}} t_{i{n_1}}
489 individuals can differ
493 individual-level model
496 (=model y_i (+ (f \theta_i \psi_i) epsilion_i ))
497 (=var \epsilion_i \sigma_between \sigma_within )
499 ;; Information Matrix for pop deisgn
501 (defparameter IM (sum (i 1 N) (MF \psi_i \phi_i)))
504 For nonlinear structureal models, expand around RE=0
506 Cramer-Rao : MF^{-1} is lower bound for estimation variance.
510 - smallest SE, but is a matrix, so
511 - criteria for matrix comparison
512 -- D-opt, (power (determinant MF) (/ 1 P))
515 find design maxing D opt, (power (determinant MF) (/ 1 P))
517 -- contin vars for smapling times within interval or set -- number of groups for cat vars
519 Stat in Med 2009, expansion around post-hoc RE est, not necessarily zero.
521 Example binary covariate C
524 (if (= i reference-class)
529 (=model (log \theta) ( ))
536 PFIM provides for a given design and values of \beta:
538 SE/RSE for \beta of each class of each covar
539 eval influence of design on SE(\beta)
541 inter-occassion variability (IOV)
542 - patients sampled more than once, H occassions
544 - additional vars to estimate
548 ;;; comparison criteria
550 functional of conc/time curve which is used for comparison, i.e.
551 (AUC conc/time-curve)
552 (Cmax conc/time-curve)
553 (Tmax conc/time-curve)
557 (defun conc/time-curve (t)
560 (let ((conc (exp (* t \beta1))))
566 (url-get "www.pfim.biostat.fr")
569 ;;; Thinking of generics...
570 (information-matrix model parameters)
571 (information-matrix variance-matrix)
572 (information-matrix model data)
573 (information-matrix list-of-individual-IMs)
576 (defun IM (loglikelihood parameters times)
577 "Does double work. Sum up the resulting IMs to form a full IM."
578 (let ((IM (make-matrix (length parameters)
580 :initial-value 0.0d0)))
581 (dolist (parameterI parameters)
582 (dolist (parameterJ parameters)
584 (differentiate (differentiate loglikelihood parameterI) parameterJ))))))
586 *** difference between empirical, fisherian, and ...? information.
587 *** Example of Integration with CL-GENOMIC
588 - State "TODO" from "" [2010-10-12 Tue 14:03]
590 CL-GENOMIC is a very interesting data-structure strategy for
591 manipulating sequence data.
595 (in-package :cl-user)
596 (asdf:oos 'asdf:compile-op :ironclad)
597 (asdf:oos 'asdf:load-op :cl-genomic)
599 (in-package :bio-sequence)
600 (make-dna "agccg") ;; fine
601 (make-aa "agccg") ;; fine
602 (make-aa "agc9zz") ;; error expected
605 ** TODO [#B] Documentation and Examples [0/3]
606 - State "TODO" from "" [2010-10-14 Thu 00:12]
608 *** TODO [#B] Docudown
609 - State "TODO" from "" [2010-11-05 Fri 15:34]
612 - State "TODO" from "" [2010-11-05 Fri 15:34]
614 *** TODO [#B] CLPDF, and literate data analysis
615 - State "TODO" from "" [2010-11-05 Fri 15:34]
618 Place proposals for features, work, etc here...
619 ** <2011-12-29 Thu> new stuff
620 First new proposal is to track proposals.
622 This project is dedicated to all the lisp hackers out there who
623 provided the basic infrastructure to get so far so fast with minimal