a bit better, but still not sure P_STOP does everything right

[dmvccm.git] / DMVCCM.org

# -*- coding: mule-utf-8-unix -*-

#+STARTUP: overview
#+TAGS: OPTIMIZE PRETTIER
#+STARTUP: hidestars
#+TITLE: DMV/CCM -- todo-list / progress
#+AUTHOR: Kevin Brubeck Unhammer
#+EMAIL: K.BrubeckUnhammer at student uva nl 
#+OPTIONS: H:4 toc:3 ^:{} 
#+LANGUAGE: en
#+SEQ_TODO: TOGROK TODO DONE

* dmvccm report and project
  DEADLINE: <2008-06-30 Mon>
But absolute, extended, really-quite-dead-now deadline: August
31... 
- [[file:src/dmv.py][dmv.py]]
- [[file:src/io.py][io.py]]
- [[file:src/harmonic.py::harmonic%20py%20initialization%20for%20dmv][harmonic.py]]


* TODO [#A] P_STOP and P_CHOOSE for IO/EM
[[file:src/dmv.py::DMV%20probabilities][dmv-P_STOP]]
Remember: The P_{STOP} formula is upside-down (left-to-right also).
(In the article..not the thesis)

Remember: Initialization makes some "short-cut" rules, these will also
have to be updated along with the other P_{STOP} updates:
- b[(NOBAR, n_{h}), 'h'] = 1.0       # always
- b[(RBAR, n_{h}), 'h'] = h_.probA  # h_ is RBAR stop rule
- b[(LRBAR, n_{h}), 'h'] = h_.probA * _ h_.probA

** P_STOP formulas for various dir and adj:
Assuming this:
- P_{STOP}(STOP|h,L,non_adj) = \sum_{corpus} \sum_{s<loc(h)} \sum_{t}
  inner(s,t,_h_, ...) / \sum_{corpus} \sum_{s<loc(h)} \sum_{t}
  inner(s,t,h_, ...)
- P_{STOP}(STOP|h,L,adj) = \sum_{corpus} \sum_{s=loc(h)} \sum_{t}
  inner(s,t,_h_, ...) / \sum_{corpus} \sum_{s=loc(h)} \sum_{t}
  inner(s,t,h_, ...)
- P_{STOP}(STOP|h,R,non_adj) = \sum_{corpus} \sum_{s} \sum_{t>loc(h)}
  inner(s,t,_h_, ...) / \sum_{corpus} \sum_{s} \sum_{t>loc(h)}
  inner(s,t,h_, ...)
- P_{STOP}(STOP|h,R,adj) = \sum_{corpus} \sum_{s} \sum_{t=loc(h)}
  inner(s,t,_h_, ...) / \sum_{corpus} \sum_{s} \sum_{t=loc(h)}
  inner(s,t,h_, ...)

(And P_{STOP}(-STOP|...) = 1 - P_{STOP}(STOP|...) )
** P_CHOOSE formula:
Assuming this:
- P_{CHOOSE}(a|h,L) = \sum_{corpus} \sum_{s<loc(h)} \sum_{t=loc(h)} \sum_{r<t}
  inner(s,r,_a_, ...) / \sum_{corpus} \sum_{s<loc(h)} \sum_{t=loc(h)}
  inner(s,t,h_,...)
  - or t >= loc(h)?
- P_{CHOOSE}(a|h,R) = \sum_{corpus} \sum_{s=loc(h)} \sum_{r>s}
  inner(s,r,_a_,...) / \sum_{corpus} \sum_{s=loc(h)} \sum_{t}
  inner(s,t,h, ...)
  - or s => loc(h)?


* TOGROK Find Most Probable Parse of given test sentence
One way of finding the MPP is to construct all complete trees, so
could this be used with P_STOP to make sure we only get counts from
complete trees? 

(But then, there are probably MPP algorithms that avoid computing _all_
complete trees for a sentence...)
* TOGROK Combine the models
* TODO Get a dependency parsed version of WSJ to test with
* Initialization   
[[file:~/Documents/Skole/V08/Probability/dmvccm/src/dmv.py::Initialization%20todo][dmv-inits]]

We do have to go through the corpus, since the probabilities are based
on how far away in the sentence arguments are from their heads.
** TOGROK CCM Initialization    
P_{SPLIT} used here... how, again?
** DONE Separate initialization to another file?                      :PRETTIER:
   CLOSED: [2008-06-08 Sun 12:51]
[[file:src/harmonic.py::harmonic%20py%20initialization%20for%20dmv][harmonic.py]]
** DONE DMV Initialization probabilities
(from initialization frequency)
** DONE DMV Initialization frequencies    
   CLOSED: [2008-05-27 Tue 20:04]
*** P_STOP    
P_{STOP} is not well defined by K&M. One possible interpretation given
the sentence [det nn vb nn] is
: f_{STOP}( STOP|det, L, adj) +1
: f_{STOP}(-STOP|det, L, adj) +0  
: f_{STOP}( STOP|det, L, non_adj) +1
: f_{STOP}(-STOP|det, L, non_adj) +0
: f_{STOP}( STOP|det, R, adj) +0
: f_{STOP}(-STOP|det, R, adj) +1
:
: f_{STOP}( STOP|nn, L, adj) +0
: f_{STOP}(-STOP|nn, L, adj) +1
: f_{STOP}( STOP|nn, L, non_adj) +1  # since there's at least one to the left
: f_{STOP}(-STOP|nn, L, non_adj) +0
**** TODO tweak
# <<pstoptweak>>
:            f[head,  'STOP', 'LN'] += (i_h <= 1)     # first two words
:            f[head, '-STOP', 'LN'] += (not i_h <= 1)     
:            f[head,  'STOP', 'LA'] += (i_h == 0)     # very first word
:            f[head, '-STOP', 'LA'] += (not i_h == 0)     
:            f[head,  'STOP', 'RN'] += (i_h >= n - 2) # last two words
:            f[head, '-STOP', 'RN'] += (not i_h >= n - 2) 
:            f[head,  'STOP', 'RA'] += (i_h == n - 1) # very last word
:            f[head, '-STOP', 'RA'] += (not i_h == n - 1) 
vs
:            # this one requires some additional rewriting since it
:            # introduces divisions by zero
:            f[head,  'STOP', 'LN'] += (i_h == 1)     # second word
:            f[head, '-STOP', 'LN'] += (not i_h <= 1) # not first two
:            f[head,  'STOP', 'LA'] += (i_h == 0)     # first word
:            f[head, '-STOP', 'LA'] += (not i_h == 0) # not first
:            f[head,  'STOP', 'RN'] += (i_h == n - 2)     # second-to-last
:            f[head, '-STOP', 'RN'] += (not i_h >= n - 2) # not last two
:            f[head,  'STOP', 'RA'] += (i_h == n - 1)     # last word
:            f[head, '-STOP', 'RA'] += (not i_h == n - 1) # not last
vs
:            f[head,  'STOP', 'LN'] += (i_h == 1)     # second word
:            f[head, '-STOP', 'LN'] += (not i_h == 1) # not second
:            f[head,  'STOP', 'LA'] += (i_h == 0)     # first word
:            f[head, '-STOP', 'LA'] += (not i_h == 0) # not first
:            f[head,  'STOP', 'RN'] += (i_h == n - 2)     # second-to-last
:            f[head, '-STOP', 'RN'] += (not i_h == n - 2) # not second-to-last
:            f[head,  'STOP', 'RA'] += (i_h == n - 1)     # last word
:            f[head, '-STOP', 'RA'] += (not i_h == n - 1) # not last
vs 
"all words take the same number of arguments" interpreted as
:for all heads:
:    p_STOP(head, 'STOP', 'LN') = 0.3
:    p_STOP(head, 'STOP', 'LA') = 0.5
:    p_STOP(head, 'STOP', 'RN') = 0.4
:    p_STOP(head, 'STOP', 'RA') = 0.7
(which we easily may tweak in init_zeros())
*** P_CHOOSE
Go through the corpus, counting distances between heads and
arguments. In [det nn vb nn], we give 
- f_{CHOOSE}(nn|det, R) +1/1 + C
- f_{CHOOSE}(vb|det, R) +1/2 + C
- f_{CHOOSE}(nn|det, R) +1/3 + C
  - If this were the full corpus, P_{CHOOSE}(nn|det, R) would have
    (1+1/3+2C) / sum_a f_{CHOOSE}(a|det, R)

The ROOT gets "each argument with equal probability", so in a sentence
of three words, 1/3 for each (in [nn vb nn], 'nn' gets 2/3). Basically
just a frequency count of the corpus...
* [#C] Deferred
** TODO if loc_h == t, no need to try right-attachment rules          :OPTIMIZE:
** TODO if loc_h == s, no need to try left-attachment rules           :OPTIMIZE:
** TODO when reestimating P_STOP etc, remove rules with p < epsilon   :OPTIMIZE:
** TODO inner_dmv, short ranges and impossible attachment             :OPTIMIZE:
If s-t <= 2, there can be only one attachment below, so don't recurse
with both Lattach=True and Rattach=True.

If s-t <= 1, there can be no attachment below, so only recurse with
Lattach=False, Rattach=False.

Put this in the loop under rewrite rules (could also do it in the STOP
section, but that would only have an effect on very short sentences).
** TODO clean up the module files                                     :PRETTIER:
Is there better way to divide dmv and harmonic? There's a two-way
dependency between the modules. Guess there could be a third file that
imports both the initialization and the actual EM stuff, while a file
containing constants and classes could be imported by all others:
: dmv.py imports dmv_EM.py imports dmv_classes.py
: dmv.py imports dmv_inits.py imports dmv_classes.py

** TOGROK Some (tagged) sentences are bound to come twice             :OPTIMIZE:
Eg, first sort and count, so that the corpus
[['nn','vbd','det','nn'],
 ['vbd','nn','det','nn'],
 ['nn','vbd','det','nn']]
becomes
[(['nn','vbd','det','nn'],2),
 (['vbd','nn','det','nn'],1)]
and then in each loop through sentences, make sure we handle the
frequency correctly.
          
Is there much to gain here?

** TOGROK tags as numbers or tags as strings?                         :OPTIMIZE:
Need to clean up the representation.

Stick with tag-strings in initialization then switch to numbers for
IO-algorithm perhaps? Can probably afford more string-matching in
initialization..
** DONE inner_dmv() should disregard rules with heads not in sent     :OPTIMIZE:
   CLOSED: [2008-06-08 Sun 10:18]
If the sentence is "nn vbd det nn", we should not even look at rules
where
: rule.head() not in "nn vbd det nn".split()
This is ruled out by getting rules from g.rules(LHS, sent).

Also, we optimize this further by saying we don't even recurse into
attachment rules where
: rule.head() not in sent[ s :r+1]
: rule.head() not in sent[r+1:t+1]
meaning, if we're looking at the span "vbd det", we only use
attachment rules where both daughters are members of ['vbd','det']
(although we don't (yet) care about removing rules that rewrite to the
same tag if there are no duplicate tags in the span, etc., that would
be a lot of trouble for little potential gain).
* Adjacency and combining it with inner()
Each DMV_Rule now has both a probN and a probA, for
adjacencies. inner() needs the correct one in each case.

Adjacency gives a problem with duplicate words/tags, eg. in the
sentence "a a b". If this has the dependency structure b->a_{0}->a_{1},
then b is non-adjacent to a_{0} and should use probN (for the LRStop and
the attachment of a_{0}), while the other rules should all use
probA. But within the e(0,2,b) we can't just say "oh, a has index 0
so it's not adjacent to 2", since there's also an a at index 1, and
there's also a dependency structure b->a_{1}->a_{0} for that. We want
both. And in possibly much more complex versions.

So now we call inner() for each location of a head, and on each
terminal, loc_h must equal s (and t). In the recursive attachment
calls, we use the locations (sentence indices) of words to the left or
right of the head in calls to inner(). loc_h lets us check whether we
need probN or probA.

In each inner() call, loc_h is the location of the root of this
dependency structure.
** Possible alternate type of adjacency
K&M's adjacency is just whether or not an argument has been generated
in the current direction yet. One could also make a stronger type of
adjacency, where h and a are not adjacent if b is in between, eg. with
the sentence "a b h" and the structure ((h->a), (a->b)), h is
K&M-adjacent to a, but not next to a, since b is in between. It's easy
to check this type of adjacency in inner(), but it needs new rules for
P_STOP reestimation.
* Expectation Maximation in IO/DMV-terms
inner(s,t,LHS) calculates the expected number of trees headed by LHS
from s to t (sentence positions). This uses the P_STOP and P_CHOOSE
values, which have been conveniently distributed into CNF rules as
probN and probA (non-adjacent and adjacent probabilites).

When re-estimating, we use the expected values from inner() to get new
values for P_STOP and P_CHOOSE. When we've re-estimated for the entire
corpus, we distribute P_STOP and P_CHOOSE into the CNF rules again, so
that in the next round we use new probN and probA to find
inner-probabilites.

The distribution of P_STOP and P_CHOOSE into CNF rules also happens in
init_normalize() (here along with the creation of P_STOP and
P_CHOOSE); P_STOP is used to create CNF rules where one branch of the
rule is STOP, P_CHOOSE is used to create rules of the form 
: h  -> h  _a_
: h_ -> h_ _a_

Since "adjacency" is not captured in regular CNF rules, we need two
probabilites for each rule, and inner() has to know when to use which.

** TODO Corpus access
** TOGROK sentences or rules as the "outer loop"?                     :OPTIMIZE:
In regard to the E/M-step, finding P_{STOP}, P_{CHOOSE}.


* Python-stuff
- [[file:src/pseudo.py][pseudo.py]]
- http://nltk.org/doc/en/structured-programming.html recursive dynamic
- http://nltk.org/doc/en/advanced-parsing.html 
- http://jaynes.colorado.edu/PythonIdioms.html



* Git
Setting up a new project:
: git init
: git add .
: git commit -m "first release"

Later on: (=-a= does =git rm= and =git add= automatically)
: git init
: git commit -a -m "some subsequent release"

Then push stuff up to the remote server:
: git push git+ssh://username@repo.or.cz/srv/git/dmvccm.git master

(=eval `ssh-agent`= and =ssh-add= to avoid having to type in keyphrase all
the time)

Make a copy of the (remote) master branch:
: git clone git://repo.or.cz/dmvccm.git 

Make and name a new branch in this folder
: git checkout -b mybranch

To save changes in =mybranch=:
: git commit -a 

Go back to the master branch (uncommitted changes from =mybranch= are
carried over):
: git checkout master

Try out:
: git add --interactive

Good tutorial:
http://www-cs-students.stanford.edu/~blynn//gitmagic/