merge from trunk

[emacs.git] / README.xwidget

-*-org-*-
* Xwidgets

This is an experimental branch to enable embedding of GTK widgets
inside an Emacs window. The Emacs abstraction is called an Xwidget,
for eXternal widget, and also in reference to the Xembed protocoll.

There is a demo file called xwidget-test.el which shows some of the
possibilities. There are some screnshots at the emacswiki.

Currently its possible to insert buttons, sliders, xembed widgets, and
webkit in the buffer. It works similar to the support for images in
Emacs.  Adding more types of widgets should be fairly straightforward,
but will require adapter code for each type.

A difference from images is that xwidgets live their own life. You
create them with an api, get a reference, and tie them to a particular
buffer with a display spec. 

Each xwidget can have several views. In MVC terms, an xwidget is the
model, and an xwidget-view is a view of the xwidget in a particular
Emacs window.

The xwidget code attempts to keep the visual appearance of the views
in sync with through an Observer pattern implementation. This is
necessary to support the Emacs window paradigm.

** building
bzr co bzr+ssh://bzr.savannah.gnu.org/emacs/xwidget/
#or:
#git clone https://github.com/jave/xwidget-emacs.git
#the below compiler flags shouldn't be strictly necessary
export CFLAGS=" -g"
./configure --with-xwidgets --enable-asserts --with-x-toolkit=gtk3
make -j4
gdb -ex run src/emacs

** try it out
If you have GTK3 and gtk-webkit installed, you should be able to
start the embedded webkit browser now:

M-X xwidget-webkit-browse-url

If that didnt work out try the minimal demonstration instead:

(load-library "xwidget-test")
(xwidget-demo-a-button)

It looks unimpressive, but it's a gtk button inside an Emacs buffer!
*** webkit hints
If you got webkit working, great! Please note, though, that the
current support is far from a full fledged browser. My focus is on
delivering a component that can be used to build a full emacs based
browser on. Since I implement a browse-url function you can get quite
far just by:

(setq browse-url-browser-function 'xwidget-webkit-browse-url)

then all Emacs browser interface systems work to a degree.
heres stuff I use currenly

- m-x anything-surfraw interfaces to search engines
- C-o in org mode opens links inside org
- m-x ffap opens links anywhere in a buffer
- m-x gtk-lookup-symbol searches gtk docs
- m-x bookmark-jump
etc..

I'll add more examples as I go along.

However theres lots of support missing, see TODO list for
information. Briefly:
- download handling
- keyboard field navigation
- isearch
- history
- sites that use flash. I dont really care about this issue so its
unlikely to be fixed. Just a heads up.

** Stability
Beginning with Summer 2011 I am now able to use Xwidget Emacs as my
primary Emacs. That is good for the project and the stability of the
code.

At the time of writing I have 24 hour Emacs uptime with several
embedded webkit browsers, Gnus, org-mode, tramp, etc. etc.

That said, there are still many improvements that needs to be done,
particularily in memory management. Expect xwidget emacs to leak
heavily for now.

** timeline for inclusion in trunk
The Emacs 24 feature freeze is passed, so xwidgets won't probably be merged
until Emacs 25. OTOH since I now use xwidget emacs as my primary
emacs, I will merge from trunk much more often than in the past.

** reporting bugs
Emacs xwidgets uses the same tracker as mainline emacs, but a
different package. To report a bug:
M-x report-xwidget-bug


browse bugs:
http://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs-xwidgets
** Thanks
emacs-devel@gnu.org. There are very helpful people there. When I
started the xwidget project I had no clue about the Emacs internals. 

* Brief overview of how xwidgets work
Xwidgets work in one way like images in Emacs. You bind a display spec very
similar to an image display spec to buffer contents. The display engine will
notice the display spec and try to display the xwidget there. The display engine
prepares space at the right place for the xwidget and so on for free, as long as
we provide proper sizes and so on back to the redisplay engine.

** Issues
The problem is that Emacs cant actually draw the widgets, as it can with
images. Emacs must notify GTK about where the widgets should be, and how they
should be clipped and so on, and this information must be given to GTK
synchronous with Emacs display changes. Ok, so why is that difficult then?

- How do we know when a widget is NOT to be drawn? The only way I found so far
  is having a flag for each xwdiget, that is reset before a redisplay. When an
  xwidget is encountered during display, the flag is set. After redisplay,
  iterate all xwidgets and hide those which hasnt been displayed. 

- The gtk socket type for embedding external applications is desirable
  but presents a lot of difficulties of its own. One difficulty is
  deciding which input events to forward, and when and how to do it.

** placement and clipping
the entire emacs frame is a gtk window. we use the fixed layout
manager to place xwidgets on the frame. coordinates are supplied by
the emacs display engine. widgets are placed inside an intermediate
window, called the widgetwindow. the widgetwindows are placed on the
emacs frame.

this way was chosen to simplify clipping of the widgets against emacs
window borders.


** different strategies
Integrating toolkit widgets(gtk in this case) and the emacs display
engine is more difficult than your plain average gui application, and
different strategies has been tested and will continue to be tested.

There was a distinction between live xwidgets and
phantom xwidgets, previous to the change to MVC.

- the first aproach was to have the live xwidget on-screen, and move
  them about. the phantoms were generated by snapshoting the live
  xwidget. 

the drawback of that aproach was that the gtk toolkit is admirably
lazy and doesnt draw the widget if its not actualy shown, meaning that
the snapshots for the phantoms will show garbage.

- the second aproach was to use composition support. that tells gtk
  that the widget should be drawn in an off-screen buffer and drawn on
  screen by the application.

this has the primary advantage that the snapshot is always
available, and enables the possibility of more eye-candy like drawing
live and phantom widgets in different colors.

the drawback is that its our own responsibility to handle drawing,
which puts more of the display optimization burden on us.

this is aproach worked so-so.

- another aproach is to have both live and phantom widgets drawn
  on-screen by proxy gtk objects. the live xwidget will be entirely
  handled in an off-screen window, and the proxy objects will redirect
  events there.

- combine on-screen and off-screen aproaches. maybe composition is the
  way to go for most cases, but on-screen xembeding is the way to go
  for particular special cases, like showing video in a
  window. off-screen rendering and whatnot, is not efficient in that
  particular case, and the user will simply have to accept that the
  phantom of a video widget isnt particularily beautiful.

- The current and seemingly sanest aproach implements a MVC pattern.

** Testing
;;test like:
;; cd /path/to/xwidgets-emacs-dir
;; make   all&&  src/emacs -q --eval "(progn (load \"`pwd`/lisp/xwidget-test.el\") (xwidget-demo-basic))"

** MVC and Xembedd
The MVC approach appears to be at least in principle robust for plain gtk
widgets. For the interesting case of gtk sockets which implements an
xembed host widget that allows for embedding other applications inside
an Emacs window, the story gets more complex.

The problem is that xembed is designed to plug an application window
inside a socket and thats it. You can't move a plug between
sockets. I tried numerous hacks to get around this but there is
nothing that works really well.

Therefore the Emacs part of the code will only expose well-defined
interfaces. cooperating applications will be able to use the interface
in a well defined manner. The problem is that there is no known xembeddable
application that implement the needed type of functionality, which is
allowing for creating new windows on the fly that plug into new
sockets.

Therefore I will attempt to provide an external application that wraps
another application and through hacks attempts to provide the needed
multi view xembed function. That way Emacs is sane and the insanity
contained.

This app will work by providing a socket that an app plugs into. The
socket window is copied efficiently by means of composition to a
number of other windows, that are then plugged into the different
Emacs sockets. 
** old notes from x_draw_xwidget_glyph_string

    BUG it seems this method for some reason is called with bad s->x and s->y sometimes.
    When this happens the xwidget doesnt move on screen as it should.
    This might be because of x_scroll_run. Emacs decides to scroll the screen by blitting sometimes.
    then emacs doesnt try to actualy call the paint routines, which means this here code will never
    run so the xwidget wont know it has been moved.

    Solved temporarily by never optimizing in try_window_reusing_current_matrix().

    BUG the phantoming code doesnt work very well when the live xwidget is off screen.
    you will get weirdo display artefacts. Composition ought to solve this, since that means the live window is
    always available in an off-screen buffer. My current attempt at composition doesnt work properly however.

    //allocation debugging. the correct values cant be expected to show upp immediately, but eventually they should get to be ok
    // this is because we dont know when the container gets around to do layout
    //GtkAllocation galloc;
    //gtk_widget_get_allocation(GTK_WIDGET (xv->widgetwindow), &galloc);
    //printf("allocation %d %d , %d %d\n", galloc.x,galloc.y,galloc.width,galloc.height);


*** old notes about the old live/phantom scheme

   //TODO:
   // 1) always draw live xwidget in selected window
   // (2) if there were no live instances of the xwidget in selected window, also draw it live)
   // 3) if there was a live xwidget previously, now phantom it.

   else
     {
       //ok, we are painting the xwidgets in non-selected window, so draw a phantom
       //printf("draw phantom xwidget at:%d %d\n",x,y);
       //xwidget_composite_draw_phantom (xw, x, y, clipx, clipy); //TODO MVC there will be very few cases of phantoming
     }


   atm this works as follows: only check if xwidgets are displayed in the
   "selected window". if not, hide them or phantom them.

   this means valid cases like xwidgets being displayed only once in
   non-selected windows, does not work well. they should also be visible
   in that case not phantomed.

* ToDo:s
** TODO webkit crash
[2013-04-13 Sat] seems to crash a lot on http://www.dilbert.com
Not always, but enough to be annoying.

** TODO optimize drawing off large offscreen widgets
Currently I just allocate as large an area as the offscreen webkit
widget desires. This works well most of the time. But a HTML page
might in principle be of infinite height so there are cases where this
doesn't work too well.

Heres a proposed strategy:
- never grow the offscreen webkit over xwidget-webkit-max-height
- allow for webkit to handle its own scrolling internally as well
- be more clever about when you have more than one emacs window
  showing the same webkit instance. 
- allow to grow the offscreen instance in steps rather than just
  allocate the entire height at once

** DONE again a trace
   CLOSED: [2011-10-28 Fri 13:48]
[2011-08-23 Tue]
the hunch is that since I still hand-wave the view storage the array
can get out of synchronous. so maybe switching to a lisp structure
will help as it did for the model. Anyway, doesnt happen at all often.
*** the trace
(gdb) bt
#0  0x0000000000685304 in xwidget_touch (xv=0x0) at xwidget.c:1225
#1  0x00000000006853e7 in xwidget_end_redisplay (w=0x11b42ca0, matrix=
    0xff9bf40) at xwidget.c:1272
#2  0x000000000041cc31 in update_window (w=0x11b42ca0, force_p=0)
    at dispnew.c:3705
#3  0x000000000041c0e5 in update_window_tree (w=0x11b42ca0, force_p=0)
    at dispnew.c:3331
#4  0x000000000041be8b in update_frame (f=0x1682a50, force_p=0, 
    inhibit_hairy_id_p=0) at dispnew.c:3258
#5  0x000000000045066f in redisplay_internal () at xdisp.c:12931
#6  0x000000000044e210 in redisplay () at xdisp.c:12110
#7  0x0000000000567e65 in read_char (commandflag=1, nmaps=7, maps=
    0x7fffffffc040, prev_event=12708226, used_mouse_menu=0x7fffffffc254, 
    end_time=0x0) at keyboard.c:2447
#8  0x000000000057613c in read_key_sequence (keybuf=0x7fffffffc4a0, bufsize=
    30, prompt=12708226, dont_downcase_last=0, can_return_switch_frame=1, 
    fix_current_buffer=1) at keyboard.c:9299
#9  0x0000000000565d45 in command_loop_1 () at keyboard.c:1448
#10 0x0000000000601008 in internal_condition_case (bfun=
    0x565962 <command_loop_1>, handlers=12760466, hfun=0x565259 <cmd_error>)
    at eval.c:1490
#11 0x0000000000565659 in command_loop_2 (ignore=12708226) at keyboard.c:1159
#12 0x0000000000600992 in internal_catch (tag=12873826, func=
---Type <return> to continue, or q <return> to quit---
    0x565633 <command_loop_2>, arg=12708226) at eval.c:1247
#13 0x00000000005655bd in command_loop () at keyboard.c:1124
#14 0x0000000000564da7 in recursive_edit_1 () at keyboard.c:759
#15 0x0000000000564f43 in Frecursive_edit () at keyboard.c:823
#16 0x000000000060444f in Ffuncall (nargs=1, args=0x7fffffffca20)
    at eval.c:2986
#17 0x00000000006507f8 in exec_byte_code (bytestr=145172929, vector=145179445, 
    maxdepth=116, args_template=12708226, nargs=0, args=0x0) at bytecode.c:785
#18 0x0000000000604eec in funcall_lambda (fun=140575909, nargs=2, arg_vector=
    0x7fffffffcfe8) at eval.c:3220
#19 0x000000000060467e in Ffuncall (nargs=3, args=0x7fffffffcfe0)
    at eval.c:3038
#20 0x00000000006035fc in Fapply (nargs=2, args=0x7fffffffd0b0) at eval.c:2494
#21 0x0000000000603b43 in apply1 (fn=12874242, arg=301666310) at eval.c:2732
#22 0x00000000005feb25 in call_debugger (arg=301666310) at eval.c:220
#23 0x0000000000601ca9 in maybe_call_debugger (conditions=9431542, sig=
    12761282, data=301666742) at eval.c:1893
#24 0x0000000000601785 in Fsignal (error_symbol=12761282, data=301666742)
    at eval.c:1714
#25 0x0000000000601898 in xsignal (error_symbol=12761282, data=301666742)
    at eval.c:1749
#26 0x0000000000601926 in xsignal2 (error_symbol=12761282, arg1=102756373, 
    arg2=0) at eval.c:1770
---Type <return> to continue, or q <return> to quit---
#27 0x0000000000604d6e in funcall_lambda (fun=102756373, nargs=0, arg_vector=
    0x7fffffffd398) at eval.c:3189
#28 0x000000000060467e in Ffuncall (nargs=1, args=0x7fffffffd390)
    at eval.c:3038
#29 0x00000000006507f8 in exec_byte_code (bytestr=54783137, vector=109656229, 
    maxdepth=12, args_template=12708226, nargs=0, args=0x0) at bytecode.c:785
#30 0x0000000000604eec in funcall_lambda (fun=109656517, nargs=0, arg_vector=
    0x7fffffffd890) at eval.c:3220
#31 0x000000000060467e in Ffuncall (nargs=1, args=0x7fffffffd888)
    at eval.c:3038
#32 0x0000000000603b08 in apply1 (fn=109656517, arg=12708226) at eval.c:2725
#33 0x00000000005fc8c9 in Fcall_interactively (function=109656517, record_flag=
    12708226, keys=12754549) at callint.c:379
#34 0x00000000006044c2 in Ffuncall (nargs=4, args=0x7fffffffdc60)
    at eval.c:2996
#35 0x0000000000603c57 in call3 (fn=12893554, arg1=109656517, arg2=12708226, 
    arg3=12708226) at eval.c:2789
#36 0x00000000005784cd in Fcommand_execute (cmd=109656517, record_flag=
    12708226, keys=12708226, special=12708226) at keyboard.c:10290
#37 0x00000000005661fb in command_loop_1 () at keyboard.c:1575
#38 0x0000000000601008 in internal_condition_case (bfun=
    0x565962 <command_loop_1>, handlers=12760466, hfun=0x565259 <cmd_error>)
    at eval.c:1490
---Type <return> to continue, or q <return> to quit---
#39 0x0000000000565659 in command_loop_2 (ignore=12708226) at keyboard.c:1159
#40 0x0000000000600992 in internal_catch (tag=12756258, func=
    0x565633 <command_loop_2>, arg=12708226) at eval.c:1247
#41 0x000000000056560c in command_loop () at keyboard.c:1138
#42 0x0000000000564da7 in recursive_edit_1 () at keyboard.c:759
#43 0x0000000000564f43 in Frecursive_edit () at keyboard.c:823
#44 0x0000000000563052 in main (argc=1, argv=0x7fffffffe678) at emacs.c:1711

Lisp Backtrace:
"recursive-edit" (0xffffca28)
"debug" (0xffffcfe8)
"image-bol" (0xffffd398)
0x68939c0 PVEC_COMPILED
"call-interactively" (0xffffdc68)
(gdb) 


** DONE new annoying trace
   CLOSED: [2011-08-13 Sat 16:16]
maybe related to scroll inhibiting or cursor inhibiting code.
It appears actually to be related to GLYPH_DEBUG=1. this flag is no
longer needed.
*** the trace
Breakpoint 1, abort () at emacs.c:383
383       kill (getpid (), SIGABRT);
Missing separate debuginfos, use: debuginfo-install hunspell-1.2.15-2.fc15.x86_64 nss-mdns-0.10-9.fc15.x86_64
(gdb) 
(gdb) 
(gdb) bt
#0  abort () at emacs.c:383
#1  0x0000000000418f01 in matrix_row (matrix=0xac29400, row=-1)
    at dispnew.c:1477
#2  0x000000000046e113 in draw_glyphs (w=0x18235c0, x=198, row=0xa3af100, area=
    TEXT_AREA, start=17, end=18, hl=DRAW_CURSOR, overlaps=0) at xdisp.c:22550
#3  0x000000000047869f in draw_phys_cursor_glyph (w=0x18235c0, row=0xa3af100, 
    hl=DRAW_CURSOR) at xdisp.c:24882
#4  0x00000000005083bb in x_draw_window_cursor (w=0x18235c0, glyph_row=
    0xa3af100, x=180, y=361, cursor_type=0, cursor_width=1, on_p=1, active_p=1)
    at xterm.c:7440
#5  0x00000000004790cd in display_and_set_cursor (w=0x18235c0, on=1, hpos=17, 
    vpos=19, x=180, y=361) at xdisp.c:25098
#6  0x00000000004fa31f in x_update_window_end (w=0x18235c0, cursor_on_p=1, 
    mouse_face_overwritten_p=0) at xterm.c:644
#7  0x000000000041ccb9 in update_window (w=0x18235c0, force_p=0)
    at dispnew.c:3694
#8  0x000000000041c165 in update_window_tree (w=0x18235c0, force_p=0)
    at dispnew.c:3331
#9  0x000000000041beee in update_frame (f=0x1658460, force_p=0, 
    inhibit_hairy_id_p=0) at dispnew.c:3258
#10 0x0000000000450a2e in redisplay_internal () at xdisp.c:12983
#11 0x000000000044e2a6 in redisplay () at xdisp.c:12099
#12 0x000000000056a60d in read_char (commandflag=1, nmaps=6, maps=

** DONE allow xwidgets to report their size
   CLOSED: [2011-07-19 Tue 14:26]
now we just hard code sizes. but webkit widgets for instance can
report sizes that suit the content. support that.
** DONE BUG xwidget view ghosts
   CLOSED: [2013-04-05 Fri 23:35]
(havent seen this in quite a while)
- xwidget-webkit-browse-url somewhere
- split window.
now theres 2 webkit views
- c-x 1
now theres 2 views but one is a ghost!
one should have been deleted when its window died but that didnt work
for some reason here.

- m-x xwidget-cleanup

the ghost goes away because we killed explicitly but this is just a workaround.

xwidget_view_delete_all_in_window(w); in delete-window-internal is not sufficient.
delete-other-windows-internal
delete_all_subwindows
unshow_buffer

Added cleanup those window configuration hook which works in practice
but feels kludgy.

*** code looks like this

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defun xwidget-cleanup ()
  "Delete zombie xwidgets."
  ;;its still pretty easy to trigger bugs with xwidgets.
  ;;this function tries to implement a workaround
  (interactive)
  (xwidget-delete-zombies) ;;kill xviews who should have been deleted but stull linger
  (redraw-display);;redraw display otherwise ghost of zombies  will remain to haunt the screen
  )



;;this is a workaround because I cant find the right place to put it in C
;;seems to work well in practice though
(add-hook 'window-configuration-change-hook 'xwidget-cleanup)

*** but it ought rather to work like this
xwidget-delete-zombies should be called from C after window
configuration has changed but before redisplay. redisplay should not
be called.


** DONE BUG annoying backtrace
   CLOSED: [2011-07-19 Tue 14:28]
(this no longer seems to happen even under heavy usage. seems merging
from trunk helped. lots were happening in redisplay at this time in trunk.)

sadly happens a lot.
- happens even with no initialized xwidgets
-                    + row->glyphs[area][i].face_id
or similar code, so row is invalid for some reason.
xwidgets currently disable some redisplay opimizations so it might be
an actual emacs bug manifesting without optimizations.

*** bt 1
      /* Compute the width of this line.  */
      row->pixel_width = row->x;
      for (i = 0; i < row->used[TEXT_AREA]; ++i)
        row->pixel_width += row->glyphs[TEXT_AREA][i].pixel_width;

(gdb) bt
#0  0x000000000045c340 in compute_line_metrics (it=0x7fffffff8a20)
    at xdisp.c:17549
#1  0x00000000004603da in display_line (it=0x7fffffff8a20) at xdisp.c:18792
#2  0x0000000000457646 in try_window (window=23403045, pos=..., flags=1)
    at xdisp.c:15399
#3  0x00000000004559c9 in redisplay_window (window=23403045, just_this_one_p=0)
    at xdisp.c:14944
#4  0x0000000000450247 in redisplay_window_0 (window=23403045) at xdisp.c:13152
#5  0x00000000005fdcd9 in internal_condition_case_1 (bfun=
    0x450208 <redisplay_window_0>, arg=23403045, handlers=12691046, hfun=
    0x4501d9 <redisplay_window_error>) at eval.c:1538
#6  0x00000000004501ba in redisplay_windows (window=23403045) at xdisp.c:13132
#7  0x000000000044f19c in redisplay_internal () at xdisp.c:12706
#8  0x000000000044f9f2 in redisplay_preserve_echo_area (from_where=7)
    at xdisp.c:12964
#9  0x0000000000568525 in swallow_events (do_display=1) at keyboard.c:4197
#10 0x0000000000422554 in sit_for (timeout=40, reading=1, do_display=1)
    at dispnew.c:5963
#11 0x000000000056512c in read_char (commandflag=1, nmaps=8, maps=
    0x7fffffffd3f0, prev_event=12720514, used_mouse_menu=0x7fffffffd604, 
    end_time=0x0) at keyboard.c:2689
#12 0x0000000000572c59 in read_key_sequence (keybuf=0x7fffffffd850, bufsize=
    30, prompt=12720514, dont_downcase_last=0, can_return_switch_frame=1, 
---Type <return> to continue, or q <return> to quit---
    fix_current_buffer=1) at keyboard.c:9291
#13 0x0000000000562897 in command_loop_1 () at keyboard.c:1446
#14 0x00000000005fdb52 in internal_condition_case (bfun=
    0x5624b4 <command_loop_1>, handlers=12772898, hfun=0x561dab <cmd_error>)
    at eval.c:1493
#15 0x00000000005621ab in command_loop_2 (ignore=12720514) at keyboard.c:1157
#16 0x00000000005fd4ce in internal_catch (tag=12768770, func=
    0x562185 <command_loop_2>, arg=12720514) at eval.c:1247
#17 0x000000000056215e in command_loop () at keyboard.c:1136
#18 0x00000000005618f9 in recursive_edit_1 () at keyboard.c:757
#19 0x0000000000561a95 in Frecursive_edit () at keyboard.c:821
#20 0x000000000055fba2 in main (argc=1, argv=0x7fffffffe188) at emacs.c:1704


*** bt 2

** DONE Examine using XComposite rather than GTK off-screen
  rendering. This would make xembed widgets work much better. This
  would probably be rathter difficult, but could open up other
  interesting possibilities for Emacs. There is an early attempt in
  xwidget.c, but the X call to redirect to offscreen rendering fails
  for unknown reasons.

  the attempt was further worked on, and the xlib calls replaced with
  gdk calls, this works better.

  In the end I abandoned this aproach. Xwidget-osr is the new aproach.

** TODO  make the keyboard event code propagation code work. 
There is an attempt to provide an api to send keyboard events to an
xwidget, but it doesnt currently work very well.

*** TODO try gtk event creation instead 
since that works fine in the webkit osr code. 
but, oh no, that didn't work for some reason.
the widgets seems to receive the event but then the embedded widgets
hangs.

http://kegel.com/gtk/button.c

*** TODO examine some library to synthesize events
xdotool
xte xautomation
crikey
libxdo

*** TODO webkit raw keyboard event escape
c-c tab could send a raw tab to the webkit instance.
** DONE remove the special-case for when the minibuffer is
  active.  I added some code to reduce the annoying problem display artefacts
  when making the minibuffer the selected window. This made xwidgets in the
  buffer go grey or black whenever one did m-x to activate the minibuffer. The
  coded tried to handle the minibuffer as a special case. That simply wasnt a
  good idea. Special-casing will never work properly. It is much better to spend
  time finding solutions that work acceptably in the general case.

** DONE disable emacs cursor drawing on top of an active xwidget. 
  This ought to be rather simple and should improve the visuals a lot.

** TODO  improve the xwidgets programming interface 
so its less of hand-waving affair. This shouldnt be too hard, but I
  have deliberatley not spent any time on it, since getting the
  visuals right is much harder. Anyway, I sort of think the interface
  should be somewhat like it is, except symbols is used instead of
  integers.
*** DONE use symbols for xwidget types rather than ints
    CLOSED: [2011-06-27 Mon 12:52]


*** TODO better lisp based structure for xwidgets
the lisp interface woud be like this:
- make-xwidget returns an xwidget object, similar to a process
  object. this object is used when creating the display spec(instead of
  the user defined id now used)

the data structure would be something like this:
- a "process" like aproach to create the xwidgets. xwidgets are
  coupled to buffers, somewhat like processes, except a buffer can
  hold several xwidgets
- an xwidget has a plist to hold the model, like a process
- an xwidget has an assoc list of xwidget views

there are some things that arent clear:
- an xwidget doesnt necessarily need to be coupled to a buffer but it
  seems to be the clearest model. xwidgets would be buffer local
- xwidget-views are by necessity coupled to a emacs window so it might
  be better to store them window locally rather than in an assoc
  coupled to the xwidget model
- for some gtk widgets that resist an mvc approach, like the webkit
  widgets, special operations are needed, similar to the old phantom
  widgets aproach. so we need to differentiate live and phantom
  instances for these troublesome widgets and let lisp manage all the trickery.

stuff that needs to work:
- do something for all views of a xwidget(resize, value change)
- do something for all xw-views in an emacs window(deletion etc)
- lookup xw-view for xwidget in emacs window(during redisplay)
(- do something for all siblings of a xw-view. not atm)

*** DONE xwidget creation interface
    CLOSED: [2011-07-18 Mon 01:59]
xwidgets are a little bit like emacs processes but also a little bit
like emacs images. Therefore its not perfectly obvious how to handle
creation. Currently I just use hardcoded identifiers. the real scheme
needs to be something else.

Heres a tentative approach:
- xwidget-create returns a xwidget object, like process creation
  functions. the xwidget will be largely uninitialized until
  discovered by redisplay. an xw belongs to a buffer
- xwidget-insert inserts the xwidget in a buffer. when discovered by
  redisplay it will be initialized and a xwidget-view allocated
- an event will be emitted when initialization is finished when
  relevant like for sockets

the problem with this aproach is that its not really legal to reuse
xwidget objects by writing several display specs who reference the
same xwidget. It could presumably be done but it would just become
weird for no real benefit. the big preblem is that the display spec
decides the on-screen size, and its not sane to have xwidget views
with different sizes. therefore such display specs would need to be
catched and dissallowed. Except it is hard because AFAIK the specs
don't have an identity as such. A flag in the structure could be set
by lookup so the first display attempt would win. but then you can't
rewrite the spec to change the size. hmmm. A third approach would be
to just allow the 1st spec refering an xw during a redisplay to take
effect, the rest are signaled as errors. this wouldnt be too bad.

the other aproach would be to work more like images:

- write the display spec with all information needed to create the
  xwidget.
- retrieve the xwidget objet from the spec with an xwidget-at-point function. It
  can be uninitalized which client code must handle. Unlike
  asynchronous process creation we dont get back a handle, because
  there is none yet.
- emitted event on initialization, when needed. Many widgets don't
  need this. for instance, a button sends an event when pressed. but
  you can't press it unless its on screen, and then its initialized
  properly. 

This approach seemed good, but how do I know which instance
generates an event if I cant set the id beforehand?

so, therefore, the first two aproach is used. 


*** DONE xwidget creation interface actually
    CLOSED: [2011-07-18 Mon 01:59]
conclusion of above ramblings:
- should be similar to make-text-button
- don't init from display spec, instead during make-xwidget call
*** TODO callbacks would be nice 
but they need to be handled initially with events for technical
reasons. C code can't call Lisp easily. The event handler can call the
callback though.

** TODO  more documentation
There should be user docs, and xwidget contributor docs. The current README
is all contributor docs there is now, apart from the code.



** CANCELLED look into more ways of displaying xwidgets, like binding them to a
   CLOSED: [2011-07-05 Tue 11:34]
window rather than a point in a buffer. This was suggested by Chidong.
This would be a useful addition to Emacs in itself, and would avoid nearly all 
display issues. I still think the general case is more interesting, but this
special case should also be added. The xwidget would then be bound to
replace the view of a particular window, and it would only show in
that window.

I got the webkit xwidget to work well enough so I dont see the need
for this now, except for sockets and I think it can better be dealt
with at the lisp level.

** DONE MVC mode for xwidgets
   CLOSED: [2011-06-27 Mon 12:53]
It appears unfruitful to chase using the same display mode for all
types of xwidgets. Composition is fun but not robust the way I
tried to do it.

Instead there should be a set of MVC xwidgets. Each on-screen instance
of an MVC widget would be a real GTK widget. The instances would
communciate state using signals. 

There are drawbacks. There is no inbuilt support for MVC in GTK, so we
have to roll our own, which is tedious if not much work for the few
cases.

MVC for xembedded application will need support from the applications
themselves. Inkscape supports multiple views to the same document,
other programs don't. In practice it might not be a big drawback.


*** DONE figure out what to do with the multiple frames case. 
    CLOSED: [2011-06-27 Mon 12:52]
This should be easier to solve with MVC.
Surprisingly, this just worked!
*** DONE how to propagate changes in views to other views?
    CLOSED: [2011-06-27 Mon 12:53]
I used gtk signals, the implementation for sliders works well!

** TODO canvas support
heres an interesting comparision of gtk canvases
http://live.gnome.org/ProjectRidley/CanvasOverview

ATM there are small hardcoded demos in the code, these should be
removed and replaced with working xwgir counterparts.
*** goocanvas
goocanvas is a gtk canvas implemented using cairo. investigate.

pros:
- it has a MVC model aproach out of the box which is nice.

http://developer.gnome.org/goocanvas/unstable/goocanvas-model-view-canvas.html

export CFLAGS="`pkg-config --cflags goocanvas` -DHAVE_GOOCANVAS"
export LDFLAGS=`pkg-config --libs goocanvas`
./configure
make

I made a hello goo world xwidget so seems doable.
I wanted to load a SVG which wasnt immediately straightforward, so I
tried clutter. but it turns out the exact same strategy could be used
with goocanvas.

*** clutter
maybe clutter can be used as a canvas? 
pros:
- seems to have a lot of traction atm. many examples
- potentialy fast and cool vector graphics
cons:
- no out of the box MVC support, but seems doable. no worse than the
  other home brew mvc support I have in xwidgets
(media-explorer in an application that employes the MVC pattern)

http://www.openismus.com/documents/clutter_tutorial/0.9/docs/tutorial/html/sec-stage-widget.html

there is also cool stuff like this:
http://gitorious.org/webkit-clutter/webkit-clutter which is an webkit actor for
clutter! hmmmmm.

I want to render svg. aparently:
  librsvg rsvg_handle_render_cairo(h, cr);
  ClutterCairoTexture
  Clutter

export CFLAGS="`pkg-config --cflags clutter-gtk-1.0` -DHAVE_CLUTTER"
export LDFLAGS=`pkg-config --libs clutter-gtk-1.0`
./configure
make

compiles but I get:
Gtk-ERROR **: GTK+ 2.x symbols detected. Using GTK+ 2.x and GTK+ 3 in
the same process is not supported

export CFLAGS="`pkg-config --cflags clutter-gtk-0.10` -DHAVE_CLUTTER"
export LDFLAGS=`pkg-config --libs clutter-gtk-0.10`
./configure
make


*** webkit html 5
expose the DOM to lisp or something. The webkit xwidget works pretty
well now, so this might be the way ahead.
** DONE mvc code crashes after a while
   CLOSED: [2011-07-12 Tue 18:52]
seemingly only when compiling with optimizations.
I have no idea why.

Doesn't seem to happen after some code cleanups.
** DONE xwidget-resize-at
   CLOSED: [2011-07-19 Tue 14:28]
reimplement so display spec is not involved
** DONE display spec validation
   CLOSED: [2011-07-19 Tue 14:44]
it is an error to reuse xwidgets in several buffers or in the same
buffer. how do we catch these errors? 
- showing the same xwidget twice in a buffer is no more wrong than
  showing in several emacs windows, just conceptually wrong, so ignore
  this case for now
- xwidgets now store a reference to the buffer they were created in,
  so use that to invalidate xwidget references in oher buffers. but
  thats not really an error either
- xwidgets should now be proper lisp objects so you dont delete them
  you await their garbage collection. so therefore there can never be
  invalid display specs

so turned out this got solved by using proper lisp objects for
xwidgets. yay!

** DONE clipping of controllers
   CLOSED: [2011-07-05 Tue 11:33]

Emacs uses a big GTK window and does its own clipping against Emacs
windows inside this area. So, in order to layout gtk widgets in emacs
windows we must clip thim ourselves. 

The following method worked well for a long time:
- make a gtk widget, say a button, xw
- make a clipping area, of type gtkfixed(many types have been tested)
- put the clip area in the main emacs gtk window
- figure out clip area changes during emacs redisplay

the only weirdness was that one has to tell gtk the clip area has a
window in order to get clipping. This is weird because all gtkwidgets
are windows in a sense and a window is almost by definition also a
clipping area.

Anyway, in GTK3 the   gtk_widget_set_has_window(GTK_WIDGET (
xv->widgetwindow), TRUE); call is ignored. 

The gtkeventbox which is documented to have its own window doesnt work
either.

http://www.lanedo.com/~carlos/gtk3-doc/chap-drawing-model.html

anyway clipping is rather complicated but seems to finally work okay.

*** DONE subclass my own clipping widget
    CLOSED: [2011-07-04 Mon 16:55]
http://www.lanedo.com/~carlos/gtk3-doc/GtkWidget.html#gtk-widget-set-has-window
mentions that it has_window can only be called inside a widget
implementation.

this wasnt really the issue. allocation was the problem
*** DONE try scrolled window
    CLOSED: [2011-07-01 Fri 10:56]
clipping does in fact work with
gtk_scrolled_window_add_with_viewport (xv->widgetwindow, xv->widget);
!!

I get unwanted scrollbars in the widget though.

  gtk_scrolled_window_set_policy      (  xv->widgetwindow,
  GTK_POLICY_NEVER, GTK_POLICY_NEVER); 

stops clipping from working! 


*** DONE try viewport
    CLOSED: [2011-07-01 Fri 10:56]
gtkviewport is used in scrolled window so in order to remove
scrollbars it should be possible to use viewport directly. however, 
viewport ignores size requests. or rather the container does.


*** DONE debug allocation
    CLOSED: [2011-07-04 Mon 16:56]
the container determines how much size to allocate to child widgets.

        GtkAllocation galloc;
        gtk_widget_get_allocation(GTK_WIDGET (xv->widgetwindow), &galloc);
        printf("allocation %d %d , %d %d\n", galloc.x,galloc.y,galloc.width,galloc.height);

after my clipping attemp shows that my size request is ignored! this
might be logical, since the container provided by emacs is a
gtkfixed. gtkfixed might choose to heed the widgets size desires and
allocate the entire widget size. but we want clipping!

since i cant reasonably expect to change the emacs main container, i
can maybe overide the setallocation method in gwfixed, and adjust
allocation to clipping if its an xwidget asking for allocation.

**** DONE subclass gtkfixed
     CLOSED: [2011-07-04 Mon 16:56]
possibly i need to subclass gtkfixed and override
#+begin_src C
  void                gtk_widget_size_allocate            (GtkWidget *widget,
                                                           GtkAllocation *allocation);
#+end_src
http://developer.gnome.org/gobject/stable/howto-gobject.html

turns out emacs already does this for gtk3 according to jan D:
>>For GTK3, Emacs already subclasses GtkFixed, see emacsgtkfixed.[ch].

- widgets may not be underallocated, aparently
http://mail.gnome.org/archives/commits-list/2011-April/msg10950.html

- how to call base class method/chain up
http://developer.gnome.org/gobject/stable/howto-gobject-chainup.html

- the allocation modification could happen in the container or the
  child. it feels more apropiate in the container

it is however unexpectedy inconvenient to modify allocation because
the needed data is private to the base class. to overcome this:

 - run base class method 1st. 
 - then, iterate all children, and modify allocation  for xwidget
   children only. x y will then be set.

JanD pointed out the GTK3 port already has its own subclass, so I
modified that one.

*** DONE clip top
    CLOSED: [2011-07-05 Tue 11:30]
there are four controller edges that potentialy need clipping. I begun
with right and bottom edges. clipping them is just a matter of setting
the right size of the widgetwindow and also ensure it gets the right
allocation from the container.

clipping top (and left) is not equally straightforward. I'm using a
viewport now and scroll it the amount that needs to be clipped.
however, the viewport is sensitive to changes in allocation, which
makes it harder to use the allocation workarounds.

see:
- gtk_widget_set_size_request
- gtkscrolledwindow

I returned to using a simple gtkfixed for the widgetwindow. with
allocation hack and set_has_window it works. Idea prefer not to have
the allocatien hack and it wasnt needed it gtk3 only gtk2. needs
further investigation.

** various code cleanups
There are many cleanups necessary before any hope of inclusion in
Emacs trunk. To begin with, the part of the patch that touches other
parts of emacs must be very clean. 
*** DONE use FRAME_GTK_WIDGET (f)
    CLOSED: [2011-07-20 Wed 20:02]
rather than gwfixed.

*** DONE support configure
    CLOSED: [2011-07-12 Tue 18:48]
*** DONE ifdef all xwidget code
    CLOSED: [2011-08-13 Sat 16:19]
so you can reliably disable the code at compiletime
** DONE translate clicks 
   CLOSED: [2011-07-03 Sun 22:12]
on onscreen webkit peer to offscreen

maybe
http://developer.gnome.org/gdk/stable/gdk-Windows.html#GdkWindow-from-embedder

turned out to be not so hard, captured events, copied them and
forwarded them offscreen!

** CANCELLED investigate gdk_window_redirect_to_drawable
   CLOSED: [2013-04-05 Fri 23:37]
(cancelled this, the current approach seems okay)
http://developer.gnome.org/gdk/stable/gdk-Windows.html#gdk-offscreen-window-set-embedder
maybe could be used in place of my own copy hacks? to work it must
support a chain of redirects, which seems unlikely. the benefit would
be that I dont have to spend time optimizing redrawing.


** DONE remove xwidget_views when emacs window is deleted
   CLOSED: [2011-07-05 Tue 11:29]
removing xwidget views when an Emacs window closes is not reliable.

- switching buffers in a window seems to hide the corresponding
  xwidget-views properly, but they might as well be deleted.

-  patching delete-window-internal could be used to delete the xwidget-views
this seems to work


** browser xwidget
although embedding a browser is not my primary concern many are
interested in this. some suitable browser component needs to be found
supporting gtk.

*** DONE webkit
    CLOSED: [2011-07-03 Sun 22:13]
there is a webkit gtk port. there is no obvious mvc support.
http://live.gnome.org/WebKitGtk
http://webkitgtk.org/

it might be possible to keep a set of webxits in artificial
synchronisation by recursive deep copy of the DOM from one webkit to
another. This will be error prone at best though. Another way might be
to just use bitmap copy of the "live"instance to the "phantom"
instances. the problem of transfering the live view remains though.

export CFLAGS="`pkg-config --cflags webkit-1.0` -DHAVE_WEBKIT -g"
export LDFLAGS=`pkg-config --libs webkit-1.0`
./configure
make

**** off screen rendering
export CFLAGS="`pkg-config --cflags webkit-1.0` -DHAVE_WEBKIT_OSR -g"
export LDFLAGS=`pkg-config --libs webkit-1.0`
./configure
make

works a little bit  but i get errors like:

(emacs:8362): GLib-GObject-WARNING **: invalid cast from `GdkOffscreenWindow' to `GdkDrawableImplX11'

set a breakpoint in g_log, backtrace seems to indicate
webkitViewportAttributesRecompute is the offender. 

maybe try gtk3 variants?
#+begin_src sh
  export CFLAGS="`pkg-config --cflags webkitgtk-3.0 ` -DHAVE_WEBKIT_OSR "
  export LDFLAGS=`pkg-config --libs webkitgtk-3.0 `
  ./configure   --with-x-toolkit=gtk3
  make
#+end_src
crash in gtk_window_get_size instead. great.

http://gtkplus-p3.0.sourcearchive.com/documentation/2.91.5-0ubuntu1/testoffscreenwindow_8c-source.html

after many attempts, the basic issue remains. for some reason the
offscreen widget isnt ok when I want to snapshot it, so i simply get
emptiness. the surface is only ok sometimes.

here is a useful debugging snippets:
#+begin_src C
  // debugging redraw:
  //  - the bg colors always change, so theres no error in signal handling
  //  - i get this error now and then:
  //(emacs:7109): GLib-GObject-WARNING **: invalid cast from `GdkOffscreenWindow' to `GdkDrawableImplX11'
  // seems to happen in webkit actually. see README
  
  if(0){ //redraw debug hack. helped a lot in fact. use the with alpha painter below also
    cairo_set_source_rgb(cr, osr_dbg_color, 1.0, 0.2);
    cairo_rectangle(cr, 0,0, xw->width, xw->height);
    cairo_fill(cr);
    osr_dbg_color+=0.1;
    if(osr_dbg_color>1.0)
      osr_dbg_color=0.0;
    
  }
#+end_src
you need to terminate drawing like this:
#+begin_src C  
  //cairo_set_source_surface (cr, src_pixmap, 0,0); 
  //cairo_set_operator (cr, CAIRO_OPERATOR_OVER);

  //cairo_paint_with_alpha (cr, 1.0);
  //cairo_paint(cr);
#+end_src

the snippets change background color on oach redraw. 

**** on-screen rendering to separate window
an alternative might be to open a separate window and snapshot it. the
idea is that whatever oddness webkit does so that offscreen rendering
doesnt work, doesnt happen on-screen. the window could be opened
somewhere not in the way.

*** CANCELLED firefox
    CLOSED: [2011-07-03 Sun 22:13]
http://www-archive.mozilla.org/unix/gtk-embedding.html
seems to be severly bitrotted

heres a newer aproach
http://hg.mozilla.org/incubator/embedding/file/29ac0fe51754/gtk/tests/test.cpp

while webkit clearly has the best traction as an embedded, the
offscreen rendering issues makes it interesting to see what ff brings
to the table. 

turned out webkit has as good offscreen support as anyone, see I went
with that in the end.


*** DONE text field support
    CLOSED: [2011-07-20 Wed 20:05]
Emacs captures all keyboard events so text field support isn't super
straightforward. 

**** propagate keyboard events
I have some old hacks for this and they are not good.
**** use the DOM model
expose document.activeElement to lisp. This is potentially more
interesting than just forwarding keyboard events.

webkit_web_view_get_dom_document ()

this is hard it seems. an idea might be to hack elisp support for swig
to machine generate the bindings. 
**** DONE inject javascript
     CLOSED: [2011-07-03 Sun 22:50]
webkit_web_view_execute_script ()

this works now:
(xwidget-webkit-execute-script 5 "document.activeElement.value='test'")

so it should be possible to do some interesting stuff.
execute-script does however not return anything at the interface level
so satisfaction is not total:

http://markmail.org/message/4yowmdgras73z3x5

maybe
https://launchpad.net/gnome-seed

or this funny hack:
<jave> im trying to understand how to interact via javascript to an embedded
       webkit gtk instance  [23:38]
<jave> i use webkit_web_view_execute_script() which is nice but doesnt return
       a value, by design aparently  [23:39]
<jave> any hints?
<lucian> jave: afaik, webkit still doesn't have full gobject bindings  [23:48]
<lucian> jave: you can hack it up by making the JS modify the title, and read
         the title from gtk-side
<jave> lucian: that was a pretty cool idea!


*** webkit_web_view_load_string ()
I would like preview of html in a buffer rather than from uri.



*** DONE simple xwidget-webkit wrapper
    CLOSED: [2011-07-22 Fri 11:01]
so that it could be used for actual browsing :)
I dont want to reinvent too many wheels so i'd like to use existing
emacs facilities here possible. use bindings similar to w3m(or info)

- m-x xwidget-webkit starts a session
- 'g' goes to a url
- use bookmark-jump i suppose. I mostly use org for bookmarks myself
- browse-url support so webkit can be the default browser
- some way of getting around the quirky keyboard interaction since
  xwidgets dont receive keyboard events because I hawe no idea how to
  do that in a sane way

... and one can of course go on bikeshedding forever. lets keep it
simple and extensible, and compatible with other Emacs packages.

the really cool ideas would need Emacs DOM integration, which is not
easy. 

** webkit related
*** TODO webkit support webkit signals

**** DONE particularly document-load-finished
     CLOSED: [2011-08-01 Mon 22:34]
http://webkitgtk.org/reference/webkitgtk-webkitwebview.html#WebKitWebView-document-load-finished
because one might need tell set a title and sizes and things when it loads.
**** TODO event bug
Debugger entered--Lisp error: (error "Two bases given in one event")

hapens sometimes with xwidget events. appears to be when the
originating xwidget is offscreen so that the event doesn't get caught
by the correct emacs event map.

maybe I need to set the originating window in the event structure.
  event.frame_or_window = Qnil; //frame; //how to get the frame here? //TODO i store it in the xwidget now

since its an offscreen xwidget the buffer local keymap isnt the right
place for the handler. some global map should be used.

onscreen widgets don't have the same issue.

anyway, seems it'll turn out like this:
- xwidget-osr stores a callback and user data
- the event is an implementation detail only and get caught in the
  topmost event map
- the event map calls the callback in the xw with the right args.

we need the event handler at some level because we can't call lisp
asynchronously. 

**** TODO navigation signal
**** TODO new window signal
*** TODO console messages
http://webkitgtk.org/reference/webkitgtk-webkitwebview.html#WebKitWebView-console-message
http://getfirebug.com/wiki/index.php/Console_API#console.count.28.5Btitle.5D.29
because maybe we can make a simple JS REPL that way.
  (xwidget-webkit-execute-script ( xwidget-webkit-last-session)
  "console.log('hello')")
prints hello to stdout but theres no way to catch stdout from webkit I
think other than receiving the signal.

*** TODO webkit flashkiller by default
while its possible to support plugins in the webkit xwidget, flash has
issues on 64 bit, and slows down emacs to a halt with off screen
rendering, and of course is not free software. its in the way for real
world usage even if its interesting to watch flash animations inside
emacs. which should be achieved with Gnash or other free software
instead.

http://stackoverflow.com/questions/4885513/prevent-flash-in-cocoa-webview

simply use this api:
http://webkitgtk.org/reference/WebKitWebPluginDatabase.html

theres an implementation now but it's not robust enough webkit often
crashes taking emacs with it.

*** TODO webkit downloads
when clicking a download link in Webkit Emacs should take over and handle it
from there. Probably need signals. There are Emacs libraries to
download things, with wget etc. an url.el facility should be made.
"download-requested"
*** TODO webkit alt-text not handled
XKCD use image-title to display a cartoon comment. These mysteriously
don't work ATM. Other mouseovers work though. Maybe webkit tries to
open a new window or something, which wont work.

*** TODO webkit isearch in webkit buffers
have a look at how docview solves it
webkit_web_view_search_text ()
*** TODO webkit relative references doesn't work
because we handle scrolling in a non-standard way. It does
work sort of when theres a html frameset and webkit scrolls by itself.

internal links (page.html#section) do not work
see xwidget-webkit-show-named-element

also did some  webkit signal work for this.

now it actually works! except for I need to know the Y coordinate of
the element to navigate to, and that can either be by "name" or "id"
attribute, currently "id"  works.



*** TODO webkit width adjustment handling issue
since there are so many levels of clipping and whatnot in xwidgets
sizing issues are difficult.

- an xwidget is told how large it can be by emacs. thats the end of
  it. if the xwidget thinks otherwise it will be clipped.
- but emacs can ask the xwidget how large it wants to be. it can then
  resize the reserved area and inform the xwidget thusly. 

That should have been enough. but webkit never reports less than what
it already has. So currently a webkit view will only growth and not
adjust to smaller sizes. 

This is not a big problem in practice but is still annoying.

to see the problem surface to http://www.slashdot.org
- xwidget-webkit-adjust-size
- xwidget-webkit-adjust-size-to-content

and then compare by resizing in Epiphany, which is also webkit based.

**** TODO try putting webkit osr inside a scrolling window
it seems webkit is supposed to behave differently while embedded in a
scrolling window. This is a bit cumbersome because the container stack
is already deep.
*** TODO xwidget webkit allow loading from string from emacs
*** DONE xwidget-webkit-last-session
    CLOSED: [2011-08-01 Mon 22:38]
was rather hurried. end result is that the lisp layer only really
allows for one webkit session.
*** TODO extract DOM to lisp
then the SHR html renderer from Gnus could render the DOM as created
by Webkit. 

made a simple oxperimental DOM tree traverser. It can be expanded to
return a lisp representation, LDOM.

in order to bring lisp and DOM closer together the LDOM can include a
mapping to the originating DOM node. so, find a node in LDOM, and the
cell maps to the original DOM. but since the LDOM is a copy it can get
out of sync. DOM events might help.
*** DONE C-X b in other buffer from webkit
    CLOSED: [2011-08-12 Fri 22:20]
bafflingly resets the webkit view to the top. Maybe the window
reconfiguration hook code? further mystification is added because it
only seems to happen with ido mode enabled.

in comparison with image-mode which does the right thing, I discovered
that image-mode has special code to handle scrolling. the browser mode
and image mode has some similarities.

I made some delegation code frrom webkit mode to image mode.
*** TODO url-browse improvement
sindikat: site.com and http://site.com should be equivalent (simple site.com
  throws error)

Yes, but its unclear at what level in Emacs to do this
properly. I added a url-tidy function as a start.

this should be further improved:
- change the call to url-tidy so its a hook
- provide a couple of demonstration hooks:
  - url-tidy, which just prepends http://
  - youtube which appends &html5=1 to urls looking like http://www.youtube.com/watch?v=DZdUgjEx_dQ
  - history which logs all visited urls like a traditional browser

*** TODO sindicat notes
Here are some comments from user "sindikat" and my replies

- http://ya.ru renders inadequatly (compare with any other browser) -
  the search text-input is way below

The problem is the size communication between Emacs and Webkit. 

- doing PageDown is endless; so if you do 100 PageDowns, you have to
  do 100 PageUps to retun to the header of the page

True, I hadn't noticed. Thanks.

- http://linux.org.ru (just an example) renders incorrectly too - it
  should stretch horizontally

Size communication.

- obviously, pointing of mouse over some link should change it to
  pointing hand cursor

Need to verify with some other webkit browser.

- when you are somewhere on the middle of a long page, than go to some
  other page, you are still in the middle, instead of being again on
  the top

This is because I inherit from Image view mode. I kind of like it so
we can add an option for it.


- changing dropdown menus cause flickering


- string entering is incorrect - by default it enters the title of the
  page, while it should be empty

The cause is the lack of return value in the webkit evaluation
API. Ive made some fixes.

- internal links (page.html#section) do not work

 ive added a rudimentary function "xwidget-webkit-show-named-element" for this

- maybe it's a good idea to implement Conkeror or some other
  keybindings, where you press 'f' then select the exact <input
  type="text"> where you want to enter text, without using mouse,
  etc.;

Indeed, this would require better DOM integration.

- pressing 'home' and 'end' puts nonsense into minibuffer

Probably because the Image mode derivative is mostly a hack.
fixed now I think.






- implement search (emacs internal isearch obviously doesn't work)

Either use the webkit search but that doesn't feel right. It would be
better to expose the DOM and search that.

- some sites intercept with keyboard; example -
  http://www.artlebedev.ru/kovodstvo/business-lynch/2011/10/03/ uses
  Ctrl+left/right/up/down to navigate between pages - this should be
  implemented too

Keyboard integration is the unloved step-child of xwidgets, unfortunately.



** TODO xwidget image display  spec compatibility
some history: the first version of the xwidget display spec was
the same as an image spec. This turned out not to be fantastic because
an xwidget is both like a process and like an image. it has a separate
existence from display. So now the xwidget display spec is just a
pointer to a xwidget. But then some useful functionality in Emacs
can't be reused for xwidget, in particular image-mode.

Maybe a new image type could be added that was a wraper on an
xwidget. Then image mode could be reused for webkit mode. 

I tried some adaptor code in xwidget.el so webkit mode now delegates
to image mode but its a kludge.

** socket related
*** TODO some flickering during redisplay of sockets
with gtk3 an size allocation workaround is used.
this seems maybe to result in flickering sizewize y-axis with the
xwidget socket type. The webkit xwidget doesn't seem similarily
afflicted.

the size allocation workaround works by 1st running the ordinary
allocation then modifying the results. its done this way to minimize
the copy paste index from the base class. it might be that the
original allocation has a brief time window to show itself.

tried to modify the allocation hack so it doesn't call allocate
twice. this doesn't seem to help flicker at all aparently so the
hypothesis falls. Maybe then a socket simply doesn't like being clipped
by gtkfixed. 

*** TODO xwidget view reaping too agressive
hide an emacs window for a while and return to it. the xwidget might
get reaped and a new socket thus created.
*** DONE try out OSR for sockets
    CLOSED: [2011-07-25 Mon 21:30]

didn't work too well in the inkscape case. it might be that some other
bitmap copy method works better though.

basically sockets doesn't like to be offscreen because they want their
own gdk window.

** DONE synchronise emacs background with xwidget color
   CLOSED: [2011-08-11 Thu 11:04]
fine-tuning to reduce flicker.

isn't needed if emacs bg erase disabled

** DONE xwidgets doesn't work during bootstrap all of a sudden
   CLOSED: [2011-08-01 Mon 22:33]
might be some annoying local issues with my install because it is not
reliably reproducible. (went away during merges)

** CANCELLED low impact xwidget based image viewer
   CLOSED: [2013-04-05 Fri 23:38]
(cancelled this because it no longer seems like a good idea)
for instance to render SVG using webkit, or some other canvas.
that way it would be possible to merge to trunk in stages.

so, webkit could be used to display the SVG. the display spec for
images would be used. multiple webkits would be used rather than
offscreen rendering, so it would be GTK2 compatible. 
** DONE xwidget movement doesn't work all of a sudden
   CLOSED: [2011-08-11 Thu 11:03]
this used to work great. now it doesn't.

suspects:
- XCopyArea
  - x_shift_glyphs_for_insert
  - x_scroll_run. this is run by the try_window* functions, and
    inhibiting them doesnt help. but also callid in scrolling_window.
  

- try_window_reusing_current_matrix
- I used to enable GLYPH_DEBUG which I currently don't. it disables
  many optimisations. this was fixed.
- lookup_xwidget then produce_xwidget_glyph gets called always but not 
x_draw_xwidget_glyph_string probably because of scroll optimization. 
movement detection could possibly be moved to produce_xwidget_glyph(not)

no longer helps:
(setq inhibit-try-window-id t)
(setq inhibit-try-window-reusing t)

workaround:
(run-with-timer 1 1 'redraw-display)

seems to work:
inhibiting scrolling_window(). and this seem to be enough to restore the
old behaviour, GLYPH_DEBUG doesn't seem needed.


** DONE GLYPH_DEBUG doesn't work
   CLOSED: [2011-08-08 Mon 17:30]
was stupid accidental line removal that was hard to spot
** TODO osc xwidget example  
a couple of xwidget sliders that control a csound/supercollider song with osc.
so, for that to work we need slider callbacks to work. when a slider
changes send an osc message. use ocssend:

 oscsend localhost 7777 /sample/address iTfs 1 3.14 hello

or better:
http://delysid.org/emacs/osc.el

sliders could be defined in csound comments or something to illustrate
the point. or if real fanciness is desired parse the csound source
with Semantic and provide a control buffer corresponding to the
defined controls.



Added: [2011-08-11 Thu 10:53]


** DONE SEB
   CLOSED: [2011-10-26 Wed 15:36]
the SEB site does something funny so I can't insert text in
fields. aparently document.activeElement.value doesn't work with framesets.

seems to work using the ugly javascript in 
xwidget-webkit-activeelement-js

** support downstreams
http://aur.archlinux.org/packages.php?ID=53902
http://gpo.zugaina.org/app-editors/emacs-xwidget/ChangeLog
** DONE the proof of concept canvas code should be disabled by default.
   CLOSED: [2011-10-12 Wed 23:03]
** TODO advi
active dvi viewer. investigate if it could be xwidgetified.
advi supports embedding inside presentations.
** cairo configuration support
gtk3 brings in cairo on Fedora, but apparently not on all plattforms.
pkg-config --cflags cairo
** TODO splint
splint   -Demacs -DHAVE_CONFIG_H  -I. -I/home/joakim/build_myprojs/emacsnew/emacs.bzr/xwidget.mint/src -I../lib -I/home/joakim/build_myprojs/emacsnew/emacs.bzr/xwidget.mint/src/../lib   -DGSEAL_ENABLE  -I/usr/include/gtk-3.0 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12   -I/usr/include/freetype2    -I/usr/include/alsa    -I/usr/include/librsvg-2.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/cairo -I/usr/include/libpng12 -I/usr/include/pixman-1 -I/usr/include/freetype2    -I/usr/include/ImageMagick   -I/usr/include/libxml2   -I/usr/include/dbus-1.0 -I/usr/lib64/dbus-1.0/include   -DGSEAL_ENABLE  -I/usr/include/webkit-3.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/gtk-3.0 -I/usr/include/libsoup-2.4 -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/pango-1.0 -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12 -I/usr/include/libxml2  -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -DORBIT2=1  -I/usr/include/gconf/2 -I/usr/include/orbit-2.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -I/usr/include/freetype2   xwidget.c

** TODO 32 bit bug
user reports that xwidgets segfaults  on the 32 bit Mint distribution
but not the 64 bit. Mint is an Ubuntu derivative. I got some
VirtualBox images to test with.
** DONE youtube 
   CLOSED: [2011-11-01 Tue 11:19]
http://www.youtube.com/watch?v=DZdUgjEx_dQ&html5=1
html5 makes it work without stupid flash plugins!
** TODO clicking on an webkit xwidgets
doesn't make the window active. this leads to problems.
** DONE "g" should default to current url
   CLOSED: [2011-11-03 Thu 22:25]
"g" runs xwidget-webkit-browse-url which gets its interactive argument
from browse-url-interactive-arg. this might need a new optional argument.

http://test
** TODO anything/helm support
hook so anything/helm can filter browser history.
** TODO new relative url code sometimes fail
http://www.dilbert.com
** TODO input field enhancements
*** password field. 
was straightforward

*** textarea 
less straightforward. I would like it to work like emacs-w3m, where a
new editing buffer is opened. on c-c, the buffer is closed and the
browser field updated. however, it's not immediately obvious how to
store the reference to the field reliably.

furthermore the current code doesn't seem to propagate linefeed
properly to text areas.

** DONE bug in current navigation handler
   CLOSED: [2011-11-09 Wed 10:04]
on www.dn.se
Debugger entered--Lisp error: (args-out-of-range "http://platform.twitter.com/widgets/hub.html" 54 357)
  match-string(1 "http://platform.twitter.com/widgets/hub.html")
  xwidget-webkit-callback(48890368 navigation-policy-decision-requested)
  xwidget-event-handler()
  call-interactively(xwidget-event-handler nil nil)
** TODO how to set the name of a webkit buffer?
not obvious because, the buffer isn't created immediately and there is
a callback that sets the buffer name automatically
** TODO how to find next field in tab order?
** TODO unique buffer names
the webkit xwidgets renames the buffer after load but not uniquely so
it sometimes fails.
** TODO kill the offscreen webkit xwidgets when last view killed
The offscreen xwidgets is currently kept around even if the xwidgets
views are all gone. this is a general problem and it requires actions
on the behalf of the application to resolve.

In the case of webkit it is currently possible to get errors like these:

Debugger entered--Lisp error: (error "Selecting deleted buffer")
  xwidget-webkit-callback(60925380 navigation-policy-decision-requested)
  xwidget-event-handler()
  call-interactively(xwidget-event-handler nil nil)


because the last view is gone and the offscreen widgets is still
generating events. 

In the case of webkit it is okay to kill the offscreen widgets
completely when the user kills the last view window because it would
be unexpected by the user to see it pop up again. This is not true in
the general case.

** DONE xwidgets debugging log
   CLOSED: [2012-01-23 Mon 14:32]
currently theres a lot of debugging traces using "message" which is
annoying. Instead put them in a separate trace buffer.
(see xwidgetbuffer)
** TODO make garbage collect work for xwidgets
when an xwidget is removed from xwidget-alist, and there are no other
references(mostly views) the xwidget should be garbage collected.

special finalization would go into gc_sweep()
** TODO embedding evince
http://developer.gnome.org/libevview/3.2/libevview-ev-view.html
would be useful for reading PDF:s and other document types.
it would work the same way webkit embedding works.

** TODO support gobject introspection
https://live.gnome.org/GObjectIntrospection/
supporting gobject introspection would mean that more gtk widgets
could be tried out with less effort, and also that the build process
and runtime support would be easier. The drawbacks are small: somewhat
slower execution, and difficulty in providing elisp bindings for
introspection. 

https://live.gnome.org/GObjectIntrospection/HowToWriteALanguageBinding

http://developer.gnome.org/gi/unstable/gi-girepository.html
http://developer.gnome.org/gi/unstable/gi-overview.html


In order for GIR to work, it needs the namespace and class of a
  widget. This is used to access the typelib file, which contains the
  introspection data. The namespace and class is stored as a property
  on the lisp symbol handle used by xwidgets to identify the widget
  class.

This snippet sets the needed :xwgir-class property, and calls the
set_zoom_level method:

M-x xwidget-webkit-browse-url RET www.emacswiki.org RET

Then eval the following:

;;load the webkit typelib
(xwgir-require-namespace "WebKit" "2.0")

;;provide the metadata needed so xwgir can work with the webkit-osr xwidget
(put 'webkit-osr :xwgir-class '("WebKit" "WebView"))
;;call the method
(xwgir-call-method (xwidget-at 1) "set_zoom_level" '(3.0))

It's also possible to create widgets dynamically, by using
introspection to call a widget constructor(from xwidget-test.el):


(defun xwgir-test ()
  (interactive)
  (xwgir-require-namespace "Gtk" "3.0")
  (put 'color-selection :xwgir-class '("Gtk" "ColorSelection"))
  
  (xwgir-demo-a-xwgir-button)
  (xwgir-call-method (xwidget-at 1) "set_label" '( "xwgir set label!"))
  )

Current limitation:
- Only 0 arg constructors are supported at the moment. Since xwidgets
  defer construction, the args needs to be stored with the xwidget.

- xwgir-call-method does indeed lisp to gobject conversion for the
  arguments, but only some primitive types are supported atm.

- next to no argument checking. If wrong type args are used with the
  xwgir methods, emacs crashes.

*** TODO xwgir create components with more advanced constructor
so this opens up an entire new can of beans.

explain by example:
lets say we want to create agtkhscale on screen. its a slider.
https://developer.gnome.org/gtk3/stable/GtkHScale.html
we can already create buttons, so sliders shouldnt be much more
advanced right? wrong.

the simplest slider constructor looks like:
GtkWidget *         gtk_hscale_new
(GtkAdjustment *adjustment);

so in order to call it, we must be able to forward arguments to the
constructor. this is almost already done, but we lack the ability to
pass object instances, only simple types atm.

we need to be able to create GtkAdjustment 
https://developer.gnome.org/gtk3/stable/GtkAdjustment.html

this we can already almost do, because an xwidget is a gir object
with some decorations. we also store the decorated gir object in an
array, retrievable from lisp.

In order for this to be usable in practice, we need some changes:
- lightweight objects should be stored un-decorated. they have no
  need for the entire graphical machinery of xwidgets
- lightweight objects should be garbage collectable, but this is the
  same for all the xwidget objects, and isnt really resolved atm.


** DONE investigate gdk_offscreen_window_set_embedder()
   CLOSED: [2013-04-06 Sat 10:45]
https://developer.gnome.org/gdk/unstable/gdk-Windows.html

,----
| Offscreen windows are more general than composited windows, since they
| allow not only to modify the rendering of the child window onto its
| parent, but also to apply coordinate transformations.
| 
| To integrate an offscreen window into a window hierarchy, one has to
| call gdk_offscreen_window_set_embedder() and handle a number of
| signals. The "pick-embedded-child" signal on the embedder window is
| used to select an offscreen child at given coordinates, and the
| "to-embedder" and "from-embedder" signals on the offscreen window are
| used to translate coordinates between the embedder and the offscreen
| window.
| 
| For rendering an offscreen window onto its embedder, the contents of
| the offscreen window are available as a pixmap, via
| gdk_offscreen_window_get_pixmap().
`----

okay, [2013-04-03 Wed] I finally suceeded in this approach!
it was pretty  hard to make it work and currently works like this:
- the on screen dravwing area is the embedder
- you must implement "pick child"
event forwarding is done automatically! 

BUT its not really super, because it only works well with a single
embedder.

perhaps the strategy could be refined:
- the window frame would be the embedder for all xwidgets. (but what
  about several frames then?)
- in the from-embedder signal handler, which maps container coords to
  embedded widget coords, find out which xw-view i clicked on, and
  compute the coords.

[2013-04-04 Thu] I had a strategy working for a xwgir button but not
a webkit. set_embedded in the motion event handler for the xv. it
even works for 2 frames! but not webkit :(

[2013-04-05 Fri] it works for xwgir osr components, but not for
webkit. Webkit retains the previous event forwarding system.

Now it works like this:
- the offscreen widget is created as before
- the on-screen views also as before, painting and copying as before.
- gdk_offscreen_window_set_embedder() is now used to embedd the
  offscreen widget in the onscreen one, upon view creation
- only one widget can embedd one other. This means that the embedding
  widget must be switched between the onscreen ones. This is now done
  in the mouse motion event handler.

The above approach has been tested for xwgir created buttons and seems
to work. it doesnt work for webkit, so the old scheme is preserved
for webkit.
** TODO investigate git-remote-bzr