2 @page publish_subscribe Publish-subscribe message passing in Tor
8 Tor has introduced a generic publish-subscribe mechanism for delivering
9 messages internally. It is meant to help us improve the modularity of
10 our code, by avoiding direct coupling between modules that don't
11 actually need to invoke one another.
13 This publish-subscribe mechanism is *not* meant for handing
14 multithreading or multiprocess issues, thought we hope that eventually
15 it might be extended and adapted for that purpose. Instead, we use
16 publish-subscribe today to decouple modules that shouldn't be calling
19 For example, there are numerous parts of our code that might need to
20 take action when a circuit is completed: a controller might need to be
21 informed, an onion service negotiation might need to be attached, a
22 guard might need to be marked as working, or a client connection might
23 need to be attached. But many of those actions occur at a higher layer
24 than circuit completion: calling them directly is a layering violation,
25 and makes our code harder to understand and analyze.
27 But with message-passing, we can invert this layering violation: circuit
28 completion can become a "message" that the circuit code publishes, and
29 to which higher-level layers subscribe. This means that circuit
30 handling can be decoupled from higher-level modules, and stay nice and
31 simple. (@ref pubsub_notyet "1")
33 > @anchor pubsub_notyet 1. Unfortunately, like most of our code, circuit
34 > handling is _not_ yet refactored to use publish-subscribe throughout.
35 > Instead, layer violations of the type described here are pretty common
36 > in Tor today. To see a small part of what happens when a circuit is
37 > completed today, have a look at circuit_build_no_more_hops() and its
40 ## Channels and delivery policies
42 To work with messages, especially when refactoring existing code, you'll
43 need to understand "channels" and "delivery policies".
45 Every message is delivered on a "message channel". Each channel
46 (conceptually) a queue-like structure that can support an arbitrarily
47 number of message types. Where channels vary is their delivery
48 mechanisms, and their guarantees about when messages are processed.
50 Currently, three delivery policies are possible:
52 - `DELIV_PROMPT` -- causes messages to be processed via a callback in
53 Tor's event loop. This is generally the best choice, since it
54 avoids unexpected growth of the stack.
56 - `DELIV_IMMEDIATE` -- causes messages to be processed immediately
57 on the call stack when they are published. This choice grows the
58 stack, and can lead to unexpected complexity in the call graph.
59 We should only use it when necessary.
61 - `DELIV_NEVER` -- causes messages not to be delivered by the message
62 dispatch system at all. Instead, some other part of the code must
63 call dispatch_flush() to get the messages delivered.
65 See mainloop_pubsub.c and mainloop_pubsub.h for more information and
66 implementation details.
68 ## Layers: Dispatch vs publish-subsubscribe vs mainloop.
70 At the lowest level, messages are sent via the "dispatcher" module in
71 @refdir{lib/dispatch}. For performance, this dispatcher works with a
72 untyped messages. Publishers, subscribers, channels, and messages are
73 distinguished by short integers. Associated data is handled as
74 dynamically-typed data pointers, and its types are also stored as short
77 Naturally, this results in a type-unsafe C API, so most other modules
78 shouldn't invoke @refdir{lib/dispatch} directly. At a higher level,
79 @refdir{lib/pubsub} defines a set of functions and macros that make
80 messages named and type-safe. This is the one that other modules should
81 use when they want to send or receive a message.
83 The two modules above do not handle message delivery. Instead, the
84 dispatch module takes a callback that it can invoke when a channel
85 becomes nonempty, and defines a dispatch_flush() function to deliver all
86 the messages queued in a channel. The work of actually making sure that
87 dispatch_flush() is called when appropriate falls to the main loop,
88 which needs to integrate the message dispatcher with the rest of our
89 events and callbacks. This work happens in mainloop_pubsub.c.
92 ## How to publish and subscribe
94 This section gives an overview of how to make new messages and how to
95 use them. For full details, see pubsub_macros.h.
97 Before anybody can publish or subscribe to a message, the message must
98 be declared, typically in a header. This uses DECLARE_MESSAGE() or
99 DECLARE_MESSAGE_INT().
101 Only subsystems can publish or subscribe messages. For more information
102 about the subsystems architecture, see @ref initialization.
104 To publish a message, you must:
105 - Include the header that declares the message.
106 - Declare a set of helper functions via DECLARE_PUBLISH(). These
107 must be visible wherever you call PUBLISH().
108 - Call PUBLISH() to actually send a message.
109 - Connect your subsystem to the dispatcher by calling
110 DISPATCH_ADD_PUB() from your subsystem's subsys_fns_t.add_pubsub
113 To subscribe to a message, you must:
114 - Include the header that declares the message.
115 - Declare a callback function to be invoked when the message is delivered.
116 - Use DISPATCH_SUBSCRIBE at file scope to define a set of wrapper
117 functions to call your callback function with the appropriate type.
118 - Connect your subsystem to the dispatcher by calling
119 DISPATCH_ADD_SUB() from your subsystem's subsys_fns_t.add_pubsub
122 Again, the file-level documentation for pubsub_macros.h describes how to
123 declare a message, how to publish it, and how to subscribe to it.
125 ## Designing good messages
128 The publish-subscribe system uses a few function calls
129 and allocations for each message sent. This makes it unsuitable for
130 very-high-bandwidth events, like "receiving a single data cell" or "a
131 socket has become writable." It's fine, however, for events that
132 ordinarily happen a bit less frequently than that, like a circuit
133 getting finished, a new connection getting opened, or so on.
136 A message should declare that something has happened or is happening,
137 not that something in particular should be done.
139 For example, suppose you want to set up a message so that onion services
140 clean up their replay caches whenever we're low on memory. The event
141 should be something like `memory_low`, not `clean_up_replay_caches`.
142 The latter name would imply that the publisher knew who was subscribing
143 to the message and what they intended to do about it, which would be a