1 Hacking Tor: An Incomplete Guide
2 ================================
7 For full information on how Tor is supposed to work, look at the files in
8 https://gitweb.torproject.org/torspec.git/tree
10 For an explanation of how to change Tor's design to work differently, look at
11 https://gitweb.torproject.org/torspec.git/blob_plain/HEAD:/proposals/001-process.txt
13 For the latest version of the code, get a copy of git, and
15 git clone https://git.torproject.org/git/tor
17 We talk about Tor on the tor-talk mailing list. Design proposals and
18 discussion belong on the tor-dev mailing list. We hang around on
19 irc.oftc.net, with general discussion happening on #tor and development
20 happening on #tor-dev.
22 How we use Git branches
23 -----------------------
25 Each main development series (like 0.2.1, 0.2.2, etc) has its main work
26 applied to a single branch. At most one series can be the development series
27 at a time; all other series are maintenance series that get bug-fixes only.
28 The development series is built in a git branch called "master"; the
29 maintenance series are built in branches called "maint-0.2.0", "maint-0.2.1",
30 and so on. We regularly merge the active maint branches forward.
32 For all series except the development series, we also have a "release" branch
33 (as in "release-0.2.1"). The release series is based on the corresponding
34 maintenance series, except that it deliberately lags the maint series for
35 most of its patches, so that bugfix patches are not typically included in a
36 maintenance release until they've been tested for a while in a development
37 release. Occasionally, we'll merge an urgent bugfix into the release branch
38 before it gets merged into maint, but that's rare.
40 If you're working on a bugfix for a bug that occurs in a particular version,
41 base your bugfix branch on the "maint" branch for the first supported series
42 that has that bug. (As of June 2013, we're supporting 0.2.3 and later.) If
43 you're working on a new feature, base it on the master branch.
49 When you do a commit that needs a ChangeLog entry, add a new file to
50 the "changes" toplevel subdirectory. It should have the format of a
51 one-entry changelog section from the current ChangeLog file, as in
54 - Fix a potential buffer overflow. Fixes bug 99999; bugfix on
57 To write a changes file, first categorize the change. Some common categories
58 are: Minor bugfixes, Major bugfixes, Minor features, Major features, Code
59 simplifications and refactoring. Then say what the change does. If
60 it's a bugfix, mention what bug it fixes and when the bug was
61 introduced. To find out which Git tag the change was introduced in,
62 you can use "git describe --contains <sha1 of commit>".
64 If at all possible, try to create this file in the same commit where
65 you are making the change. Please give it a distinctive name that no
66 other branch will use for the lifetime of your change.
68 When we go to make a release, we will concatenate all the entries
69 in changes to make a draft changelog, and clear the directory. We'll
70 then edit the draft changelog into a nice readable format.
72 What needs a changes file?::
73 A not-exhaustive list: Anything that might change user-visible
74 behavior. Anything that changes internals, documentation, or the build
75 system enough that somebody could notice. Big or interesting code
76 rewrites. Anything about which somebody might plausibly wonder "when
77 did that happen, and/or why did we do that" 6 months down the line.
79 Why use changes files instead of Git commit messages?::
80 Git commit messages are written for developers, not users, and they
81 are nigh-impossible to revise after the fact.
83 Why use changes files instead of entries in the ChangeLog?::
84 Having every single commit touch the ChangeLog file tended to create
85 zillions of merge conflicts.
90 These aren't strictly necessary for hacking on Tor, but they can help track
96 https://jenkins.torproject.org
101 The dmalloc library will keep track of memory allocation, so you can find out
102 if we're leaking memory, doing any double-frees, or so on.
104 dmalloc -l ~/dmalloc.log
105 (run the commands it tells you)
106 ./configure --with-dmalloc
111 valgrind --leak-check=yes --error-limit=no --show-reachable=yes src/or/tor
113 (Note that if you get a zillion openssl warnings, you will also need to
114 pass --undef-value-errors=no to valgrind, or rebuild your openssl
117 Running lcov for unit test coverage
118 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
120 Lcov is a utility that generates pretty HTML reports of test code coverage.
121 To generate such a report:
124 ./configure --enable-coverage
127 $BROWSER ./coverage_html/index.html
130 This will run the tor unit test suite `./src/test/test` and generate the HTML
131 coverage code report under the directory ./coverage_html/. To change the
132 output directory, use `make coverage-html HTML_COVER_DIR=./funky_new_cov_dir`.
134 Coverage diffs using lcov are not currently implemented, but are being
135 investigated (as of July 2014).
137 Running the unit tests
138 ~~~~~~~~~~~~~~~~~~~~~~
140 To quickly run all tests:
145 To run unit tests only:
150 To selectively run just some tests (the following can be combined
153 ./src/test/test <name_of_test> [<name of test 2>] ...
154 ./src/test/test <prefix_of_name_of_test>.. [<prefix_of_name_of_test2>..] ...
155 ./src/test/test :<name_of_excluded_test> [:<name_of_excluded_test2]...
158 Running gcov for unit test coverage
159 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
162 ./configure --enable-coverage
165 mkdir coverage-output
166 ./scripts/test/coverage coverage-output
169 (On OSX, you'll need to start with "--enable-coverage CC=clang".)
171 Then, look at the .gcov files in coverage-output. '-' before a line means
172 that the compiler generated no code for that line. '######' means that the
173 line was never reached. Lines with numbers were called that number of times.
175 If that doesn't work:
176 * Try configuring Tor with --disable-gcc-hardening
177 * You might need to run 'make clean' after you run './configure'.
179 If you make changes to Tor and want to get another set of coverage results,
180 you can run "make reset-gcov" to clear the intermediary gcov output.
182 If you have two different "coverage-output" directories, and you want to see
183 a meaningful diff between them, you can run:
186 ./scripts/test/cov-diff coverage-output1 coverage-output2 | less
189 In this diff, any lines that were visited at least once will have coverage
190 "1". This lets you inspect what you (probably) really want to know: which
191 untested lines were changed? Are there any new untested lines?
193 Running integration tests
194 ~~~~~~~~~~~~~~~~~~~~~~~~~
196 We have the beginnings of a set of scripts to run integration tests using
197 Chutney. To try them, set CHUTNEY_PATH to your chutney source directory, and
198 run "make test-network".
200 Profiling Tor with oprofile
201 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
203 The oprofile tool runs (on Linux only!) to tell you what functions Tor is
204 spending its CPU time in, so we can identify berformance pottlenecks.
206 Here are some basic instructions
208 - Build tor with debugging symbols (you probably already have, unless
209 you messed with CFLAGS during the build process).
210 - Build all the libraries you care about with debugging symbols
211 (probably you only care about libssl, maybe zlib and Libevent).
212 - Copy this tor to a new directory
213 - Copy all the libraries it uses to that dir too (ldd ./tor will
215 - Set LD_LIBRARY_PATH to include that dir. ldd ./tor should now
216 show you it's using the libs in that dir
218 - Reset oprofiles counters/start it
219 * "opcontrol --reset; opcontrol --start", if Nick remembers right.
220 - After a while, have it dump the stats on tor and all the libs
221 in that dir you created.
222 * "opcontrol --dump;"
223 * "opreport -l that_dir/*"
233 If possible, send your patch as one of these (in descending order of
236 - A git branch we can pull from
237 - Patches generated by git format-patch
242 - To build your code while configured with --enable-gcc-warnings?
243 - To run "make check-spaces" on your code?
244 - To run "make check-docs" to see whether all new options are on
246 - To write unit tests, as possible?
247 - To base your code on the appropriate branch?
248 - To include a file in the "changes" directory as appropriate?
250 Whitespace and C conformance
251 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
253 Invoke "make check-spaces" from time to time, so it can tell you about
254 deviations from our C whitespace style. Generally, we use:
256 - Unix-style line endings
257 - K&R-style indentation
258 - No space before newlines
259 - A blank line at the end of each file
260 - Never more than one blank line in a row
261 - Always spaces, never tabs
262 - No more than 79-columns per line.
263 - Two spaces per indent.
264 - A space between control keywords and their corresponding paren
265 "if (x)", "while (x)", and "switch (x)", never "if(x)", "while(x)", or
267 - A space between anything and an open brace.
268 - No space between a function name and an opening paren. "puts(x)", not
270 - Function declarations at the start of the line.
272 We try hard to build without warnings everywhere. In particular, if you're
273 using gcc, you should invoke the configure script with the option
274 "--enable-gcc-warnings". This will give a bunch of extra warning flags to
275 the compiler, and help us find divergences from our preferred C style.
277 Getting emacs to edit Tor source properly
278 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
280 Nick likes to put the following snippet in his .emacs file:
283 (add-hook 'c-mode-hook
286 (set-variable 'show-trailing-whitespace t)
288 (let ((fname (expand-file-name (buffer-file-name))))
290 ((string-match "^/home/nickm/src/libevent" fname)
291 (set-variable 'indent-tabs-mode t)
292 (set-variable 'c-basic-offset 4)
293 (set-variable 'tab-width 4))
294 ((string-match "^/home/nickm/src/tor" fname)
295 (set-variable 'indent-tabs-mode nil)
296 (set-variable 'c-basic-offset 2))
297 ((string-match "^/home/nickm/src/openssl" fname)
298 (set-variable 'indent-tabs-mode t)
299 (set-variable 'c-basic-offset 8)
300 (set-variable 'tab-width 8))
304 You'll note that it defaults to showing all trailing whitespace. The "cond"
305 test detects whether the file is one of a few C free software projects that I
306 often edit, and sets up the indentation level and tab preferences to match
309 If you want to try this out, you'll need to change the filename regex
310 patterns to match where you keep your Tor files.
312 If you use emacs for editing Tor and nothing else, you could always just say:
315 (add-hook 'c-mode-hook
318 (set-variable 'show-trailing-whitespace t)
319 (set-variable 'indent-tabs-mode nil)
320 (set-variable 'c-basic-offset 2)))
323 There is probably a better way to do this. No, we are probably not going
324 to clutter the files with emacs stuff.
330 We have some wrapper functions like tor_malloc, tor_free, tor_strdup, and
331 tor_gettimeofday; use them instead of their generic equivalents. (They
332 always succeed or exit.)
334 You can get a full list of the compatibility functions that Tor provides by
335 looking through src/common/util.h and src/common/compat.h. You can see the
336 available containers in src/common/containers.h. You should probably
337 familiarize yourself with these modules before you write too much code, or
338 else you'll wind up reinventing the wheel.
340 Use 'INLINE' instead of 'inline', so that we work properly on Windows.
342 Calling and naming conventions
343 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
345 Whenever possible, functions should return -1 on error and 0 on success.
347 For multi-word identifiers, use lowercase words combined with
348 underscores. (e.g., "multi_word_identifier"). Use ALL_CAPS for macros and
351 Typenames should end with "_t".
353 Function names should be prefixed with a module name or object name. (In
354 general, code to manipulate an object should be a module with the same name
355 as the object, so it's hard to tell which convention is used.)
357 Functions that do things should have imperative-verb names
358 (e.g. buffer_clear, buffer_resize); functions that return booleans should
359 have predicate names (e.g. buffer_is_empty, buffer_needs_resizing).
361 If you find that you have four or more possible return code values, it's
362 probably time to create an enum. If you find that you are passing three or
363 more flags to a function, it's probably time to create a flags argument that
369 Don't optimize anything if it's not in the critical path. Right now, the
370 critical path seems to be AES, logging, and the network itself. Feel free to
371 do your own profiling to determine otherwise.
376 https://trac.torproject.org/projects/tor/wiki/doc/TorFAQ#loglevel
378 No error or warning messages should be expected during normal OR or OP
381 If a library function is currently called such that failure always means ERR,
382 then the library function should log WARN and let the caller log ERR.
384 Every message of severity INFO or higher should either (A) be intelligible
385 to end-users who don't know the Tor source; or (B) somehow inform the
386 end-users that they aren't expected to understand the message (perhaps
387 with a string like "internal error"). Option (A) is to be preferred to
393 We use the 'doxygen' utility to generate documentation from our
394 source code. Here's how to use it:
396 1. Begin every file that should be documented with
399 * \brief Short description of the file.
402 (Doxygen will recognize any comment beginning with /** as special.)
404 2. Before any function, structure, #define, or variable you want to
405 document, add a comment of the form:
407 /** Describe the function's actions in imperative sentences.
409 * Use blank lines for paragraph breaks
415 * Write <b>argument_names</b> in boldface.
418 * place_example_code();
419 * between_code_and_endcode_commands();
423 3. Make sure to escape the characters "<", ">", "\", "%" and "#" as "\<",
424 "\>", "\\", "\%", and "\#".
426 4. To document structure members, you can use two forms:
429 /** You can put the comment before an element; */
431 int b; /**< Or use the less-than symbol to put the comment
432 * after the element. */
435 5. To generate documentation from the Tor source code, type:
439 To generate a file called 'Doxyfile'. Edit that file and run
440 'doxygen' to generate the API documentation.
442 6. See the Doxygen manual for more information; this summary just
443 scratches the surface.
445 Doxygen comment conventions
446 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
448 Say what functions do as a series of one or more imperative sentences, as
449 though you were telling somebody how to be the function. In other words, DO
452 /** The strtol function parses a number.
454 * nptr -- the string to parse. It can include whitespace.
455 * endptr -- a string pointer to hold the first thing that is not part
456 * of the number, if present.
457 * base -- the numeric base.
458 * returns: the resulting number.
460 long strtol(const char *nptr, char **nptr, int base);
462 Instead, please DO say:
464 /** Parse a number in radix <b>base</b> from the string <b>nptr</b>,
465 * and return the result. Skip all leading whitespace. If
466 * <b>endptr</b> is not NULL, set *<b>endptr</b> to the first character
467 * after the number parsed.
469 long strtol(const char *nptr, char **nptr, int base);
471 Doxygen comments are the contract in our abstraction-by-contract world: if
472 the functions that call your function rely on it doing something, then your
473 function should mention that it does that something in the documentation. If
474 you rely on a function doing something beyond what is in its documentation,
475 then you should watch out, or it might do something else later.
477 Putting out a new release
478 -------------------------
480 Here are the steps Roger takes when putting out a new Tor release:
482 1) Use it for a while, as a client, as a relay, as a hidden service,
483 and as a directory authority. See if it has any obvious bugs, and
486 1.5) As applicable, merge the maint-X branch into the release-X branch.
488 2) Gather the changes/* files into a changelog entry, rewriting many
489 of them and reordering to focus on what users and funders would find
490 interesting and understandable.
492 2.1) Make sure that everything that wants a bug number has one.
493 Make sure that everything which is a bugfix says what version
495 2.2) Concatenate them.
496 2.3) Sort them by section. Within each section, sort by "version it's
497 a bugfix on", else by numerical ticket order.
502 "Fixes bug 9999; bugfix on 0.3.3.3-alpha."
504 One space after a period.
506 Make stuff very terse
508 Make sure each section name ends with a colon
510 Describe the user-visible problem right away
512 Mention relevant config options by name. If they're rare or unusual,
513 remind people what they're for
515 Avoid starting lines with open-paren
517 Present and imperative tense: not past.
519 'Relays', not 'servers' or 'nodes' or 'Tor relays'.
521 "Stop FOOing", not "Fix a bug where we would FOO".
523 Try not to let any given section be longer than about a page. Break up
524 long sections into subsections by some sort of common subtopic. This
525 guideline is especially important when organizing Release Notes for
528 If a given changes stanza showed up in a different release (e.g.
529 maint-0.2.1), be sure to make the stanzas identical (so people can
530 distinguish if these are the same change).
534 2.6) Clean everything one last time.
536 2.7) Run ./scripts/maint/format_changelog.py to make it prettier.
538 3) Compose a short release blurb to highlight the user-facing
539 changes. Insert said release blurb into the ChangeLog stanza. If it's
540 a stable release, add it to the ReleaseNotes file too. If we're adding
541 to a release-0.2.x branch, manually commit the changelogs to the later
544 4) Bump the version number in configure.ac and rebuild.
546 5) Make dist, put the tarball up somewhere, and tell #tor about it. Wait
547 a while to see if anybody has problems building it. Try to get Sebastian
548 or somebody to try building it on Windows.
550 6) Get at least two of weasel/arma/sebastian to put the new version number
551 in their approved versions list.
553 7) Sign the tarball, then sign and push the git tag:
554 gpg -ba <the_tarball>
555 git tag -u <keyid> tor-0.2.x.y-status
556 git push origin tag tor-0.2.x.y-status
558 8a) scp the tarball and its sig to the dist website, i.e.
559 /srv/dist-master.torproject.org/htdocs/ on dist-master. When you want
560 it to go live, you run "static-update-component dist.torproject.org"
563 8b) Edit "include/versions.wmi" and "Makefile" to note the new version.
565 9) Email the packagers (cc'ing tor-assistants) that a new tarball is up.
567 10) Add the version number to Trac. To do this, go to Trac, log in,
568 select "Admin" near the top of the screen, then select "Versions" from
569 the menu on the left. At the right, there will be an "Add version"
570 box. By convention, we enter the version in the form "Tor:
571 0.2.2.23-alpha" (or whatever the version is), and we select the date as
572 the date in the ChangeLog.
574 11) Forward-port the ChangeLog.
576 12) Wait up to a day or two (for a development release), or until most
577 packages are up (for a stable release), and mail the release blurb and
578 changelog to tor-talk or tor-announce.
580 (We might be moving to faster announcements, but don't announce until
581 the website is at least updated.)
583 13) If it's a stable release, bump the version number in the maint-x.y.z
584 branch to "newversion-dev", and do a "merge -s ours" merge to avoid
585 taking that change into master. Do a similar 'merge -s theirs'
586 merge to get the change (and only that change) into release. (Some
587 of the build scripts require that maint merge cleanly into release.)