Section on peer-to-peer bandwidth estimation

[tor/rransom.git] / doc / TODO.021

$Id$
Legend:
SPEC!!  - Not specified
SPEC    - Spec not finalized
N       - nick claims
R       - arma claims
P       - phobos claims
S       - Steven claims
E       - Matt claims
M       - Mike claims
J       - Jeff claims
I       - ioerror claims
W       - weasel claims
K       - Karsten claims
        - Not done
        * Top priority
        . Partially done
        o Done
        d Deferrable
        D Deferred
        X Abandoned

=======================================================================

Things Roger would be excited to see:

Nick
  * Look at Roger's proposal 141 discussions on or-dev, and help us
    decide how to proceed.
  . Tors start believing the contents of NETINFO cells.
  - respond to Steven's red-team TLS testing (a.k.a, look at a packet
    dump and compare)

Matt
  - Fit Vidalia in 640x480 again.
  - Vidalia should avoid stomping on your custom exit policy lines
    just because you click on 'save' for a totally different config thing.
  - How much space do we save in TBB by stripping symbols from Vidalia
    first? Good idea or crazy idea?

ioerror
  * weather.torproject.org should go live.
  - Keep advocating new Tor servers and working with orgs like Mozilla
    to let them like Tor.
  - Find out what happened to the buildbot and get it back up:
    http://tor-buildbot.freehaven.net:8010/
  - Learn about locking memory pages that have sensitive content. Get
    that started in Tor.
  - Translation portal
    - Vidalia html help files
    - should we i18nize polipo's error messages too?
    - how to get our diagrams translated, and how to get our screenshots
      from the right language?
    - Some of our translated wml files are very old -- so old that they
      are harmful to leave in place. We need some sort of way to notice
      this and disable them.

Steven
  - Move proposal 131 or equivalent forward.
  - Keep bugging us about exploits on the .exit notation.
  - Mike's question #3 on https://www.torproject.org/volunteer#Research
  - Worthwhile shipping TBB with some local html help files that come
    as bookmarks?

Andrew
  - Which bundles include Torbutton? Change the docs/tor-doc-foo pages
    so they admit that Torbutton is in them too. Change the download
    page too.
  - The OS X bundle screenshots are from forever ago -- they don't
    include Torbutton, they still say it's tor.eff.org, etc.
  - Should we still be telling you how to use Safari on OS X for Tor,
    given all the holes that Torbutton-dev solves on Firefox?

Weasel
  - Figure out how to make Vidalia and Tor play nicely on Debian, make
    the necessary modifications, and make some Vidalia debs that pass
    muster.
  - Fix bug 393.
  - Get oftc to switch to Tor dns bulk exitlist. Or tell us why it's
    not suitable yet.
  - Move proposal 134 forward.
  - putting port predictions in state file
  - if tor hasn't been used in a while it stops fetching consensus
    documents.  Retain that state over restarts.

Roger
  - Finish tor-doc-bridge.wml
  . Fix FAQ entry on setting up private Tor network
  - Did we actually apply Steven's dkimproxy patch?
  - Brainstorm about safe but effective ways for vidalia to
    auto-update its user's bridges via Tor in the background.
  - it doesn't count as successfully opening a circuit if it's not
    an exit circuit.

Mike:
  - Roger wants to get an email every time there's a blog change,
    e.g. a comment. That way spam doesn't go undetected for weeks.
    - Or, maybe just disable linking from blog comments entirely?

=======================================================================

Bugs/issues for Tor 0.2.0.x:
  . we should have an off-by-default way for relays to dump geoip data to
    a file in their data directory, for measurement purposes.
    o Basic implementation
N   - Include probability-of-selection
R d let bridges set relaybandwidthrate as low as 5kb

Documentation for Tor 0.2.0.x:
  o Proposals:
    o 111: Prioritize local traffic over relayed.
    o 113: mark as closed close.
  o document the "3/4 and 7/8" business in the clients fetching consensus
    documents timeline.
R   - then document the bridge user download timeline.
  - HOWTO for DNSPort. See tup's wiki page.
  . Document transport and natdport in a good HOWTO.
  - Quietly document NT Service options: revise (or create) FAQ entry

=======================================================================

For 0.2.1.x-alpha:
R d bug: if we launch using bridges, and then stop using bridges, we
    still have our bridges in our entryguards section, and may use them.
  o add an event to report geoip summaries to vidalia for bridge relays,
    so vidalia can say "recent activity (1-8 users) from sa".
R - investigate: it looks like if the bridge authority is unreachable,
    we're not falling back on querying bridges directly?
R - if "no running bridges known", an application request should make
    us retry all our bridges.

For 0.2.1.x:
  - Proposals to do:
    o 110: avoid infinite-length circuits
      * Figure out the right value for max RELAY_EARLY cells (Bug 878)
    - 117: IPv6 Exits
      - Internal code support for ipv6:
        o Clone ipv6 functions (inet_ntop, inet_pton) where they don't exist.
        o Many address variables need to become tor_addr_t
          o addr in connection_t
          o n_addr in extend_info_t
        - Teach resolving code how to handle ipv6.
        . Teach exit policies about ipv6 (consider ipv4/ipv6 interaction!)
        o Use IPv6 in connect/connected/failed-exitpolicy cells
        o accept ipv6 from socks
        o Generate END_REASON_EXITPOLICY cells right
          . ... and parse them right
        . Generate new BEGIN cell types and parse them right
        - Detect availability of ipv6
        - Advertise availability of ipv6.
        - Geoip support, if only to add a zone called "ipv6"

K   . 121: Hidden service authentication:
      - missing: delayed descriptor publication for 'stealth' mode.
    o 128: families of private bridges
    o 135: simplify configuration of private tor networks.
K   - 143: Improvements of Distributed Hidden Service Descriptor Storage:
           only easy parts for 0.2.1.x, defer complex ones to 0.2.2.x.
    o 148: Stream end reasons from the client side should be uniform.
K   o 155: Four Improvements of Hidden Service Performance
    - 145: Separate "suitable from a guard" from "suitable as a new guard"
    - 146: Adding new flag to reflect long-term stability
    - 149: Using data from NETINFO cells
      o Don't extend a circuit over a noncanonical connection with
        mismatched address.
        o Apply rovv's bugfixes wrt preferring canonical connections.
        o Make sure that having a non-canonical connection doesn't count
          as _having_ a connection for the purpose of connecting to others,
          and that when no canonical connection exists, we make one.
      - Learn our outgoing IP address from netinfo cells?
      - Learn skew from netinfo cells?
    o 157: Make certificate downloads specific.

  - Proposals to write:
    - Fix voting to handle bug 608 case when multiple servers get
      Named.
N   . Draft proposal for GeoIP aggregation (see external constraints *)
    . Figure out how to make good use of the fallback consensus file. Right
      now many of the addresses in the fallback consensus will be stale,
      so it will take dozens of minutes to bootstrap from it. This is a
      bad first Tor experience. But if we check the fallback consensus
      file *after* we fail to connect to any authorities, then it may
      still be valuable as a blocking-resistance step.
      o Write the proposal.
      - Patch our tor.spec rpm package so it knows where to put the fallback
        consensus file.
    . Put bandwidth weights in the networkstatus? So clients get weight
      their choices even before they have the descriptors; and so
      authorities can put in more accurate numbers in the future.

  - Spec compliance:
    * Make sure that clients could do the new handshake without sending any
      certs, if they wanted.

  - Tiny designs to write:
    - If a relay publishes a new descriptor with a significantly lower
      uptime or with a new IP address, then we should consider its current
      "running" interval to have ended even if it hadn't yet failed its
      third reachability test. the interval ended when the new descriptor
      appeared, and a new interval began then too.

  - Authority improvements:
R   - authorities should initiate a reachability test upon first
      glimpsing a new descriptor.

  - Use less bandwidth
    - Use if-modified-since to download consensuses

  - Testing
    - Better unit test coverage
    - Verify that write limits to linked connections work.

  - Security improvements
    - make is-consensus-fresh-enough check tighter.
    - If we haven't tried downloading a consensus for ages since we're tired,
      try getting a new one before we use old descriptors for a circuit.
      Related to bug 401. [What does "since we're tired" mean? -RD]
      [I don't know. -NM]

  - Feature removals and deprecations:
    - Get rid of the v1 directory stuff (making, serving, and caching)
      . First verify that the caches won't flip out?
        o If they will, just stop the caches from caching for now
      . perhaps replace it with a "this is a tor server" stock webpage.
        - Get the debs to set DirPortFrontPage in the default.
        - Decide how to handle DirPortFrontPage files with image links.
    - Can we deprecate controllers that don't use both features?
      - Both TorK and Vidalia use VERBOSE_NAMES.
      - TorK uses EXTENDED_EVENTS.  Vidalia does not. (As of 9 Dec.)
      - Matt is checking whether Vidalia would break if we started to use
        EXTENDED_EVENTS by default.  He says no.

External tool improvements:
  - Get IOCP patches into libevent

Nice to have for 0.2.1.x:
  - Proposals, time permitting
    - 134: handle authority fragmentation.
    - 140: Provide diffs betweeen consensuses

  - Handle multi-core cpus better
    - Split circuit AES across cores
      - Split cell_queue_t into a new structure with a processed subqueue,
        an unprocessed subqueue, and a symmetric key.
      - Write a function to pull cells from the unprocessed subqueue,
        en/decrypt them, and place them on the processed subqueue.
      - When a cell is added to a queue that previously had no
        unprocessed cells, put that queue into a set of queues that
        need to be processed.  When the last cell is processed in a
        queue, remove it from the set of queues that need to be
        processed.
      - Worker code to process queues in round-robin fashion.
      - Think about how to be fair to differet circuits _and_ about to get
        CPU-affinity, if that matters.
      - When a cell is processed and placed onto a processed subqueue
        that was previously empty, _and_ the or_conn output buffer
        that the queue is targetting is empty, stick the buffer onto a
        list of buffers that need attention and notify the main
        thread if it was not already on the list.
      - When the main thread gets notified, it pumps those buffers.
        (i.e., it puts cells onto them from some of their circuits).
      - To free a queue that is not currently processing, grab its lock
        and free it.
      - To free a queue that _is_ processing, .... ?

  - Documentation
P   - Make documentation realize that location of system configuration file
      will depend on location of system defaults, and isn't always /etc/torrc.

  - Small controller features
    - A status event for when tor decides to stop fetching directory info
      if the client hasn't clicked recently: then make the onion change too.
    o Add a status event when new consensus arrives

  - Windows build
P   - Figure out why dll's compiled in mingw don't work right in WinXP.
P   - create a "make win32-bundle" for vidalia-privoxy-tor-torbutton bundle

  - Refactor bad code:
    - connection_or_get_by_identity_digest() and connection_good_enough_for
      _extend() could be merged into a smarter variant, perhaps.
    - Refactor the HTTP logic so the functions aren't so large.
    - Refactor buf_read and buf_write to have sensible ways to return
      error codes after partial writes
    - deprecate router_digest_is_trusted_dir() in favor of
      router_get_trusteddirserver_by_digest()

  - Should be trivial
    - Tor logs the libevent version on startup, for debugging purposes.
      This is great. But it does this before configuring the logs, so
      it only goes to stdout and is then lost.

  - Deprecations
    - Even clients run rep_hist_load_mtbf_data().  This doesn't waste memory
      unless they had previously been non-clients collecting MTBF data.
      Dump it anyway?
    - Unless we start using ftime functions, dump them.
    - can we deprecate the FastFirstHopPK config option?
    - The v2dir flag isn't used for anything anymore, right? If so, dump it.
    - can we deprecate 'getinfo network-status'?
    - Dump most uint32_t addr functions.

  - do the part of the "abandon .exit" proposal that involves isolating
    circuits which have used a .exit stream from those that haven't

Defer:
  - Proposals
    - 118: Listen on and advertise multiple ports:
      - Tor should be able to have a pool of outgoing IP addresses that it is
        able to rotate through. (maybe.  Possible overlap with proposal 118.)
      - config option to publish what ports you listen on, beyond
        ORPort/DirPort.  It should support ranges and bit prefixes (?) too.
      - Need to figure out the right format for routerinfo_t on this.
    - 147: Eliminate the need for v2 directories in generating v3 directories

  - Proposals to write.
    d Something for bug 469, to limit connections per IP.
    d Do we want to maintain our own set of entryguards that we use as
      next hop after the bridge?
    d Possibly: revise link protocol to allow big circuit IDs,
      variable-length cells, proposal-110 stuff, and versioned CREATES?
    d Fetch an updated geoip file from the directory authorities.
R   - bridge communities (revive proposal 128)
      . spec
      . deploy
        - man page entries for Alternate*Authority config options

  - Tiny designs to write
    - Better estimate of clock skew; has anonymity implications.  Clients
      should estimate their skew as median of skew from servers over last
      N seconds, but for servers this is not so easy, since a server does
      not choose who it connects to.
    - Do TLS connection rotation more often than "once a week" in the
      extra-stable case.
      (One reason not to do it more often is because the old TLS conn
       probably has a circuit on it, and we don't really want to build up
       dozens of TCP connections to all the other extra-stable relays.)


  - Use less RAM
    - Optimize cell pool allocation.
    - Support (or just always use) jemalloc (if it helps)
    - mmap more files.
    - Pull serverdescs off buffers as they arrive.
    - Allocate routerstatus_t objects on a per-networkstatus memchunk.

  - Split TLS across multiple cores

  - "In the future, we should migrate to LOCAL_APPDATA entirely."

  - Use more mid-level and high-level libevent APIs
    - For dns?
    - For http?
    - For buffers?

  - Proposals to write
    - steven's plan for replacing check.torproject.org with a built-in
      answer by tor itself.

  - Refactor bad code:
    - Streamline how we pick entry nodes: Make choose_random_entry() have
      less magic and less control logic.
    - Move all status info out of routerinfo into local_routerstatus.  Make
      "who can change what" in local_routerstatus explicit.  Make
      local_routerstatus (or equivalent) subsume all places to go for "what
      router is this?"
    o Don't call time(NULL) so much; instead have a static time_t field
      that gets updated only a handful of times per second.
    - Refactor unit tests into multiple files

  - Make Tor able to chroot itself
    o allow it to load an entire config file from control interface
    - document LOADCONF
    - log rotation (and FD passing) via control interface
    - chroot yourself, including inhibit trying to read config file
      and reopen logs, unless they are under datadir.

  - Should be trivial:
    - Base relative control socket paths (and other stuff in torrc) on datadir.
    o enforce a lower limit on MaxCircuitDirtiness and CircuitBuildTimeout.
    - Make 'safelogging' extend to info-level logs too.
    - don't do dns hijacking tests if we're reject *:* exit policy?
      (deferred until 0.1.1.x is less common)
    - More consistent error checking in router_parse_entry_from_string().
      I can say "banana" as my bandwidthcapacity, and it won't even squeak.

  d Interface for letting SOAT modify flags that authorities assign.
    (How to keep the authority from clobbering them afterwards?