Sideline is about improvements to TopGit that make "sideline development" possible. TopGit already makes it relatively easy to continuously integrate changes from mainline and other branches into patch development branches. This makes sure that patches are always applicable to mainline. Sideline is trying to remove (some of) the pains when the patch development branches are at least partially evolving or experimental and dependencies need to be changed, and when infrastructure changes are made, especially on the patch development branches. Infrastructure changes are changes that touch the code in many places, but which are often semantically mostly independent of other changes. Examples for these kinds of changes are:

• unifying code variants for 64-bit and 32-bit machines,
• reorganizing data structures,
• changing the locking scheme,
• changing the memory management scheme,
• re-designing the interfaces,
• refactoring the code.

Infrastructure changes usually take some time to mature, especially when they are mainly useful together with other features under development. The complete set of changes may even constitute an alternative to the current mainline version, which may or may not turn out to be a better alternative, or it may end up in a separate subsystem, or be discarded with some lessons learned. For these reasons, these change sets are usually not applied to mainline directly. These changes are developed alongside the mainline, or in a sideline. The difference to a fork is that "normal" mainline features, fixes and infrastructure changes in mainline are still continuously integrated into the sideline. (Even other branches can be integrated.) That means that the whole set of changes of the sideline remains available as patches against the current state of mainline, and they can at any time be tested and used with the current mainline version. Therefore no split in the developer and user communities is necessary, and a fork is avoided.

There are two main problems with sideline development. One is that it must be possible to easily change patch dependencies. The other problem is that continuously merging new changes from mainline to the sideline becomes difficult when at least one of them has infrastructure changes that the other one does not have.

Support for changing patch dependencies in TopGit is still evolving, and work is done in the tsort repository (the parent repository for this one). The sideline repository adds a patch that is a different approach to relatively cheaply add dependencies to a TopGit branch, while making it possible to bring in a previously removed dependency (which current TopGit cannot do).

Changing patch dependencies is especially important for sidelines, because sidelines tend to be experimental (otherwise they could have been developed in a normal branch against mainline, instead of in a TopGit-managed patch set that is used to prepare patches that need to go through the patch review process), and basically everything can change. As experience with the code grows, patches may need to be dropped, re-added or rewritten. (Typically there are several areas to explore: finding an adequate solution to the problem, finding out which parts of the solution need to be written, and which parts can be covered with existing code, how can the solution be made competetive, etc. Depending on the experience with the problem and the existing code, and also based on how well upcoming features by others can be anticipated, the initial designs may or may not need to be changed.) Another factor is the assessment by others, which may not be predictable before the sideline has matured to a certain degree (e.g. prototype). For example, mainline maintainers may want to cherry-pick some features up in the dependency hierarchy earlier, while features on a lower level may need to mature more before they can be used in mainline, and this may change patch dependencies.

Developers still need to be able to create (and commit) code quickly and to try it out, because that is often the easiest way to collect the experience that leads to better designs. Results from previous approaches should be reproducable, and design decisions are likely to change and variants saved in the SCM may need to be re-used later on. So on the design level, it must be possible to easily (and cheaply) change dependencies later on, after many commits have already been made.

The merge problem arises when infrastructure changes are made in the mainline, the sideline or in both. (Infrastructure changes are common in sidelines, because they are the main reason why sidelines are not merged into the mainline early on: they need to prove their merits, as nobody wants to rely on infrastructure that does not meet everyone's requirements.) Infrastructure changes change code in many places, remove lines of code in many places and may even rearrange code by moving many pieces of code to other places. When new mainline changes are merged in on top of such infrastructure changes, the infrastructure changes are likely to overlap with the new mainline changes, and conflicts are detected for all of these cases. (git rerere does not help here, as the changes are new to git rerere as well, and no previously encountered conflict matches.)

Here is a summary of issues and how to address them:

• Changes can be experimental, so it should be possible to remove them or to use their changes as the basis for other changes, if they turn out to be useful, or to reorganize TopGit branches when requirements, assumptions and code change. TopGit needs support for removing a patch branch by folding the changes into another branch and removing it from other branches' dependencies, or instead of folding the changes into another branch it should be possible to simply drop them. And support is needed for adding, removing and maybe even reordering dependencies in a cheap and simple way.
• Infrastructure changes make updating the sideline with mainline changes prone to produce many merge conflicts whenever the sideline is updated with mainline changes that touch the same subsystem. One idea to handle this is to add a hook for a pre-merge script that then pre-processes the mainline files before a merge is attempted. The user can create this script, and it's pre-processing should change the mainline files in a way that makes it possible for the built-in content tracking in git to match up mainline content with sideline content. This can be as simple as changing variable names systematically.
• If the project supports modules, it should be possible to put the sideline into a separate module (in a different set of source code files), so that the mainline variant remains available. This makes testing easier (the mainline variant may provide necessary functionality), and a comparison of features is easier as well. The merge hooks should make this possible, by enabling git's content tracking to detect that there is a copy of the content which needs the mainline change merged in as well.