4 A cvs2svn run consists of eight passes. Each pass saves the data it
5 produces to files on disk, so that a) we don't hold huge amounts of
6 state in memory, and b) the conversion process is resumable.
11 The goal of this pass is to write to 'cvs2svn-data.revs' a summary of
12 all the revisions for each RCS file. Each revision will be
13 represented by one line. At the end of this stage, the revisions
14 (i.e., the lines) will be grouped by RCS file, not by logical commits.
16 We walk over the repository, collecting data about the RCS files into
17 an instance of CollectData. Each RCS file is processed with
18 rcsparse.parse(), which invokes callbacks from an instance of
19 cvs2svn's FileDataCollector class (which is a subclass of
22 For each RCS file, the first thing the parser encounters is the
23 administrative header, including the head revision, the principal
24 branch, symbolic names, RCS comments, etc. The main thing that
25 happens here is that FileDataCollector.define_tag() is invoked on each
26 symbolic name and its attached revision, so all the tags and branches
27 of this file get collected.
29 Next, the parser hits the revision summary section. That's the part
30 of the RCS file that looks like this:
33 date 2002.06.12.04.54.12; author captnmark; state Exp;
39 date 2002.05.28.18.02.11; author captnmark; state Exp;
45 For each revision summary, FileDataCollector.define_revision() is
46 invoked, recording that revision's metadata in various variables of
47 the FileDataCollector class instance.
49 After finishing the revision summaries, the parser invokes
50 FileDataCollector.tree_completed(), which loops over the revision
51 information stored, determining if there are instances where a higher
52 revision was committed "before" a lower one (rare, but it can happen
53 when there was clock skew on the repository machine). If there are
54 any, it "resyncs" the timestamp of the earlier rev to be just before
55 that of the later rev, but saves the original timestamp in
56 self.rev_data[blah][2], so we can later write out a record to the
57 resync file indicating that an adjustment was made (this makes it
58 possible to catch the other parts of this commit and resync them
59 similarly, more details below).
61 Next, the parser encounters the *real* revision data, which has the
62 log messages and file contents. For each revision, it invokes
63 FileDataCollector.set_revision_info(), which writes a new line to
64 cvs2svn-data.revs. The line is constructed by the CVSRevision class -
65 one of its many roles. Here is an example:
67 3dc32955 5afe9b4ba41843d8eb52ae7db47a43eaa9573254 3dc32954 3dc32956 C 1.1 1.2 1.3 1 1 1024 N * 0 0 foo/bar,v
71 1. a fixed-width timestamp
72 2. a digest of the log message + author
73 3. a fixed-width timestamp indicating the timestamp of this
74 revision's previous revision (or "*", if it's the first
75 revision on this line of development).
76 4. a fixed-width timestamp indicating the timestamp of this
77 revision's next revision (or "*", if it's the last revision on
78 this line of development).
79 5. the type of change ("A"dd, "C"hange, or "D"elete)
80 6. the revision number of the previous revision along this line of
81 development (or "*", if it's the first revision on this line of
83 7. the revision number
84 8. the revision number of the next revision along this line of
85 development (or "*", if it's the last revision on this line of
87 9. 1 if the RCS file is in the Attic, "*" if it isn't.
88 10. 1 if the RCS file has the executable bit set, "*" if not.
89 12. The size of the RCS file, in bytes.
90 12. "N" if this revision has non-empty deltatext, else "E" for empty
91 13. the RCS keyword substitution mode ("k", "b", etc), or "*" if none
92 14. the branch on which this commit happened, or "*" if not on a branch
93 15. the number of tags rooted at this revision (followed by their
94 names, space-delimited)
95 16. the number of branches rooted at this revision (followed by
96 their names, space-delimited)
97 17. the path of the RCS file in the repository
99 (Of course, in the above example, fields 15 and 16 are "0", so they have
102 Also, for resync'd revisions, a line like this is written out to
103 'cvs2svn-data.resync':
105 3d6c1329 18a215a05abea1c6c155dcc7283b88ae7ce23502 3d6c1328
109 NEW_TIMESTAMP DIGEST OLD_TIMESTAMP
111 (The resync file will be explained later.)
113 That's it -- the RCS file is done.
115 When every RCS file is done, Pass 1 is complete, and:
117 - cvs2svn-data.revs contains a summary of every RCS file's
118 revisions. All the revisions for a given RCS file are grouped
119 together, but note that the groups are in no particular order.
120 In other words, you can't yet identify the commits from looking
121 at these lines; a multi-file commit will be scattered all over
124 - cvs2svn-data.resync contains a small amount of resync data, in
130 This is where the resync file is used. The goal of this pass is to
131 convert cvs2svn-data.revs to a new file, 'cvs2svn-data.c-revs' (clean
132 revs). It's the same as the original file, except for some resync'd
135 First, read the whole resync file into a hash table that maps each
136 author+log digest to a list of lists. Each sublist represents one of
137 the timestamp adjustments from Pass 1, and looks like this:
139 [old_time_lower, old_time_upper, new_time]
141 The reason to map each digest to a list of sublists, instead of to one
142 list, is that sometimes you'll get the same digest for unrelated
143 commits (for example, the same author commits many times using the
144 empty log message, or a log message that just says "Doc tweaks."). So
145 each digest may need to "fan out" to cover multiple commits, but
146 without accidentally unifying those commits.
148 Now we loop over cvs2svn-data.revs, writing each line out to
149 'cvs2svn-data.c-revs'. Most lines are written out unchanged, but
150 those whose digest matches some resync entry, and appear to be part of
151 the same commit as one of the sublists in that entry, get tweaked.
152 The tweak is to adjust the commit time of the line to the new_time,
153 which is taken from the resync hash and results from the adjustment
156 The way we figure out whether a given line needs to be tweaked is to
157 loop over all the sublists, seeing if this commit's original time
158 falls within the old<-->new time range for the current sublist. If it
159 does, we tweak the line before writing it out, and then conditionally
160 adjust the sublist's range to account for the timestamp we just
161 adjusted (since it could be an outlier). Note that this could, in
162 theory, result in separate commits being accidentally unified, since
163 we might gradually adjust the two sides of the range such that they are
164 eventually more than COMMIT_THRESHOLD seconds apart. However, this is
165 really a case of CVS not recording enough information to disambiguate
166 the commits; we'd know we have a time range that exceeds the
167 COMMIT_THRESHOLD, but we wouldn't necessarily know where to divide it
168 up. We could try some clever heuristic, but for now it's not
169 important -- after all, we're talking about commits that weren't
170 important enough to have a distinctive log message anyway, so does it
171 really matter if a couple of them accidentally get unified? Probably
177 This is where we deduce the changesets, that is, the grouping of file
178 changes into single commits.
180 It's very simple -- run 'sort' on cvs2svn-data.c-revs, converting it
181 to 'cvs2svn-data.s-revs'. Because of the way the data is laid out,
182 this causes commits with the same digest (that is, the same author and
183 log message) to be grouped together. Poof! We now have the CVS
184 changes grouped by logical commit.
186 In some cases, the changes in a given commit may be interleaved with
187 other commits that went on at the same time, because the sort gives
188 precedence to date before log digest. However, Pass 4 detects this by
189 seeing that the log digest is different, and reseparates the commits.
194 This pass has two primary objectives:
196 1. Create a database that maps CVSRevision unique keys to the actual
197 CVSRevision string from the revs file (whose format is described
198 above in pass 1). This results in a database containing one
199 key-value pair for each line in the revs file. This gives us the
200 ability to pass around these smaller keys instead of whole CVS
201 revisions (which look like lines from the s-revs file). See the
202 CVSRevision class for more details on what the unique key is.
204 2. Find and create a database containing the last CVS revision that is
205 a source (also referred to as an "opening" revision) for all
206 symbolic names. This will result in a database containing
207 key-value pairs whose key is the unique key for a CVSRevision, and
208 whose value is a list of symbolic names for which that CVSRevision
209 is the last "opening."
211 The format for this file is:
213 cvs-symname-last-revs.db:
215 CVS Revision array of Symbolic names
219 1.38/foo/bar/baz.txt,v --> [TAG11, BRANCH38]
220 1.93/foo/qux/bat.c,v --> [TAG39]
221 1.4/foo/bar/baz.txt,v --> [BRANCH48, BRANCH37]
222 1.18/foo/bar/quux.txt,v --> [TAG320, TAG1178]
227 Primarily, this pass gathers CVS revisions into Subversion revisions
228 (a Subversion revision is comprised of one or more CVS revisions)
229 before we actually begin committing (where "committing" means either
230 to a Subversion repository or to a dump file).
232 This pass does the following:
234 1. Creates a database file to map Subversion Revision numbers to their
235 corresponding CVS Revisions (cvs2svn-svn-revnums-to-cvs-revs.db).
236 Creates another database file to map CVS Revisions to their
237 Subversion Revision numbers (cvs2svn-cvs-revs-to-svn-revnums.db).
239 2. When a file is copied to a symbolic name in cvs2svn, there are a
240 range of valid Subversion revisions that we can copy the file from.
241 The first valid Subversion revision number for a symbolic name is
242 called the "Opening", and the first *invalid* Subversion revision
243 number encountered after the "Opening" is called the "Closing". In
244 this pass, the SymbolingsLogger class writes one line to
245 cvs2svn-symbolic-names.txt per CVS file, per symbolic name, per
248 3. For each CVS Revision in s-revs, we write out a line (for each
249 symbolic name that it opens) to cvs2svn-symbolic-names.txt if it is
250 the first possible source revision (the "opening" revision) for a
251 copy to create a branch or tag, or if it is the last possible
252 revision (the "closing" revision) for a copy to create a branch or
253 tag. Not every opening will have a corresponding closing.
255 The format of each line is:
257 SYMBOLIC_NAME SVN_REVNUM TYPE BRANCH_NAME CVS_PATH
261 MY_TAG1 234 O * foo/bar/baz.txt,v
262 MY_BRANCH3 245 O * foo/qux/bat.c,v
263 MY_TAG1 241 C MY_BRANCH1 foo/bar/baz.txt,v
264 MY_BRANCH_BLAH 201 O MY_BRANCH1 foo/bar/quux.txt,v
266 Here is what the columns mean:
268 SYMBOLIC_NAME: The name of the branch or tag that starts or ends
269 in this CVS Revision (there can be multiples per
272 SVN_REVNUM: The Subversion revision number that is the opening or
273 closing for this SYMBOLIC_NAME.
275 TYPE: "O" for Openings and "C" for Closings.
277 BRANCH_NAME: The (uncleaned) branch name where this opening or
278 closing happened. '*' denotes the default branch.
280 CVS_PATH: The CVS path where this opening or closing happened.
282 See SymbolingsLogger for more details.
287 This pass merely sorts cvs2svn-symbolic-names.txt into
288 cvs2svn-symbolic-names-s.txt. This orders the file first by symbolic
289 name, and second by Subversion revision number, thus grouping all
290 openings and closings for each symbolic name together.
295 This pass iterates through all the lines in
296 cvs2svn-symbolic-names-s.txt, writing out a database file mapping
297 SYMBOLIC_NAME to the file offset in SYMBOL_OPENINGS_CLOSINGS_SORTED
298 where SYMBOLIC_NAME is first encountered. This will allow us to seek
299 to the various offsets in the file and sequentially read only the
300 openings and closings that we need.
305 The 8th pass has very little "thinking" to do--it basically opens the
306 svn-nums-to-cvs-revs.db and, starting with Subversion revision 2
307 (revision 1 creates /trunk, /tags, and /branches), sequentially plays
308 out all the commits to either a Subversion repository or to a
311 In --dump-only mode, the result of this pass is a Subversion
312 repository dumpfile (suitable for input to 'svnadmin load'). The
313 dumpfile is the data's last static stage: last chance to check over
314 the data, run it through svndumpfilter, move the dumpfile to another
317 However, when not in --dump-only mode, no full dumpfile is created for
318 subsequent load into a Subversion repository. Instead, miniature
319 dumpfiles representing a single revision are created, loaded into the
320 repository, and then removed.
322 In both modes, the dumpfile revisions are created by walking through
325 ===============================
326 Branches and Tags Plan.
327 ===============================
329 This pass is also where tag and branch creation is done. Since
330 subversion does tags and branches by copying from existing revisions
331 (then maybe editing the copy, making subcopies underneath, etc), the
332 big question for cvs2svn is how to achieve the minimum number of
333 operations per creation. For example, if it's possible to get the
334 right tag by just copying revision 53, then it's better to do that
335 than, say, copying revision 51 and then sub-copying in bits of
338 Also, since CVS does not version symbolic names, there is the
339 secondary question of *when* to create a particular tag or branch.
340 For example, a tag might have been made at any time after the youngest
341 commit included in it, or might even have been made piecemeal; and the
342 same is true for a branch, with the added constraint that for any
343 particular file, the branch must have been created before the first
344 commit on the branch.
346 Answering the second question first: cvs2svn creates tags as soon as
347 possible and branches as late as possible.
349 Tags are created as soon as cvs2svn encounters the last CVS Revision
350 that is a source for that tag. The whole tag is created in one
353 For branches, this is "just in time" creation -- the moment it sees
354 the first commit on a branch, it snaps the entire branch into
355 existence (or as much of it as possible), and then outputs the branch
358 The reason we say "as much of it as possible" is that it's possible to
359 have a branch where some files have branch commits occuring earlier
360 than the other files even have the source revisions from which the
361 branch sprouts (this can happen if the branch was created piecemeal,
362 for example). In this case, we create as much of the branch as we
363 can, that is, as much of it as there are source revisions available to
364 copy, and leave the rest for later. "Later" might mean just until
365 other branch commits come in, or else during a cleanup stage that
366 happens at the end of this pass (about which more later).
368 How just-in-time branch creation works:
370 In order to make the "best" set of copies/deletes when creating a
371 branch, cvs2svn keeps track of two sets of trees while it's making
374 1. A skeleton mirror of the subversion repository, that is, an
375 array of revisions, with a tree hanging off each revision. (The
376 "array" is actually implemented as an anydbm database itself,
377 mapping string representations of numbers to root keys.)
379 2. A tree for each CVS symbolic name, and the svn file/directory
380 revisions from which various parts of that tree could be copied.
382 Both tree sets live in anydbm databases, using the same basic schema:
383 unique keys map to marshal.dumps() representations of dictionaries,
384 which in turn map entry names to other unique keys:
386 root_key ==> { entryname1 : entrykey1, entryname2 : entrykey2, ... }
387 entrykey1 ==> { entrynameX : entrykeyX, ... }
388 entrykey2 ==> { entrynameY : entrykeyY, ... }
389 entrykeyX ==> { etc, etc ...}
390 entrykeyY ==> { etc, etc ...}
392 (The leaf nodes -- files -- are also dictionaries, for simplicity.)
394 The repository mirror allows cvs2svn to remember what paths exist in
397 For details on how branches and tags are created, please see the
398 docstring the SymbolingsLogger class (and its methods).
400 -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*-
401 - -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -
402 -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*- -*-
404 Some older notes and ideas about cvs2svn. Not deleted, because they
405 may contain suggestions for future improvements in design.
407 -----------------------------------------------------------------------
409 An email from John Gardiner Myers <jgmyers@speakeasy.net> about some
410 considerations for the tool.
413 From: John Gardiner Myers <jgmyers@speakeasy.net>
414 Subject: Thoughts on CVS to SVN conversion
416 Date: Sun, 15 Apr 2001 17:47:10 -0700
418 Some things you may want to consider for a CVS to SVN conversion utility:
420 If converting a CVS repository to SVN takes days, it would be good for
421 the conversion utility to keep its progress state on disk. If the
422 conversion fails halfway through due to a network outage or power
423 failure, that would allow the conversion to be resumed where it left off
424 instead of having to start over from an empty SVN repository.
426 It is a short step from there to allowing periodic updates of a
427 read-only SVN repository from a read/write CVS repository. This allows
428 the more relaxed conversion procedure:
430 1) Create SVN repository writable only by the conversion tool.
431 2) Update SVN repository from CVS repository.
432 3) Announce the time of CVS to SVN cutover.
433 4) Repeat step (2) as needed.
434 5) Disable commits to CVS repository, making it read-only.
436 7) Enable commits to SVN repository.
437 8) Wait for developers to move their workspaces to SVN.
438 9) Decomission the CVS repository.
440 You may forward this message or parts of it as you seem fit.
443 -----------------------------------------------------------------------
445 Further design thoughts from Greg Stein <gstein@lyra.org>
447 * timestamp the beginning of the process. ignore any commits that
448 occur after that timestamp; otherwise, you could miss portions of a
449 commit (e.g. scan A; commit occurs to A and B; scan B; create SVN
450 revision for items in B; we missed A)
452 * the above timestamp can also be used for John's "grab any updates
453 that were missed in the previous pass."
455 * for each file processed, watch out for simultaneous commits. this
456 may cause a problem during the reading/scanning/parsing of the file,
457 or the parse succeeds but the results are garbaged. this could be
458 fixed with a CVS lock, but I'd prefer read-only access.
460 algorithm: get the mtime before opening the file. if an error occurs
461 during reading, and the mtime has changed, then restart the file. if
462 the read is successful, but the mtime changed, then restart the
465 * use a separate log to track unique branches and non-branched forks
466 of revision history (Q: is it possible to create, say, 1.4.1.3
467 without a "real" branch?). this log can then be used to create a
468 /branches/ directory in the SVN repository.
470 Note: we want to determine some way to coalesce branches across
471 files. It can't be based on name, though, since the same branch name
472 could be used in multiple places, yet they are semantically
473 different branches. Given files R, S, and T with branch B, we can
474 tie those files' branch B into a "semantic group" whenever we see
475 commit groups on a branch touching multiple files. Files that are
476 have a (named) branch but no commits on it are simply ignored. For
477 each "semantic group" of a branch, we'd create a branch based on
478 their common ancestor, then make the changes on the children as
479 necessary. For single-file commits to a branch, we could use
480 heuristics (pathname analysis) to add these to a group (and log what
481 we did), or we could put them in a "reject" kind of file for a human
482 to tell us what to do (the human would edit a config file of some
483 kind to instruct the converter).
485 * if we have access to the CVSROOT/history, then we could process tags
486 properly. otherwise, we can only use heuristics or configuration
487 info to group up tags (branches can use commits; there are no
488 commits associated with tags)
490 * ideally, we store every bit of data from the ,v files to enable a
491 complete restoration of the CVS repository. this could be done by
492 storing properties with CVS revision numbers and stuff (i.e. all
493 metadata not already embodied by SVN would go into properties)
495 * how do we track the "states"? I presume "dead" is simply deleting
496 the entry from SVN. what are the other legal states, and do we need
497 to do anything with them?
499 * where do we put the "description"? how about locks, access list,
502 * note that using something like the SourceForge repository will be an
503 ideal test case. people *move* their repositories there, which means
504 that all kinds of stuff can be found in those repositories, from
505 wherever people used to run them, and under whatever development
506 policies may have been used.
508 For example: I found one of the projects with a "permissions 644;"
509 line in the "gnuplot" repository. Most RCS releases issue warnings
510 about that (although they properly handle/skip the lines), and CVS
511 ignores RCS newphrases altogether.