| commit | c7b6b9ca55b8a259c8f380a69297762756674ce4 | |
| author | Shawn O. Pearce <spearce@spearce.org> | |
| Thu, 23 Apr 2009 03:36:23 +0000 (22 20:36 -0700) | ||
| committer | Robin Rosenberg <robin.rosenberg@dewire.com> | |
| Thu, 23 Apr 2009 22:25:58 +0000 (24 00:25 +0200) | ||
| tree | 53afefb999d07e860248a12355ba49974ba52e1d | treesnapshot (tar.gz zip) |
| parent | 45f5de89c4b0bbd587bfaf523a92c15dba0f470d | commitdiff |
Teach PackWriter to recover from removed/replaced packs
A concurrently running "git gc" in the same repository could cause a
PackFile that was previously identified for object reuse to disappear
(or be rewritten) between the time that a segment was selected to
be reused, and when we actually need to copy the raw data from the
pack to the output stream.
We now peg the pack file open during the reuse period, ensuring
that the underlying file descriptor cannot be closed while we are
copying data, even if memory pressure gets high and windows are
evicted from the WindowCache.
If the pack file is gone (or has been rewritten) since we originally
picked it for reuse we throw away that reuse decision and make
it again. This is a relative waste of CPU and disk IO, as we have
to do work twice, but it should be fairly infrequent as repositories
are not repacked that often. If we do have to recompute one object,
it is likely that we may need to recompute all reuse decisions for
the remainder of this pack output stream, but doing a bit more work
and succeeding is better than failing outright with an obtuse error.
The way we handle the recovery is subject to livelock. We could
pick a new reuse location, see it disappear before we can get a pin,
and need to select another one. Livelock however is not likely
here, as the situation can only happen when the selected pack
file has been deleted or overwritten. Repacking takes some time
on any repository, typically longer than a single PackWriter may
need to stream to a network client. It is highly improbable that
the repository administator is running "while true; git gc; done",
as that would suck up all system resources and generally make the
host unresponsive anyway. So, long story short, we should be OK
against livelock if the repository administrator isn't hellbent
on otherwise sucking up CPU usage through repeat git gc attempts.
And if they are, we'll just help them out by falling for the obvious
livelock case.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
A concurrently running "git gc" in the same repository could cause a
PackFile that was previously identified for object reuse to disappear
(or be rewritten) between the time that a segment was selected to
be reused, and when we actually need to copy the raw data from the
pack to the output stream.
We now peg the pack file open during the reuse period, ensuring
that the underlying file descriptor cannot be closed while we are
copying data, even if memory pressure gets high and windows are
evicted from the WindowCache.
If the pack file is gone (or has been rewritten) since we originally
picked it for reuse we throw away that reuse decision and make
it again. This is a relative waste of CPU and disk IO, as we have
to do work twice, but it should be fairly infrequent as repositories
are not repacked that often. If we do have to recompute one object,
it is likely that we may need to recompute all reuse decisions for
the remainder of this pack output stream, but doing a bit more work
and succeeding is better than failing outright with an obtuse error.
The way we handle the recovery is subject to livelock. We could
pick a new reuse location, see it disappear before we can get a pin,
and need to select another one. Livelock however is not likely
here, as the situation can only happen when the selected pack
file has been deleted or overwritten. Repacking takes some time
on any repository, typically longer than a single PackWriter may
need to stream to a network client. It is highly improbable that
the repository administator is running "while true; git gc; done",
as that would suck up all system resources and generally make the
host unresponsive anyway. So, long story short, we should be OK
against livelock if the repository administrator isn't hellbent
on otherwise sucking up CPU usage through repeat git gc attempts.
And if they are, we'll just help them out by falling for the obvious
livelock case.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>