author | Augie Fackler <raf@durin42.com> |
Fri, 26 Jul 2013 21:35:56 -0400 | |
changeset 793 | fa746ef46e8a |
parent 792 | 36d0e71aa9e4 |
child 794 | 089755743050 |
permissions | -rw-r--r-- |
791
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
1 |
"""Extension to try and speed up transfer of obsolete markers. |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
2 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
3 |
Mercurial 2.6 transfers obsolete markers in the dumbest way possible: |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
4 |
it simply transfers all of them to the server on every |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
5 |
operation. While this /works/, it's not ideal because it's a large |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
6 |
amount of extra data for users to pull down (1.9M for the 17k obsolete |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
7 |
markers in hg-crew as of this writing in late July 2013). It's also |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
8 |
frustrating because this transfer takes a nontrivial amount of time. |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
9 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
10 |
You can specify a strategy with the config knob |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
11 |
obsolete.syncstrategy. Current strategies are "stock" and |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
12 |
"boxfill". Default strategy is presently boxfill. |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
13 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
14 |
TODO(durin42): consider better names for sync strategies. |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
15 |
""" |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
16 |
import sys |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
17 |
|
793
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
18 |
from mercurial import base85 |
791
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
19 |
from mercurial import commands |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
20 |
from mercurial import extensions |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
21 |
from mercurial import obsolete |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
22 |
from mercurial import node |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
23 |
from mercurial.i18n import _ |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
24 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
25 |
_strategies = { |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
26 |
'stock': obsolete.syncpush, |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
27 |
} |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
28 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
29 |
def _strategy(name, default=False): |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
30 |
def inner(func): |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
31 |
_strategies[name] = func |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
32 |
if default: |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
33 |
_strategies[None] = func |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
34 |
return func |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
35 |
return inner |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
36 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
37 |
def syncpushwrapper(orig, repo, remote): |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
38 |
stratfn = _strategies[repo.ui.config('obsolete', 'syncstrategy')] |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
39 |
return stratfn(repo, remote) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
40 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
41 |
extensions.wrapfunction(obsolete, 'syncpush', syncpushwrapper) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
42 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
43 |
def pushmarkerwrapper(orig, repo, *args): |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
44 |
if repo.ui.config('obsolete', 'syncstrategy') == 'stock': |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
45 |
return orig(repo, *args) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
46 |
# We shouldn't need to do this, since we transmit markers |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
47 |
# effectively during push in localrepo. Just return success. |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
48 |
return 1 |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
49 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
50 |
def _getoutgoing(): |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
51 |
f = sys._getframe(4) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
52 |
return f.f_locals['outgoing'] |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
53 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
54 |
@_strategy('boxfill', default=True) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
55 |
def boxfill(repo, remote): |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
56 |
"""The "fill in the box" strategy from the 2.6 sprint. |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
57 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
58 |
See the notes[0] from the 2.6 sprint for what "fill in the box" |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
59 |
means here. It's a fairly subtle algorithm, which may have |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
60 |
surprising behavior at times, but was the least-bad option |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
61 |
proposed at the sprint. |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
62 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
63 |
[0]: https://bitbucket.org/durin42/2.6sprint-notes/src/tip/mercurial26-obsstore-rev.1398.txt |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
64 |
""" |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
65 |
outgoing = _getoutgoing() |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
66 |
urepo = repo.unfiltered() |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
67 |
# need to collect obsolete markers which list any of |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
68 |
# outgoing.missing as a successor (transitively), as well as any |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
69 |
# kill markers for dead nodes descended from any of the precursors |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
70 |
# of outgoing.missing. |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
71 |
boxedges = urepo.revs( |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
72 |
'(descendants(precursors(%ln)) or descendants(%ln)) and hidden()', |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
73 |
outgoing.missing, outgoing.missing) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
74 |
transmit = [] |
793
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
75 |
for node in outgoing.missing: |
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
76 |
transmit.extend(obsolete.precursormarkers(urepo[node])) |
791
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
77 |
for node in boxedges: |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
78 |
transmit.extend(obsolete.successormarkers(urepo[node])) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
79 |
xmit, total = len(transmit), len(repo.obsstore._all) |
792
36d0e71aa9e4
fastobs: clean up logging a little
Augie Fackler <raf@durin42.com>
parents:
791
diff
changeset
|
80 |
repo.ui.status( |
36d0e71aa9e4
fastobs: clean up logging a little
Augie Fackler <raf@durin42.com>
parents:
791
diff
changeset
|
81 |
'boxpush: about to transmit %d obsolete markers (%d markers total)\n' |
36d0e71aa9e4
fastobs: clean up logging a little
Augie Fackler <raf@durin42.com>
parents:
791
diff
changeset
|
82 |
% (xmit, total)) |
791
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
83 |
parts, size, chunk = [], 0, 0 |
793
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
84 |
def transmitmarks(): |
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
85 |
repo.ui.note( |
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
86 |
'boxpush: sending a chunk of obsolete markers\n') |
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
87 |
data = ''.join([obsolete._pack('>B', obsolete._fmversion)] + parts) |
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
88 |
remote.pushkey('obsolete', 'dump-%d' % chunk, '', |
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
89 |
base85.b85encode(data)) |
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
90 |
|
791
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
91 |
for marker in transmit: |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
92 |
enc = obsolete._encodeonemarker(_markertuple(marker)) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
93 |
parts.append(enc) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
94 |
size += len(enc) |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
95 |
if size > obsolete._maxpayload: |
793
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
96 |
transmitmarks() |
791
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
97 |
parts, size = [], 0 |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
98 |
chunk += 1 |
793
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
99 |
if parts: |
fa746ef46e8a
fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents:
792
diff
changeset
|
100 |
transmitmarks() |
791
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
101 |
|
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
102 |
def _markertuple(marker): |
5f3b53d74b7f
fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
103 |
return marker._data |