hgfastobs.py
author Augie Fackler <raf@durin42.com>
Fri, 26 Jul 2013 16:04:40 -0400
changeset 791 5f3b53d74b7f
child 792 36d0e71aa9e4
permissions -rw-r--r--
fastobs: first commit of an extension to test obsolete marker exchange methods This currently implements the "fill in the box" approach discussed at the 2.6 sprint, which seems to handle common cases correctly with significantly less data transferred.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     1
"""Extension to try and speed up transfer of obsolete markers.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     2
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     3
Mercurial 2.6 transfers obsolete markers in the dumbest way possible:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     4
it simply transfers all of them to the server on every
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     5
operation. While this /works/, it's not ideal because it's a large
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     6
amount of extra data for users to pull down (1.9M for the 17k obsolete
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     7
markers in hg-crew as of this writing in late July 2013). It's also
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     8
frustrating because this transfer takes a nontrivial amount of time.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     9
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    10
You can specify a strategy with the config knob
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    11
obsolete.syncstrategy. Current strategies are "stock" and
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    12
"boxfill". Default strategy is presently boxfill.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    13
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    14
TODO(durin42): consider better names for sync strategies.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    15
"""
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    16
import sys
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    17
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    18
from mercurial import commands
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    19
from mercurial import extensions
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    20
from mercurial import obsolete
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    21
from mercurial import node
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    22
from mercurial.i18n import _
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    23
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    24
_strategies = {
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    25
    'stock': obsolete.syncpush,
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    26
    }
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    27
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    28
def _strategy(name, default=False):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    29
    def inner(func):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    30
        _strategies[name] = func
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    31
        if default:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    32
            _strategies[None] = func
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    33
        return func
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    34
    return inner
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    35
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    36
def syncpushwrapper(orig, repo, remote):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    37
    stratfn = _strategies[repo.ui.config('obsolete', 'syncstrategy')]
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    38
    return stratfn(repo, remote)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    39
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    40
extensions.wrapfunction(obsolete, 'syncpush', syncpushwrapper)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    41
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    42
def pushmarkerwrapper(orig, repo, *args):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    43
    if repo.ui.config('obsolete', 'syncstrategy') == 'stock':
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    44
        return orig(repo, *args)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    45
    # We shouldn't need to do this, since we transmit markers
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    46
    # effectively during push in localrepo. Just return success.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    47
    return 1
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    48
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    49
def _getoutgoing():
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    50
    f = sys._getframe(4)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    51
    return f.f_locals['outgoing']
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    52
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    53
@_strategy('boxfill', default=True)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    54
def boxfill(repo, remote):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    55
    """The "fill in the box" strategy from the 2.6 sprint.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    56
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    57
    See the notes[0] from the 2.6 sprint for what "fill in the box"
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    58
    means here. It's a fairly subtle algorithm, which may have
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    59
    surprising behavior at times, but was the least-bad option
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    60
    proposed at the sprint.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    61
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    62
    [0]: https://bitbucket.org/durin42/2.6sprint-notes/src/tip/mercurial26-obsstore-rev.1398.txt
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    63
    """
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    64
    outgoing = _getoutgoing()
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    65
    urepo = repo.unfiltered()
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    66
    # need to collect obsolete markers which list any of
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    67
    # outgoing.missing as a successor (transitively), as well as any
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    68
    # kill markers for dead nodes descended from any of the precursors
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    69
    # of outgoing.missing.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    70
    boxedges = urepo.revs(
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    71
        '(descendants(precursors(%ln)) or descendants(%ln)) and hidden()',
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    72
        outgoing.missing, outgoing.missing)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    73
    transmit = []
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    74
    for node in boxedges:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    75
        transmit.extend(obsolete.successormarkers(urepo[node]))
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    76
    xmit, total = len(transmit), len(repo.obsstore._all)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    77
    repo.ui.status('about to transmit %d obsolete markers (%d markers total)\n'
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    78
                   % (xmit, total))
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    79
    parts, size, chunk = [], 0, 0
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    80
    for marker in transmit:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    81
        enc = obsolete._encodeonemarker(_markertuple(marker))
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    82
        parts.append(enc)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    83
        size += len(enc)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    84
        if size > obsolete._maxpayload:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    85
            repo.ui.note(
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    86
                'obsolete boxpush: sending a chunk of obsolete markers\n')
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    87
            data = ''.join([obsolete._pack('>B', _fmversion)], parts)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    88
            remote.pushkey('obsolete', 'dump%d' % chunk, base85.b85encode(data))
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    89
            parts, size = [], 0
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    90
            chunk += 1
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    91
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    92
def _markertuple(marker):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    93
    return marker._data