hgext/hgfastobs.py
author Pierre-Yves David <pierre-yves.david@fb.com>
Fri, 28 Feb 2014 13:38:19 -0800
changeset 824 fed090e07621
parent 815 916bebf91c41
permissions -rw-r--r--
exchange: pull markers relevant to the pulled subset only With the command recently introduced we can select to pull only markers relevant to some nodes. We are now pull all markers for all node in the relevant subset. We'll try to pull less (just markers for node where local and remote marker diverge) later, but we need some marker discovery mechanism for that. which are not easy.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     1
"""Extension to try and speed up transfer of obsolete markers.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     2
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     3
Mercurial 2.6 transfers obsolete markers in the dumbest way possible:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     4
it simply transfers all of them to the server on every
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     5
operation. While this /works/, it's not ideal because it's a large
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     6
amount of extra data for users to pull down (1.9M for the 17k obsolete
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     7
markers in hg-crew as of this writing in late July 2013). It's also
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     8
frustrating because this transfer takes a nontrivial amount of time.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     9
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    10
You can specify a strategy with the config knob
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    11
obsolete.syncstrategy. Current strategies are "stock" and
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    12
"boxfill". Default strategy is presently boxfill.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    13
794
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    14
:stock: use the default strategy of mercurial explaned above
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    15
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    16
:boxfill: transmit obsolete markers which list any of transmitted changesets as
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    17
          a successor (transitively), as well as any kill markers for dead
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    18
          nodes descended from any of the precursors of outgoing.missing.
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    19
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    20
TODO(durin42): consider better names for sync strategies.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    21
"""
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    22
import sys
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    23
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
    24
from mercurial import base85
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    25
from mercurial import commands
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    26
from mercurial import extensions
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    27
from mercurial import node
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    28
from mercurial import obsolete
799
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    29
from mercurial import exchange
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    30
from mercurial import revset
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    31
from mercurial.i18n import _
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    32
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    33
_strategies = {
799
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    34
    'stock': exchange._pushobsolete,
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    35
    }
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    36
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    37
def _strategy(name, default=False):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    38
    def inner(func):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    39
        _strategies[name] = func
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    40
        if default:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    41
            _strategies[None] = func
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    42
        return func
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    43
    return inner
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    44
799
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    45
def _pushobsoletewrapper(orig, pushop):
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    46
    stratfn = _strategies[pushop.repo.ui.config('obsolete', 'syncstrategy')]
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    47
    return stratfn(pushop)
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    48
799
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    49
extensions.wrapfunction(exchange, '_pushobsolete', _pushobsoletewrapper)
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    50
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    51
def _precursors(repo, s):
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    52
    """Precursor of a changeset"""
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    53
    cs = set()
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    54
    nm = repo.changelog.nodemap
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    55
    markerbysubj = repo.obsstore.precursors
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    56
    for r in s:
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    57
        for p in markerbysubj.get(repo[r].node(), ()):
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    58
            pr = nm.get(p[0])
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    59
            if pr is not None:
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    60
                cs.add(pr)
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    61
    return cs
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    62
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    63
def _revsetprecursors(repo, subset, x):
800
ad2060da7ffa fastobs: revset refactor related fixes
Augie Fackler <raf@durin42.com>
parents: 799
diff changeset
    64
    s = revset.getset(repo, revset.baseset(range(len(repo))), x)
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    65
    cs = _precursors(repo, s)
800
ad2060da7ffa fastobs: revset refactor related fixes
Augie Fackler <raf@durin42.com>
parents: 799
diff changeset
    66
    return revset.baseset([r for r in subset if r in cs])
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    67
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    68
revset.symbols['_fastobs_precursors'] = _revsetprecursors
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    69
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    70
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    71
@_strategy('boxfill', default=True)
799
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    72
def boxfill(pushop):
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    73
    """The "fill in the box" strategy from the 2.6 sprint.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    74
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    75
    See the notes[0] from the 2.6 sprint for what "fill in the box"
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    76
    means here. It's a fairly subtle algorithm, which may have
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    77
    surprising behavior at times, but was the least-bad option
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    78
    proposed at the sprint.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    79
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    80
    [0]: https://bitbucket.org/durin42/2.6sprint-notes/src/tip/mercurial26-obsstore-rev.1398.txt
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    81
    """
799
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    82
    repo = pushop.repo
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    83
    remote = pushop.remote
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    84
    outgoing = pushop.outgoing
a398478e8d86 adapt fastopt extension to mercurial.exchange module
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 798
diff changeset
    85
    urepo = pushop.repo.unfiltered()
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    86
    # need to collect obsolete markers which list any of
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    87
    # outgoing.missing as a successor (transitively), as well as any
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    88
    # kill markers for dead nodes descended from any of the precursors
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    89
    # of outgoing.missing.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    90
    boxedges = urepo.revs(
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    91
        '(descendants(_fastobs_precursors(%ln)) or '
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    92
        ' descendants(%ln)) and hidden()',
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    93
        outgoing.missing, outgoing.missing)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    94
    transmit = []
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
    95
    for node in outgoing.missing:
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
    96
        transmit.extend(obsolete.precursormarkers(urepo[node]))
815
916bebf91c41 fastobs: rename var from `node` to `rev`
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 814
diff changeset
    97
    for rev in boxedges:
916bebf91c41 fastobs: rename var from `node` to `rev`
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 814
diff changeset
    98
        transmit.extend(obsolete.successormarkers(urepo[rev]))
796
443e563f0943 Deduplicate markers. Depends on __hash__ and __eq__ on obsolete.marker
Augie Fackler <raf@durin42.com>
parents: 795
diff changeset
    99
    transmit = list(set(transmit))
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   100
    xmit, total = len(transmit), len(repo.obsstore._all)
792
36d0e71aa9e4 fastobs: clean up logging a little
Augie Fackler <raf@durin42.com>
parents: 791
diff changeset
   101
    repo.ui.status(
36d0e71aa9e4 fastobs: clean up logging a little
Augie Fackler <raf@durin42.com>
parents: 791
diff changeset
   102
        'boxpush: about to transmit %d obsolete markers (%d markers total)\n'
36d0e71aa9e4 fastobs: clean up logging a little
Augie Fackler <raf@durin42.com>
parents: 791
diff changeset
   103
        % (xmit, total))
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   104
    parts, size, chunk = [], 0, 0
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   105
    def transmitmarks():
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   106
            repo.ui.note(
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   107
                'boxpush: sending a chunk of obsolete markers\n')
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   108
            data = ''.join([obsolete._pack('>B', obsolete._fmversion)] + parts)
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   109
            remote.pushkey('obsolete', 'dump-%d' % chunk, '',
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   110
                           base85.b85encode(data))
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   111
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   112
    for marker in transmit:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   113
        enc = obsolete._encodeonemarker(_markertuple(marker))
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   114
        parts.append(enc)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   115
        size += len(enc)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   116
        if size > obsolete._maxpayload:
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   117
            transmitmarks()
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   118
            parts, size = [], 0
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   119
            chunk += 1
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   120
    if parts:
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   121
        transmitmarks()
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   122
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   123
def _markertuple(marker):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   124
    return marker._data