hgfastobs.py
author Pierre-Yves David <pierre-yves.david@logilab.fr>
Tue, 18 Feb 2014 15:30:54 -0800
changeset 798 eb0d18490c14
parent 796 443e563f0943
child 799 a398478e8d86
permissions -rw-r--r--
drop unused function `pushmarkerwrapper` I can find any user for this function.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     1
"""Extension to try and speed up transfer of obsolete markers.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     2
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     3
Mercurial 2.6 transfers obsolete markers in the dumbest way possible:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     4
it simply transfers all of them to the server on every
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     5
operation. While this /works/, it's not ideal because it's a large
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     6
amount of extra data for users to pull down (1.9M for the 17k obsolete
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     7
markers in hg-crew as of this writing in late July 2013). It's also
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     8
frustrating because this transfer takes a nontrivial amount of time.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     9
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    10
You can specify a strategy with the config knob
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    11
obsolete.syncstrategy. Current strategies are "stock" and
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    12
"boxfill". Default strategy is presently boxfill.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    13
794
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    14
:stock: use the default strategy of mercurial explaned above
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    15
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    16
:boxfill: transmit obsolete markers which list any of transmitted changesets as
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    17
          a successor (transitively), as well as any kill markers for dead
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    18
          nodes descended from any of the precursors of outgoing.missing.
089755743050 fastobs: details strategies in the main help
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 793
diff changeset
    19
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    20
TODO(durin42): consider better names for sync strategies.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    21
"""
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    22
import sys
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    23
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
    24
from mercurial import base85
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    25
from mercurial import commands
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    26
from mercurial import extensions
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    27
from mercurial import node
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    28
from mercurial import obsolete
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    29
from mercurial import revset
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    30
from mercurial.i18n import _
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    31
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    32
_strategies = {
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    33
    'stock': obsolete.syncpush,
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    34
    }
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    35
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    36
def _strategy(name, default=False):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    37
    def inner(func):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    38
        _strategies[name] = func
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    39
        if default:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    40
            _strategies[None] = func
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    41
        return func
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    42
    return inner
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    43
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    44
def syncpushwrapper(orig, repo, remote):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    45
    stratfn = _strategies[repo.ui.config('obsolete', 'syncstrategy')]
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    46
    return stratfn(repo, remote)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    47
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    48
extensions.wrapfunction(obsolete, 'syncpush', syncpushwrapper)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    49
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    50
def _getoutgoing():
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    51
    f = sys._getframe(4)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    52
    return f.f_locals['outgoing']
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    53
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    54
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    55
def _precursors(repo, s):
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    56
    """Precursor of a changeset"""
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    57
    cs = set()
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    58
    nm = repo.changelog.nodemap
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    59
    markerbysubj = repo.obsstore.precursors
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    60
    for r in s:
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    61
        for p in markerbysubj.get(repo[r].node(), ()):
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    62
            pr = nm.get(p[0])
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    63
            if pr is not None:
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    64
                cs.add(pr)
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    65
    return cs
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    66
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    67
def _revsetprecursors(repo, subset, x):
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    68
    s = revset.getset(repo, range(len(repo)), x)
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    69
    cs = _precursors(repo, s)
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    70
    return [r for r in subset if r in cs]
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    71
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    72
revset.symbols['_fastobs_precursors'] = _revsetprecursors
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    73
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    74
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    75
@_strategy('boxfill', default=True)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    76
def boxfill(repo, remote):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    77
    """The "fill in the box" strategy from the 2.6 sprint.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    78
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    79
    See the notes[0] from the 2.6 sprint for what "fill in the box"
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    80
    means here. It's a fairly subtle algorithm, which may have
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    81
    surprising behavior at times, but was the least-bad option
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    82
    proposed at the sprint.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    83
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    84
    [0]: https://bitbucket.org/durin42/2.6sprint-notes/src/tip/mercurial26-obsstore-rev.1398.txt
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    85
    """
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    86
    outgoing = _getoutgoing()
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    87
    urepo = repo.unfiltered()
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    88
    # need to collect obsolete markers which list any of
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    89
    # outgoing.missing as a successor (transitively), as well as any
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    90
    # kill markers for dead nodes descended from any of the precursors
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    91
    # of outgoing.missing.
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    92
    boxedges = urepo.revs(
795
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    93
        '(descendants(_fastobs_precursors(%ln)) or '
c6888ec28e9e copy precursor revset from evolve so we don't depend on involve
Augie Fackler <raf@durin42.com>
parents: 794
diff changeset
    94
        ' descendants(%ln)) and hidden()',
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    95
        outgoing.missing, outgoing.missing)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    96
    transmit = []
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
    97
    for node in outgoing.missing:
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
    98
        transmit.extend(obsolete.precursormarkers(urepo[node]))
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    99
    for node in boxedges:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   100
        transmit.extend(obsolete.successormarkers(urepo[node]))
796
443e563f0943 Deduplicate markers. Depends on __hash__ and __eq__ on obsolete.marker
Augie Fackler <raf@durin42.com>
parents: 795
diff changeset
   101
    transmit = list(set(transmit))
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   102
    xmit, total = len(transmit), len(repo.obsstore._all)
792
36d0e71aa9e4 fastobs: clean up logging a little
Augie Fackler <raf@durin42.com>
parents: 791
diff changeset
   103
    repo.ui.status(
36d0e71aa9e4 fastobs: clean up logging a little
Augie Fackler <raf@durin42.com>
parents: 791
diff changeset
   104
        'boxpush: about to transmit %d obsolete markers (%d markers total)\n'
36d0e71aa9e4 fastobs: clean up logging a little
Augie Fackler <raf@durin42.com>
parents: 791
diff changeset
   105
        % (xmit, total))
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   106
    parts, size, chunk = [], 0, 0
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   107
    def transmitmarks():
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   108
            repo.ui.note(
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   109
                'boxpush: sending a chunk of obsolete markers\n')
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   110
            data = ''.join([obsolete._pack('>B', obsolete._fmversion)] + parts)
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   111
            remote.pushkey('obsolete', 'dump-%d' % chunk, '',
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   112
                           base85.b85encode(data))
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   113
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   114
    for marker in transmit:
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   115
        enc = obsolete._encodeonemarker(_markertuple(marker))
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   116
        parts.append(enc)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   117
        size += len(enc)
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   118
        if size > obsolete._maxpayload:
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   119
            transmitmarks()
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   120
            parts, size = [], 0
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   121
            chunk += 1
793
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   122
    if parts:
fa746ef46e8a fastobs: fix a bunch of stupid errors that prevented it from working at all
Augie Fackler <raf@durin42.com>
parents: 792
diff changeset
   123
        transmitmarks()
791
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   124
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   125
def _markertuple(marker):
5f3b53d74b7f fastobs: first commit of an extension to test obsolete marker exchange methods
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   126
    return marker._data