hgext3rd/pullbundle.py
author Anton Shestakov <av6@dwimlabs.net>
Fri, 20 Dec 2019 14:08:37 +0700
changeset 5047 a8ac174010a5
parent 5046 ebd0f5c6098f
child 5193 a4d081923c81
permissions -rw-r--r--
pullbundle: random.sample() chokes on a spanset, turn it into a list instead On Python 3.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     1
# Extension to provide automatic caching of bundle server for pull
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     2
#
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     3
# Copyright 2018 Pierre-Yves David <pierre-yves.david@ens-lyon.org>
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     4
#
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     5
# This software may be used and distributed according to the terms of the
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     6
# GNU General Public License version 2 or any later version.
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
     7
"""pullbundle: automatic server side bundle caching
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     8
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
     9
General principle
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    10
=================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    11
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    12
This extension provides a means for server to use pre-computed bundle for
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    13
serving arbitrary pulls. If missing, the necessary pre-computed bundle will be
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    14
generated on demand.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    15
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    16
To maximize usage of existing cached bundle, each pull will be served through
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    17
multiple bundles. The bundle will be created using "standard range" from the
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    18
"stablerange" principle. The "stablerange" concept if already used for
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    19
obsmarkers discovery in the evolve extensions.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    20
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    21
Using pull Bundle
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    22
=================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    23
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    24
All configuration is only required server side.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    25
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    26
The "stablerange" code currently still live in the evolve extensions, so for
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    27
now enabling that extensions is required:
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    28
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    29
You need at minimum the following configuration:
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    30
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    31
    [extensions]
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    32
    evolve=yes
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    33
    pullbundle=yes
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    34
    [experimental]
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    35
    obshashrange.warm-cache = yes
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    36
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    37
If you do not want to use evolution server side, you should disable obsmarkers exchange:
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    38
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    39
    [experimental]
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    40
    evolution.exchange=no
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    41
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    42
Extra Configuration
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    43
===================
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    44
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    45
  [pullbundle]
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    46
  # By default bundles are stored `.hg/cache/pullbundles/.
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    47
  # This can be changed with the following config:
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    48
  cache-directory=/absolute/path
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    49
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    50
Implementation status
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    51
=====================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    52
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    53
Both for stablerange and pullbundle use "simple" initial implementations.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    54
Theses implemenations focus on testing the algorithms and proving the features
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    55
works. Yet they are already useful and used in production.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    56
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    57
Performances are expected to greatly improved in the final implementation,
4141
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    58
especially if some of it end up being compiled code.
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    59
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    60
This first implementation lacks the ability to server the cached bundle from a
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    61
CDN. We'll want this limitation to be lifted quickly.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    62
4141
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    63
The way mercurial core report progress is designed for the receival of a single
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    64
changegroup. So currently using pullbundle means flooding the user with output.
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    65
This will have to be fixed.
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    66
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    67
Why is does this live in the same repository as evolve
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    68
======================================================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    69
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    70
There is no fundamental reasons for live in the same repository. However, the
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    71
stablerange data-structure lives in evolve, so it was simpler to put this new
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    72
extensions next to it. As soon as stable range have been upstreamed, we won't
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    73
need the dependency to the evolve extension anymore.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    74
"""
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    75
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    76
import collections
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    77
import errno
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    78
import random
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    79
import os
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    80
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    81
from mercurial import (
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    82
    changegroup,
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    83
    discovery,
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    84
    error,
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    85
    exchange,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    86
    narrowspec,
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    87
    node as nodemod,
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    88
    registrar,
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    89
    scmutil,
5044
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
    90
    ui as uimod,
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    91
    util,
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    92
)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    93
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    94
from mercurial.i18n import _
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    95
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
    96
__version__ = b'0.1.1'
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
    97
testedwith = b'4.4 4.5 4.6 4.7.1'
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
    98
minimumhgversion = b'4.4'
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
    99
buglink = b'https://bz.mercurial-scm.org/'
4135
47f1d7b4305d pullbundle: add basic extension metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4134
diff changeset
   100
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   101
cmdtable = {}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   102
command = registrar.command(cmdtable)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   103
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   104
configtable = {}
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   105
configitem = registrar.configitem(configtable)
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   106
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   107
configitem(b'pullbundle', b'cache-directory',
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   108
           default=None,
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   109
)
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   110
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   111
# generic wrapping
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   112
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   113
def uisetup(ui):
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   114
    exchange.getbundle2partsmapping[b'changegroup'] = _getbundlechangegrouppart
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   115
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   116
def _getbundlechangegrouppart(bundler, repo, source, bundlecaps=None,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   117
                              b2caps=None, heads=None, common=None, **kwargs):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   118
    """add a changegroup part to the requested bundle"""
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   119
    if not kwargs.get(r'cg', True):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   120
        return
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   121
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   122
    version = b'01'
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   123
    cgversions = b2caps.get(b'changegroup')
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   124
    if cgversions:  # 3.1 and 3.2 ship with an empty value
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   125
        cgversions = [v for v in cgversions
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   126
                      if v in changegroup.supportedoutgoingversions(repo)]
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   127
        if not cgversions:
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   128
            raise ValueError(_(b'no common changegroup version'))
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   129
        version = max(cgversions)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   130
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   131
    outgoing = exchange._computeoutgoing(repo, heads, common)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   132
    if not outgoing.missing:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   133
        return
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   134
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   135
    if kwargs.get(r'narrow', False):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   136
        include = sorted(filter(bool, kwargs.get(r'includepats', [])))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   137
        exclude = sorted(filter(bool, kwargs.get(r'excludepats', [])))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   138
        filematcher = narrowspec.match(repo.root, include=include,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   139
                                       exclude=exclude)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   140
    else:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   141
        filematcher = None
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   142
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   143
    # START OF ALTERED PART
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   144
    makeallcgpart(bundler.newpart, repo, outgoing, version, source, bundlecaps,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   145
                  filematcher, cgversions)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   146
    # END OF ALTERED PART
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   147
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   148
    if kwargs.get(r'narrow', False) and (include or exclude):
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   149
        narrowspecpart = bundler.newpart(b'narrow:spec')
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   150
        if include:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   151
            narrowspecpart.addparam(
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   152
                b'include', b'\n'.join(include), mandatory=True)
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   153
        if exclude:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   154
            narrowspecpart.addparam(
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   155
                b'exclude', b'\n'.join(exclude), mandatory=True)
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   156
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   157
def makeallcgpart(newpart, repo, outgoing, version, source,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   158
                  bundlecaps, filematcher, cgversions):
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   159
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   160
    pullbundle = not filematcher
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   161
    if pullbundle and not util.safehasattr(repo, 'stablerange'):
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   162
        repo.ui.warn(b'pullbundle: required extension "evolve" are missing, skipping pullbundle\n')
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   163
        pullbundle = False
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   164
    if filematcher:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   165
        makeonecgpart(newpart, repo, None, outgoing, version, source, bundlecaps,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   166
                      filematcher, cgversions)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   167
    else:
4133
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   168
        start = util.timer()
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   169
        slices = sliceoutgoing(repo, outgoing)
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   170
        end = util.timer()
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   171
        msg = _(b'pullbundle-cache: "missing" set sliced into %d subranges '
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   172
                b'in %f seconds\n')
4133
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   173
        repo.ui.write(msg % (len(slices), end - start))
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   174
        for sliceid, sliceout in slices:
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   175
            makeonecgpart(newpart, repo, sliceid, sliceout, version, source, bundlecaps,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   176
                          filematcher, cgversions)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   177
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   178
# stable range slicing
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   179
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   180
DEBUG = False
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   181
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   182
def sliceoutgoing(repo, outgoing):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   183
    cl = repo.changelog
4929
bb2b4f6c99dc compat: compatibility for cl.nodemap.get vs cl.index.get_rev
Anton Shestakov <av6@dwimlabs.net>
parents: 4814
diff changeset
   184
    rev = getgetrev(cl)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   185
    node = cl.node
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   186
    revsort = repo.stablesort
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   187
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   188
    missingrevs = set(rev(n) for n in outgoing.missing)
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   189
    if DEBUG:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   190
        ms = missingrevs.copy()
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   191
        ss = []
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   192
    allslices = []
4147
88e922eca4e2 pullbundle: deal with head in a predicable order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4145
diff changeset
   193
    missingheads = [rev(n) for n in sorted(outgoing.missingheads, reverse=True)]
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   194
    for head in missingheads:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   195
        localslices = []
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   196
        localmissing = set(repo.revs(b'%ld and ::%d', missingrevs, head))
4149
4abec3a1208f pullbundle: fix style warning
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4148
diff changeset
   197
        thisrunmissing = localmissing.copy()
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   198
        while localmissing:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   199
            slicerevs = []
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   200
            for r in revsort.walkfrom(repo, head):
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   201
                if r not in thisrunmissing:
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   202
                    break
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   203
                slicerevs.append(r)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   204
            slicenodes = [node(r) for r in slicerevs]
4140
9b71aa222f8e pullbundle: issue local branches in the right order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4139
diff changeset
   205
            localslices.append(canonicalslices(repo, slicenodes))
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   206
            if DEBUG:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   207
                ss.append(slicerevs)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   208
            missingrevs.difference_update(slicerevs)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   209
            localmissing.difference_update(slicerevs)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   210
            if localmissing:
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   211
                heads = list(repo.revs(b'heads(%ld)', localmissing))
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   212
                heads.sort(key=node)
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   213
                head = heads.pop()
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   214
                if heads:
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   215
                    thisrunmissing = repo.revs(b'%ld and only(%d, %ld)',
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   216
                                               localmissing,
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   217
                                               head,
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   218
                                               heads)
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   219
                else:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   220
                    thisrunmissing = localmissing.copy()
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   221
        if DEBUG:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   222
            for s in reversed(ss):
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   223
                ms -= set(s)
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   224
                missingbase = repo.revs(b'parents(%ld) and %ld', s, ms)
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   225
                if missingbase:
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   226
                    repo.ui.write_err(b'!!! rev bundled while parents missing\n')
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   227
                    repo.ui.write_err(b'    parent: %s\n' % list(missingbase))
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   228
                    pb = repo.revs(b'%ld and children(%ld)', s, missingbase)
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   229
                    repo.ui.write_err(b'    children: %s\n' % list(pb))
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   230
                    h = repo.revs(b'heads(%ld)', s)
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   231
                    repo.ui.write_err(b'    heads: %s\n' % list(h))
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   232
                    raise error.ProgrammingError(b'issuing a range before its parents')
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   233
4140
9b71aa222f8e pullbundle: issue local branches in the right order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4139
diff changeset
   234
        for s in reversed(localslices):
9b71aa222f8e pullbundle: issue local branches in the right order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4139
diff changeset
   235
            allslices.extend(s)
4136
be3a94d3105f pullbundle: attempt to save stablerange cache after each computation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4135
diff changeset
   236
    # unknown subrange might had to be computed
be3a94d3105f pullbundle: attempt to save stablerange cache after each computation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4135
diff changeset
   237
    repo.stablerange.save(repo)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   238
    return [(rangeid, outgoingfromnodes(repo, nodes))
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   239
            for rangeid, nodes in allslices]
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   240
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   241
def canonicalslices(repo, nodes):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   242
    depth = repo.depthcache.get
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   243
    stablerange = repo.stablerange
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   244
    rangelength = lambda x: stablerange.rangelength(repo, x)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   245
    headrev = repo.changelog.rev(nodes[0])
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   246
    nbrevs = len(nodes)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   247
    headdepth = depth(headrev)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   248
    skipped = headdepth - nbrevs
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   249
    rangeid = (headrev, skipped)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   250
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   251
    subranges = canonicalsubranges(repo, stablerange, rangeid)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   252
    idx = 0
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   253
    slices = []
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   254
    nodes.reverse()
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   255
    for rangeid in subranges:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   256
        size = rangelength(rangeid)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   257
        slices.append((rangeid, nodes[idx:idx + size]))
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   258
        idx += size
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   259
    ### slow code block to validate ranges content
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   260
    # rev = repo.changelog.nodemap.get
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   261
    # for ri, ns in slices:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   262
    #     a = set(rev(n) for n in ns)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   263
    #     b = set(repo.stablerange.revsfromrange(repo, ri))
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   264
    #     l = repo.stablerange.rangelength(repo, ri)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   265
    #     repo.ui.write('range-length: %d-%d %s %s\n' % (ri[0], ri[1], l, len(a)))
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   266
    #     if a != b:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   267
    #         d =  (ri[0], ri[1], b - a, a - b)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   268
    #         repo.ui.write("mismatching content: %d-%d -%s +%s\n" % d)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   269
    return slices
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   270
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   271
def canonicalsubranges(repo, stablerange, rangeid):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   272
    """slice a size of nodes into most reusable subranges
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   273
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   274
    We try to slice a range into a set of "largest" and "canonical" stable
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   275
    range.
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   276
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   277
    It might make sense to move this function as a 'stablerange' method.
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   278
    """
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   279
    headrev, skip = rangeid
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   280
    rangedepth = stablerange.depthrev(repo, rangeid[0])
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   281
    canonicals = []
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   282
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   283
    # 0. find the first power of 2 higher than this range depth
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   284
    cursor = 1
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   285
    while cursor <= rangedepth:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   286
        cursor *= 2
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   287
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   288
    # 1. find first cupt
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   289
    precut = cut = 0
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   290
    while True:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   291
        if skip <= cut:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   292
            break
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   293
        if cut + cursor < rangedepth:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   294
            precut = cut
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   295
            cut += cursor
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   296
        if cursor == 1:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   297
            break
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   298
        cursor //= 2
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   299
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   300
    # 2. optimise, bottom part
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   301
    if skip != cut:
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   302
        currentsize = tailsize = cut - skip
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   303
        assert 0 < tailsize, tailsize
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   304
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   305
        # we need to take several "standard cut" in the bottom part
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   306
        #
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   307
        # This is similar to what we will do for the top part, we reusing the
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   308
        # existing structure is a bit more complex.
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   309
        allcuts = list(reversed(standardcut(tailsize)))
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   310
        prerange = (headrev, precut)
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   311
        ### slow code block to check we operate on the right data
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   312
        # rev = repo.changelog.nodemap.get
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   313
        # allrevs = [rev(n) for n in nodes]
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   314
        # allrevs.reverse()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   315
        # prerevs = repo.stablerange.revsfromrange(repo, prerange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   316
        # assert allrevs == prerevs[(len(prerevs) - len(allrevs)):]
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   317
        # end of check
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   318
        sub = list(stablerange.subranges(repo, prerange)[:-1])
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   319
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   320
        bottomranges = []
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   321
        # XXX we might be able to reuse core stable-range logic instead of
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   322
        # redoing this manually
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   323
        currentrange = sub.pop()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   324
        currentsize = stablerange.rangelength(repo, currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   325
        currentcut = None
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   326
        while allcuts or currentcut is not None:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   327
            # get the next cut if needed
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   328
            if currentcut is None:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   329
                currentcut = allcuts.pop()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   330
            # deal attemp a cut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   331
            if currentsize == currentcut:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   332
                bottomranges.append(currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   333
                currentcut = None
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   334
            elif currentsize < currentcut:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   335
                bottomranges.append(currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   336
                currentcut -= currentsize
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   337
            else: # currentsize > currentcut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   338
                newskip = currentrange[1] + (currentsize - currentcut)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   339
                currentsub = stablerange._slicesrangeat(repo, currentrange, newskip)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   340
                bottomranges.append(currentsub.pop())
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   341
                sub.extend(currentsub)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   342
                currentcut = None
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   343
            currentrange = sub.pop()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   344
            currentsize = stablerange.rangelength(repo, currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   345
        bottomranges.reverse()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   346
        canonicals.extend(bottomranges)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   347
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   348
    # 3. take recursive subrange until we get to a power of two size?
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   349
    current = (headrev, cut)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   350
    while not poweroftwo(stablerange.rangelength(repo, current)):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   351
        sub = stablerange.subranges(repo, current)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   352
        canonicals.extend(sub[:-1])
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   353
        current = sub[-1]
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   354
    canonicals.append(current)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   355
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   356
    return canonicals
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   357
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   358
def standardcut(size):
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   359
    assert 0 < size
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   360
    # 0. find the first power of 2 higher than this range depth
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   361
    cut = 1
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   362
    while cut <= size:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   363
        cut *= 2
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   364
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   365
    allcuts = []
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   366
    # 1. find all standard expected cut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   367
    while 1 < cut and size:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   368
        cut //= 2
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   369
        if cut <= size:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   370
            allcuts.append(cut)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   371
            size -= cut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   372
    return allcuts
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   373
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   374
def poweroftwo(num):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   375
    return num and not num & (num - 1)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   376
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   377
def outgoingfromnodes(repo, nodes):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   378
    return discovery.outgoing(repo,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   379
                              missingroots=nodes,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   380
                              missingheads=nodes)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   381
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   382
# changegroup part construction
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   383
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   384
def _changegroupinfo(repo, nodes, source):
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   385
    if repo.ui.verbose or source == b'bundle':
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   386
        repo.ui.status(_(b"%d changesets found\n") % len(nodes))
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   387
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   388
def _makenewstream(newpart, repo, outgoing, version, source,
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   389
                   bundlecaps, filematcher, cgversions):
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   390
    old = changegroup._changegroupinfo
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   391
    try:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   392
        changegroup._changegroupinfo = _changegroupinfo
4145
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   393
        if filematcher is not None:
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   394
            cgstream = changegroup.makestream(repo, outgoing, version, source,
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   395
                                              bundlecaps=bundlecaps,
5034
1015a1dbaf7c pullbundle: kwarg to changegroup.makestream() is called matcher
Anton Shestakov <av6@dwimlabs.net>
parents: 4929
diff changeset
   396
                                              matcher=filematcher)
4145
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   397
        else:
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   398
            cgstream = changegroup.makestream(repo, outgoing, version, source,
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   399
                                              bundlecaps=bundlecaps)
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   400
    finally:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   401
        changegroup._changegroupinfo = old
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   402
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   403
    nbchanges = len(outgoing.missing)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   404
    pversion = None
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   405
    if cgversions:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   406
        pversion = version
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   407
    return (cgstream, nbchanges, pversion)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   408
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   409
def _makepartfromstream(newpart, repo, cgstream, nbchanges, version):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   410
    # same as upstream code
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   411
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   412
    part = newpart(b'changegroup', data=cgstream)
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   413
    if version:
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   414
        part.addparam(b'version', version)
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   415
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   416
    part.addparam(b'nbchanges', b'%d' % nbchanges,
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   417
                  mandatory=False)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   418
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   419
    if b'treemanifest' in repo.requirements:
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   420
        part.addparam(b'treemanifest', b'1')
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   421
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   422
# cache management
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   423
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   424
def cachedir(repo):
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   425
    cachedir = repo.ui.config(b'pullbundle', b'cache-directory')
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   426
    if cachedir is not None:
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   427
        return cachedir
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   428
    return repo.cachevfs.join(b'pullbundles')
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   429
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   430
def getcache(repo, bundlename):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   431
    cdir = cachedir(repo)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   432
    bundlepath = os.path.join(cdir, bundlename)
4139
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   433
    if not os.path.exists(bundlepath):
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   434
        return None
4139
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   435
    # delay file opening as much as possible this introduce a small race
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   436
    # condition if someone remove the file before we actually use it. However
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   437
    # opening too many file will not work.
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   438
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   439
    def data():
4807
ea8da5aa23c6 python3: add raw prefix to open()-like functions
Raphaël Gomès <rgomes@octobus.net>
parents: 4756
diff changeset
   440
        with open(bundlepath, r'rb') as fd:
4139
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   441
            for chunk in util.filechunkiter(fd):
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   442
                yield chunk
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   443
    return data()
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   444
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   445
def cachewriter(repo, bundlename, stream):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   446
    cdir = cachedir(repo)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   447
    bundlepath = os.path.join(cdir, bundlename)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   448
    try:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   449
        os.makedirs(cdir)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   450
    except OSError as exc:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   451
        if exc.errno == errno.EEXIST:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   452
            pass
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   453
    with util.atomictempfile(bundlepath) as cachefile:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   454
        for chunk in stream:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   455
            cachefile.write(chunk)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   456
            yield chunk
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   457
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   458
BUNDLEMASK = b"%s-%s-%010iskip-%010isize.hg"
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   459
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   460
def makeonecgpart(newpart, repo, rangeid, outgoing, version, source,
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   461
                  bundlecaps, filematcher, cgversions):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   462
    bundlename = cachedata = None
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   463
    if rangeid is not None:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   464
        nbchanges = repo.stablerange.rangelength(repo, rangeid)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   465
        headnode = nodemod.hex(repo.changelog.node(rangeid[0]))
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   466
        # XXX do we need to use cgversion in there?
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   467
        bundlename = BUNDLEMASK % (version, headnode, rangeid[1], nbchanges)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   468
        cachedata = getcache(repo, bundlename)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   469
    if cachedata is None:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   470
        partdata = _makenewstream(newpart, repo, outgoing, version, source,
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   471
                                  bundlecaps, filematcher, cgversions)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   472
        if bundlename is not None:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   473
            cgstream = cachewriter(repo, bundlename, partdata[0])
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   474
            partdata = (cgstream,) + partdata[1:]
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   475
    else:
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   476
        if repo.ui.verbose or source == b'bundle':
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   477
            repo.ui.status(_(b"%d changesets found in caches\n") % nbchanges)
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   478
        pversion = None
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   479
        if cgversions:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   480
            pversion = version
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   481
        partdata = (cachedata, nbchanges, pversion)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   482
    return _makepartfromstream(newpart, repo, *partdata)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   483
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   484
@command(b'debugpullbundlecacheoverlap',
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   485
         [(b'', b'count', 100, _(b'of "client" pulling')),
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   486
          (b'', b'min-cache', 1, _(b'minimum size of cached bundle')),
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   487
          ],
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   488
         _(b'hg debugpullbundlecacheoverlap [--client 100] REVSET'))
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   489
def debugpullbundlecacheoverlap(ui, repo, *revs, **opts):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   490
    '''Display statistic on bundle cache hit
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   491
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   492
    This command "simulate pulls from multiple clients. Each using a random
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   493
    subset of revisions defined by REVSET. And display statistic about the
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   494
    overlap in bundle necessary to serve them.
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   495
    '''
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   496
    actionrevs = scmutil.revrange(repo, revs)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   497
    if not revs:
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   498
        raise error.Abort(b'No revision selected')
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   499
    count = opts['count']
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   500
    min_cache = opts['min_cache']
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   501
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   502
    bundlehits = collections.defaultdict(lambda: 0)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   503
    pullstats = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   504
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   505
    rlen = lambda rangeid: repo.stablerange.rangelength(repo, rangeid)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   506
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   507
    repo.ui.write(b"gathering %d sample pulls within %d revisions\n"
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   508
                  % (count, len(actionrevs)))
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   509
    if 1 < min_cache:
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   510
        repo.ui.write(b"  not caching ranges smaller than %d changesets\n" % min_cache)
4737
46c990705a96 py3: replace xrange() by range()
Martin von Zweigbergk <martinvonz@google.com>
parents: 4208
diff changeset
   511
    for i in range(count):
5044
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   512
        progress(repo.ui, b'gathering data', i, total=count)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   513
        outgoing = takeonesample(repo, actionrevs)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   514
        ranges = sliceoutgoing(repo, outgoing)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   515
        hitranges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   516
        hitchanges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   517
        totalchanges = 0
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   518
        largeranges = []
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   519
        for rangeid, __ in ranges:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   520
            length = rlen(rangeid)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   521
            totalchanges += length
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   522
            if bundlehits[rangeid]:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   523
                hitranges += 1
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   524
                hitchanges += rlen(rangeid)
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   525
            if min_cache <= length:
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   526
                bundlehits[rangeid] += 1
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   527
                largeranges.append(rangeid)
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   528
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   529
        stats = (len(outgoing.missing),
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   530
                 totalchanges,
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   531
                 hitchanges,
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   532
                 len(largeranges),
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   533
                 hitranges,
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   534
                 )
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   535
        pullstats.append(stats)
5044
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   536
    progress(repo.ui, b'gathering data', None)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   537
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   538
    sizes = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   539
    changesmissing = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   540
    totalchanges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   541
    totalcached = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   542
    changesratio = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   543
    rangesratio = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   544
    bundlecount = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   545
    for entry in pullstats:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   546
        sizes.append(entry[0])
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   547
        changesmissing.append(entry[1] - entry[2])
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   548
        changesratio.append(entry[2] / float(entry[1]))
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   549
        if entry[3]:
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   550
            rangesratio.append(entry[4] / float(entry[3]))
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   551
        else:
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   552
            rangesratio.append(1)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   553
        bundlecount.append(entry[3])
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   554
        totalchanges += entry[1]
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   555
        totalcached += entry[2]
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   556
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   557
    cachedsizes = []
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   558
    cachedhits = []
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   559
    for rangeid, hits in bundlehits.items():
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   560
        if hits <= 0:
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   561
            continue
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   562
        length = rlen(rangeid)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   563
        cachedsizes.append(length)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   564
        cachedhits.append(hits)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   565
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   566
    sizesdist = distribution(sizes)
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   567
    repo.ui.write(fmtdist(b'pull size', sizesdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   568
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   569
    changesmissingdist = distribution(changesmissing)
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   570
    repo.ui.write(fmtdist(b'non-cached changesets', changesmissingdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   571
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   572
    changesratiodist = distribution(changesratio)
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   573
    repo.ui.write(fmtdist(b'ratio of cached changesets', changesratiodist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   574
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   575
    bundlecountdist = distribution(bundlecount)
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   576
    repo.ui.write(fmtdist(b'bundle count', bundlecountdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   577
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   578
    rangesratiodist = distribution(rangesratio)
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   579
    repo.ui.write(fmtdist(b'ratio of cached bundles', rangesratiodist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   580
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   581
    repo.ui.write(b'changesets served:\n')
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   582
    repo.ui.write(b'  total:      %7d\n' % totalchanges)
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   583
    repo.ui.write(b'  from cache: %7d (%2d%%)\n'
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   584
                  % (totalcached, (totalcached * 100 // totalchanges)))
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   585
    repo.ui.write(b'  bundle:     %7d\n' % sum(bundlecount))
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   586
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   587
    cachedsizesdist = distribution(cachedsizes)
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   588
    repo.ui.write(fmtdist(b'size of cached bundles', cachedsizesdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   589
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   590
    cachedhitsdist = distribution(cachedhits)
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   591
    repo.ui.write(fmtdist(b'hit on cached bundles', cachedhitsdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   592
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   593
def takeonesample(repo, revs):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   594
    node = repo.changelog.node
5047
a8ac174010a5 pullbundle: random.sample() chokes on a spanset, turn it into a list instead
Anton Shestakov <av6@dwimlabs.net>
parents: 5046
diff changeset
   595
    revs = list(revs)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   596
    pulled = random.sample(revs, max(4, len(revs) // 1000))
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   597
    pulled = repo.revs(b'%ld::%ld', pulled, pulled)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   598
    nodes = [node(r) for r in pulled]
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   599
    return outgoingfromnodes(repo, nodes)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   600
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   601
def distribution(data):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   602
    data.sort()
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   603
    length = len(data)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   604
    return {
4814
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   605
        b'min': data[0],
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   606
        b'10%': data[length // 10],
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   607
        b'25%': data[length // 4],
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   608
        b'50%': data[length // 2],
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   609
        b'75%': data[(length // 4) * 3],
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   610
        b'90%': data[(length // 10) * 9],
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   611
        b'95%': data[(length // 20) * 19],
48b30ff742cb python3: use format-source to run byteify-strings in .py files
Raphaël Gomès <rgomes@octobus.net>
parents: 4807
diff changeset
   612
        b'max': data[-1],
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   613
    }
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   614
5046
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   615
STATSFORMAT = b"""%(name)s:
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   616
  min: %(min)r
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   617
  10%%: %(10%)r
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   618
  25%%: %(25%)r
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   619
  50%%: %(50%)r
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   620
  75%%: %(75%)r
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   621
  90%%: %(90%)r
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   622
  95%%: %(95%)r
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   623
  max: %(max)r
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   624
"""
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   625
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   626
def fmtdist(name, data):
5046
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   627
    data[b'name'] = name
ebd0f5c6098f pullbundle: use % instead of .format() because % exists for bytes
Anton Shestakov <av6@dwimlabs.net>
parents: 5044
diff changeset
   628
    return STATSFORMAT % data
4929
bb2b4f6c99dc compat: compatibility for cl.nodemap.get vs cl.index.get_rev
Anton Shestakov <av6@dwimlabs.net>
parents: 4814
diff changeset
   629
5044
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   630
# hg <= 4.6 (bec1212eceaa)
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   631
if util.safehasattr(uimod.ui, 'makeprogress'):
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   632
    def progress(ui, topic, pos, item=b"", unit=b"", total=None):
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   633
        progress = ui.makeprogress(topic, unit, total)
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   634
        if pos is not None:
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   635
            progress.update(pos, item=item)
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   636
        else:
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   637
            progress.complete()
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   638
else:
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   639
    def progress(ui, topic, pos, item=b"", unit=b"", total=None):
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   640
        ui.progress(topic, pos, item, unit, total)
b9179a034005 pullbundle: compatibility for progress in hg <= 4.6
Anton Shestakov <av6@dwimlabs.net>
parents: 5034
diff changeset
   641
4929
bb2b4f6c99dc compat: compatibility for cl.nodemap.get vs cl.index.get_rev
Anton Shestakov <av6@dwimlabs.net>
parents: 4814
diff changeset
   642
# nodemap.get and index.[has_node|rev|get_rev]
bb2b4f6c99dc compat: compatibility for cl.nodemap.get vs cl.index.get_rev
Anton Shestakov <av6@dwimlabs.net>
parents: 4814
diff changeset
   643
# hg <= 5.3 (02802fa87b74)
bb2b4f6c99dc compat: compatibility for cl.nodemap.get vs cl.index.get_rev
Anton Shestakov <av6@dwimlabs.net>
parents: 4814
diff changeset
   644
def getgetrev(cl):
bb2b4f6c99dc compat: compatibility for cl.nodemap.get vs cl.index.get_rev
Anton Shestakov <av6@dwimlabs.net>
parents: 4814
diff changeset
   645
    """Returns index.get_rev or nodemap.get (for pre-5.3 Mercurial)."""
bb2b4f6c99dc compat: compatibility for cl.nodemap.get vs cl.index.get_rev
Anton Shestakov <av6@dwimlabs.net>
parents: 4814
diff changeset
   646
    if util.safehasattr(cl.index, 'get_rev'):
bb2b4f6c99dc compat: compatibility for cl.nodemap.get vs cl.index.get_rev
Anton Shestakov <av6@dwimlabs.net>
parents: 4814
diff changeset
   647
        return cl.index.get_rev
bb2b4f6c99dc compat: compatibility for cl.nodemap.get vs cl.index.get_rev
Anton Shestakov <av6@dwimlabs.net>
parents: 4814
diff changeset
   648
    return cl.nodemap.get