hgext3rd/pullbundle.py
author Pierre-Yves David <pierre-yves.david@octobus.net>
Wed, 10 Apr 2019 20:24:55 +0200
changeset 4496 02d3177085e1
parent 4208 7d4c80c3f386
child 4737 46c990705a96
permissions -rw-r--r--
test: rename test about content divergent meta data The test now use a name in line with the other content divergence test cases.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     1
# Extension to provide automatic caching of bundle server for pull
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     2
#
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     3
# Copyright 2018 Pierre-Yves David <pierre-yves.david@ens-lyon.org>
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     4
#
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     5
# This software may be used and distributed according to the terms of the
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     6
# GNU General Public License version 2 or any later version.
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
     7
"""pullbundle: automatic server side bundle caching
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     8
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
     9
General principle
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    10
=================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    11
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    12
This extension provides a means for server to use pre-computed bundle for
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    13
serving arbitrary pulls. If missing, the necessary pre-computed bundle will be
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    14
generated on demand.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    15
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    16
To maximize usage of existing cached bundle, each pull will be served through
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    17
multiple bundles. The bundle will be created using "standard range" from the
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    18
"stablerange" principle. The "stablerange" concept if already used for
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    19
obsmarkers discovery in the evolve extensions.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    20
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    21
Using pull Bundle
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    22
=================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    23
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    24
All configuration is only required server side.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    25
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    26
The "stablerange" code currently still live in the evolve extensions, so for
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    27
now enabling that extensions is required:
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    28
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    29
You need at minimum the following configuration:
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    30
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    31
    [extensions]
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    32
    evolve=yes
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    33
    pullbundle=yes
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    34
    [experimental]
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    35
    obshashrange.warm-cache = yes
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    36
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    37
If you do not want to use evolution server side, you should disable obsmarkers exchange:
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    38
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    39
    [experimental]
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    40
    evolution.exchange=no
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    41
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    42
Extra Configuration
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    43
===================
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    44
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    45
  [pullbundle]
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    46
  # By default bundles are stored `.hg/cache/pullbundles/.
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    47
  # This can be changed with the following config:
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    48
  cache-directory=/absolute/path
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    49
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    50
Implementation status
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    51
=====================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    52
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    53
Both for stablerange and pullbundle use "simple" initial implementations.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    54
Theses implemenations focus on testing the algorithms and proving the features
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    55
works. Yet they are already useful and used in production.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    56
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    57
Performances are expected to greatly improved in the final implementation,
4141
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    58
especially if some of it end up being compiled code.
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    59
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    60
This first implementation lacks the ability to server the cached bundle from a
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    61
CDN. We'll want this limitation to be lifted quickly.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    62
4141
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    63
The way mercurial core report progress is designed for the receival of a single
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    64
changegroup. So currently using pullbundle means flooding the user with output.
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    65
This will have to be fixed.
1421ff5c5c96 pullbundle: point out the output flooding issue from core
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4140
diff changeset
    66
4132
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    67
Why is does this live in the same repository as evolve
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    68
======================================================
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    69
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    70
There is no fundamental reasons for live in the same repository. However, the
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    71
stablerange data-structure lives in evolve, so it was simpler to put this new
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    72
extensions next to it. As soon as stable range have been upstreamed, we won't
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    73
need the dependency to the evolve extension anymore.
afc933d32085 pullbundle: add extension documentation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4130
diff changeset
    74
"""
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    75
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    76
import collections
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    77
import errno
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    78
import random
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    79
import os
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    80
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    81
from mercurial import (
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    82
    changegroup,
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    83
    discovery,
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    84
    error,
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    85
    exchange,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    86
    narrowspec,
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    87
    node as nodemod,
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
    88
    registrar,
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
    89
    scmutil,
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
    90
    util,
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    91
)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    92
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    93
from mercurial.i18n import _
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    94
4201
46cef25fbd96 pullbundle: drop erroneous flagging of the command as basic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4176
diff changeset
    95
__version__ = '0.1.1'
4175
03c3206f44f3 pullbundle: document working version and update changelog to mention it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4149
diff changeset
    96
testedwith = '4.4 4.5 4.6 4.7.1'
03c3206f44f3 pullbundle: document working version and update changelog to mention it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4149
diff changeset
    97
minimumhgversion = '4.4'
4135
47f1d7b4305d pullbundle: add basic extension metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4134
diff changeset
    98
buglink = 'https://bz.mercurial-scm.org/'
47f1d7b4305d pullbundle: add basic extension metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4134
diff changeset
    99
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   100
cmdtable = {}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   101
command = registrar.command(cmdtable)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   102
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   103
configtable = {}
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   104
configitem = registrar.configitem(configtable)
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   105
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   106
configitem('pullbundle', 'cache-directory',
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   107
           default=None,
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   108
)
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   109
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   110
# generic wrapping
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   111
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   112
def uisetup(ui):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   113
    exchange.getbundle2partsmapping['changegroup'] = _getbundlechangegrouppart
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   114
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   115
def _getbundlechangegrouppart(bundler, repo, source, bundlecaps=None,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   116
                              b2caps=None, heads=None, common=None, **kwargs):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   117
    """add a changegroup part to the requested bundle"""
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   118
    if not kwargs.get(r'cg', True):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   119
        return
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   120
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   121
    version = '01'
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   122
    cgversions = b2caps.get('changegroup')
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   123
    if cgversions:  # 3.1 and 3.2 ship with an empty value
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   124
        cgversions = [v for v in cgversions
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   125
                      if v in changegroup.supportedoutgoingversions(repo)]
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   126
        if not cgversions:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   127
            raise ValueError(_('no common changegroup version'))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   128
        version = max(cgversions)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   129
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   130
    outgoing = exchange._computeoutgoing(repo, heads, common)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   131
    if not outgoing.missing:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   132
        return
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   133
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   134
    if kwargs.get(r'narrow', False):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   135
        include = sorted(filter(bool, kwargs.get(r'includepats', [])))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   136
        exclude = sorted(filter(bool, kwargs.get(r'excludepats', [])))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   137
        filematcher = narrowspec.match(repo.root, include=include,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   138
                                       exclude=exclude)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   139
    else:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   140
        filematcher = None
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   141
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   142
    # START OF ALTERED PART
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   143
    makeallcgpart(bundler.newpart, repo, outgoing, version, source, bundlecaps,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   144
                  filematcher, cgversions)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   145
    # END OF ALTERED PART
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   146
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   147
    if kwargs.get(r'narrow', False) and (include or exclude):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   148
        narrowspecpart = bundler.newpart('narrow:spec')
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   149
        if include:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   150
            narrowspecpart.addparam(
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   151
                'include', '\n'.join(include), mandatory=True)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   152
        if exclude:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   153
            narrowspecpart.addparam(
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   154
                'exclude', '\n'.join(exclude), mandatory=True)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   155
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   156
def makeallcgpart(newpart, repo, outgoing, version, source,
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   157
                  bundlecaps, filematcher, cgversions):
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   158
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   159
    pullbundle = not filematcher
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   160
    if pullbundle and not util.safehasattr(repo, 'stablerange'):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   161
        repo.ui.warn('pullbundle: required extension "evolve" are missing, skipping pullbundle\n')
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   162
        pullbundle = False
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   163
    if filematcher:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   164
        makeonecgpart(newpart, repo, None, outgoing, version, source, bundlecaps,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   165
                      filematcher, cgversions)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   166
    else:
4133
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   167
        start = util.timer()
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   168
        slices = sliceoutgoing(repo, outgoing)
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   169
        end = util.timer()
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   170
        msg = _('pullbundle-cache: "missing" set sliced into %d subranges '
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   171
                'in %s seconds\n')
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   172
        repo.ui.write(msg % (len(slices), end - start))
1293625d274d pullbundle: add an output about the time spent slicing the set
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4132
diff changeset
   173
        for sliceid, sliceout in slices:
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   174
            makeonecgpart(newpart, repo, sliceid, sliceout, version, source, bundlecaps,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   175
                          filematcher, cgversions)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   176
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   177
# stable range slicing
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   178
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   179
DEBUG = False
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   180
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   181
def sliceoutgoing(repo, outgoing):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   182
    cl = repo.changelog
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   183
    rev = cl.nodemap.get
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   184
    node = cl.node
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   185
    revsort = repo.stablesort
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   186
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   187
    missingrevs = set(rev(n) for n in outgoing.missing)
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   188
    if DEBUG:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   189
        ms = missingrevs.copy()
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   190
        ss = []
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   191
    allslices = []
4147
88e922eca4e2 pullbundle: deal with head in a predicable order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4145
diff changeset
   192
    missingheads = [rev(n) for n in sorted(outgoing.missingheads, reverse=True)]
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   193
    for head in missingheads:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   194
        localslices = []
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   195
        localmissing = set(repo.revs('%ld and ::%d', missingrevs, head))
4149
4abec3a1208f pullbundle: fix style warning
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4148
diff changeset
   196
        thisrunmissing = localmissing.copy()
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   197
        while localmissing:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   198
            slicerevs = []
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   199
            for r in revsort.walkfrom(repo, head):
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   200
                if r not in thisrunmissing:
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   201
                    break
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   202
                slicerevs.append(r)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   203
            slicenodes = [node(r) for r in slicerevs]
4140
9b71aa222f8e pullbundle: issue local branches in the right order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4139
diff changeset
   204
            localslices.append(canonicalslices(repo, slicenodes))
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   205
            if DEBUG:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   206
                ss.append(slicerevs)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   207
            missingrevs.difference_update(slicerevs)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   208
            localmissing.difference_update(slicerevs)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   209
            if localmissing:
4148
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   210
                heads = list(repo.revs('heads(%ld)', localmissing))
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   211
                heads.sort(key=node)
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   212
                head = heads.pop()
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   213
                if heads:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   214
                    thisrunmissing = repo.revs('%ld and only(%d, %ld)',
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   215
                                               localmissing,
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   216
                                               head,
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   217
                                               heads)
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   218
                else:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   219
                    thisrunmissing = localmissing.copy()
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   220
        if DEBUG:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   221
            for s in reversed(ss):
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   222
                ms -= set(s)
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   223
                missingbase = repo.revs('parents(%ld) and %ld', s, ms)
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   224
                if missingbase:
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   225
                    repo.ui.write_err('!!! rev bundled while parents missing\n')
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   226
                    repo.ui.write_err('    parent: %s\n' % list(missingbase))
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   227
                    pb = repo.revs('%ld and children(%ld)', s, missingbase)
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   228
                    repo.ui.write_err('    children: %s\n' % list(pb))
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   229
                    h = repo.revs('heads(%ld)', s)
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   230
                    repo.ui.write_err('    heads: %s\n' % list(h))
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   231
                    raise error.ProgrammingError('issuing a range before its parents')
5ef93dbace5e pullbundle: improves management of multiple branch while dealing with a head
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4147
diff changeset
   232
4140
9b71aa222f8e pullbundle: issue local branches in the right order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4139
diff changeset
   233
        for s in reversed(localslices):
9b71aa222f8e pullbundle: issue local branches in the right order
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4139
diff changeset
   234
            allslices.extend(s)
4136
be3a94d3105f pullbundle: attempt to save stablerange cache after each computation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4135
diff changeset
   235
    # unknown subrange might had to be computed
be3a94d3105f pullbundle: attempt to save stablerange cache after each computation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4135
diff changeset
   236
    repo.stablerange.save(repo)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   237
    return [(rangeid, outgoingfromnodes(repo, nodes))
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   238
            for rangeid, nodes in allslices]
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   239
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   240
def canonicalslices(repo, nodes):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   241
    depth = repo.depthcache.get
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   242
    stablerange = repo.stablerange
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   243
    rangelength = lambda x: stablerange.rangelength(repo, x)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   244
    headrev = repo.changelog.rev(nodes[0])
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   245
    nbrevs = len(nodes)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   246
    headdepth = depth(headrev)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   247
    skipped = headdepth - nbrevs
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   248
    rangeid = (headrev, skipped)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   249
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   250
    subranges = canonicalsubranges(repo, stablerange, rangeid)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   251
    idx = 0
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   252
    slices = []
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   253
    nodes.reverse()
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   254
    for rangeid in subranges:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   255
        size = rangelength(rangeid)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   256
        slices.append((rangeid, nodes[idx:idx + size]))
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   257
        idx += size
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   258
    ### slow code block to validate ranges content
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   259
    # rev = repo.changelog.nodemap.get
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   260
    # for ri, ns in slices:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   261
    #     a = set(rev(n) for n in ns)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   262
    #     b = set(repo.stablerange.revsfromrange(repo, ri))
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   263
    #     l = repo.stablerange.rangelength(repo, ri)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   264
    #     repo.ui.write('range-length: %d-%d %s %s\n' % (ri[0], ri[1], l, len(a)))
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   265
    #     if a != b:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   266
    #         d =  (ri[0], ri[1], b - a, a - b)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   267
    #         repo.ui.write("mismatching content: %d-%d -%s +%s\n" % d)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   268
    return slices
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   269
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   270
def canonicalsubranges(repo, stablerange, rangeid):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   271
    """slice a size of nodes into most reusable subranges
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   272
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   273
    We try to slice a range into a set of "largest" and "canonical" stable
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   274
    range.
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   275
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   276
    It might make sense to move this function as a 'stablerange' method.
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   277
    """
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   278
    headrev, skip = rangeid
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   279
    rangedepth = stablerange.depthrev(repo, rangeid[0])
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   280
    canonicals = []
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   281
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   282
    # 0. find the first power of 2 higher than this range depth
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   283
    cursor = 1
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   284
    while cursor <= rangedepth:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   285
        cursor *= 2
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   286
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   287
    # 1. find first cupt
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   288
    precut = cut = 0
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   289
    while True:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   290
        if skip <= cut:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   291
            break
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   292
        if cut + cursor < rangedepth:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   293
            precut = cut
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   294
            cut += cursor
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   295
        if cursor == 1:
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   296
            break
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   297
        cursor //= 2
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   298
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   299
    # 2. optimise, bottom part
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   300
    if skip != cut:
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   301
        currentsize = tailsize = cut - skip
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   302
        assert 0 < tailsize, tailsize
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   303
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   304
        # we need to take several "standard cut" in the bottom part
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   305
        #
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   306
        # This is similar to what we will do for the top part, we reusing the
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   307
        # existing structure is a bit more complex.
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   308
        allcuts = list(reversed(standardcut(tailsize)))
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   309
        prerange = (headrev, precut)
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   310
        ### slow code block to check we operate on the right data
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   311
        # rev = repo.changelog.nodemap.get
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   312
        # allrevs = [rev(n) for n in nodes]
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   313
        # allrevs.reverse()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   314
        # prerevs = repo.stablerange.revsfromrange(repo, prerange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   315
        # assert allrevs == prerevs[(len(prerevs) - len(allrevs)):]
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   316
        # end of check
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   317
        sub = list(stablerange.subranges(repo, prerange)[:-1])
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   318
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   319
        bottomranges = []
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   320
        # XXX we might be able to reuse core stable-range logic instead of
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   321
        # redoing this manually
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   322
        currentrange = sub.pop()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   323
        currentsize = stablerange.rangelength(repo, currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   324
        currentcut = None
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   325
        while allcuts or currentcut is not None:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   326
            # get the next cut if needed
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   327
            if currentcut is None:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   328
                currentcut = allcuts.pop()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   329
            # deal attemp a cut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   330
            if currentsize == currentcut:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   331
                bottomranges.append(currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   332
                currentcut = None
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   333
            elif currentsize < currentcut:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   334
                bottomranges.append(currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   335
                currentcut -= currentsize
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   336
            else: # currentsize > currentcut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   337
                newskip = currentrange[1] + (currentsize - currentcut)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   338
                currentsub = stablerange._slicesrangeat(repo, currentrange, newskip)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   339
                bottomranges.append(currentsub.pop())
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   340
                sub.extend(currentsub)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   341
                currentcut = None
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   342
            currentrange = sub.pop()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   343
            currentsize = stablerange.rangelength(repo, currentrange)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   344
        bottomranges.reverse()
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   345
        canonicals.extend(bottomranges)
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   346
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   347
    # 3. take recursive subrange until we get to a power of two size?
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   348
    current = (headrev, cut)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   349
    while not poweroftwo(stablerange.rangelength(repo, current)):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   350
        sub = stablerange.subranges(repo, current)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   351
        canonicals.extend(sub[:-1])
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   352
        current = sub[-1]
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   353
    canonicals.append(current)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   354
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   355
    return canonicals
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   356
4138
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   357
def standardcut(size):
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   358
    assert 0 < size
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   359
    # 0. find the first power of 2 higher than this range depth
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   360
    cut = 1
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   361
    while cut <= size:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   362
        cut *= 2
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   363
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   364
    allcuts = []
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   365
    # 1. find all standard expected cut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   366
    while 1 < cut and size:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   367
        cut //= 2
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   368
        if cut <= size:
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   369
            allcuts.append(cut)
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   370
            size -= cut
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   371
    return allcuts
cfdc6f55599b pullbundle: improve slicing of the lower part of range
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4136
diff changeset
   372
4129
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   373
def poweroftwo(num):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   374
    return num and not num & (num - 1)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   375
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   376
def outgoingfromnodes(repo, nodes):
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   377
    return discovery.outgoing(repo,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   378
                              missingroots=nodes,
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   379
                              missingheads=nodes)
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   380
bc4e62a1cb82 pullbundle: slice pull into multiple ranges
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4128
diff changeset
   381
# changegroup part construction
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   382
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   383
def _changegroupinfo(repo, nodes, source):
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   384
    if repo.ui.verbose or source == 'bundle':
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   385
        repo.ui.status(_("%d changesets found\n") % len(nodes))
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   386
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   387
def _makenewstream(newpart, repo, outgoing, version, source,
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   388
                   bundlecaps, filematcher, cgversions):
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   389
    old = changegroup._changegroupinfo
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   390
    try:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   391
        changegroup._changegroupinfo = _changegroupinfo
4145
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   392
        if filematcher is not None:
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   393
            cgstream = changegroup.makestream(repo, outgoing, version, source,
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   394
                                              bundlecaps=bundlecaps,
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   395
                                              filematcher=filematcher)
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   396
        else:
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   397
            cgstream = changegroup.makestream(repo, outgoing, version, source,
08b3c370e8b3 pullbundle: compat with Mercurial 4.7
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4144
diff changeset
   398
                                              bundlecaps=bundlecaps)
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   399
    finally:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   400
        changegroup._changegroupinfo = old
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   401
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   402
    nbchanges = len(outgoing.missing)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   403
    pversion = None
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   404
    if cgversions:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   405
        pversion = version
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   406
    return (cgstream, nbchanges, pversion)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   407
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   408
def _makepartfromstream(newpart, repo, cgstream, nbchanges, version):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   409
    # same as upstream code
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   410
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   411
    part = newpart('changegroup', data=cgstream)
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   412
    if version:
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   413
        part.addparam('version', version)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   414
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   415
    part.addparam('nbchanges', '%d' % nbchanges,
4128
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   416
                  mandatory=False)
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   417
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   418
    if 'treemanifest' in repo.requirements:
4e5ec9ae682e pullbundle: initial commit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
   419
        part.addparam('treemanifest', '1')
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   420
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   421
# cache management
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   422
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   423
def cachedir(repo):
4134
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   424
    cachedir = repo.ui.config('pullbundle', 'cache-directory')
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   425
    if cachedir is not None:
ab77f37fedf3 pullbundle: add a config option for the cache directory
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4133
diff changeset
   426
        return cachedir
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   427
    return repo.cachevfs.join('pullbundles')
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   428
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   429
def getcache(repo, bundlename):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   430
    cdir = cachedir(repo)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   431
    bundlepath = os.path.join(cdir, bundlename)
4139
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   432
    if not os.path.exists(bundlepath):
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   433
        return None
4139
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   434
    # delay file opening as much as possible this introduce a small race
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   435
    # condition if someone remove the file before we actually use it. However
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   436
    # opening too many file will not work.
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   437
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   438
    def data():
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   439
        with open(bundlepath, 'rb') as fd:
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   440
            for chunk in util.filechunkiter(fd):
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   441
                yield chunk
2bd652bece97 pullbundle: delay cache file opening
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4138
diff changeset
   442
    return data()
4130
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   443
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   444
def cachewriter(repo, bundlename, stream):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   445
    cdir = cachedir(repo)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   446
    bundlepath = os.path.join(cdir, bundlename)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   447
    try:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   448
        os.makedirs(cdir)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   449
    except OSError as exc:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   450
        if exc.errno == errno.EEXIST:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   451
            pass
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   452
    with util.atomictempfile(bundlepath) as cachefile:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   453
        for chunk in stream:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   454
            cachefile.write(chunk)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   455
            yield chunk
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   456
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   457
BUNDLEMASK = "%s-%s-%010iskip-%010isize.hg"
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   458
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   459
def makeonecgpart(newpart, repo, rangeid, outgoing, version, source,
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   460
                  bundlecaps, filematcher, cgversions):
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   461
    bundlename = cachedata = None
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   462
    if rangeid is not None:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   463
        nbchanges = repo.stablerange.rangelength(repo, rangeid)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   464
        headnode = nodemod.hex(repo.changelog.node(rangeid[0]))
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   465
        # XXX do we need to use cgversion in there?
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   466
        bundlename = BUNDLEMASK % (version, headnode, rangeid[1], nbchanges)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   467
        cachedata = getcache(repo, bundlename)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   468
    if cachedata is None:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   469
        partdata = _makenewstream(newpart, repo, outgoing, version, source,
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   470
                                  bundlecaps, filematcher, cgversions)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   471
        if bundlename is not None:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   472
            cgstream = cachewriter(repo, bundlename, partdata[0])
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   473
            partdata = (cgstream,) + partdata[1:]
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   474
    else:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   475
        if repo.ui.verbose or source == 'bundle':
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   476
            repo.ui.status(_("%d changesets found in caches\n") % nbchanges)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   477
        pversion = None
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   478
        if cgversions:
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   479
            pversion = version
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   480
        partdata = (cachedata, nbchanges, pversion)
a1f6b8211016 pullbundle: add caching logic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4129
diff changeset
   481
    return _makepartfromstream(newpart, repo, *partdata)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   482
4201
46cef25fbd96 pullbundle: drop erroneous flagging of the command as basic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4176
diff changeset
   483
@command('debugpullbundlecacheoverlap',
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   484
         [('', 'count', 100, _('of "client" pulling')),
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   485
          ('', 'min-cache', 1, _('minimum size of cached bundle')),
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   486
         ],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   487
         _('hg debugpullbundlecacheoverlap [--client 100] REVSET'))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   488
def debugpullbundlecacheoverlap(ui, repo, *revs, **opts):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   489
    '''Display statistic on bundle cache hit
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   490
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   491
    This command "simulate pulls from multiple clients. Each using a random
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   492
    subset of revisions defined by REVSET. And display statistic about the
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   493
    overlap in bundle necessary to serve them.
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   494
    '''
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   495
    actionrevs = scmutil.revrange(repo, revs)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   496
    if not revs:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   497
        raise error.Abort('No revision selected')
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   498
    count = opts['count']
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   499
    min_cache = opts['min_cache']
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   500
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   501
    bundlehits = collections.defaultdict(lambda: 0)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   502
    pullstats = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   503
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   504
    rlen = lambda rangeid: repo.stablerange.rangelength(repo, rangeid)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   505
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   506
    repo.ui.write("gathering %d sample pulls within %d revisions\n"
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   507
                  % (count, len(actionrevs)))
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   508
    if 1 < min_cache:
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   509
        repo.ui.write("  not caching ranges smaller than %d changesets\n" % min_cache)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   510
    for i in xrange(count):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   511
        repo.ui.progress('gathering data', i, total=count)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   512
        outgoing = takeonesample(repo, actionrevs)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   513
        ranges = sliceoutgoing(repo, outgoing)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   514
        hitranges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   515
        hitchanges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   516
        totalchanges = 0
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   517
        largeranges = []
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   518
        for rangeid, __ in ranges:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   519
            length = rlen(rangeid)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   520
            totalchanges += length
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   521
            if bundlehits[rangeid]:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   522
                hitranges += 1
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   523
                hitchanges += rlen(rangeid)
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   524
            if min_cache <= length:
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   525
                bundlehits[rangeid] += 1
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   526
                largeranges.append(rangeid)
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   527
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   528
        stats = (len(outgoing.missing),
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   529
                 totalchanges,
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   530
                 hitchanges,
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   531
                 len(largeranges),
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   532
                 hitranges,
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   533
                 )
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   534
        pullstats.append(stats)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   535
    repo.ui.progress('gathering data', None)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   536
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   537
    sizes = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   538
    changesmissing = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   539
    totalchanges = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   540
    totalcached = 0
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   541
    changesratio = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   542
    rangesratio = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   543
    bundlecount = []
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   544
    for entry in pullstats:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   545
        sizes.append(entry[0])
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   546
        changesmissing.append(entry[1] - entry[2])
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   547
        changesratio.append(entry[2] / float(entry[1]))
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   548
        if entry[3]:
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   549
            rangesratio.append(entry[4] / float(entry[3]))
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   550
        else:
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   551
            rangesratio.append(1)
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   552
        bundlecount.append(entry[3])
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   553
        totalchanges += entry[1]
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   554
        totalcached += entry[2]
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   555
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   556
    cachedsizes = []
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   557
    cachedhits = []
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   558
    for rangeid, hits in bundlehits.items():
4144
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   559
        if hits <= 0:
b5cd26712e4b pullbundle: add some information on the cached change
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4143
diff changeset
   560
            continue
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   561
        length = rlen(rangeid)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   562
        cachedsizes.append(length)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   563
        cachedhits.append(hits)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   564
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   565
    sizesdist = distribution(sizes)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   566
    repo.ui.write(fmtdist('pull size', sizesdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   567
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   568
    changesmissingdist = distribution(changesmissing)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   569
    repo.ui.write(fmtdist('non-cached changesets', changesmissingdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   570
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   571
    changesratiodist = distribution(changesratio)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   572
    repo.ui.write(fmtdist('ratio of cached changesets', changesratiodist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   573
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   574
    bundlecountdist = distribution(bundlecount)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   575
    repo.ui.write(fmtdist('bundle count', bundlecountdist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   576
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   577
    rangesratiodist = distribution(rangesratio)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   578
    repo.ui.write(fmtdist('ratio of cached bundles', rangesratiodist))
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   579
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   580
    repo.ui.write('changesets served:\n')
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   581
    repo.ui.write('  total:      %7d\n' % totalchanges)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   582
    repo.ui.write('  from cache: %7d (%2d%%)\n'
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   583
                  % (totalcached, (totalcached * 100 // totalchanges)))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   584
    repo.ui.write('  bundle:     %7d\n' % sum(bundlecount))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   585
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   586
    cachedsizesdist = distribution(cachedsizes)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   587
    repo.ui.write(fmtdist('size of cached bundles', cachedsizesdist))
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   588
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   589
    cachedhitsdist = distribution(cachedhits)
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   590
    repo.ui.write(fmtdist('hit on cached bundles', cachedhitsdist))
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   591
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   592
def takeonesample(repo, revs):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   593
    node = repo.changelog.node
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   594
    pulled = random.sample(revs, max(4, len(revs) // 1000))
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   595
    pulled = repo.revs('%ld::%ld', pulled, pulled)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   596
    nodes = [node(r) for r in pulled]
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   597
    return outgoingfromnodes(repo, nodes)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   598
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   599
def distribution(data):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   600
    data.sort()
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   601
    length = len(data)
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   602
    return {
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   603
        'min': data[0],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   604
        '10%': data[length // 10],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   605
        '25%': data[length // 4],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   606
        '50%': data[length // 2],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   607
        '75%': data[(length // 4) * 3],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   608
        '90%': data[(length // 10) * 9],
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   609
        '95%': data[(length // 20) * 19],
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   610
        'max': data[-1],
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   611
    }
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   612
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   613
STATSFORMAT = """{name}:
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   614
  min: {min}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   615
  10%: {10%}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   616
  25%: {25%}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   617
  50%: {50%}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   618
  75%: {75%}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   619
  90%: {90%}
4143
2ddd8785f8d7 pullbundle: add data about the cached bundle size and hit
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4142
diff changeset
   620
  95%: {95%}
4142
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   621
  max: {max}
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   622
"""
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   623
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   624
def fmtdist(name, data):
683ceec8d37e pullbundle: add a command to generate cache hit statistic
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 4141
diff changeset
   625
    return STATSFORMAT.format(name=name, **data)