pullbundle: add data about the cached bundle size and hit

pullbundle: add a command to generate cache hit statistic This new command should help checking that the stable range produce reusable bundle.

pullbundle: point out the output flooding issue from core Right now, we don't have a reliable progress report when using pullbundle.

pullbundle: issue local branches in the right order

pullbundle: delay cache file opening Otherwise we can end-up with a too many file open at the same time.

pullbundle: improve slicing of the lower part of range The previous method could get confuse by merge and overslice. The new method is better at using sticking on power of two boundaries.

stablerange: fix slicing of arbitrary ranges Pull bundle trigger slicing from range with arbitrary initial skip. We have to adjust the current slicing to take this into account.

pullbundle: attempt to save stablerange cache after each computation Since we do not serve the full repository, we use more stablerange starting from a random point. These are probably not loaded yet and should be cached. This is best effort caching, if we can't save them we won't.

pullbundle: add basic extension metadata

pullbundle: add a config option for the cache directory

pullbundle: add an output about the time spent slicing the set This will be useful to monitor overall performance.

pullbundle: add extension documentation

pullbundle: add an extra test showing how cache are reusable We issue a single final pull, most of its item are reusable.

pullbundle: add caching logic We now only generate a bundle once (and store it to disk). If we need it again, we use it directly from disk.

pullbundle: slice pull into multiple ranges We subdivide the set of missing revisions into multiple "range" using the "stable range" data structure. This slicing aims at maximizing the capability of the resulting ranges.

pullbundle: initial commit Right now, we do not do any slicing, but we install the wrapping and the tests.

test: update output to make streamv2 capabilities CORE-TEST-OUTPUT-UPDATE: 4bd6e444c76f

branching: merge with stable

topic: respect preexisting 'topic' value on workingcommitctx Mercurial's core changeset: 6c8ceebce309 revealed an issue. If the creator of a workingcommitctx explicitly assigned a 'topic', it was silently overridden by the current active on. This break import of patch with topic information. We now respect preexisting topic information.

topic: add a compatibility to access transaction's names The attribute was made private in 4024c363cd33.

topic: add a compatibility to access transaction's validator The attribute was made private in ebbba3ba3f66.

evovle: remove redundancy in evolve output Copying the discription of this redundancy issue given by Pierre Yves David: When running `hg evolve` to stabilize orphan changeset output about the currently stabilized changeset is issued. For example: $ hg evolve move:[3] a3 atop:[4] a2 working directory is now at 7c5649f73d11 This output can become quite repetitive when orphan are stabilized atop each other. For example: $ hg evolve --all move:[8] dansk 2! atop:[10] dansk! merging main-file-1 move:[9] dansk 3! atop:[11] dansk 2! In this case it would be smoother to issue: $ hg evolve --all move:[8] dansk 2! atop:[10] dansk! merging main-file-1 move:[9] dansk 3! Since we are moving "dansk 3!" atop the changeset we just stabilized. When adding this be careful that we still want to issue the "atop" message in various cases: 1. first changesets in a stack 2. when the orphan is not stabilized atop previous one 3. when using hg evolve --continue to resume an evolution So, I have made the changes which also respect above listed three points. And changes in tests/test-evovle*.t reflecting the changed behavior.

amend: handle crash before transaction initialization with --patch The `tr` object can be None.

branching: merge with stable

obsdiscovery: adopt to calling convention change Upstream commits 71d83b315778 and abce899c985f changed the calling convention of setdiscovery._takefullsample(). We inspect the signature of the function before calling it so we can pass the proper arguments.

branching: merge stable into default

packaging: mark as development version This avoid confusion when people install version from source.

Added tag 8.2.1 for changeset c6362c4abd69

packagin: prepare release 8.2.1

test-compat: merge mercurial-4.4 into mercurial-4.3

test-compat: merge mercurial-4.5 into mercurial-4.4

test-compat: merge mercurial-4.6 into mercurial-4.5

test-compat: merge stable into mercurial-4.6

branching: merge stable into default

caches: issue both debug and blackbox log message This increase the odd someone looking for issue will find them.

cmdrewrite: rename variable and move it closer to where it is used Renaming to satisfy pep8-naming linter plugin, which complained on this variable because all-uppercase names are reserved for constants, and constants need to be module-level. This variable is not a good choice for moving up to the module level because it's used only in one place. Additionally, let's have it closer to the place it's actually used to potentially avoid an extra call to _() function (however cheap it may be).

changelog: mention the cache improvements

obshashrangecache: be more paranoid in the errors we catch It seems like there error were already covered. Better safe than sorry.

stablerangecache: be more paranoid in the errors we catch It seems like there error were already covered. Better safe than sorry.

stablesortcache: update the variable tracking on-disk state after write Since we updated the on disk content, we should update that value. In practice the object will likely be discarded after the write, but there is nothing wrong in being more correct.

stablesortcache: ignore permission and OS errors when writing This cache is related to the obshashrange one and we update it lazily by default. This can be an issue when pulling locally from a read only repository that was not configured for a more aggressive cache warming. The raised permission error was uncaught and could crash the whole process. Errors during cache update should not block Mercurial operations.

firstmergecache: update the variable tracking on-disk state after write Since we updated the on disk content, we should update that value. In practice the object will likely be discarded after the write, but there is nothing wrong in being more correct.

firstmergecache: ignore permission and OS errors when writing This cache is related to the obshashrange one and we update it lazily by default. This can be an issue when pulling locally from a read only repository that was not configured for a more aggressive cache warming. The raised permission error was uncaught and could crash the whole process. Errors during cache update should not block Mercurial operations.

depthcache: update the variable tracking on disk state after write Since we updated the on disk content, we should update that value. In practice the object will likely be discarded after the write, but there is nothing wrong in being more correct.

depthcache: ignore permission and OS errors when writing This cache is related to the obshashrange one and we update it lazily by default. This can be an issue when pulling locally from a read only repository that was not configured for a more aggressive cache warming. The raised permission error was uncaught and could crash the whole process. Errors during Cache update should not block Mercurial operations.

obscache: update the variable tracking on disk state after write Since we updated the on disk content, we should update that value. In practice the object will likely be discarded after the write, but there is nothing wrong in being more correct.

obscache: ignore permission and OS errors when writing In practice we always update this cache along side a new transaction, with lock taken so permission issues should not happens. However, staying on the safe side is better. Errors during Cache update should not block Mercurial operations.

obshashrange: force reset if many changes If there is many new markers (or the markers affect many changesets) resetting affected changeset can get slow. A common case for that is then changeset and markers are added in different transaction, (eg: when using clone bundle). When we detect such case, we reset the obshashrange cache, rewarming it should be faster. Here many is arbitrarily set to 10 000. The number might get adjusted.

topic: refactor revset.py slightly Checking both kind == 'literal' and topic.startswith('literal:') seems to be redundant, because of how stringutil.stringmatcher() works.

test: update output to match new `hg verify` reports CORE-TEST-OUTPUT-UPDATE: f1186c292d03

test: update output to match new "pull" phase report CORE-TEST-OUTPUT-UPDATE: 89630d0b3e23

test: adapt to change in revert messages order CORE-TEST-OUTPUT-UPDATE: cb70501d8b71

rewind: fix help text (mostly grammar, but also s/precursor/predecessor)

evolve: fix use of "'%d' % ctx" ctx has not been convertible to int since Mercurial commit 38f480502043 (context: remove basectx.__int__ (API), 2018-02-24). I decided to use "%s" instead to match other similar messages in evolvecmd.py. I also added a space to match the other messages.

stablerange: only issue the message once There is a flag to prevent issuing the long load message multiple time. However, it was never set. This is now fixed

changelog: fix version number

branching: merge back with stable

packaing: mark as development version This help avoid confusion when installing in progress work.

Added tag 8.2.0 for changeset 8d8f08245f97

branching: merge back with stable

packaging: prepare new release

test-compat: merge mercurial-4.4 into mercurial-4.3

test-compat: merge mercurial-4.5 into mercurial-4.4

test-compat: merge mercurial-4.6 into mercurial-4.5

test-compat: merge stable into mercurial-4.6

topic: rework topic templatekw declaration to be 4.5 compatible again Mercurial prior to 4.5 does not have a requires keyword, so we need a compatibility layer.

test-compat: merge mercurial-4.4 into mercurial-4.3

test-compat: merge mercurial-4.5 into mercurial-4.4

test-compat: merge mercurial-4.6 into mercurial-4.5

test-compat: merge stable into mercurial-4.6

branching: merge the two stable heads

test-output: backout test output update in 8fcee1e65e29

test-output: backout test output update in 503ae2689df8

topic: update the changelog with the 's#' alias change

topic: update other occurrence of 't0' is 's0' in the code

topic: mention t0 in topic documentation

stack: display 's#' instead of 't#' and 'b#' If we want people to use it, we should display it.

stack: clarify s# error message in case of branch A branch might have a lots of changeset, with all of them public.

alias: allow reference through 's#' We keep the 't#' option, but we hope to be able to deprecate it at some point. We move to 's#' because it refers to `stack` a more generic concept. than topic. In addition, this allows to phase out 'b#' that collide with normal hex-node.

topic-revset: get the list of revset through the official localrepo API This is simpler and faster. In addition we tighter the logic to be used in case of empty topic. We do not need to fall back to revision search.

topic-revset: strictly read string Making a clear distinction between string and symbol passed to topic is important when it comes to empty topic. Filtering an empty topic name is harmless. Searching a non existent revision is not.

test: search for empty topic as a string The reduce the ambiguity in the test to help with the next change.

topic: handle ambiguous arguments to topic() revset These arguments can be interpreted as either string or a revset. The decision is made based on existence of topic with such a name. This matches the behavior of branch() revset. The code needs to know all topics that ever existed in the repo, because some commands report "disappearance" of topics after certain operations, using this revset (e.g. via stack.stack or repo.revs).

topic: handle revsets passed to topic() revset This handles cases when the argument can't be interpreted as a string at all.

topic: handle string argument to topic() revset earlier

topic: prepare to handle non-string arguments to topic() revset

topic: return result early if there are no arguments to topic() revset

obshashrange: enable by default We have been testing this in various placed and specific stress test have been performed. It is time to enable this by default

obshashrange: issue a warning if the cache takes a long time to load The -current- implementation of the obshashrange cache can get slow. This will be fixed eventually, in the mean time we inform people on large repository of what have been going on. Since we are about to enable obshashrange by default, we need to make sure nobody will have an (unexplained) bad surprise.

pick: fix last references of grab in code Grab was still referenced on comments and variable names.

pick: replace `hg grab` invocation in test with `hg pick`

pick: replace "grabstate" with "pickstate"

pick: use "pick" as operation name in obsolescence markers

pick: use "pick" instead of "grab" for various internal API This should not much user visible impact but this is cleaner.

pick: update command output to mention pick instead of grab

pick: rename the grab command to pick Grab is ambiguous with "graft", "pick" match the associated verb in histedit. The "grab" name is preserved as an alias for now.

grab: prefix function with 'cmd' as for the other command in the extension

readme: document the purpose of the evolve extension Another place for the explanation to live.

prune: replace --biject with --pair People have been complaining about `--biject` being to obscure for a while. A recent discussion with Pulkit Goyal and Dan Villiom Podlaski Christiansen produce some new concrete proposal that we liked. We eventually settle on using `--pair`. The `--biject` flag will be kept as an alias for a while.

topic: check that repo.currenttopic is set before using it as an argument repo.currenttopic can be an empty string, which means there's no current topic set. This fact can be used to short-circuit some checks. Also, this avoids passing an empty string as an argument to topic() revset.

test: update output after merge message change CORE-TEST-OUTPUT-UPDATE: f785073f792c CORE-TEST-OUTPUT-UPDATE: a3fd84f4fb38

readme: document the tag we use on output change

branching: merge with stable

packaging: fix debian version numbers

packaging: fix debian version numbers

branching: merge stable into default

packaging: mark in progress work as development version This avoid confusion when people install development version.

Added tag 8.1.2 for changeset f1cde4c97806

packagin: prepare version 8.1.2

obshashrange: force obshashrange invalidation by bumping schema The fix in 8.1.1 deserve recomputing the obs hash range cache. So we bump its schema version to declare the older cache invalid. The other caches, including the expensive stablerange are unaffected.

test-compat: merge mercurial-4.4 into mercurial-4.3

test-compat: merge mercurial-4.5 into mercurial-4.4

test-compat: merge mercurial-4.6 into mercurial-4.5

test-compat: merge stable into mercurial-4.6

branching: merge with stable

changelog: mention the database robutness fix in the changelog

sqlcache: also catch malformed database error This is apparently another way for sqlite to fail at concurrency.

stablerange: rework saving logic to be clearer and safer

obshashrange: rework saving branching to be clearer and safer

sqlcache: protect read query too Some error (like locked database) can even happens when doing readonly operation. So we protect them too. At that point, it seems that pysqlite3 is not the right tool for this job.

sqlite: fast path when nothing to save If we have no new data to save, we should try to do anything. Doing something concurrently is hard enough. If we can avoid doing it at all that would be great.

sqlcache: initialize meta table last It turns out that pysqlite commit after each table creation. This commit is an hopeless attemps to make the caches more concurrent friendly.

obshashrange: always check in base cachekey against the recorded one Ignoring the check in the none case is just disaster waiting to happens.

stablerangecache: fix output in the drifted case The node were not hex, and the field had different order in the pair. This is now fixed.

sqlcache: also ignore integrity error integrity error can happens when multiple client tries to warm similar cache. Given we leave multiple hole in the cache that can be warmed by anyone, this is harder to avoid that we would like. We simply catch the error in this case. Someone will warm the missing entry later.

sqlcache: cache OperationError when saving This is a cache, so we should not crash when trying to operate on it. OperationError can happens when the database is seen as readonly or locked.

sqlcache: passe better connection option These new options should help with handling transaction consistency and database access on high load.

contrib: introduce a small script to stress tests obsolescence exchange Tester have been reporting some error that seems to originate from concurrent access/update to the cache. This script will be useful to reproduce these situations locally.