docs/obs-implementation.rst
author Pierre-Yves David <pierre-yves.david@fb.com>
Tue, 22 Apr 2014 17:00:24 -0700
changeset 905 b177fde040dd
parent 519 9825c7da5b54
permissions -rw-r--r--
exchange: use fixed order for obstore content The obsstore content seems unstable from one version of python to another (we most certainly use some dict) we fix the order once and for all.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
519
9825c7da5b54 ensure all file have a copyright notice
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 501
diff changeset
     1
.. Copyright 2011 Pierre-Yves David <pierre-yves.david@ens-lyon.org>
9825c7da5b54 ensure all file have a copyright notice
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 501
diff changeset
     2
..                Logilab SA        <contact@logilab.fr>
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
     3
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
     4
-----------------------------------------------------
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
     5
Implementation of Obsolete Marker
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
     6
-----------------------------------------------------
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
     7
.. warning:: This document is still in heavy work in progress
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
     8
244
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
     9
Main questions about Obsolete Marker Implementation
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    10
-----------------------------------------------------
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    11
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    12
246
1e8e32d3871c doc: idea about OnDisk Storage
Pierre-Yves.David@ens-lyon.org
parents: 244
diff changeset
    13
244
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    14
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    15
How shall we exchange Marker over the Wire ?
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    16
`````````````````````````````````````````````````````````
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    17
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    18
We can have a lot of markers. We do not want to exchange data for the one we
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    19
already know. Listkey() is not very appropriate there as you get everything.
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    20
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    21
Moreover, we might want to only hear about Marker that impact changeset we are
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    22
pulling.
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    23
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    24
pushkey is not batchable yet (could be fixed)
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    25
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    26
A dedicated discovery and exchange protocol seems mandatory here.
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    27
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    28
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    29
Various technical details
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    30
-----------------------------------------------------
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    31
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    32
Some stuff that worse to note. some may deserve their own section later.
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    33
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    34
storing old changeset
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    35
``````````````````````
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    36
244
9d37254031fc [doc] add Question and Answer about Obsolet Marker Implementation
Pierre-Yves.David@ens-lyon.org
parents: 228
diff changeset
    37
The new general delta format allows a very efficient storage of two very similar
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    38
changesets. Storing obsolete children using general delta takes no more place
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    39
than storing the obsolete diff. Reverted file will even we reused. The whole
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    40
operation will take much less space the strip backup.
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    41
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    42
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    43
Abstraction from history rewriting UI
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    44
```````````````````````````````````````````
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    45
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    46
How Mercurial handles obsolete marker is independent from what decides
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    47
to create them and what actual operation solves the error case. Any of
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    48
the existing history rewriting UI (rebase, mq, histedit) can lay
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    49
obsolete markers and resolve situation created by others. To go
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    50
further, a hook system of obsolete marker creation would allow each
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    51
mechanism to collaborate with other though a standard and central
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    52
mechanism.
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    53
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    54
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    55
Obsolete marker storage
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    56
```````````````````````````
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    57
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    58
The Obsolete marker will most likely be stored outside standard
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    59
history. They are multiple reasons for this:
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    60
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    61
First, obsolete markers are really perpendicular to standard history
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    62
there is no strong reason to include it here other than convenience.
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    63
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    64
Second, storing obsolete marker inside standard history means:
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    65
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    66
* A changeset must be created every time an obsolete relation is added. Very
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    67
  inconvenient for delete operation.
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    68
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    69
* Obsolete marker must be forged at the creation of the new changeset. This
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    70
  is very inconvenient for split operation. And in general it becomes
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    71
  complicated to fix history afterward in particular when working with older
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    72
  clients.
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    73
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    74
Storing obsolete marker outside history have several pros:
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    75
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    76
* It eases Exchange of obsolete markers without unnecessary obsolete
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    77
  changeset contents.
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    78
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    79
* It allows tuning the actual storage and protocol exchange while maintaining
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    80
  compatibility with older clients through the wire (as we do the repository
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    81
  format).
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    82
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    83
* It eases the exchange of obsolete related information during
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    84
  discovery to exchange obsolete changeset relevant to conflict
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    85
  resolution. Exchanging such information deserves a dedicated
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    86
  protocol.
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    87
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    88
Persistent
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    89
```````````````````````
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    90
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    91
*Extinct* changeset and obsolete marker will most likely be garbage collected as
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    92
some point. However, archive server may decide to keep them forever in order to
357
b398e9c2dbd1 doc: English proofreading
Jordi Gutiérrez Hermoso <jordigh@octave.org>
parents: 311
diff changeset
    93
keep a fully auditable history in its finest conception.
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    94
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    95
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    96
Current status
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    97
-----------------------------------------------------
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
    98
501
9a17c48c2099 update implementation details
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 357
diff changeset
    99
Obsolete marker are partialy in core.
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
   100
501
9a17c48c2099 update implementation details
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 357
diff changeset
   101
2.3:
161
4e3f25ba5401 More doc and index with sphynx
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
diff changeset
   102
501
9a17c48c2099 update implementation details
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 357
diff changeset
   103
- storage over obsolete marker
9a17c48c2099 update implementation details
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 357
diff changeset
   104
- exchange suing pushkey
9a17c48c2099 update implementation details
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 357
diff changeset
   105
- extinct changeset are properly hidden
9a17c48c2099 update implementation details
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 357
diff changeset
   106
- extinct changeset are excluded from exchange