0
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
1 |
"""Migration test script |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
2 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
3 |
* migration will be played into a chroot of the local machine |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
4 |
* the database server used can be configured |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
5 |
* test tested instance may be on another host |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
6 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
7 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
8 |
We are using postgres'.pgpass file. Here is a copy of postgres documentation |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
9 |
about that: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
10 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
11 |
The file .pgpass in a user's home directory or the file referenced by |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
12 |
PGPASSFILE can contain passwords to be used if the connection requires |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
13 |
a password (and no password has been specified otherwise). |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
14 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
15 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
16 |
This file should contain lines of the following format: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
17 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
18 |
hostname:port:database:username:password |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
19 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
20 |
Each of the first four fields may be a literal value, or *, which |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
21 |
matches anything. The password field from the first line that matches |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
22 |
the current connection parameters will be used. (Therefore, put |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
23 |
more-specific entries first when you are using wildcards.) If an entry |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
24 |
needs to contain : or \, escape this character with \. A hostname of |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
25 |
localhost matches both host (TCP) and local (Unix domain socket) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
26 |
connections coming from the local machine. |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
27 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
28 |
The permissions on .pgpass must disallow any access to world or group; |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
29 |
achieve this by the command chmod 0600 ~/.pgpass. If the permissions |
1977
|
30 |
are less strict than this, the file will be ignored. |
0
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
31 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
32 |
:organization: Logilab |
1977
|
33 |
:copyright: 2001-2009 LOGILAB S.A. (Paris, FRANCE), license is LGPL v2. |
0
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
34 |
:contact: http://www.logilab.fr/ -- mailto:contact@logilab.fr |
1977
|
35 |
:license: GNU Lesser General Public License, v2.1 - http://www.gnu.org/licenses |
0
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
36 |
""" |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
37 |
__docformat__ = "restructuredtext en" |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
38 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
39 |
from os import system |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
40 |
from os.path import join, basename |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
41 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
42 |
from logilab.common.shellutils import cp, rm |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
43 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
44 |
from cubicweb.toolsutils import read_config |
2105
92ea410806fe
refactor sources configuration, add source to sources when using a cube defining
Sylvain Thénault <sylvain.thenault@logilab.fr>
diff
changeset
|
45 |
from cubicweb.server.utils import generate_sources_file |
0
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
46 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
47 |
# XXXX use db-copy instead |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
48 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
49 |
# test environment configuration |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
50 |
chrootpath = '/sandbox/cubicwebtest' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
51 |
tmpdbhost = 'crater' |
2105
92ea410806fe
refactor sources configuration, add source to sources when using a cube defining
Sylvain Thénault <sylvain.thenault@logilab.fr>
diff
changeset
|
52 |
tmpdbuser = 'syt' |
0
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
53 |
tmpdbpasswd = 'syt' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
54 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
55 |
def play_migration(applhome, applhost='', sudo=False): |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
56 |
applid = dbname = basename(applhome) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
57 |
testapplhome = join(chrootpath, applhome) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
58 |
# copy instance into the chroot |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
59 |
if applhost: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
60 |
system('scp -r %s:%s %s' % (applhost, applhome, testapplhome)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
61 |
else: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
62 |
cp(applhome, testapplhome) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
63 |
## # extract db parameters |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
64 |
## sources = read_config(join(testapplhome, 'sources')) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
65 |
## dbname = sources['system']['db-name'] |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
66 |
## dbhost = sources['system'].get('db-host') or '' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
67 |
## dbuser = sources['system'].get('db-user') or '' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
68 |
## dbpasswd = sources['system'].get('db-password') or '' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
69 |
# generate sources file |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
70 |
# XXX multisources |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
71 |
sources = {'system': {}} |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
72 |
sources['system']['db-encoding'] = 'UTF8' # XXX |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
73 |
sources['system']['db-name'] = dbname |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
74 |
sources['system']['db-host'] = None |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
75 |
sources['system']['db-user'] = tmpdbuser |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
76 |
sources['system']['db-password'] = None |
3904
dc7d315165fc
default_instance id should also be available when silently rebuilding soures file
Sylvain Thénault <sylvain.thenault@logilab.fr>
diff
changeset
|
77 |
generate_sources_file(applid, join(testapplhome, 'sources'), sources) |
0
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
78 |
## # create postgres password file so we won't need anymore passwords |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
79 |
## # XXX may exist! |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
80 |
## pgpassfile = expanduser('~/.pgpass') |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
81 |
## pgpass = open(pgpassfile, 'w') |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
82 |
## if dbpasswd: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
83 |
## pgpass.write('%s:*:%s:%s:%s\n' % (dbhost or applhost or 'localhost', |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
84 |
## dbname, dbuser, dbpasswd)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
85 |
## if tmpdbpasswd: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
86 |
## pgpass.write('%s:*:%s:%s:%s\n' % (tmpdbhost or 'localhost', dbname, |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
87 |
## tmpdbuser, tmpdbpasswd)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
88 |
## pgpass.close() |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
89 |
## chmod(pgpassfile, 0600) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
90 |
# dump db |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
91 |
## dumpcmd = 'pg_dump -Fc -U %s -f /tmp/%s.dump %s' % ( |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
92 |
## dbuser, dbname, dbname) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
93 |
## if dbhost: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
94 |
## dumpcmd += ' -h %s' % dbhost |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
95 |
dumpfile = '/tmp/%s.dump' % applid |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
96 |
dumpcmd = 'cubicweb-ctl db-dump --output=%s %s' % (dumpfile, applid) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
97 |
if sudo: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
98 |
dumpcmd = 'sudo %s' % dumpcmd |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
99 |
if applhost: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
100 |
dumpcmd = 'ssh %s "%s"' % (applhost, dumpcmd) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
101 |
if system(dumpcmd): |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
102 |
raise Exception('error while dumping the database') |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
103 |
## if not dbhost and applhost: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
104 |
if applhost: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
105 |
# retrieve the dump |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
106 |
if system('scp %s:%s %s' % (applhost, dumpfile, dumpfile)): |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
107 |
raise Exception('error while retreiving the dump') |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
108 |
# move the dump into the chroot |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
109 |
system('mv %s %s%s' % (dumpfile, chrootpath, dumpfile)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
110 |
# locate installed versions |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
111 |
vcconf = read_config(join(testapplhome, 'vc.conf')) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
112 |
template = vcconf['TEMPLATE'] |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
113 |
cubicwebversion = vcconf['CW'] |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
114 |
templversion = vcconf['TEMPLATE_VERSION'] |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
115 |
# install the same versions cubicweb and template versions into the chroot |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
116 |
system('sudo chroot %s apt-get update' % chrootpath) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
117 |
system('sudo chroot %s apt-get install cubicweb-server=%s cubicweb-client=%s' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
118 |
% (chrootpath, cubicwebversion, cubicwebversion)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
119 |
system('sudo chroot %s apt-get install cubicweb-%s-appl-server=%s cubicweb-%s-appl-client=%s' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
120 |
% (chrootpath, template, templversion, template, templversion)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
121 |
# update and upgrade to the latest version |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
122 |
system('sudo chroot %s apt-get install cubicweb-server cubicweb-client' % chrootpath) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
123 |
system('sudo chroot %s apt-get install cubicweb-%s-appl-server cubicweb-%s-appl-client' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
124 |
% (chrootpath, template, template)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
125 |
# create and fill the database |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
126 |
system('sudo chroot cubicweb-ctl db-restore %s %s' % (applid, dumpfile)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
127 |
## if not tmpdbhost: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
128 |
## system('createdb -U %s -T template0 -E UTF8 %s' % (tmpdbuser, dbname)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
129 |
## system('pg_restore -U %s -O -Fc -d %s /tmp/%s.dump' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
130 |
## % (tmpdbuser, dbname, dbname)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
131 |
## else: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
132 |
## system('createdb -h %s -U %s -T template0 -E UTF8 %s' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
133 |
## % (tmpdbhost, tmpdbuser, dbname)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
134 |
## system('pg_restore -h %s -U %s -O -Fc -d %s /tmp/%s.dump' |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
135 |
## % (tmpdbhost, tmpdbuser, dbname, dbname)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
136 |
# launch upgrade |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
137 |
system('sudo chroot %s cubicweb-ctl upgrade %s' % (chrootpath, applid)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
138 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
139 |
# cleanup |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
140 |
rm(testapplhome) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
141 |
## rm(pgpassfile) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
142 |
## if tmpdbhost: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
143 |
## system('dropdb -h %s -U %s %s' % (tmpdbuser, tmpdbhost, dbname)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
144 |
## else: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
145 |
## system('dropdb -U %s %s' % (tmpdbuser, dbname)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
146 |
## if not dbhost and applhost: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
147 |
if applhost: |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
148 |
system('ssh %s rm %s' % (applhost, dumpfile)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
149 |
rm('%s%s' % (chrootpath, dumpfile)) |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
150 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
151 |
|
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
152 |
if __name__ == '__main__': |
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff
changeset
|
153 |
play_migration('/etc/cubicweb.d/jpl', 'lepus') |