author | Sylvain Thénault <sylvain.thenault@logilab.fr> |
Fri, 02 Jul 2010 09:09:59 +0200 | |
branch | stable |
changeset 5856 | a02129508378 |
parent 5693 | 8af6623f3d4e |
child 5696 | 98d390c28edb |
child 5857 | 1a24c62aefc5 |
permissions | -rw-r--r-- |
5421
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
1 |
# copyright 2003-2010 LOGILAB S.A. (Paris, FRANCE), all rights reserved. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
2 |
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
3 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
4 |
# This file is part of CubicWeb. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
5 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
6 |
# CubicWeb is free software: you can redistribute it and/or modify it under the |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
7 |
# terms of the GNU Lesser General Public License as published by the Free |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
8 |
# Software Foundation, either version 2.1 of the License, or (at your option) |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
9 |
# any later version. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
10 |
# |
5424
8ecbcbff9777
replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5421
diff
changeset
|
11 |
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT |
5421
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
12 |
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
13 |
# FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
14 |
# details. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
15 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
16 |
# You should have received a copy of the GNU Lesser General Public License along |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
17 |
# with CubicWeb. If not, see <http://www.gnu.org/licenses/>. |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
18 |
"""custom storages for the system source""" |
5693
8af6623f3d4e
[pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5625
diff
changeset
|
19 |
|
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
20 |
from os import unlink, path as osp |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
21 |
|
5218
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
22 |
from yams.schema import role_name |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
23 |
|
5693
8af6623f3d4e
[pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5625
diff
changeset
|
24 |
from cubicweb import Binary, ValidationError |
5396
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
25 |
from cubicweb.server import hook |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
26 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
27 |
def set_attribute_storage(repo, etype, attr, storage): |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4831
diff
changeset
|
28 |
repo.system_source.set_storage(etype, attr, storage) |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
29 |
|
4512
e7ac20bf3629
unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4349
diff
changeset
|
30 |
def unset_attribute_storage(repo, etype, attr): |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4831
diff
changeset
|
31 |
repo.system_source.unset_storage(etype, attr) |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
32 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
33 |
class Storage(object): |
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
34 |
"""abstract storage |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
35 |
|
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
36 |
* If `source_callback` is true (by default), the callback will be run during |
5625
6ee2a7b6f194
[external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5599
diff
changeset
|
37 |
query result process of fetched attribute's value and should have the |
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
38 |
following prototype:: |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
39 |
|
5625
6ee2a7b6f194
[external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5599
diff
changeset
|
40 |
callback(self, source, session, value) |
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
41 |
|
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
42 |
where `value` is the value actually stored in the backend. None values |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
43 |
will be skipped (eg callback won't be called). |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
44 |
|
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
45 |
* if `source_callback` is false, the callback will be run during sql |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
46 |
generation when some attribute with a custom storage is accessed and |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
47 |
should have the following prototype:: |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
48 |
|
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
49 |
callback(self, generator, relation, linkedvar) |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
50 |
|
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
51 |
where `generator` is the sql generator, `relation` the current rql syntax |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
52 |
tree relation and linkedvar the principal syntax tree variable holding the |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
53 |
attribute. |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
54 |
""" |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
55 |
is_source_callback = True |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
56 |
|
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
57 |
def callback(self, *args): |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
58 |
"""see docstring for prototype, which vary according to is_source_callback |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
59 |
""" |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
60 |
raise NotImplementedError() |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
61 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
62 |
def entity_added(self, entity, attr): |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
63 |
"""an entity using this storage for attr has been added""" |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
64 |
raise NotImplementedError() |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
65 |
def entity_updated(self, entity, attr): |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
66 |
"""an entity using this storage for attr has been updatded""" |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
67 |
raise NotImplementedError() |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
68 |
def entity_deleted(self, entity, attr): |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
69 |
"""an entity using this storage for attr has been deleted""" |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
70 |
raise NotImplementedError() |
5397
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
71 |
def migrate_entity(self, entity, attribute): |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
72 |
"""migrate an entity attribute to the storage""" |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
73 |
raise NotImplementedError() |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
74 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
75 |
# TODO |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
76 |
# * make it configurable without code |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
77 |
# * better file path attribution |
4329
815e08c53548
add a reminder
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4322
diff
changeset
|
78 |
# * handle backup/restore |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
79 |
|
5218
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
80 |
def uniquify_path(dirpath, basename): |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
81 |
"""return a unique file name for `basename` in `dirpath`, or None |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
82 |
if all attemps failed. |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
83 |
|
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
84 |
XXX subject to race condition. |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
85 |
""" |
5549
ab3a69a34626
[bfss] ensure base name doesn't contain path separator
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
86 |
path = osp.join(dirpath, basename.replace(osp.sep, '-')) |
5218
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
87 |
if not osp.isfile(path): |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
88 |
return path |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
89 |
base, ext = osp.splitext(path) |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
90 |
for i in xrange(1, 256): |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
91 |
path = '%s%s%s' % (base, i, ext) |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
92 |
if not osp.isfile(path): |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
93 |
return path |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
94 |
return None |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
95 |
|
5398
b9e1abe1bdfe
[repo] cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5397
diff
changeset
|
96 |
|
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
97 |
class BytesFileSystemStorage(Storage): |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
98 |
"""store Bytes attribute value on the file system""" |
5219
35d44017c72b
[storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5218
diff
changeset
|
99 |
def __init__(self, defaultdir, fsencoding='utf-8'): |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
100 |
self.default_directory = defaultdir |
5219
35d44017c72b
[storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5218
diff
changeset
|
101 |
self.fsencoding = fsencoding |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
102 |
|
5625
6ee2a7b6f194
[external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5599
diff
changeset
|
103 |
def callback(self, source, session, value): |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
104 |
"""sql generator callback when some attribute with a custom storage is |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
105 |
accessed |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
106 |
""" |
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
107 |
fpath = source.binary_to_str(value) |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
108 |
try: |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
109 |
return Binary(file(fpath).read()) |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
110 |
except OSError, ex: |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
111 |
source.critical("can't open %s: %s", value, ex) |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5012
diff
changeset
|
112 |
return None |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
113 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
114 |
def entity_added(self, entity, attr): |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
115 |
"""an entity using this storage for attr has been added""" |
5131
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
116 |
if entity._cw.transaction_data.get('fs_importing'): |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
117 |
binary = Binary(file(entity[attr].getvalue()).read()) |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
118 |
else: |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
119 |
binary = entity.pop(attr) |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
120 |
fpath = self.new_fs_path(entity, attr) |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
121 |
# bytes storage used to store file's path |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
122 |
entity[attr] = Binary(fpath) |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
123 |
file(fpath, 'w').write(binary.getvalue()) |
5396
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
124 |
hook.set_operation(entity._cw, 'bfss_added', fpath, AddFileOp) |
5131
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
125 |
return binary |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
126 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
127 |
def entity_updated(self, entity, attr): |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
128 |
"""an entity using this storage for attr has been updatded""" |
5599
be94157bd754
[bfss] Rename filenames according to their metadata on entity update.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5549
diff
changeset
|
129 |
oldpath = self.current_fs_path(entity, attr) |
5183
8d66003351f8
[storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5131
diff
changeset
|
130 |
if entity._cw.transaction_data.get('fs_importing'): |
8d66003351f8
[storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5131
diff
changeset
|
131 |
fpath = entity[attr].getvalue() |
8d66003351f8
[storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5131
diff
changeset
|
132 |
binary = Binary(file(fpath).read()) |
8d66003351f8
[storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5131
diff
changeset
|
133 |
else: |
8d66003351f8
[storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5131
diff
changeset
|
134 |
binary = entity.pop(attr) |
5599
be94157bd754
[bfss] Rename filenames according to their metadata on entity update.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5549
diff
changeset
|
135 |
fpath = self.new_fs_path(entity, attr) |
5183
8d66003351f8
[storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5131
diff
changeset
|
136 |
UpdateFileOp(entity._cw, filepath=fpath, filedata=binary.getvalue()) |
5599
be94157bd754
[bfss] Rename filenames according to their metadata on entity update.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5549
diff
changeset
|
137 |
if oldpath != fpath: |
be94157bd754
[bfss] Rename filenames according to their metadata on entity update.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5549
diff
changeset
|
138 |
entity[attr] = Binary(fpath) |
be94157bd754
[bfss] Rename filenames according to their metadata on entity update.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5549
diff
changeset
|
139 |
hook.set_operation(entity._cw, 'bfss_deleted', oldpath, |
be94157bd754
[bfss] Rename filenames according to their metadata on entity update.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5549
diff
changeset
|
140 |
DeleteFileOp) |
5131
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
141 |
return binary |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
142 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
143 |
def entity_deleted(self, entity, attr): |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
144 |
"""an entity using this storage for attr has been deleted""" |
5396
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
145 |
fpath = self.current_fs_path(entity, attr) |
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
146 |
hook.set_operation(entity._cw, 'bfss_deleted', fpath, DeleteFileOp) |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
147 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
148 |
def new_fs_path(self, entity, attr): |
5219
35d44017c72b
[storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5218
diff
changeset
|
149 |
# We try to get some hint about how to name the file using attribute's |
35d44017c72b
[storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5218
diff
changeset
|
150 |
# name metadata, so we use the real file name and extension when |
35d44017c72b
[storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5218
diff
changeset
|
151 |
# available. Keeping the extension is useful for example in the case of |
35d44017c72b
[storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5218
diff
changeset
|
152 |
# PIL processing that use filename extension to detect content-type, as |
35d44017c72b
[storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5218
diff
changeset
|
153 |
# well as providing more understandable file names on the fs. |
5218
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
154 |
basename = [str(entity.eid), attr] |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
155 |
name = entity.attr_metadata(attr, 'name') |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
156 |
if name is not None: |
5219
35d44017c72b
[storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5218
diff
changeset
|
157 |
basename.append(name.encode(self.fsencoding)) |
5218
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
158 |
fspath = uniquify_path(self.default_directory, '_'.join(basename)) |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
159 |
if fspath is None: |
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
160 |
msg = entity._cw._('failed to uniquify path (%s, %s)') % ( |
5693
8af6623f3d4e
[pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5625
diff
changeset
|
161 |
self.default_directory, '_'.join(basename)) |
5218
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
162 |
raise ValidationError(entity.eid, {role_name(attr, 'subject'): msg}) |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
163 |
return fspath |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
164 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
165 |
def current_fs_path(self, entity, attr): |
4349
48dadeeacfa5
[bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4329
diff
changeset
|
166 |
sysource = entity._cw.pool.source('system') |
48dadeeacfa5
[bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4329
diff
changeset
|
167 |
cu = sysource.doexec(entity._cw, |
48dadeeacfa5
[bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4329
diff
changeset
|
168 |
'SELECT cw_%s FROM cw_%s WHERE cw_eid=%s' % ( |
5218
aebd00a2d316
[fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
5183
diff
changeset
|
169 |
attr, entity.__regid__, entity.eid)) |
5131
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
170 |
rawvalue = cu.fetchone()[0] |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
171 |
if rawvalue is None: # no previous value |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
172 |
return self.new_fs_path(entity, attr) |
88b5ca8da928
[storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5013
diff
changeset
|
173 |
return sysource._process_value(rawvalue, cu.description[0], |
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4721
diff
changeset
|
174 |
binarywrap=str) |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
175 |
|
5397
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
176 |
def migrate_entity(self, entity, attribute): |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
177 |
"""migrate an entity attribute to the storage""" |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
178 |
entity.edited_attributes = set() |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
179 |
self.entity_added(entity, attribute) |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
180 |
session = entity._cw |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
181 |
source = session.repo.system_source |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
182 |
attrs = source.preprocess_entity(entity) |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
183 |
sql = source.sqlgen.update('cw_' + entity.__regid__, attrs, |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
184 |
['cw_eid']) |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5396
diff
changeset
|
185 |
source.doexec(session, sql, attrs) |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
186 |
|
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
187 |
|
5396
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
188 |
class AddFileOp(hook.Operation): |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
189 |
def rollback_event(self): |
5396
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
190 |
for filepath in self.session.transaction_data.pop('bfss_added'): |
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
191 |
try: |
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
192 |
unlink(filepath) |
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
193 |
except Exception, ex: |
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
194 |
self.error('cant remove %s: %s' % (filepath, ex)) |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
195 |
|
5396
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
196 |
class DeleteFileOp(hook.Operation): |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
197 |
def commit_event(self): |
5396
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
198 |
for filepath in self.session.transaction_data.pop('bfss_deleted'): |
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
199 |
try: |
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
200 |
unlink(filepath) |
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
201 |
except Exception, ex: |
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
202 |
self.error('cant remove %s: %s' % (filepath, ex)) |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
203 |
|
5396
78d92a47a4e5
[bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5219
diff
changeset
|
204 |
class UpdateFileOp(hook.Operation): |
4322
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
205 |
def precommit_event(self): |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
206 |
try: |
f65743cc53e4
first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
207 |
file(self.filepath, 'w').write(self.filedata) |
4349
48dadeeacfa5
[bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4329
diff
changeset
|
208 |
except Exception, ex: |
48dadeeacfa5
[bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4329
diff
changeset
|
209 |
self.exception(str(ex)) |