server/sources/storages.py
author Julien Cristau <julien.cristau@logilab.fr>
Thu, 10 Dec 2015 16:58:45 +0100
changeset 10997 da712d3f0601
parent 10996 dc572d116731
permissions -rw-r--r--
Bring back the separate web-side entity cache Prior to changeset 635cfac73d28 "[repoapi] fold ClientConnection into Connection", we had two entity caches: one on the client/dbapi/web side, and one on the server/repo side. This is a waste, but it is actually needed as long as we have the magic _cw attribute on entities which must sometimes be a request and sometimes a cnx. Removing the duplication caused weird problems with entity._cw alternating between both types of objects, which is unexpected by both the repo and the web sides. We add an entity cache on ConnectionCubicWebRequestBase, separate from the Connection's, and bring back the _cw_update_attr_cache/dont-cache-attrs mechanism, to let the server/edition code let caches know which attributes have been modified Entity.as_rset can be cached again, as ResultSet no longer modifies the entity when fetching it from a cache. Contrary to the pre-3.21 code, _cw_update_attr_cache now handles web requests and connections in the same way (otherwise the cache ends up with wrong values if a hook modifies attributes), but dont-cache-attrs is never set for (inlined) relations. Closes #6863543
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9463
diff changeset
     1
# copyright 2003-2013 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    18
"""custom storages for the system source"""
5693
8af6623f3d4e [pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5625
diff changeset
    19
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
    20
import os
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
    21
import sys
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    22
from os import unlink, path as osp
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
    23
from contextlib import contextmanager
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
    24
import tempfile
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    25
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
    26
from six import PY2, PY3, text_type, binary_type
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
    27
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
    28
from logilab.common import nullobject
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
    29
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    30
from yams.schema import role_name
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    31
5693
8af6623f3d4e [pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5625
diff changeset
    32
from cubicweb import Binary, ValidationError
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    33
from cubicweb.server import hook
7118
e094b3d4eb95 [server] move EditedEntity class to its own module, to avoid cyclic dependency when needed from e.g. session.py
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
    34
from cubicweb.server.edition import EditedEntity
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
    35
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    36
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    37
def set_attribute_storage(repo, etype, attr, storage):
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4831
diff changeset
    38
    repo.system_source.set_storage(etype, attr, storage)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    39
4512
e7ac20bf3629 unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4349
diff changeset
    40
def unset_attribute_storage(repo, etype, attr):
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4831
diff changeset
    41
    repo.system_source.unset_storage(etype, attr)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    42
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
    43
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    44
class Storage(object):
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    45
    """abstract storage
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    46
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    47
    * If `source_callback` is true (by default), the callback will be run during
5625
6ee2a7b6f194 [external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5599
diff changeset
    48
      query result process of fetched attribute's value and should have the
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    49
      following prototype::
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    50
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
    51
        callback(self, source, cnx, value)
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    52
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    53
      where `value` is the value actually stored in the backend. None values
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    54
      will be skipped (eg callback won't be called).
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    55
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    56
    * if `source_callback` is false, the callback will be run during sql
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    57
      generation when some attribute with a custom storage is accessed and
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    58
      should have the following prototype::
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    59
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    60
        callback(self, generator, relation, linkedvar)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    61
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    62
      where `generator` is the sql generator, `relation` the current rql syntax
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    63
      tree relation and linkedvar the principal syntax tree variable holding the
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    64
      attribute.
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    65
    """
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    66
    is_source_callback = True
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    67
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    68
    def callback(self, *args):
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    69
        """see docstring for prototype, which vary according to is_source_callback
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    70
        """
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    71
        raise NotImplementedError()
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    72
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    73
    def entity_added(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    74
        """an entity using this storage for attr has been added"""
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    75
        raise NotImplementedError()
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    76
    def entity_updated(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    77
        """an entity using this storage for attr has been updatded"""
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    78
        raise NotImplementedError()
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    79
    def entity_deleted(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    80
        """an entity using this storage for attr has been deleted"""
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    81
        raise NotImplementedError()
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
    82
    def migrate_entity(self, entity, attribute):
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
    83
        """migrate an entity attribute to the storage"""
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
    84
        raise NotImplementedError()
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    85
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    86
# TODO
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    87
# * make it configurable without code
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    88
# * better file path attribution
4329
815e08c53548 add a reminder
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4322
diff changeset
    89
# * handle backup/restore
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    90
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    91
def uniquify_path(dirpath, basename):
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
    92
    """return a file descriptor and unique file name for `basename` in `dirpath`
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    93
    """
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
    94
    path = basename.replace(osp.sep, '-')
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    95
    base, ext = osp.splitext(path)
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
    96
    return tempfile.mkstemp(prefix=base, suffix=ext, dir=dirpath)
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    97
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
    98
@contextmanager
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
    99
def fsimport(cnx):
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   100
    present = 'fs_importing' in cnx.transaction_data
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   101
    old_value = cnx.transaction_data.get('fs_importing')
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   102
    cnx.transaction_data['fs_importing'] = True
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   103
    yield
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   104
    if present:
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   105
        cnx.transaction_data['fs_importing'] = old_value
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   106
    else:
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   107
        del cnx.transaction_data['fs_importing']
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   108
5398
b9e1abe1bdfe [repo] cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5397
diff changeset
   109
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   110
_marker = nullobject()
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   111
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   112
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   113
class BytesFileSystemStorage(Storage):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   114
    """store Bytes attribute value on the file system"""
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   115
    def __init__(self, defaultdir, fsencoding=_marker, wmode=0o444):
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   116
        if PY3:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   117
            if not isinstance(defaultdir, text_type):
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   118
                raise TypeError('defaultdir must be a unicode object in python 3')
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   119
            if fsencoding is not _marker:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   120
                raise ValueError('fsencoding is no longer supported in python 3')
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   121
        else:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   122
            self.fsencoding = fsencoding or 'utf-8'
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   123
            if isinstance(defaultdir, text_type):
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   124
                defaultdir = defaultdir.encode(fsencoding)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   125
        self.default_directory = defaultdir
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   126
        # extra umask to use when creating file
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   127
        # 0444 as in "only allow read bit in permission"
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   128
        self._wmode = wmode
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   129
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   130
    def _writecontent(self, fd, binary):
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   131
        """write the content of a binary in readonly file
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   132
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   133
        As the bfss never alters an existing file it does not prevent it from
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   134
        working as intended. This is a better safe than sorry approach.
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   135
        """
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   136
        os.fchmod(fd, self._wmode)
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   137
        fileobj = os.fdopen(fd, 'wb')
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   138
        binary.to_file(fileobj)
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   139
        fileobj.close()
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   140
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   141
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   142
    def callback(self, source, cnx, value):
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   143
        """sql generator callback when some attribute with a custom storage is
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   144
        accessed
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   145
        """
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   146
        fpath = source.binary_to_str(value)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   147
        try:
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   148
            return Binary.from_file(fpath)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8662
diff changeset
   149
        except EnvironmentError as ex:
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   150
            source.critical("can't open %s: %s", value, ex)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   151
            return None
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   152
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   153
    def entity_added(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   154
        """an entity using this storage for attr has been added"""
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   155
        if entity._cw.transaction_data.get('fs_importing'):
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   156
            binary = Binary.from_file(entity.cw_edited[attr].getvalue())
10997
da712d3f0601 Bring back the separate web-side entity cache
Julien Cristau <julien.cristau@logilab.fr>
parents: 10996
diff changeset
   157
            entity._cw_dont_cache_attribute(attr, repo_side=True)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   158
        else:
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   159
            binary = entity.cw_edited.pop(attr)
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   160
            fd, fpath = self.new_fs_path(entity, attr)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   161
            # bytes storage used to store file's path
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   162
            binary_obj = Binary(fpath if PY2 else fpath.encode('utf-8'))
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   163
            entity.cw_edited.edited_attribute(attr, binary_obj)
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   164
            self._writecontent(fd, binary)
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   165
            AddFileOp.get_instance(entity._cw).add_data(fpath)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   166
        return binary
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   167
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   168
    def entity_updated(self, entity, attr):
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   169
        """an entity using this storage for attr has been updated"""
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   170
        # get the name of the previous file containing the value
5599
be94157bd754 [bfss] Rename filenames according to their metadata on entity update.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5549
diff changeset
   171
        oldpath = self.current_fs_path(entity, attr)
5183
8d66003351f8 [storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5131
diff changeset
   172
        if entity._cw.transaction_data.get('fs_importing'):
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   173
            # If we are importing from the filesystem, the file already exists.
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   174
            # We do not need to create it but we need to fetch the content of
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   175
            # the file as the actual content of the attribute
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   176
            fpath = entity.cw_edited[attr].getvalue()
10997
da712d3f0601 Bring back the separate web-side entity cache
Julien Cristau <julien.cristau@logilab.fr>
parents: 10996
diff changeset
   177
            entity._cw_dont_cache_attribute(attr, repo_side=True)
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   178
            assert fpath is not None
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   179
            binary = Binary.from_file(fpath)
5183
8d66003351f8 [storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5131
diff changeset
   180
        else:
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   181
            # We must store the content of the attributes
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   182
            # into a file to stay consistent with the behaviour of entity_add.
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   183
            # Moreover, the BytesFileSystemStorage expects to be able to
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   184
            # retrieve the current value of the attribute at anytime by reading
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   185
            # the file on disk. To be able to rollback things, use a new file
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   186
            # and keep the old one that will be removed on commit if everything
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   187
            # went ok.
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   188
            #
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   189
            # fetch the current attribute value in memory
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   190
            binary = entity.cw_edited.pop(attr)
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   191
            if binary is None:
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   192
                fpath = None
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   193
            else:
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   194
                # Get filename for it
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   195
                fd, fpath = self.new_fs_path(entity, attr)
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   196
                # write attribute value on disk
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   197
                self._writecontent(fd, binary)
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   198
                # Mark the new file as added during the transaction.
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   199
                # The file will be removed on rollback
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   200
                AddFileOp.get_instance(entity._cw).add_data(fpath)
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   201
            # reinstall poped value
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   202
            if fpath is None:
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   203
                entity.cw_edited.edited_attribute(attr, None)
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   204
            else:
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   205
                # register the new location for the file.
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   206
                binary_obj = Binary(fpath if PY2 else fpath.encode('utf-8'))
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   207
                entity.cw_edited.edited_attribute(attr, binary_obj)
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   208
        if oldpath is not None and oldpath != fpath:
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   209
            # Mark the old file as useless so the file will be removed at
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   210
            # commit.
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   211
            DeleteFileOp.get_instance(entity._cw).add_data(oldpath)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   212
        return binary
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   213
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   214
    def entity_deleted(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   215
        """an entity using this storage for attr has been deleted"""
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   216
        fpath = self.current_fs_path(entity, attr)
7449
5e45e32071aa [bfss] Make ``current_fs_path`` return None when an attribute has no value
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7118
diff changeset
   217
        if fpath is not None:
5e45e32071aa [bfss] Make ``current_fs_path`` return None when an attribute has no value
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7118
diff changeset
   218
            DeleteFileOp.get_instance(entity._cw).add_data(fpath)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   219
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   220
    def new_fs_path(self, entity, attr):
5219
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   221
        # We try to get some hint about how to name the file using attribute's
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   222
        # name metadata, so we use the real file name and extension when
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   223
        # available. Keeping the extension is useful for example in the case of
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   224
        # PIL processing that use filename extension to detect content-type, as
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   225
        # well as providing more understandable file names on the fs.
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   226
        if PY2:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   227
            attr = attr.encode('ascii')
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   228
        basename = [str(entity.eid), attr]
5576
08c6d4d6c50c [deprecation] fix 3.9 deprecation warning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5549
diff changeset
   229
        name = entity.cw_attr_metadata(attr, 'name')
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   230
        if name is not None:
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   231
            basename.append(name.encode(self.fsencoding) if PY2 else name)
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   232
        fd, fspath = uniquify_path(self.default_directory,
8148
b7a195d54fd4 [bfss] convert root BFSS directory to str in constructor (closes #2137793)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 8131
diff changeset
   233
                               '_'.join(basename))
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   234
        if fspath is None:
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   235
            msg = entity._cw._('failed to uniquify path (%s, %s)') % (
5693
8af6623f3d4e [pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5625
diff changeset
   236
                self.default_directory, '_'.join(basename))
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   237
            raise ValidationError(entity.eid, {role_name(attr, 'subject'): msg})
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   238
        assert isinstance(fspath, str)  # bytes on py2, unicode on py3
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   239
        return fd, fspath
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   240
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   241
    def current_fs_path(self, entity, attr):
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   242
        """return the current fs_path of the attribute, or None is the attr is
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   243
        not stored yet.
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   244
        """
9463
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8900
diff changeset
   245
        sysource = entity._cw.repo.system_source
4349
48dadeeacfa5 [bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4329
diff changeset
   246
        cu = sysource.doexec(entity._cw,
48dadeeacfa5 [bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4329
diff changeset
   247
                             'SELECT cw_%s FROM cw_%s WHERE cw_eid=%s' % (
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8695
diff changeset
   248
                             attr, entity.cw_etype, entity.eid))
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   249
        rawvalue = cu.fetchone()[0]
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   250
        if rawvalue is None: # no previous value
7449
5e45e32071aa [bfss] Make ``current_fs_path`` return None when an attribute has no value
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7118
diff changeset
   251
            return None
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   252
        fspath = sysource._process_value(rawvalue, cu.description[0],
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   253
                                         binarywrap=binary_type)
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   254
        if PY3:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   255
            fspath = fspath.decode('utf-8')
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   256
        assert isinstance(fspath, str)  # bytes on py2, unicode on py3
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   257
        return fspath
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   258
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   259
    def migrate_entity(self, entity, attribute):
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   260
        """migrate an entity attribute to the storage"""
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   261
        entity.cw_edited = EditedEntity(entity, **entity.cw_attr_cache)
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   262
        self.entity_added(entity, attribute)
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   263
        cnx = entity._cw
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   264
        source = cnx.repo.system_source
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   265
        attrs = source.preprocess_entity(entity)
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8695
diff changeset
   266
        sql = source.sqlgen.update('cw_' + entity.cw_etype, attrs,
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   267
                                   ['cw_eid'])
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   268
        source.doexec(cnx, sql, attrs)
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   269
        entity.cw_edited = None
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   270
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   271
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   272
class AddFileOp(hook.DataOperationMixIn, hook.Operation):
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   273
    def rollback_event(self):
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   274
        for filepath in self.get_data():
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   275
            assert isinstance(filepath, str)  # bytes on py2, unicode on py3
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   276
            try:
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   277
                unlink(filepath)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8662
diff changeset
   278
            except Exception as ex:
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   279
                self.error("can't remove %s: %s" % (filepath, ex))
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   280
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   281
class DeleteFileOp(hook.DataOperationMixIn, hook.Operation):
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   282
    def postcommit_event(self):
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   283
        for filepath in self.get_data():
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   284
            assert isinstance(filepath, str)  # bytes on py2, unicode on py3
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   285
            try:
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   286
                unlink(filepath)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8662
diff changeset
   287
            except Exception as ex:
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   288
                self.error("can't remove %s: %s" % (filepath, ex))