server/sources/storages.py
author Rémi Cardona <remi.cardona@logilab.fr>
Fri, 18 Sep 2015 11:58:29 +0200
changeset 10707 3f4f190f8e35
parent 10706 b261d90149d0
child 10996 dc572d116731
permissions -rw-r--r--
[py3k] unicode → six.text_type
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9463
diff changeset
     1
# copyright 2003-2013 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    18
"""custom storages for the system source"""
5693
8af6623f3d4e [pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5625
diff changeset
    19
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
    20
import os
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
    21
import sys
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    22
from os import unlink, path as osp
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
    23
from contextlib import contextmanager
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
    24
import tempfile
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    25
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
    26
from six import PY2, PY3, text_type, binary_type
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
    27
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
    28
from logilab.common import nullobject
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
    29
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    30
from yams.schema import role_name
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    31
5693
8af6623f3d4e [pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5625
diff changeset
    32
from cubicweb import Binary, ValidationError
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    33
from cubicweb.server import hook
7118
e094b3d4eb95 [server] move EditedEntity class to its own module, to avoid cyclic dependency when needed from e.g. session.py
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
    34
from cubicweb.server.edition import EditedEntity
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
    35
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    36
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    37
def set_attribute_storage(repo, etype, attr, storage):
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4831
diff changeset
    38
    repo.system_source.set_storage(etype, attr, storage)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    39
4512
e7ac20bf3629 unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4349
diff changeset
    40
def unset_attribute_storage(repo, etype, attr):
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4831
diff changeset
    41
    repo.system_source.unset_storage(etype, attr)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    42
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
    43
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    44
class Storage(object):
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    45
    """abstract storage
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    46
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    47
    * If `source_callback` is true (by default), the callback will be run during
5625
6ee2a7b6f194 [external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5599
diff changeset
    48
      query result process of fetched attribute's value and should have the
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    49
      following prototype::
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    50
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
    51
        callback(self, source, cnx, value)
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    52
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    53
      where `value` is the value actually stored in the backend. None values
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    54
      will be skipped (eg callback won't be called).
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    55
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    56
    * if `source_callback` is false, the callback will be run during sql
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    57
      generation when some attribute with a custom storage is accessed and
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    58
      should have the following prototype::
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    59
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    60
        callback(self, generator, relation, linkedvar)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    61
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    62
      where `generator` is the sql generator, `relation` the current rql syntax
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    63
      tree relation and linkedvar the principal syntax tree variable holding the
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    64
      attribute.
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    65
    """
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    66
    is_source_callback = True
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    67
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    68
    def callback(self, *args):
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    69
        """see docstring for prototype, which vary according to is_source_callback
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    70
        """
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    71
        raise NotImplementedError()
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    72
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    73
    def entity_added(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    74
        """an entity using this storage for attr has been added"""
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    75
        raise NotImplementedError()
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    76
    def entity_updated(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    77
        """an entity using this storage for attr has been updatded"""
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    78
        raise NotImplementedError()
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    79
    def entity_deleted(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    80
        """an entity using this storage for attr has been deleted"""
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    81
        raise NotImplementedError()
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
    82
    def migrate_entity(self, entity, attribute):
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
    83
        """migrate an entity attribute to the storage"""
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
    84
        raise NotImplementedError()
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    85
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    86
# TODO
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    87
# * make it configurable without code
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    88
# * better file path attribution
4329
815e08c53548 add a reminder
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4322
diff changeset
    89
# * handle backup/restore
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    90
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    91
def uniquify_path(dirpath, basename):
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
    92
    """return a file descriptor and unique file name for `basename` in `dirpath`
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    93
    """
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
    94
    path = basename.replace(osp.sep, '-')
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    95
    base, ext = osp.splitext(path)
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
    96
    return tempfile.mkstemp(prefix=base, suffix=ext, dir=dirpath)
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    97
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
    98
@contextmanager
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
    99
def fsimport(cnx):
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   100
    present = 'fs_importing' in cnx.transaction_data
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   101
    old_value = cnx.transaction_data.get('fs_importing')
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   102
    cnx.transaction_data['fs_importing'] = True
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   103
    yield
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   104
    if present:
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   105
        cnx.transaction_data['fs_importing'] = old_value
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   106
    else:
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   107
        del cnx.transaction_data['fs_importing']
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   108
5398
b9e1abe1bdfe [repo] cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5397
diff changeset
   109
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   110
_marker = nullobject()
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   111
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   112
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   113
class BytesFileSystemStorage(Storage):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   114
    """store Bytes attribute value on the file system"""
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   115
    def __init__(self, defaultdir, fsencoding=_marker, wmode=0o444):
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   116
        if PY3:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   117
            if not isinstance(defaultdir, text_type):
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   118
                raise TypeError('defaultdir must be a unicode object in python 3')
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   119
            if fsencoding is not _marker:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   120
                raise ValueError('fsencoding is no longer supported in python 3')
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   121
        else:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   122
            self.fsencoding = fsencoding or 'utf-8'
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   123
            if isinstance(defaultdir, text_type):
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   124
                defaultdir = defaultdir.encode(fsencoding)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   125
        self.default_directory = defaultdir
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   126
        # extra umask to use when creating file
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   127
        # 0444 as in "only allow read bit in permission"
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   128
        self._wmode = wmode
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   129
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   130
    def _writecontent(self, fd, binary):
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   131
        """write the content of a binary in readonly file
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   132
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   133
        As the bfss never alters an existing file it does not prevent it from
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   134
        working as intended. This is a better safe than sorry approach.
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   135
        """
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   136
        os.fchmod(fd, self._wmode)
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   137
        fileobj = os.fdopen(fd, 'wb')
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   138
        binary.to_file(fileobj)
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   139
        fileobj.close()
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   140
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   141
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   142
    def callback(self, source, cnx, value):
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   143
        """sql generator callback when some attribute with a custom storage is
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   144
        accessed
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   145
        """
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   146
        fpath = source.binary_to_str(value)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   147
        try:
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   148
            return Binary.from_file(fpath)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8662
diff changeset
   149
        except EnvironmentError as ex:
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   150
            source.critical("can't open %s: %s", value, ex)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   151
            return None
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   152
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   153
    def entity_added(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   154
        """an entity using this storage for attr has been added"""
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   155
        if entity._cw.transaction_data.get('fs_importing'):
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   156
            binary = Binary.from_file(entity.cw_edited[attr].getvalue())
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   157
        else:
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   158
            binary = entity.cw_edited.pop(attr)
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   159
            fd, fpath = self.new_fs_path(entity, attr)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   160
            # bytes storage used to store file's path
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   161
            binary_obj = Binary(fpath if PY2 else fpath.encode('utf-8'))
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   162
            entity.cw_edited.edited_attribute(attr, binary_obj)
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   163
            self._writecontent(fd, binary)
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   164
            AddFileOp.get_instance(entity._cw).add_data(fpath)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   165
        return binary
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   166
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   167
    def entity_updated(self, entity, attr):
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   168
        """an entity using this storage for attr has been updated"""
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   169
        # get the name of the previous file containing the value
5599
be94157bd754 [bfss] Rename filenames according to their metadata on entity update.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5549
diff changeset
   170
        oldpath = self.current_fs_path(entity, attr)
5183
8d66003351f8 [storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5131
diff changeset
   171
        if entity._cw.transaction_data.get('fs_importing'):
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   172
            # If we are importing from the filesystem, the file already exists.
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   173
            # We do not need to create it but we need to fetch the content of
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   174
            # the file as the actual content of the attribute
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   175
            fpath = entity.cw_edited[attr].getvalue()
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   176
            assert fpath is not None
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   177
            binary = Binary.from_file(fpath)
5183
8d66003351f8 [storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5131
diff changeset
   178
        else:
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   179
            # We must store the content of the attributes
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   180
            # into a file to stay consistent with the behaviour of entity_add.
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   181
            # Moreover, the BytesFileSystemStorage expects to be able to
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   182
            # retrieve the current value of the attribute at anytime by reading
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   183
            # the file on disk. To be able to rollback things, use a new file
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   184
            # and keep the old one that will be removed on commit if everything
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   185
            # went ok.
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   186
            #
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   187
            # fetch the current attribute value in memory
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   188
            binary = entity.cw_edited.pop(attr)
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   189
            if binary is None:
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   190
                fpath = None
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   191
            else:
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   192
                # Get filename for it
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   193
                fd, fpath = self.new_fs_path(entity, attr)
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   194
                # write attribute value on disk
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   195
                self._writecontent(fd, binary)
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   196
                # Mark the new file as added during the transaction.
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   197
                # The file will be removed on rollback
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   198
                AddFileOp.get_instance(entity._cw).add_data(fpath)
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   199
            # reinstall poped value
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   200
            if fpath is None:
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   201
                entity.cw_edited.edited_attribute(attr, None)
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   202
            else:
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   203
                # register the new location for the file.
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   204
                binary_obj = Binary(fpath if PY2 else fpath.encode('utf-8'))
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   205
                entity.cw_edited.edited_attribute(attr, binary_obj)
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   206
        if oldpath is not None and oldpath != fpath:
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   207
            # Mark the old file as useless so the file will be removed at
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   208
            # commit.
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   209
            DeleteFileOp.get_instance(entity._cw).add_data(oldpath)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   210
        return binary
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   211
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   212
    def entity_deleted(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   213
        """an entity using this storage for attr has been deleted"""
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   214
        fpath = self.current_fs_path(entity, attr)
7449
5e45e32071aa [bfss] Make ``current_fs_path`` return None when an attribute has no value
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7118
diff changeset
   215
        if fpath is not None:
5e45e32071aa [bfss] Make ``current_fs_path`` return None when an attribute has no value
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7118
diff changeset
   216
            DeleteFileOp.get_instance(entity._cw).add_data(fpath)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   217
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   218
    def new_fs_path(self, entity, attr):
5219
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   219
        # We try to get some hint about how to name the file using attribute's
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   220
        # name metadata, so we use the real file name and extension when
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   221
        # available. Keeping the extension is useful for example in the case of
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   222
        # PIL processing that use filename extension to detect content-type, as
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   223
        # well as providing more understandable file names on the fs.
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   224
        if PY2:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   225
            attr = attr.encode('ascii')
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   226
        basename = [str(entity.eid), attr]
5576
08c6d4d6c50c [deprecation] fix 3.9 deprecation warning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5549
diff changeset
   227
        name = entity.cw_attr_metadata(attr, 'name')
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   228
        if name is not None:
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   229
            basename.append(name.encode(self.fsencoding) if PY2 else name)
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   230
        fd, fspath = uniquify_path(self.default_directory,
8148
b7a195d54fd4 [bfss] convert root BFSS directory to str in constructor (closes #2137793)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 8131
diff changeset
   231
                               '_'.join(basename))
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   232
        if fspath is None:
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   233
            msg = entity._cw._('failed to uniquify path (%s, %s)') % (
5693
8af6623f3d4e [pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5625
diff changeset
   234
                self.default_directory, '_'.join(basename))
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   235
            raise ValidationError(entity.eid, {role_name(attr, 'subject'): msg})
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   236
        assert isinstance(fspath, str)  # bytes on py2, unicode on py3
10453
76f601a90aa7 [storage] use mkstemp to create files in bfss
Julien Cristau <julien.cristau@logilab.fr>
parents: 9468
diff changeset
   237
        return fd, fspath
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   238
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   239
    def current_fs_path(self, entity, attr):
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   240
        """return the current fs_path of the attribute, or None is the attr is
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   241
        not stored yet.
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   242
        """
9463
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8900
diff changeset
   243
        sysource = entity._cw.repo.system_source
4349
48dadeeacfa5 [bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4329
diff changeset
   244
        cu = sysource.doexec(entity._cw,
48dadeeacfa5 [bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4329
diff changeset
   245
                             'SELECT cw_%s FROM cw_%s WHERE cw_eid=%s' % (
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8695
diff changeset
   246
                             attr, entity.cw_etype, entity.eid))
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   247
        rawvalue = cu.fetchone()[0]
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   248
        if rawvalue is None: # no previous value
7449
5e45e32071aa [bfss] Make ``current_fs_path`` return None when an attribute has no value
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7118
diff changeset
   249
            return None
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   250
        fspath = sysource._process_value(rawvalue, cu.description[0],
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   251
                                         binarywrap=binary_type)
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   252
        if PY3:
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   253
            fspath = fspath.decode('utf-8')
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   254
        assert isinstance(fspath, str)  # bytes on py2, unicode on py3
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   255
        return fspath
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   256
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   257
    def migrate_entity(self, entity, attribute):
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   258
        """migrate an entity attribute to the storage"""
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   259
        entity.cw_edited = EditedEntity(entity, **entity.cw_attr_cache)
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   260
        self.entity_added(entity, attribute)
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   261
        cnx = entity._cw
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   262
        source = cnx.repo.system_source
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   263
        attrs = source.preprocess_entity(entity)
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8695
diff changeset
   264
        sql = source.sqlgen.update('cw_' + entity.cw_etype, attrs,
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   265
                                   ['cw_eid'])
10570
c0501d4f67f1 [server/storage] rename 'session' variables to 'cnx'
Julien Cristau <julien.cristau@logilab.fr>
parents: 10522
diff changeset
   266
        source.doexec(cnx, sql, attrs)
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   267
        entity.cw_edited = None
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   268
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   269
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   270
class AddFileOp(hook.DataOperationMixIn, hook.Operation):
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   271
    def rollback_event(self):
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   272
        for filepath in self.get_data():
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   273
            assert isinstance(filepath, str)  # bytes on py2, unicode on py3
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   274
            try:
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   275
                unlink(filepath)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8662
diff changeset
   276
            except Exception as ex:
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   277
                self.error("can't remove %s: %s" % (filepath, ex))
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   278
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   279
class DeleteFileOp(hook.DataOperationMixIn, hook.Operation):
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   280
    def postcommit_event(self):
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   281
        for filepath in self.get_data():
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   282
            assert isinstance(filepath, str)  # bytes on py2, unicode on py3
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   283
            try:
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   284
                unlink(filepath)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8662
diff changeset
   285
            except Exception as ex:
10706
b261d90149d0 [server] Port BFSS to py3k
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10591
diff changeset
   286
                self.error("can't remove %s: %s" % (filepath, ex))