server/sources/storages.py
author Aurelien Campeas <aurelien.campeas@logilab.fr>
Tue, 10 Jun 2014 18:17:07 +0200
changeset 10357 59a79300f213
parent 10354 635cfac73d28
child 10522 1660a0fa4f43
permissions -rw-r--r--
[entity,storage] remove hackish code from the bfss path The ClientConnection/Connection duality brought some painful hacks that are no longer needed. Hence we can send entity._cw_update_attr_cache and ._cw_dont_cache_attribute to the grave. Related to #3837233
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9463
diff changeset
     1
# copyright 2003-2013 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    18
"""custom storages for the system source"""
5693
8af6623f3d4e [pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5625
diff changeset
    19
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
    20
import os
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
    21
import sys
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    22
from os import unlink, path as osp
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
    23
from contextlib import contextmanager
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    24
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    25
from yams.schema import role_name
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    26
5693
8af6623f3d4e [pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5625
diff changeset
    27
from cubicweb import Binary, ValidationError
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
    28
from cubicweb.server import hook
7118
e094b3d4eb95 [server] move EditedEntity class to its own module, to avoid cyclic dependency when needed from e.g. session.py
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
    29
from cubicweb.server.edition import EditedEntity
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
    30
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    31
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    32
def set_attribute_storage(repo, etype, attr, storage):
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4831
diff changeset
    33
    repo.system_source.set_storage(etype, attr, storage)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    34
4512
e7ac20bf3629 unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4349
diff changeset
    35
def unset_attribute_storage(repo, etype, attr):
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4831
diff changeset
    36
    repo.system_source.unset_storage(etype, attr)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    37
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
    38
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    39
class Storage(object):
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    40
    """abstract storage
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    41
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    42
    * If `source_callback` is true (by default), the callback will be run during
5625
6ee2a7b6f194 [external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5599
diff changeset
    43
      query result process of fetched attribute's value and should have the
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    44
      following prototype::
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    45
5625
6ee2a7b6f194 [external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5599
diff changeset
    46
        callback(self, source, session, value)
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    47
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    48
      where `value` is the value actually stored in the backend. None values
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    49
      will be skipped (eg callback won't be called).
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    50
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    51
    * if `source_callback` is false, the callback will be run during sql
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    52
      generation when some attribute with a custom storage is accessed and
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    53
      should have the following prototype::
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    54
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    55
        callback(self, generator, relation, linkedvar)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    56
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    57
      where `generator` is the sql generator, `relation` the current rql syntax
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    58
      tree relation and linkedvar the principal syntax tree variable holding the
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    59
      attribute.
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    60
    """
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    61
    is_source_callback = True
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    62
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    63
    def callback(self, *args):
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
    64
        """see docstring for prototype, which vary according to is_source_callback
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    65
        """
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    66
        raise NotImplementedError()
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    67
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    68
    def entity_added(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    69
        """an entity using this storage for attr has been added"""
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    70
        raise NotImplementedError()
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    71
    def entity_updated(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    72
        """an entity using this storage for attr has been updatded"""
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    73
        raise NotImplementedError()
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    74
    def entity_deleted(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    75
        """an entity using this storage for attr has been deleted"""
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    76
        raise NotImplementedError()
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
    77
    def migrate_entity(self, entity, attribute):
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
    78
        """migrate an entity attribute to the storage"""
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
    79
        raise NotImplementedError()
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    80
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    81
# TODO
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    82
# * make it configurable without code
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    83
# * better file path attribution
4329
815e08c53548 add a reminder
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4322
diff changeset
    84
# * handle backup/restore
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    85
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    86
def uniquify_path(dirpath, basename):
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    87
    """return a unique file name for `basename` in `dirpath`, or None
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    88
    if all attemps failed.
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    89
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    90
    XXX subject to race condition.
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    91
    """
5549
ab3a69a34626 [bfss] ensure base name doesn't contain path separator
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
    92
    path = osp.join(dirpath, basename.replace(osp.sep, '-'))
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    93
    if not osp.isfile(path):
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    94
        return path
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    95
    base, ext = osp.splitext(path)
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    96
    for i in xrange(1, 256):
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    97
        path = '%s%s%s' % (base, i, ext)
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    98
        if not osp.isfile(path):
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
    99
            return path
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   100
    return None
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   101
6383
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   102
@contextmanager
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   103
def fsimport(session):
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   104
    present = 'fs_importing' in session.transaction_data
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   105
    old_value = session.transaction_data.get('fs_importing')
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   106
    session.transaction_data['fs_importing'] = True
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   107
    yield
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   108
    if present:
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   109
        session.transaction_data['fs_importing'] = old_value
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   110
    else:
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   111
        del session.transaction_data['fs_importing']
19ebe0b994d6 Add a fsimport context manage to localy enable fsimporting.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5863
diff changeset
   112
5398
b9e1abe1bdfe [repo] cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5397
diff changeset
   113
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   114
class BytesFileSystemStorage(Storage):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   115
    """store Bytes attribute value on the file system"""
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   116
    def __init__(self, defaultdir, fsencoding='utf-8', wmode=0444):
8148
b7a195d54fd4 [bfss] convert root BFSS directory to str in constructor (closes #2137793)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 8131
diff changeset
   117
        if type(defaultdir) is unicode:
b7a195d54fd4 [bfss] convert root BFSS directory to str in constructor (closes #2137793)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 8131
diff changeset
   118
            defaultdir = defaultdir.encode(fsencoding)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   119
        self.default_directory = defaultdir
5219
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   120
        self.fsencoding = fsencoding
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   121
        # extra umask to use when creating file
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   122
        # 0444 as in "only allow read bit in permission"
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   123
        self._wmode = wmode
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   124
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   125
    def _writecontent(self, path, binary):
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   126
        """write the content of a binary in readonly file
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   127
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   128
        As the bfss never alter a create file it does not prevent it to work as
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   129
        intended. This is a beter safe than sorry approach.
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   130
        """
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   131
        write_flag = os.O_WRONLY | os.O_CREAT | os.O_EXCL
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   132
        if sys.platform == 'win32':
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   133
            write_flag |= os.O_BINARY
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   134
        fd = os.open(path, write_flag, self._wmode)
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   135
        fileobj = os.fdopen(fd, 'wb')
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   136
        binary.to_file(fileobj)
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   137
        fileobj.close()
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   138
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   139
5625
6ee2a7b6f194 [external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5599
diff changeset
   140
    def callback(self, source, session, value):
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   141
        """sql generator callback when some attribute with a custom storage is
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   142
        accessed
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   143
        """
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   144
        fpath = source.binary_to_str(value)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   145
        try:
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   146
            return Binary.from_file(fpath)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8662
diff changeset
   147
        except EnvironmentError as ex:
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   148
            source.critical("can't open %s: %s", value, ex)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5012
diff changeset
   149
            return None
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   150
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   151
    def entity_added(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   152
        """an entity using this storage for attr has been added"""
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   153
        if entity._cw.transaction_data.get('fs_importing'):
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   154
            binary = Binary.from_file(entity.cw_edited[attr].getvalue())
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   155
        else:
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   156
            binary = entity.cw_edited.pop(attr)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   157
            fpath = self.new_fs_path(entity, attr)
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   158
            # bytes storage used to store file's path
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   159
            entity.cw_edited.edited_attribute(attr, Binary(fpath))
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   160
            self._writecontent(fpath, binary)
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   161
            AddFileOp.get_instance(entity._cw).add_data(fpath)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   162
        return binary
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   163
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   164
    def entity_updated(self, entity, attr):
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   165
        """an entity using this storage for attr has been updated"""
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   166
        # get the name of the previous file containing the value
5599
be94157bd754 [bfss] Rename filenames according to their metadata on entity update.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5549
diff changeset
   167
        oldpath = self.current_fs_path(entity, attr)
5183
8d66003351f8 [storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5131
diff changeset
   168
        if entity._cw.transaction_data.get('fs_importing'):
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   169
            # If we are importing from the filesystem, the file already exists.
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   170
            # We do not need to create it but we need to fetch the content of
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   171
            # the file as the actual content of the attribute
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   172
            fpath = entity.cw_edited[attr].getvalue()
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   173
            assert fpath is not None
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   174
            binary = Binary.from_file(fpath)
5183
8d66003351f8 [storage] consider fs_importing on update operations too
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5131
diff changeset
   175
        else:
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   176
            # We must store the content of the attributes
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   177
            # into a file to stay consistent with the behaviour of entity_add.
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   178
            # Moreover, the BytesFileSystemStorage expects to be able to
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   179
            # retrieve the current value of the attribute at anytime by reading
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   180
            # the file on disk. To be able to rollback things, use a new file
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   181
            # and keep the old one that will be removed on commit if everything
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   182
            # went ok.
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   183
            #
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   184
            # fetch the current attribute value in memory
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   185
            binary = entity.cw_edited.pop(attr)
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   186
            if binary is None:
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   187
                fpath = None
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   188
            else:
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   189
                # Get filename for it
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   190
                fpath = self.new_fs_path(entity, attr)
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   191
                assert not osp.exists(fpath)
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   192
                # write attribute value on disk
8180
1f6ba9afb925 [storage] BFSS now create read only file (closes #2151672)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   193
                self._writecontent(fpath, binary)
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   194
                # Mark the new file as added during the transaction.
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   195
                # The file will be removed on rollback
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   196
                AddFileOp.get_instance(entity._cw).add_data(fpath)
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   197
            # reinstall poped value
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   198
            if fpath is None:
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   199
                entity.cw_edited.edited_attribute(attr, None)
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   200
            else:
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   201
                # register the new location for the file.
7694
bd56a29acaa8 [bfss] Fix update of BFSS attribute to None (close #1875289)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7477
diff changeset
   202
                entity.cw_edited.edited_attribute(attr, Binary(fpath))
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   203
        if oldpath is not None and oldpath != fpath:
5857
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   204
            # Mark the old file as useless so the file will be removed at
1a24c62aefc5 [bfss] fix file update to ensure file's content is available on the fs asap...
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   205
            # commit.
8300
87c72dccf7b9 [storage/bfss] ensure bfss never remove a cw_edited item (closes #2236287)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8148
diff changeset
   206
            DeleteFileOp.get_instance(entity._cw).add_data(oldpath)
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   207
        return binary
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   208
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   209
    def entity_deleted(self, entity, attr):
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   210
        """an entity using this storage for attr has been deleted"""
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   211
        fpath = self.current_fs_path(entity, attr)
7449
5e45e32071aa [bfss] Make ``current_fs_path`` return None when an attribute has no value
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7118
diff changeset
   212
        if fpath is not None:
5e45e32071aa [bfss] Make ``current_fs_path`` return None when an attribute has no value
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7118
diff changeset
   213
            DeleteFileOp.get_instance(entity._cw).add_data(fpath)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   214
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   215
    def new_fs_path(self, entity, attr):
5219
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   216
        # We try to get some hint about how to name the file using attribute's
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   217
        # name metadata, so we use the real file name and extension when
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   218
        # available. Keeping the extension is useful for example in the case of
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   219
        # PIL processing that use filename extension to detect content-type, as
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   220
        # well as providing more understandable file names on the fs.
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   221
        basename = [str(entity.eid), attr]
5576
08c6d4d6c50c [deprecation] fix 3.9 deprecation warning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5549
diff changeset
   222
        name = entity.cw_attr_metadata(attr, 'name')
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   223
        if name is not None:
5219
35d44017c72b [storage] missing qrefresh in previous patch applied: fix comment, error message, and use a storage specified encoding, not cubicweb's encoding
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5218
diff changeset
   224
            basename.append(name.encode(self.fsencoding))
8148
b7a195d54fd4 [bfss] convert root BFSS directory to str in constructor (closes #2137793)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 8131
diff changeset
   225
        fspath = uniquify_path(self.default_directory,
b7a195d54fd4 [bfss] convert root BFSS directory to str in constructor (closes #2137793)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 8131
diff changeset
   226
                               '_'.join(basename))
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   227
        if fspath is None:
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   228
            msg = entity._cw._('failed to uniquify path (%s, %s)') % (
5693
8af6623f3d4e [pylint] fix detected name errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5625
diff changeset
   229
                self.default_directory, '_'.join(basename))
5218
aebd00a2d316 [fix] fix path unicity process in BytesFileSystemStorage.new_fs_path
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 5183
diff changeset
   230
            raise ValidationError(entity.eid, {role_name(attr, 'subject'): msg})
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   231
        return fspath
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   232
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   233
    def current_fs_path(self, entity, attr):
8131
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   234
        """return the current fs_path of the attribute, or None is the attr is
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   235
        not stored yet.
a6654712ad50 fix potential problems when BFSS uses a Windows SMB share (closes #2131435)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7695
diff changeset
   236
        """
9463
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8900
diff changeset
   237
        sysource = entity._cw.repo.system_source
4349
48dadeeacfa5 [bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4329
diff changeset
   238
        cu = sysource.doexec(entity._cw,
48dadeeacfa5 [bfss] make it works when adding/updating entities with an attribute using bfss
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4329
diff changeset
   239
                             'SELECT cw_%s FROM cw_%s WHERE cw_eid=%s' % (
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8695
diff changeset
   240
                             attr, entity.cw_etype, entity.eid))
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   241
        rawvalue = cu.fetchone()[0]
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   242
        if rawvalue is None: # no previous value
7449
5e45e32071aa [bfss] Make ``current_fs_path`` return None when an attribute has no value
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 7118
diff changeset
   243
            return None
5131
88b5ca8da928 [storages] fix fs_importing side-effect on entity.data
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5013
diff changeset
   244
        return sysource._process_value(rawvalue, cu.description[0],
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4721
diff changeset
   245
                                       binarywrap=str)
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   246
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   247
    def migrate_entity(self, entity, attribute):
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   248
        """migrate an entity attribute to the storage"""
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   249
        entity.cw_edited = EditedEntity(entity, **entity.cw_attr_cache)
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   250
        self.entity_added(entity, attribute)
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   251
        session = entity._cw
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   252
        source = session.repo.system_source
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   253
        attrs = source.preprocess_entity(entity)
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8695
diff changeset
   254
        sql = source.sqlgen.update('cw_' + entity.cw_etype, attrs,
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   255
                                   ['cw_eid'])
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5396
diff changeset
   256
        source.doexec(session, sql, attrs)
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   257
        entity.cw_edited = None
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   258
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   259
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   260
class AddFileOp(hook.DataOperationMixIn, hook.Operation):
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   261
    def rollback_event(self):
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   262
        for filepath in self.get_data():
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   263
            try:
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   264
                unlink(filepath)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8662
diff changeset
   265
            except Exception as ex:
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   266
                self.error('cant remove %s: %s' % (filepath, ex))
4322
f65743cc53e4 first draft for a simple hooks based custom attribute storage,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   267
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   268
class DeleteFileOp(hook.DataOperationMixIn, hook.Operation):
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5863
diff changeset
   269
    def postcommit_event(self):
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6401
diff changeset
   270
        for filepath in self.get_data():
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   271
            try:
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   272
                unlink(filepath)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8662
diff changeset
   273
            except Exception as ex:
5396
78d92a47a4e5 [bfss] use set_operation for AddFileOp/DeleteFileOp
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5219
diff changeset
   274
                self.error('cant remove %s: %s' % (filepath, ex))