1 .. _TutosPhotoWebSiteSecurity:

     2 

     3 Security, testing and migration

     4 -------------------------------

     5 

     6 This part will cover various topics:

     7 

     8 * configuring security

     9 * migrating existing instance

    10 * writing some unit tests

    11 

    12 Here is the ``read`` security model I want:

    13 

    14 * folders, files, images and comments should have one of the following visibility:

    15 

    16   - ``public``, everyone can see it

    17   - ``authenticated``, only authenticated users can see it

    18   - ``restricted``, only a subset of authenticated users can see it

    19 

    20 * managers (e.g. me) can see everything

    21 * only authenticated users can see people

    22 * everyone can see classifier entities, such as tag and zone

    23 

    24 Also, unless explicitly specified, the visibility of an image should be the same as

    25 its parent folder, as well as visibility of a comment should be the same as the

    26 commented entity. If there is no parent entity, the default visibility is

    27 ``authenticated``.

    28 

    29 Regarding write security, that's much easier:

    30 * anonymous can't write anything

    31 * authenticated users can only add comment

    32 * managers will add the remaining stuff

    33 

    34 Now, let's implement that!

    35 

    36 Proper security in CubicWeb is done at the schema level, so you don't have to

    37 bother with it in views: users will only see what they can see automatically.

    38 

    39 .. _adv_tuto_security:

    40 

    41 Step 1: configuring security into the schema

    42 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    43 

    44 In schema, you can grant access according to groups, or to some RQL expressions:

    45 users get access if the expression returns some results. To implement the read

    46 security defined earlier, groups are not enough, we'll need some RQL expression. Here

    47 is the idea:

    48 

    49 * add a `visibility` attribute on Folder, Image and Comment, which may be one of

    50   the value explained above

    51 

    52 * add a `may_be_read_by` relation from Folder, Image and Comment to users,

    53   which will define who can see the entity

    54 

    55 * security propagation will be done in hook.

    56 

    57 So the first thing to do is to modify my cube's schema.py to define those

    58 relations:

    59 

    60 .. sourcecode:: python

    61 

    62     from yams.constraints import StaticVocabularyConstraint

    63 

    64     class visibility(RelationDefinition):

    65 	subject = ('Folder', 'File', 'Image', 'Comment')

    66 	object = 'String'

    67 	constraints = [StaticVocabularyConstraint(('public', 'authenticated',

    68 						   'restricted', 'parent'))]

    69 	default = 'parent'

    70 	cardinality = '11' # required

    71 

    72     class may_be_read_by(RelationDefinition):

    73         __permissions__ = {

    74 	    'read':   ('managers', 'users'),

    75 	    'add':    ('managers',),

    76 	    'delete': ('managers',),

    77 	    }

    78 

    79 	subject = ('Folder', 'File', 'Image', 'Comment',)

    80 	object = 'CWUser'

    81 

    82 We can note the following points:

    83 

    84 * we've added a new `visibility` attribute to folder, file, image and comment

    85   using a `RelationDefinition`

    86 

    87 * `cardinality = '11'` means this attribute is required. This is usually hidden

    88   under the `required` argument given to the `String` constructor, but we can

    89   rely on this here (same thing for StaticVocabularyConstraint, which is usually

    90   hidden by the `vocabulary` argument)

    91 

    92 * the `parent` possible value will be used for visibility propagation

    93 

    94 * think to secure the `may_be_read_by` permissions, else any user can add/delete it

    95   by default, which somewhat breaks our security model...

    96 

    97 Now, we should be able to define security rules in the schema, based on these new

    98 attribute and relation. Here is the code to add to *schema.py*:

    99 

   100 .. sourcecode:: python

   101 

   102     from cubicweb.schema import ERQLExpression

   103 

   104     VISIBILITY_PERMISSIONS = {

   105 	'read':   ('managers',

   106 		   ERQLExpression('X visibility "public"'),

   107 		   ERQLExpression('X may_be_read_by U')),

   108 	'add':    ('managers',),

   109 	'update': ('managers', 'owners',),

   110 	'delete': ('managers', 'owners'),

   111 	}

   112     AUTH_ONLY_PERMISSIONS = {

   113 	    'read':   ('managers', 'users'),

   114 	    'add':    ('managers',),

   115 	    'update': ('managers', 'owners',),

   116 	    'delete': ('managers', 'owners'),

   117 	    }

   118     CLASSIFIERS_PERMISSIONS = {

   119 	    'read':   ('managers', 'users', 'guests'),

   120 	    'add':    ('managers',),

   121 	    'update': ('managers', 'owners',),

   122 	    'delete': ('managers', 'owners'),

   123 	    }

   124 

   125     from cubes.folder.schema import Folder

   126     from cubes.file.schema import File, Image

   127     from cubes.comment.schema import Comment

   128     from cubes.person.schema import Person

   129     from cubes.zone.schema import Zone

   130     from cubes.tag.schema import Tag

   131 

   132     Folder.__permissions__ = VISIBILITY_PERMISSIONS

   133     File.__permissions__ = VISIBILITY_PERMISSIONS

   134     Image.__permissions__ = VISIBILITY_PERMISSIONS

   135     Comment.__permissions__ = VISIBILITY_PERMISSIONS.copy()

   136     Comment.__permissions__['add'] = ('managers', 'users',)

   137     Person.__permissions__ = AUTH_ONLY_PERMISSIONS

   138     Zone.__permissions__ = CLASSIFIERS_PERMISSIONS

   139     Tag.__permissions__ = CLASSIFIERS_PERMISSIONS

   140 

   141 What's important in there:

   142 

   143 * `VISIBILITY_PERMISSIONS` provides read access to managers group, if

   144   `visibility` attribute's value is 'public', or if user (designed by the 'U'

   145   variable in the expression) is linked to the entity (the 'X' variable) through

   146   the `may_read` permission

   147 

   148 * we modify permissions of the entity types we use by importing them and

   149   modifying their `__permissions__` attribute

   150 

   151 * notice the `.copy()`: we only want to modify 'add' permission for `Comment`,

   152   not for all entity types using `VISIBILITY_PERMISSIONS`!

   153 

   154 * the remaining part of the security model is done using regular groups:

   155 

   156   - `users` is the group to which all authenticated users will belong

   157   - `guests` is the group of anonymous users

   158 

   159 

   160 .. _adv_tuto_security_propagation:

   161 

   162 Step 2: security propagation in hooks

   163 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   164 

   165 To fullfill the requirements, we have to implement::

   166 

   167   Also, unless explicity specified, visibility of an image should be the same as

   168   its parent folder, as well as visibility of a comment should be the same as the

   169   commented entity.

   170 

   171 This kind of `active` rule will be done using CubicWeb's hook

   172 system. Hooks are triggered on database event such as addition of new

   173 entity or relation.

   174 

   175 The tricky part of the requirement is in *unless explicitly specified*, notably

   176 because when the entity is added, we don't know yet its 'parent'

   177 entity (e.g. Folder of an Image, Image commented by a Comment). To handle such things,

   178 CubicWeb provides `Operation`, which allow to schedule things to do at commit time.

   179 

   180 In our case we will:

   181 

   182 * on entity creation, schedule an operation that will set default visibility

   183 

   184 * when a "parent" relation is added, propagate parent's visibility unless the

   185   child already has a visibility set

   186 

   187 Here is the code in cube's *hooks.py*:

   188 

   189 .. sourcecode:: python

   190 

   191     from cubicweb.selectors import is_instance

   192     from cubicweb.server import hook

   193 

   194     class SetVisibilityOp(hook.Operation):

   195 	def precommit_event(self):

   196 	    for eid in self.session.transaction_data.pop('pending_visibility'):

   197 		entity = self.session.entity_from_eid(eid)

   198 		if entity.visibility == 'parent':

   199 		    entity.set_attributes(visibility=u'authenticated')

   200 

   201     class SetVisibilityHook(hook.Hook):

   202 	__regid__ = 'sytweb.setvisibility'

   203 	__select__ = hook.Hook.__select__ & is_instance('Folder', 'File', 'Image', 'Comment')

   204 	events = ('after_add_entity',)

   205 	def __call__(self):

   206 	    hook.set_operation(self._cw, 'pending_visibility', self.entity.eid,

   207 			       SetVisibilityOp)

   208 

   209     class SetParentVisibilityHook(hook.Hook):

   210 	__regid__ = 'sytweb.setparentvisibility'

   211 	__select__ = hook.Hook.__select__ & hook.match_rtype('filed_under', 'comments')

   212 	events = ('after_add_relation',)

   213 

   214 	def __call__(self):

   215 	    parent = self._cw.entity_from_eid(self.eidto)

   216 	    child = self._cw.entity_from_eid(self.eidfrom)

   217 	    if child.visibility == 'parent':

   218 		child.set_attributes(visibility=parent.visibility)

   219 

   220 Notice:

   221 

   222 * hooks are application objects, hence have selectors that should match entity or

   223   relation types to which the hook applies. To match a relation type, we use the

   224   hook specific `match_rtype` selector.

   225 

   226 * usage of `set_operation`: instead of adding an operation for each added entity,

   227   set_operation allows to create a single one and to store entity's eids to be

   228   processed in session's transaction data. This is a good pratice to avoid heavy

   229   operations manipulation cost when creating a lot of entities in the same

   230   transaction.

   231 

   232 * the `precommit_event` method of the operation will be called at transaction's

   233   commit time.

   234 

   235 * in a hook, `self._cw` is the repository session, not a web request as usually

   236   in views

   237 

   238 * according to hook's event, you have access to different attributes on the hook

   239   instance. Here:

   240 

   241   - `self.entity` is the newly added entity on 'after_add_entity' events

   242 

   243   - `self.eidfrom` / `self.eidto` are the eid of the subject / object entity on

   244     'after_add_relatiohn' events (you may also get the relation type using

   245     `self.rtype`)

   246 

   247 The `parent` visibility value is used to tell "propagate using parent security"

   248 because we want that attribute to be required, so we can't use None value else

   249 we'll get an error before we get any chance to propagate...

   250 

   251 Now, we also want to propagate the `may_be_read_by` relation. Fortunately,

   252 CubicWeb provides some base hook classes for such things, so we only have to add

   253 the following code to *hooks.py*:

   254 

   255 .. sourcecode:: python

   256 

   257     # relations where the "parent" entity is the subject

   258     S_RELS = set()

   259     # relations where the "parent" entity is the object

   260     O_RELS = set(('filed_under', 'comments',))

   261 

   262     class AddEntitySecurityPropagationHook(hook.PropagateSubjectRelationHook):

   263 	"""propagate permissions when new entity are added"""

   264 	__regid__ = 'sytweb.addentity_security_propagation'

   265 	__select__ = (hook.PropagateSubjectRelationHook.__select__

   266 		      & hook.match_rtype_sets(S_RELS, O_RELS))

   267 	main_rtype = 'may_be_read_by'

   268 	subject_relations = S_RELS

   269 	object_relations = O_RELS

   270 

   271     class AddPermissionSecurityPropagationHook(hook.PropagateSubjectRelationAddHook):

   272 	"""propagate permissions when new entity are added"""

   273 	__regid__ = 'sytweb.addperm_security_propagation'

   274 	__select__ = (hook.PropagateSubjectRelationAddHook.__select__

   275 		      & hook.match_rtype('may_be_read_by',))

   276 	subject_relations = S_RELS

   277 	object_relations = O_RELS

   278 

   279     class DelPermissionSecurityPropagationHook(hook.PropagateSubjectRelationDelHook):

   280 	__regid__ = 'sytweb.delperm_security_propagation'

   281 	__select__ = (hook.PropagateSubjectRelationDelHook.__select__

   282 		      & hook.match_rtype('may_be_read_by',))

   283 	subject_relations = S_RELS

   284 	object_relations = O_RELS

   285 

   286 * the `AddEntitySecurityPropagationHook` will propagate the relation

   287   when `filed_under` or `comments` relations are added

   288 

   289   - the `S_RELS` and `O_RELS` set as well as the `match_rtype_sets` selector are

   290     used here so that if my cube is used by another one, it'll be able to

   291     configure security propagation by simply adding relation to one of the two

   292     sets.

   293 

   294 * the two others will propagate permissions changes on parent entities to

   295   children entities

   296 

   297 

   298 .. _adv_tuto_tesing_security:

   299 

   300 Step 3: testing our security

   301 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   302 

   303 Security is tricky. Writing some tests for it is a very good idea. You should

   304 even write them first, as Test Driven Development recommends!

   305 

   306 Here is a small test case that will check the basis of our security

   307 model, in *test/unittest_sytweb.py*:

   308 

   309 .. sourcecode:: python

   310 

   311     from cubicweb.devtools.testlib import CubicWebTC

   312     from cubicweb import Binary

   313 

   314     class SecurityTC(CubicWebTC):

   315 

   316 	def test_visibility_propagation(self):

   317 	    # create a user for later security checks

   318 	    toto = self.create_user('toto')

   319 	    # init some data using the default manager connection

   320 	    req = self.request()

   321 	    folder = req.create_entity('Folder',

   322 				       name=u'restricted',

   323 				       visibility=u'restricted')

   324 	    photo1 = req.create_entity('Image',

   325 				       data_name=u'photo1.jpg',

   326 				       data=Binary('xxx'),

   327 				       filed_under=folder)

   328 	    self.commit()

   329 	    photo1.clear_all_caches() # good practice, avoid request cache effects

   330 	    # visibility propagation

   331 	    self.assertEquals(photo1.visibility, 'restricted')

   332 	    # unless explicitly specified

   333 	    photo2 = req.create_entity('Image',

   334 				       data_name=u'photo2.jpg',

   335 				       data=Binary('xxx'),

   336 				       visibility=u'public',

   337 				       filed_under=folder)

   338 	    self.commit()

   339 	    self.assertEquals(photo2.visibility, 'public')

   340 	    # test security

   341 	    self.login('toto')

   342 	    req = self.request()

   343 	    self.assertEquals(len(req.execute('Image X')), 1) # only the public one

   344 	    self.assertEquals(len(req.execute('Folder X')), 0) # restricted...

   345 	    # may_be_read_by propagation

   346 	    self.restore_connection()

   347 	    folder.set_relations(may_be_read_by=toto)

   348 	    self.commit()

   349 	    photo1.clear_all_caches()

   350 	    self.failUnless(photo1.may_be_read_by)

   351 	    # test security with permissions

   352 	    self.login('toto')

   353 	    req = self.request()

   354 	    self.assertEquals(len(req.execute('Image X')), 2) # now toto has access to photo2

   355 	    self.assertEquals(len(req.execute('Folder X')), 1) # and to restricted folder

   356 

   357     if __name__ == '__main__':

   358 	from logilab.common.testlib import unittest_main

   359 	unittest_main()

   360 

   361 It's not complete, but show most things you'll want to do in tests: adding some

   362 content, creating users and connecting as them in the test, etc...

   363 

   364 To run it type:

   365 

   366 .. sourcecode:: bash

   367 

   368     $ pytest unittest_sytweb.py

   369     ========================  unittest_sytweb.py  ========================

   370     -> creating tables [....................]

   371     -> inserting default user and default groups.

   372     -> storing the schema in the database [....................]

   373     -> database for instance data initialized.

   374     .

   375     ----------------------------------------------------------------------

   376     Ran 1 test in 22.547s

   377 

   378     OK

   379 

   380 

   381 The first execution is taking time, since it creates a sqlite database for the

   382 test instance. The second one will be much quicker:

   383 

   384 .. sourcecode:: bash

   385     

   386     $ pytest unittest_sytweb.py

   387     ========================  unittest_sytweb.py  ========================

   388     .

   389     ----------------------------------------------------------------------

   390     Ran 1 test in 2.662s

   391 

   392     OK

   393 

   394 If you do some changes in your schema, you'll have to force regeneration of that

   395 database. You do that by removing the tmpdb files before running the test: ::

   396 

   397     $ rm data/tmpdb*

   398 

   399 

   400 .. Note::

   401   pytest is a very convenient utility used to control test execution. It is available from the `logilab-common`_ package.

   402 

   403 .. _`logilab-common`: http://www.logilab.org/project/logilab-common

   404 

   405 .. _adv_tuto_migration_script:

   406 

   407 Step 4: writing the migration script and migrating the instance

   408 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   409 

   410 Prior to those changes, I  created an instance, feeded it with some data, so I

   411 don't want to create a new one, but to migrate the existing one. Let's see how to

   412 do that.

   413 

   414 Migration commands should be put in the cube's *migration* directory, in a

   415 file named file:`<X.Y.Z>_Any.py` ('Any' being there mostly for historical reason).

   416 

   417 Here I'll create a *migration/0.2.0_Any.py* file containing the following

   418 instructions:

   419 

   420 .. sourcecode:: python

   421 

   422   add_relation_type('may_be_read_by')

   423   add_relation_type('visibility')

   424   sync_schema_props_perms()

   425 

   426 Then I update the version number in cube's *__pkginfo__.py* to 0.2.0. And

   427 that's it! Those instructions will:

   428 

   429 * update the instance's schema by adding our two new relations and update the

   430   underlying database tables accordingly (the two first instructions)

   431 

   432 * update schema's permissions definition (the last instruction)

   433 

   434 

   435 To migrate my instance I simply type::

   436 

   437    cubicweb-ctl upgrade sytweb

   438 

   439 You'll then be asked some questions to do the migration step by step. You should say

   440 YES when it asks if a backup of your database should be done, so you can get back

   441 to initial state if anything goes wrong...