1 .. _advanced_tutorial:

    41 .. _adv_tuto_assemble_cubes:

    42 

    43 Step 2: pick building blocks into existing cubes

    44 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    45 

    46 Almost everything I want to handle in my web-site is somehow already modelized in

    47 existing cubes that I'll extend for my need. So I'll pick the following cubes:

    48 

    49 * `folder`, containing the `Folder` entity type, which will be used as

    50   both 'album' and a way to map file system folders. Entities are

    51   added to a given folder using the `filed_under` relation.

    52 

    53 * `file`, containing `File` and `Image` entity types, gallery view,

    54   and a file system import utility.

    55 

    56 * `zone`, containing the `Zone` entity type for hierarchical geographical

    57   zones. Entities (including sub-zones) are added to a given zone using the

    58   `situated_in` relation.

    59 

    60 * `person`, containing the `Person` entity type plus some basic views.

    61 

    62 * `comment`, providing a full commenting system allowing one to comment entity types

    63   supporting the `comments` relation by adding a `Comment` entity.

    64 

    65 * `tag`, providing a full tagging system as an easy and powerful way to classify

    66   entities supporting the `tags` relation by linking the to `Tag` entities. This

    67   will allows navigation into a large number of picture.

    68 

    69 Ok, now I'll tell my cube requires all this by editing :file:`cubes/sytweb/__pkginfo__.py`:

    70 

    71   .. sourcecode:: python

    72 

    73     __depends__ = {'cubicweb': '>= 3.8.0',

    74                    'cubicweb-file': '>= 1.2.0',

    75 		   'cubicweb-folder': '>= 1.1.0',

    76 		   'cubicweb-person': '>= 1.2.0',

    77 		   'cubicweb-comment': '>= 1.2.0',

    78 		   'cubicweb-tag': '>= 1.2.0',

    79 		   'cubicweb-zone': None}

    80 

    81 Notice that you can express minimal version of the cube that should be used,

    82 `None` meaning whatever version available. All packages starting with 'cubicweb-'

    83 will be recognized as being cube, not bare python packages. You can still specify

    84 this explicitly using instead the `__depends_cubes__` dictionary which should

    85 contains cube's name without the prefix. So the example below would be written

    86 as:

    87 

    88   .. sourcecode:: python

    89 

    90     __depends__ = {'cubicweb': '>= 3.8.0'}

    91     __depends_cubes__ = {'file': '>= 1.2.0',

    92 		         'folder': '>= 1.1.0',

    93 		   	 'person': '>= 1.2.0',

    94 		   	 'comment': '>= 1.2.0',

    95 		   	 'tag': '>= 1.2.0',

    96 		   	 'zone': None}

    97 

    98 

    99 Step 3: glue everything together in my cube's schema

   100 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   101 

   102 .. sourcecode:: python

   103 

   104     from yams.buildobjs import RelationDefinition

   105 

   106     class comments(RelationDefinition):

   107 	subject = 'Comment'

   108 	object = ('File', 'Image')

   109 	cardinality = '1*'

   110 	composite = 'object'

   111 

   112     class tags(RelationDefinition):

   113 	subject = 'Tag'

   114 	object = ('File', 'Image')

   115 

   116     class filed_under(RelationDefinition):

   117 	subject = ('File', 'Image')

   118 	object = 'Folder'

   119 

   120     class situated_in(RelationDefinition):

   121 	subject = 'Image'

   122 	object = 'Zone'

   123 

   124     class displayed_on(RelationDefinition):

   125 	subject = 'Person'

   126 	object = 'Image'

   127 

   128 

   129 This schema:

   130 

   131 * allows to comment and tag on `File` and `Image` entity types by adding the

   132   `comments` and `tags` relations. This should be all we've to do for this

   133   feature since the related cubes provide 'pluggable section' which are

   134   automatically displayed on the primary view of entity types supporting the

   135   relation.

   136 

   137 * adds a `situated_in` relation definition so that image entities can be

   138   geolocalized.

   139 

   140 * add a new relation `displayed_on` relation telling who can be seen on a

   141   picture.

   142 

   143 This schema will probably have to evolve as time goes (for security handling at

   144 least), but since the possibility to let a schema evolve is one of CubicWeb's

   145 features (and goals), we won't worry about it for now and see that later when needed.

   146 

   147 

   148 Step 4: creating the instance

   149 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   150 

   151 Now that I have a schema, I want to create an instance. To

   152 do so using this new 'sytweb' cube, I run::

   153 

   154   cubicweb-ctl create sytweb sytweb_instance

   155 

   156 Hint: if you get an error while the database is initialized, you can

   157 avoid having to answer the questions again by running::

   158 

   159    cubicweb-ctl db-create sytweb_instance

   160 

   161 This will use your already configured instance and start directly from the create

   162 database step, thus skipping questions asked by the 'create' command.

   163 

   164 Once the instance and database are fully initialized, run ::

   165 

   166   cubicweb-ctl start sytweb_instance

   167 

   168 to start the instance, check you can connect on it, etc...

   169 

   170 

   171 Security, testing and migration

   172 -------------------------------

   173 

   174 This part will cover various topics:

   175 

   176 * configuring security

   177 * migrating existing instance

   178 * writing some unit tests

   179 

   180 Here is the ``read`` security model I want:

   181 

   182 * folders, files, images and comments should have one of the following visibility:

   183 

   184   - ``public``, everyone can see it

   185   - ``authenticated``, only authenticated users can see it

   186   - ``restricted``, only a subset of authenticated users can see it

   187 

   188 * managers (e.g. me) can see everything

   189 * only authenticated users can see people

   190 * everyone can see classifier entities, such as tag and zone

   191 

   192 Also, unless explicitly specified, the visibility of an image should be the same as

   193 its parent folder, as well as visibility of a comment should be the same as the

   194 commented entity. If there is no parent entity, the default visibility is

   195 ``authenticated``.

   196 

   197 Regarding write security, that's much easier:

   198 * anonymous can't write anything

   199 * authenticated users can only add comment

   200 * managers will add the remaining stuff

   201 

   202 Now, let's implement that!

   203 

   204 Proper security in CubicWeb is done at the schema level, so you don't have to

   205 bother with it in views: users will only see what they can see automatically.

   206 

   207 .. _adv_tuto_security:

   208 

   209 Step 1: configuring security into the schema

   210 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   211 

   212 In schema, you can grant access according to groups, or to some RQL expressions:

   213 users get access if the expression returns some results. To implement the read

   214 security defined earlier, groups are not enough, we'll need some RQL expression. Here

   215 is the idea:

   216 

   217 * add a `visibility` attribute on Folder, Image and Comment, which may be one of

   218   the value explained above

   219 

   220 * add a `may_be_read_by` relation from Folder, Image and Comment to users,

   221   which will define who can see the entity

   222 

   223 * security propagation will be done in hook.

   224 

   225 So the first thing to do is to modify my cube's schema.py to define those

   226 relations:

   227 

   228 .. sourcecode:: python

   229 

   230     from yams.constraints import StaticVocabularyConstraint

   231 

   232     class visibility(RelationDefinition):

   233 	subject = ('Folder', 'File', 'Image', 'Comment')

   234 	object = 'String'

   235 	constraints = [StaticVocabularyConstraint(('public', 'authenticated',

   236 						   'restricted', 'parent'))]

   237 	default = 'parent'

   238 	cardinality = '11' # required

   239 

   240     class may_be_read_by(RelationDefinition):

   241         __permissions__ = {

   242 	    'read':   ('managers', 'users'),

   243 	    'add':    ('managers',),

   244 	    'delete': ('managers',),

   245 	    }

   246 

   247 	subject = ('Folder', 'File', 'Image', 'Comment',)

   248 	object = 'CWUser'

   249 

   250 We can note the following points:

   251 

   252 * we've added a new `visibility` attribute to folder, file, image and comment

   253   using a `RelationDefinition`

   254 

   255 * `cardinality = '11'` means this attribute is required. This is usually hidden

   256   under the `required` argument given to the `String` constructor, but we can

   257   rely on this here (same thing for StaticVocabularyConstraint, which is usually

   258   hidden by the `vocabulary` argument)

   259 

   260 * the `parent` possible value will be used for visibility propagation

   261 

   262 * think to secure the `may_be_read_by` permissions, else any user can add/delte it

   263   by default, which somewhat breaks our security model...

   264 

   265 Now, we should be able to define security rules in the schema, based on these new

   266 attribute and relation. Here is the code to add to *schema.py*:

   267 

   268 .. sourcecode:: python

   269 

   270     from cubicweb.schema import ERQLExpression

   271 

   272     VISIBILITY_PERMISSIONS = {

   273 	'read':   ('managers',

   274 		   ERQLExpression('X visibility "public"'),

   275 		   ERQLExpression('X may_be_read_by U')),

   276 	'add':    ('managers',),

   277 	'update': ('managers', 'owners',),

   278 	'delete': ('managers', 'owners'),

   279 	}

   280     AUTH_ONLY_PERMISSIONS = {

   281 	    'read':   ('managers', 'users'),

   282 	    'add':    ('managers',),

   283 	    'update': ('managers', 'owners',),

   284 	    'delete': ('managers', 'owners'),

   285 	    }

   286     CLASSIFIERS_PERMISSIONS = {

   287 	    'read':   ('managers', 'users', 'guests'),

   288 	    'add':    ('managers',),

   289 	    'update': ('managers', 'owners',),

   290 	    'delete': ('managers', 'owners'),

   291 	    }

   292 

   293     from cubes.folder.schema import Folder

   294     from cubes.file.schema import File, Image

   295     from cubes.comment.schema import Comment

   296     from cubes.person.schema import Person

   297     from cubes.zone.schema import Zone

   298     from cubes.tag.schema import Tag

   299 

   300     Folder.__permissions__ = VISIBILITY_PERMISSIONS

   301     File.__permissions__ = VISIBILITY_PERMISSIONS

   302     Image.__permissions__ = VISIBILITY_PERMISSIONS

   303     Comment.__permissions__ = VISIBILITY_PERMISSIONS.copy()

   304     Comment.__permissions__['add'] = ('managers', 'users',)

   305     Person.__permissions__ = AUTH_ONLY_PERMISSIONS

   306     Zone.__permissions__ = CLASSIFIERS_PERMISSIONS

   307     Tag.__permissions__ = CLASSIFIERS_PERMISSIONS

   308 

   309 What's important in there:

   310 

   311 * `VISIBILITY_PERMISSIONS` provides read access to managers group, if

   312   `visibility` attribute's value is 'public', or if user (designed by the 'U'

   313   variable in the expression) is linked to the entity (the 'X' variable) through

   314   the `may_read` permission

   315 

   316 * we modify permissions of the entity types we use by importing them and

   317   modifying their `__permissions__` attribute

   318 

   319 * notice the `.copy()`: we only want to modify 'add' permission for `Comment`,

   320   not for all entity types using `VISIBILITY_PERMISSIONS`!

   321 

   322 * the remaining part of the security model is done using regular groups:

   323 

   324   - `users` is the group to which all authenticated users will belong

   325   - `guests` is the group of anonymous users

   326 

   327 

   328 .. _adv_tuto_security_propagation:

   329 

   330 Step 2: security propagation in hooks

   331 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   332 

   333 To fullfill the requirements, we have to implement::

   334 

   335   Also, unless explicity specified, visibility of an image should be the same as

   336   its parent folder, as well as visibility of a comment should be the same as the

   337   commented entity.

   338 

   339 This kind of `active` rule will be done using CubicWeb's hook

   340 system. Hooks are triggered on database event such as addition of new

   341 entity or relation.

   342 

   343 The tricky part of the requirement is in *unless explicitly specified*, notably

   344 because when the entity is added, we don't know yet its 'parent'

   345 entity (e.g. Folder of an Image, Image commented by a Comment). To handle such things,

   346 CubicWeb provides `Operation`, which allow to schedule things to do at commit time.

   347 

   348 In our case we will:

   349 

   350 * on entity creation, schedule an operation that will set default visibility

   351 

   352 * when a "parent" relation is added, propagate parent's visibility unless the

   353   child already has a visibility set

   354 

   355 Here is the code in cube's *hooks.py*:

   356 

   357 .. sourcecode:: python

   358 

   359     from cubicweb.selectors import is_instance

   360     from cubicweb.server import hook

   361 

   362     class SetVisibilityOp(hook.Operation):

   363 	def precommit_event(self):

   364 	    for eid in self.session.transaction_data.pop('pending_visibility'):

   365 		entity = self.session.entity_from_eid(eid)

   366 		if entity.visibility == 'parent':

   367 		    entity.set_attributes(visibility=u'authenticated')

   368 

   369     class SetVisibilityHook(hook.Hook):

   370 	__regid__ = 'sytweb.setvisibility'

   371 	__select__ = hook.Hook.__select__ & is_instance('Folder', 'File', 'Image', 'Comment')

   372 	events = ('after_add_entity',)

   373 	def __call__(self):

   374 	    hook.set_operation(self._cw, 'pending_visibility', self.entity.eid,

   375 			       SetVisibilityOp)

   376 

   377     class SetParentVisibilityHook(hook.Hook):

   378 	__regid__ = 'sytweb.setparentvisibility'

   379 	__select__ = hook.Hook.__select__ & hook.match_rtype('filed_under', 'comments')

   380 	events = ('after_add_relation',)

   381 

   382 	def __call__(self):

   383 	    parent = self._cw.entity_from_eid(self.eidto)

   384 	    child = self._cw.entity_from_eid(self.eidfrom)

   385 	    if child.visibility == 'parent':

   386 		child.set_attributes(visibility=parent.visibility)

   387 

   388 Notice:

   389 

   390 * hooks are application objects, hence have selectors that should match entity or

   391   relation types to which the hook applies. To match a relation type, we use the

   392   hook specific `match_rtype` selector.

   393 

   394 * usage of `set_operation`: instead of adding an operation for each added entity,

   395   set_operation allows to create a single one and to store entity's eids to be

   396   processed in session's transaction data. This is a good pratice to avoid heavy

   397   operations manipulation cost when creating a lot of entities in the same

   398   transaction.

   399 

   400 * the `precommit_event` method of the operation will be called at transaction's

   401   commit time.

   402 

   403 * in a hook, `self._cw` is the repository session, not a web request as usually

   404   in views

   405 

   406 * according to hook's event, you have access to different attributes on the hook

   407   instance. Here:

   408 

   409   - `self.entity` is the newly added entity on 'after_add_entity' events

   410 

   411   - `self.eidfrom` / `self.eidto` are the eid of the subject / object entity on

   412     'after_add_relatiohn' events (you may also get the relation type using

   413     `self.rtype`)

   414 

   415 The `parent` visibility value is used to tell "propagate using parent security"

   416 because we want that attribute to be required, so we can't use None value else

   417 we'll get an error before we get any chance to propagate...

   418 

   419 Now, we also want to propagate the `may_be_read_by` relation. Fortunately,

   420 CubicWeb provides some base hook classes for such things, so we only have to add

   421 the following code to *hooks.py*:

   422 

   423 .. sourcecode:: python

   424 

   425     # relations where the "parent" entity is the subject

   426     S_RELS = set()

   427     # relations where the "parent" entity is the object

   428     O_RELS = set(('filed_under', 'comments',))

   429 

   430     class AddEntitySecurityPropagationHook(hook.PropagateSubjectRelationHook):

   431 	"""propagate permissions when new entity are added"""

   432 	__regid__ = 'sytweb.addentity_security_propagation'

   433 	__select__ = (hook.PropagateSubjectRelationHook.__select__

   434 		      & hook.match_rtype_sets(S_RELS, O_RELS))

   435 	main_rtype = 'may_be_read_by'

   436 	subject_relations = S_RELS

   437 	object_relations = O_RELS

   438 

   439     class AddPermissionSecurityPropagationHook(hook.PropagateSubjectRelationAddHook):

   440 	"""propagate permissions when new entity are added"""

   441 	__regid__ = 'sytweb.addperm_security_propagation'

   442 	__select__ = (hook.PropagateSubjectRelationAddHook.__select__

   443 		      & hook.match_rtype('may_be_read_by',))

   444 	subject_relations = S_RELS

   445 	object_relations = O_RELS

   446 

   447     class DelPermissionSecurityPropagationHook(hook.PropagateSubjectRelationDelHook):

   448 	__regid__ = 'sytweb.delperm_security_propagation'

   449 	__select__ = (hook.PropagateSubjectRelationDelHook.__select__

   450 		      & hook.match_rtype('may_be_read_by',))

   451 	subject_relations = S_RELS

   452 	object_relations = O_RELS

   453 

   454 * the `AddEntitySecurityPropagationHook` will propagate the relation

   455   when `filed_under` or `comments` relations are added

   456 

   457   - the `S_RELS` and `O_RELS` set as well as the `match_rtype_sets` selector are

   458     used here so that if my cube is used by another one, it'll be able to

   459     configure security propagation by simply adding relation to one of the two

   460     sets.

   461 

   462 * the two others will propagate permissions changes on parent entities to

   463   children entities

   464 

   465 

   466 .. _adv_tuto_tesing_security:

   467 

   468 Step 3: testing our security

   469 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   470 

   471 Security is tricky. Writing some tests for it is a very good idea. You should

   472 even write them first, as Test Driven Development recommends!

   473 

   474 Here is a small test case that will check the basis of our security

   475 model, in *test/unittest_sytweb.py*:

   476 

   477 .. sourcecode:: python

   478 

   479     from cubicweb.devtools.testlib import CubicWebTC

   480     from cubicweb import Binary

   481 

   482     class SecurityTC(CubicWebTC):

   483 

   484 	def test_visibility_propagation(self):

   485 	    # create a user for later security checks

   486 	    toto = self.create_user('toto')

   487 	    # init some data using the default manager connection

   488 	    req = self.request()

   489 	    folder = req.create_entity('Folder',

   490 				       name=u'restricted',

   491 				       visibility=u'restricted')

   492 	    photo1 = req.create_entity('Image',

   493 				       data_name=u'photo1.jpg',

   494 				       data=Binary('xxx'),

   495 				       filed_under=folder)

   496 	    self.commit()

   497 	    photo1.clear_all_caches() # good practice, avoid request cache effects

   498 	    # visibility propagation

   499 	    self.assertEquals(photo1.visibility, 'restricted')

   500 	    # unless explicitly specified

   501 	    photo2 = req.create_entity('Image',

   502 				       data_name=u'photo2.jpg',

   503 				       data=Binary('xxx'),

   504 				       visibility=u'public',

   505 				       filed_under=folder)

   506 	    self.commit()

   507 	    self.assertEquals(photo2.visibility, 'public')

   508 	    # test security

   509 	    self.login('toto')

   510 	    req = self.request()

   511 	    self.assertEquals(len(req.execute('Image X')), 1) # only the public one

   512 	    self.assertEquals(len(req.execute('Folder X')), 0) # restricted...

   513 	    # may_be_read_by propagation

   514 	    self.restore_connection()

   515 	    folder.set_relations(may_be_read_by=toto)

   516 	    self.commit()

   517 	    photo1.clear_all_caches()

   518 	    self.failUnless(photo1.may_be_read_by)

   519 	    # test security with permissions

   520 	    self.login('toto')

   521 	    req = self.request()

   522 	    self.assertEquals(len(req.execute('Image X')), 2) # now toto has access to photo2

   523 	    self.assertEquals(len(req.execute('Folder X')), 1) # and to restricted folder

   524 

   525     if __name__ == '__main__':

   526 	from logilab.common.testlib import unittest_main

   527 	unittest_main()

   528 

   529 It's not complete, but show most things you'll want to do in tests: adding some

   530 content, creating users and connecting as them in the test, etc...

   531 

   532 To run it type:

   533 

   534 .. sourcecode:: bash

   535 

   536     $ pytest unittest_sytweb.py

   537     ========================  unittest_sytweb.py  ========================

   538     -> creating tables [....................]

   539     -> inserting default user and default groups.

   540     -> storing the schema in the database [....................]

   541     -> database for instance data initialized.

   542     .

   543     ----------------------------------------------------------------------

   544     Ran 1 test in 22.547s

   545 

   546     OK

   547 

   548 

   549 The first execution is taking time, since it creates a sqlite database for the

   550 test instance. The second one will be much quicker:

   551 

   552 .. sourcecode:: bash

   553     

   554     $ pytest unittest_sytweb.py

   555     ========================  unittest_sytweb.py  ========================

   556     .

   557     ----------------------------------------------------------------------

   558     Ran 1 test in 2.662s

   559 

   560     OK

   561 

   562 If you do some changes in your schema, you'll have to force regeneration of that

   563 database. You do that by removing the tmpdb files before running the test: ::

   564 

   565     $ rm data/tmpdb*

   566 

   567 

   568 .. Note::

   569   pytest is a very convenient utility used to control test execution. It is available from the `logilab-common`_ package.

   570 

   571 .. _`logilab-common`: http://www.logilab.org/project/logilab-common

   572 

   573 .. _adv_tuto_migration_script:

   574 

   575 Step 4: writing the migration script and migrating the instance

   576 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   577 

   578 Prior to those changes, I  created an instance, feeded it with some data, so I

   579 don't want to create a new one, but to migrate the existing one. Let's see how to

   580 do that.

   581 

   582 Migration commands should be put in the cube's *migration* directory, in a

   583 file named file:`<X.Y.Z>_Any.py` ('Any' being there mostly for historical reason).

   584 

   585 Here I'll create a *migration/0.2.0_Any.py* file containing the following

   586 instructions:

   587 

   588 .. sourcecode:: python

   589 

   590   add_relation_type('may_be_read_by')

   591   add_relation_type('visibility')

   592   sync_schema_props_perms()

   593 

   594 Then I update the version number in cube's *__pkginfo__.py* to 0.2.0. And

   595 that's it! Those instructions will:

   596 

   597 * update the instance's schema by adding our two new relations and update the

   598   underlying database tables accordingly (the two first instructions)

   599 

   600 * update schema's permissions definition (the last instruction)

   601 

   602 

   603 To migrate my instance I simply type::

   604 

   605    cubicweb-ctl upgrade sytweb

   606 

   607 I'll then be asked some questions to do the migration step by step. You should say

   608 YES when it asks if a backup of your database should be done, so you can get back

   609 to initial state if anything goes wrong...

   610