diff -r 76ab3c71aff2 -r c67bcee93248 doc/tutorials/advanced/part02_security.rst
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/tutorials/advanced/part02_security.rst	Thu Jan 08 22:11:06 2015 +0100
@@ -0,0 +1,440 @@
+.. _TutosPhotoWebSiteSecurity:
+
+Security, testing and migration
+-------------------------------
+
+This part will cover various topics:
+
+* configuring security
+* migrating existing instance
+* writing some unit tests
+
+Here is the ``read`` security model I want:
+
+* folders, files, images and comments should have one of the following visibility:
+
+  - ``public``, everyone can see it
+  - ``authenticated``, only authenticated users can see it
+  - ``restricted``, only a subset of authenticated users can see it
+
+* managers (e.g. me) can see everything
+* only authenticated users can see people
+* everyone can see classifier entities, such as tag and zone
+
+Also, unless explicitly specified, the visibility of an image should be the same as
+its parent folder, as well as visibility of a comment should be the same as the
+commented entity. If there is no parent entity, the default visibility is
+``authenticated``.
+
+Regarding write security, that's much easier:
+* anonymous can't write anything
+* authenticated users can only add comment
+* managers will add the remaining stuff
+
+Now, let's implement that!
+
+Proper security in CubicWeb is done at the schema level, so you don't have to
+bother with it in views: users will only see what they can see automatically.
+
+.. _adv_tuto_security:
+
+Step 1: configuring security into the schema
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In schema, you can grant access according to groups, or to some RQL expressions:
+users get access if the expression returns some results. To implement the read
+security defined earlier, groups are not enough, we'll need some RQL expression. Here
+is the idea:
+
+* add a `visibility` attribute on Folder, File and Comment, which may be one of
+  the value explained above
+
+* add a `may_be_read_by` relation from Folder, File and Comment to users,
+  which will define who can see the entity
+
+* security propagation will be done in hook.
+
+So the first thing to do is to modify my cube's schema.py to define those
+relations:
+
+.. sourcecode:: python
+
+    from yams.constraints import StaticVocabularyConstraint
+
+    class visibility(RelationDefinition):
+	subject = ('Folder', 'File', 'Comment')
+	object = 'String'
+	constraints = [StaticVocabularyConstraint(('public', 'authenticated',
+						   'restricted', 'parent'))]
+	default = 'parent'
+	cardinality = '11' # required
+
+    class may_be_read_by(RelationDefinition):
+        __permissions__ = {
+	    'read':   ('managers', 'users'),
+	    'add':    ('managers',),
+	    'delete': ('managers',),
+	    }
+
+	subject = ('Folder', 'File', 'Comment',)
+	object = 'CWUser'
+
+We can note the following points:
+
+* we've added a new `visibility` attribute to folder, file, image and comment
+  using a `RelationDefinition`
+
+* `cardinality = '11'` means this attribute is required. This is usually hidden
+  under the `required` argument given to the `String` constructor, but we can
+  rely on this here (same thing for StaticVocabularyConstraint, which is usually
+  hidden by the `vocabulary` argument)
+
+* the `parent` possible value will be used for visibility propagation
+
+* think to secure the `may_be_read_by` permissions, else any user can add/delete it
+  by default, which somewhat breaks our security model...
+
+Now, we should be able to define security rules in the schema, based on these new
+attribute and relation. Here is the code to add to *schema.py*:
+
+.. sourcecode:: python
+
+    from cubicweb.schema import ERQLExpression
+
+    VISIBILITY_PERMISSIONS = {
+	'read':   ('managers',
+		   ERQLExpression('X visibility "public"'),
+		   ERQLExpression('X may_be_read_by U')),
+	'add':    ('managers',),
+	'update': ('managers', 'owners',),
+	'delete': ('managers', 'owners'),
+	}
+    AUTH_ONLY_PERMISSIONS = {
+	    'read':   ('managers', 'users'),
+	    'add':    ('managers',),
+	    'update': ('managers', 'owners',),
+	    'delete': ('managers', 'owners'),
+	    }
+    CLASSIFIERS_PERMISSIONS = {
+	    'read':   ('managers', 'users', 'guests'),
+	    'add':    ('managers',),
+	    'update': ('managers', 'owners',),
+	    'delete': ('managers', 'owners'),
+	    }
+
+    from cubes.folder.schema import Folder
+    from cubes.file.schema import File
+    from cubes.comment.schema import Comment
+    from cubes.person.schema import Person
+    from cubes.zone.schema import Zone
+    from cubes.tag.schema import Tag
+
+    Folder.__permissions__ = VISIBILITY_PERMISSIONS
+    File.__permissions__ = VISIBILITY_PERMISSIONS
+    Comment.__permissions__ = VISIBILITY_PERMISSIONS.copy()
+    Comment.__permissions__['add'] = ('managers', 'users',)
+    Person.__permissions__ = AUTH_ONLY_PERMISSIONS
+    Zone.__permissions__ = CLASSIFIERS_PERMISSIONS
+    Tag.__permissions__ = CLASSIFIERS_PERMISSIONS
+
+What's important in there:
+
+* `VISIBILITY_PERMISSIONS` provides read access to managers group, if
+  `visibility` attribute's value is 'public', or if user (designed by the 'U'
+  variable in the expression) is linked to the entity (the 'X' variable) through
+  the `may_be_read_by` permission
+
+* we modify permissions of the entity types we use by importing them and
+  modifying their `__permissions__` attribute
+
+* notice the `.copy()`: we only want to modify 'add' permission for `Comment`,
+  not for all entity types using `VISIBILITY_PERMISSIONS`!
+
+* the remaining part of the security model is done using regular groups:
+
+  - `users` is the group to which all authenticated users will belong
+  - `guests` is the group of anonymous users
+
+
+.. _adv_tuto_security_propagation:
+
+Step 2: security propagation in hooks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To fullfill the requirements, we have to implement::
+
+  Also, unless explicity specified, visibility of an image should be the same as
+  its parent folder, as well as visibility of a comment should be the same as the
+  commented entity.
+
+This kind of `active` rule will be done using CubicWeb's hook
+system. Hooks are triggered on database events such as addition of a new
+entity or relation.
+
+The tricky part of the requirement is in *unless explicitly specified*, notably
+because when the entity is added, we don't know yet its 'parent'
+entity (e.g. Folder of an File, File commented by a Comment). To handle such things,
+CubicWeb provides `Operation`, which allow to schedule things to do at commit time.
+
+In our case we will:
+
+* on entity creation, schedule an operation that will set default visibility
+
+* when a "parent" relation is added, propagate parent's visibility unless the
+  child already has a visibility set
+
+Here is the code in cube's *hooks.py*:
+
+.. sourcecode:: python
+
+    from cubicweb.predicates import is_instance
+    from cubicweb.server import hook
+
+    class SetVisibilityOp(hook.DataOperationMixIn, hook.Operation):
+
+	def precommit_event(self):
+	    for eid in self.get_data():
+		entity = self.session.entity_from_eid(eid)
+		if entity.visibility == 'parent':
+		    entity.cw_set(visibility=u'authenticated')
+
+    class SetVisibilityHook(hook.Hook):
+	__regid__ = 'sytweb.setvisibility'
+	__select__ = hook.Hook.__select__ & is_instance('Folder', 'File', 'Comment')
+	events = ('after_add_entity',)
+
+	def __call__(self):
+	    SetVisibilityOp.get_instance(self._cw).add_data(self.entity.eid)
+
+    class SetParentVisibilityHook(hook.Hook):
+	__regid__ = 'sytweb.setparentvisibility'
+	__select__ = hook.Hook.__select__ & hook.match_rtype('filed_under', 'comments')
+	events = ('after_add_relation',)
+
+	def __call__(self):
+	    parent = self._cw.entity_from_eid(self.eidto)
+	    child = self._cw.entity_from_eid(self.eidfrom)
+	    if child.visibility == 'parent':
+		child.cw_set(visibility=parent.visibility)
+
+Notice:
+
+* hooks are application objects, hence have selectors that should match entity or
+  relation types to which the hook applies. To match a relation type, we use the
+  hook specific `match_rtype` selector.
+
+* usage of `DataOperationMixIn`: instead of adding an operation for each added entity,
+  DataOperationMixIn allows to create a single one and to store entity's eids to be
+  processed in the transaction data. This is a good pratice to avoid heavy
+  operations manipulation cost when creating a lot of entities in the same
+  transaction.
+
+* the `precommit_event` method of the operation will be called at transaction's
+  commit time.
+
+* in a hook, `self._cw` is the repository session, not a web request as usually
+  in views
+
+* according to hook's event, you have access to different attributes on the hook
+  instance. Here:
+
+  - `self.entity` is the newly added entity on 'after_add_entity' events
+
+  - `self.eidfrom` / `self.eidto` are the eid of the subject / object entity on
+    'after_add_relation' events (you may also get the relation type using
+    `self.rtype`)
+
+The `parent` visibility value is used to tell "propagate using parent security"
+because we want that attribute to be required, so we can't use None value else
+we'll get an error before we get any chance to propagate...
+
+Now, we also want to propagate the `may_be_read_by` relation. Fortunately,
+CubicWeb provides some base hook classes for such things, so we only have to add
+the following code to *hooks.py*:
+
+.. sourcecode:: python
+
+    # relations where the "parent" entity is the subject
+    S_RELS = set()
+    # relations where the "parent" entity is the object
+    O_RELS = set(('filed_under', 'comments',))
+
+    class AddEntitySecurityPropagationHook(hook.PropagateRelationHook):
+	"""propagate permissions when new entity are added"""
+	__regid__ = 'sytweb.addentity_security_propagation'
+	__select__ = (hook.PropagateRelationHook.__select__
+		      & hook.match_rtype_sets(S_RELS, O_RELS))
+	main_rtype = 'may_be_read_by'
+	subject_relations = S_RELS
+	object_relations = O_RELS
+
+    class AddPermissionSecurityPropagationHook(hook.PropagateRelationAddHook):
+	"""propagate permissions when new entity are added"""
+	__regid__ = 'sytweb.addperm_security_propagation'
+	__select__ = (hook.PropagateRelationAddHook.__select__
+		      & hook.match_rtype('may_be_read_by',))
+	subject_relations = S_RELS
+	object_relations = O_RELS
+
+    class DelPermissionSecurityPropagationHook(hook.PropagateRelationDelHook):
+	__regid__ = 'sytweb.delperm_security_propagation'
+	__select__ = (hook.PropagateRelationDelHook.__select__
+		      & hook.match_rtype('may_be_read_by',))
+	subject_relations = S_RELS
+	object_relations = O_RELS
+
+* the `AddEntitySecurityPropagationHook` will propagate the relation
+  when `filed_under` or `comments` relations are added
+
+  - the `S_RELS` and `O_RELS` set as well as the `match_rtype_sets` selector are
+    used here so that if my cube is used by another one, it'll be able to
+    configure security propagation by simply adding relation to one of the two
+    sets.
+
+* the two others will propagate permissions changes on parent entities to
+  children entities
+
+
+.. _adv_tuto_tesing_security:
+
+Step 3: testing our security
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Security is tricky. Writing some tests for it is a very good idea. You should
+even write them first, as Test Driven Development recommends!
+
+Here is a small test case that will check the basis of our security
+model, in *test/unittest_sytweb.py*:
+
+.. sourcecode:: python
+
+    from cubicweb.devtools.testlib import CubicWebTC
+    from cubicweb import Binary
+
+    class SecurityTC(CubicWebTC):
+
+        def test_visibility_propagation(self):
+            with self.admin_access.repo_cnx() as cnx:
+                # create a user for later security checks
+                toto = self.create_user(cnx, 'toto')
+                cnx.commit()
+                # init some data using the default manager connection
+                folder = cnx.create_entity('Folder',
+                                           name=u'restricted',
+                                           visibility=u'restricted')
+                photo1 = cnx.create_entity('File',
+                                           data_name=u'photo1.jpg',
+                                           data=Binary('xxx'),
+                                           filed_under=folder)
+                cnx.commit()
+                # visibility propagation
+                self.assertEquals(photo1.visibility, 'restricted')
+                # unless explicitly specified
+                photo2 = cnx.create_entity('File',
+                                           data_name=u'photo2.jpg',
+                                           data=Binary('xxx'),
+                                           visibility=u'public',
+                                           filed_under=folder)
+                cnx.commit()
+                self.assertEquals(photo2.visibility, 'public')
+            with self.new_access('toto').repo_cnx() as cnx:
+                # test security
+                self.assertEqual(1, len(cnx.execute('File X'))) # only the public one
+                self.assertEqual(0, len(cnx.execute('Folder X'))) # restricted...
+            with self.admin_access.repo_cnx() as cnx:
+                # may_be_read_by propagation
+                folder = cnx.entity_from_eid(folder.eid)
+                folder.cw_set(may_be_read_by=toto)
+                cnx.commit()
+            with self.new_access('toto').repo_cnx() as cnx:
+                photo1 = cnx.entity_from_eid(photo1.eid)
+                self.failUnless(photo1.may_be_read_by)
+                # test security with permissions
+                self.assertEquals(2, len(cnx.execute('File X'))) # now toto has access to photo2
+                self.assertEquals(1, len(cnx.execute('Folder X'))) # and to restricted folder
+
+    if __name__ == '__main__':
+        from logilab.common.testlib import unittest_main
+        unittest_main()
+
+It's not complete, but shows most things you'll want to do in tests: adding some
+content, creating users and connecting as them in the test, etc...
+
+To run it type:
+
+.. sourcecode:: bash
+
+    $ pytest unittest_sytweb.py
+    ========================  unittest_sytweb.py  ========================
+    -> creating tables [....................]
+    -> inserting default user and default groups.
+    -> storing the schema in the database [....................]
+    -> database for instance data initialized.
+    .
+    ----------------------------------------------------------------------
+    Ran 1 test in 22.547s
+
+    OK
+
+
+The first execution is taking time, since it creates a sqlite database for the
+test instance. The second one will be much quicker:
+
+.. sourcecode:: bash
+
+    $ pytest unittest_sytweb.py
+    ========================  unittest_sytweb.py  ========================
+    .
+    ----------------------------------------------------------------------
+    Ran 1 test in 2.662s
+
+    OK
+
+If you do some changes in your schema, you'll have to force regeneration of that
+database. You do that by removing the tmpdb files before running the test: ::
+
+    $ rm data/database/tmpdb*
+
+
+.. Note::
+  pytest is a very convenient utility used to control test execution. It is available from the `logilab-common`_ package.
+
+.. _`logilab-common`: http://www.logilab.org/project/logilab-common
+
+.. _adv_tuto_migration_script:
+
+Step 4: writing the migration script and migrating the instance
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Prior to those changes, I created an instance, fed it with some data, so I
+don't want to create a new one, but to migrate the existing one. Let's see how to
+do that.
+
+Migration commands should be put in the cube's *migration* directory, in a
+file named file:`<X.Y.Z>_Any.py` ('Any' being there mostly for historical reasons).
+
+Here I'll create a *migration/0.2.0_Any.py* file containing the following
+instructions:
+
+.. sourcecode:: python
+
+  add_relation_type('may_be_read_by')
+  add_relation_type('visibility')
+  sync_schema_props_perms()
+
+Then I update the version number in the cube's *__pkginfo__.py* to 0.2.0. And
+that's it! Those instructions will:
+
+* update the instance's schema by adding our two new relations and update the
+  underlying database tables accordingly (the first two instructions)
+
+* update schema's permissions definition (the last instruction)
+
+
+To migrate my instance I simply type::
+
+   cubicweb-ctl upgrade sytweb_instance
+
+You'll then be asked some questions to do the migration step by step. You should say
+YES when it asks if a backup of your database should be done, so you can get back
+to initial state if anything goes wrong...