--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/book/devrepo/testing.rst Thu Jan 08 22:11:06 2015 +0100
@@ -0,0 +1,559 @@
+.. -*- coding: utf-8 -*-
+
+Tests
+=====
+
+Unit tests
+----------
+
+The *CubicWeb* framework provides the
+:class:`cubicweb.devtools.testlib.CubicWebTC` test base class .
+
+Tests shall be put into the mycube/test directory. Additional test
+data shall go into mycube/test/data.
+
+It is much advised to write tests concerning entities methods,
+actions, hooks and operations, security. The
+:class:`~cubicweb.devtools.testlib.CubicWebTC` base class has
+convenience methods to help test all of this.
+
+In the realm of views, automatic tests check that views are valid
+XHTML. See :ref:`automatic_views_tests` for details.
+
+Most unit tests need a live database to work against. This is achieved
+by CubicWeb using automatically sqlite (bundled with Python, see
+http://docs.python.org/library/sqlite3.html) as a backend.
+
+The database is stored in the mycube/test/tmpdb,
+mycube/test/tmpdb-template files. If it does not (yet) exist, it will
+be built automatically when the test suite starts.
+
+.. warning::
+
+ Whenever the schema changes (new entities, attributes, relations)
+ one must delete these two files. Changes concerned only with entity
+ or relation type properties (constraints, cardinalities,
+ permissions) and generally dealt with using the
+ `sync_schema_props_perms()` function of the migration environment do
+ not need a database regeneration step.
+
+.. _hook_test:
+
+Unit test by example
+````````````````````
+
+We start with an example extracted from the keyword cube (available
+from http://www.cubicweb.org/project/cubicweb-keyword).
+
+.. sourcecode:: python
+
+ from cubicweb.devtools.testlib import CubicWebTC
+ from cubicweb import ValidationError
+
+ class ClassificationHooksTC(CubicWebTC):
+
+ def setup_database(self):
+ with self.admin_access.repo_cnx() as cnx:
+ group_etype = cnx.find('CWEType', name='CWGroup').one()
+ c1 = cnx.create_entity('Classification', name=u'classif1',
+ classifies=group_etype)
+ user_etype = cnx.find('CWEType', name='CWUser').one()
+ c2 = cnx.create_entity('Classification', name=u'classif2',
+ classifies=user_etype)
+ self.kw1eid = cnx.create_entity('Keyword', name=u'kwgroup', included_in=c1).eid
+ cnx.commit()
+
+ def test_cannot_create_cycles(self):
+ with self.admin_access.repo_cnx() as cnx:
+ kw1 = cnx.entity_from_eid(self.kw1eid)
+ # direct obvious cycle
+ with self.assertRaises(ValidationError):
+ kw1.cw_set(subkeyword_of=kw1)
+ cnx.rollback()
+ # testing indirect cycles
+ kw3 = cnx.execute('INSERT Keyword SK: SK name "kwgroup2", SK included_in C, '
+ 'SK subkeyword_of K WHERE C name "classif1", K eid %(k)s'
+ {'k': kw1}).get_entity(0,0)
+ kw3.cw_set(reverse_subkeyword_of=kw1)
+ self.assertRaises(ValidationError, cnx.commit)
+
+The test class defines a :meth:`setup_database` method which populates the
+database with initial data. Each test of the class runs with this
+pre-populated database.
+
+The test case itself checks that an Operation does its job of
+preventing cycles amongst Keyword entities.
+
+The `create_entity` method of connection (or request) objects allows
+to create an entity. You can link this entity to other entities, by
+specifying as argument, the relation name, and the entity to link, as
+value. In the above example, the `Classification` entity is linked to
+a `CWEtype` via the relation `classifies`. Conversely, if you are
+creating a `CWEtype` entity, you can link it to a `Classification`
+entity, by adding `reverse_classifies` as argument.
+
+.. note::
+
+ the :meth:`commit` method is not called automatically. You have to
+ call it explicitly if needed (notably to test operations). It is a
+ good practice to regenerate entities with :meth:`entity_from_eid`
+ after a commit to avoid request cache effects.
+
+You can see an example of security tests in the
+:ref:`adv_tuto_security`.
+
+It is possible to have these tests run continuously using `apycot`_.
+
+.. _apycot: http://www.cubicweb.org/project/apycot
+
+.. _securitytest:
+
+Managing connections or users
++++++++++++++++++++++++++++++
+
+Since unit tests are done with the SQLITE backend and this does not
+support multiple connections at a time, you must be careful when
+simulating security, changing users.
+
+By default, tests run with a user with admin privileges. Connections
+using these credentials are accessible through the `admin_access` object
+of the test classes.
+
+The `repo_cnx()` method returns a connection object that can be used as a
+context manager:
+
+.. sourcecode:: python
+
+ # admin_access is a pre-cooked session wrapping object
+ # it is built with:
+ # self.admin_access = self.new_access('admin')
+ with self.admin_access.repo_cnx() as cnx:
+ cnx.execute(...)
+ self.create_user(cnx, login='user1')
+ cnx.commit()
+
+ user1access = self.new_access('user1')
+ with user1access.web_request() as req:
+ req.execute(...)
+ req.cnx.commit()
+
+On exit of the context manager, a rollback is issued, which releases
+the connection. Don't forget to issue the `cnx.commit()` calls!
+
+.. warning::
+
+ Do not use references kept to the entities created with a
+ connection from another one!
+
+Email notifications tests
+`````````````````````````
+
+When running tests, potentially generated e-mails are not really sent
+but are found in the list `MAILBOX` of module
+:mod:`cubicweb.devtools.testlib`.
+
+You can test your notifications by analyzing the contents of this list, which
+contains objects with two attributes:
+
+* `recipients`, the list of recipients
+* `msg`, email.Message object
+
+Let us look at a simple example from the ``blog`` cube.
+
+.. sourcecode:: python
+
+ from cubicweb.devtools.testlib import CubicWebTC, MAILBOX
+
+ class BlogTestsCubicWebTC(CubicWebTC):
+ """test blog specific behaviours"""
+
+ def test_notifications(self):
+ with self.admin_access.web_request() as req:
+ cubicweb_blog = req.create_entity('Blog', title=u'cubicweb',
+ description=u'cubicweb is beautiful')
+ blog_entry_1 = req.create_entity('BlogEntry', title=u'hop',
+ content=u'cubicweb hop')
+ blog_entry_1.cw_set(entry_of=cubicweb_blog)
+ blog_entry_2 = req.create_entity('BlogEntry', title=u'yes',
+ content=u'cubicweb yes')
+ blog_entry_2.cw_set(entry_of=cubicweb_blog)
+ self.assertEqual(len(MAILBOX), 0)
+ req.cnx.commit()
+ self.assertEqual(len(MAILBOX), 2)
+ mail = MAILBOX[0]
+ self.assertEqual(mail.subject, '[data] hop')
+ mail = MAILBOX[1]
+ self.assertEqual(mail.subject, '[data] yes')
+
+Visible actions tests
+`````````````````````
+
+It is easy to write unit tests to test actions which are visible to
+a user or to a category of users. Let's take an example in the
+`conference cube`_.
+
+.. _`conference cube`: http://www.cubicweb.org/project/cubicweb-conference
+.. sourcecode:: python
+
+ class ConferenceActionsTC(CubicWebTC):
+
+ def setup_database(self):
+ with self.admin_access.repo_cnx() as cnx:
+ self.confeid = cnx.create_entity('Conference',
+ title=u'my conf',
+ url_id=u'conf',
+ start_on=date(2010, 1, 27),
+ end_on = date(2010, 1, 29),
+ call_open=True,
+ reverse_is_chair_at=chair,
+ reverse_is_reviewer_at=reviewer).eid
+
+ def test_admin(self):
+ with self.admin_access.web_request() as req:
+ rset = req.find('Conference').one()
+ self.assertListEqual(self.pactions(req, rset),
+ [('workflow', workflow.WorkflowActions),
+ ('edit', confactions.ModifyAction),
+ ('managepermission', actions.ManagePermissionsAction),
+ ('addrelated', actions.AddRelatedActions),
+ ('delete', actions.DeleteAction),
+ ('generate_badge_action', badges.GenerateBadgeAction),
+ ('addtalkinconf', confactions.AddTalkInConferenceAction)
+ ])
+ self.assertListEqual(self.action_submenu(req, rset, 'addrelated'),
+ [(u'add Track in_conf Conference object',
+ u'http://testing.fr/cubicweb/add/Track'
+ u'?__linkto=in_conf%%3A%(conf)s%%3Asubject&'
+ u'__redirectpath=conference%%2Fconf&'
+ u'__redirectvid=' % {'conf': self.confeid}),
+ ])
+
+You just have to execute a rql query corresponding to the view you want to test,
+and to compare the result of
+:meth:`~cubicweb.devtools.testlib.CubicWebTC.pactions` with the list of actions
+that must be visible in the interface. This is a list of tuples. The first
+element is the action's `__regid__`, the second the action's class.
+
+To test actions in a submenu, you just have to test the result of
+:meth:`~cubicweb.devtools.testlib.CubicWebTC.action_submenu` method. The last
+parameter of the method is the action's category. The result is a list of
+tuples. The first element is the action's title, and the second element the
+action's url.
+
+
+.. _automatic_views_tests:
+
+Automatic views testing
+-----------------------
+
+This is done automatically with the :class:`cubicweb.devtools.testlib.AutomaticWebTest`
+class. At cube creation time, the mycube/test/test_mycube.py file
+contains such a test. The code here has to be uncommented to be
+usable, without further modification.
+
+The ``auto_populate`` method uses a smart algorithm to create
+pseudo-random data in the database, thus enabling the views to be
+invoked and tested.
+
+Depending on the schema, hooks and operations constraints, it is not
+always possible for the automatic auto_populate to proceed.
+
+It is possible of course to completely redefine auto_populate. A
+lighter solution is to give hints (fill some class attributes) about
+what entities and relations have to be skipped by the auto_populate
+mechanism. These are:
+
+* `no_auto_populate`, may contain a list of entity types to skip
+* `ignored_relations`, may contain a list of relation types to skip
+* `application_rql`, may contain a list of rql expressions that
+ auto_populate cannot guess by itself; these must yield resultsets
+ against which views may be selected.
+
+.. warning::
+
+ Take care to not let the imported `AutomaticWebTest` in your test module
+ namespace, else both your subclass *and* this parent class will be run.
+
+Cache heavy database setup
+-------------------------------
+
+Some test suites require a complex setup of the database that takes
+seconds (or even minutes) to complete. Doing the whole setup for each
+individual test makes the whole run very slow. The ``CubicWebTC``
+class offer a simple way to prepare a specific database once for
+multiple tests. The `test_db_id` class attribute of your
+``CubicWebTC`` subclass must be set to a unique identifier and the
+:meth:`pre_setup_database` class method must build the cached content. As
+the :meth:`pre_setup_database` method is not garanteed to be called
+every time a test method is run, you must not set any class attribute
+to be used during test *there*. Databases for each `test_db_id` are
+automatically created if not already in cache. Clearing the cache is
+up to the user. Cache files are found in the :file:`data/database`
+subdirectory of your test directory.
+
+.. warning::
+
+ Take care to always have the same :meth:`pre_setup_database`
+ function for all classes with a given `test_db_id` otherwise your
+ tests will have unpredictable results depending on the first
+ encountered one.
+
+
+Testing on a real-life database
+-------------------------------
+
+The ``CubicWebTC`` class uses the `cubicweb.devtools.ApptestConfiguration`
+configuration class to setup its testing environment (database driver,
+user password, application home, and so on). The `cubicweb.devtools`
+module also provides a `RealDatabaseConfiguration`
+class that will read a regular cubicweb sources file to fetch all
+this information but will also prevent the database to be initalized
+and reset between tests.
+
+For a test class to use a specific configuration, you have to set
+the `_config` class attribute on the class as in:
+
+.. sourcecode:: python
+
+ from cubicweb.devtools import RealDatabaseConfiguration
+ from cubicweb.devtools.testlib import CubicWebTC
+
+ class BlogRealDatabaseTC(CubicWebTC):
+ _config = RealDatabaseConfiguration('blog',
+ sourcefile='/path/to/realdb_sources')
+
+ def test_blog_rss(self):
+ with self.admin_access.web_request() as req:
+ rset = req.execute('Any B ORDERBY D DESC WHERE B is BlogEntry, '
+ 'B created_by U, U login "logilab", B creation_date D')
+ self.view('rss', rset, req=req)
+
+
+Testing with other cubes
+------------------------
+
+Sometimes a small component cannot be tested all by itself, so one
+needs to specify other cubes to be used as part of the the unit test
+suite. This is handled by the ``bootstrap_cubes`` file located under
+``mycube/test/data``. One example from the `preview` cube::
+
+ card, file, preview
+
+The format is:
+
+* possibly several empy lines or lines starting with ``#`` (comment lines)
+* one line containing a comma-separated list of cube names.
+
+It is also possible to add a ``schema.py`` file in
+``mycube/test/data``, which will be used by the testing framework,
+therefore making new entity types and relations available to the
+tests.
+
+Literate programming
+--------------------
+
+CubicWeb provides some literate programming capabilities. The :ref:`cubicweb-ctl`
+`shell` command accepts different file formats. If your file ends with `.txt`
+or `.rst`, the file will be parsed by :mod:`doctest.testfile` with CubicWeb's
+:ref:`migration` API enabled in it.
+
+Create a `scenario.txt` file in the `test/` directory and fill with some content.
+Refer to the :mod:`doctest.testfile` `documentation`_.
+
+.. _documentation: http://docs.python.org/library/doctest.html
+
+Then, you can run it directly by::
+
+ $ cubicweb-ctl shell <cube_instance> test/scenario.txt
+
+When your scenario file is ready, put it in a new test case to be able to run
+it automatically.
+
+.. sourcecode:: python
+
+ from os.path import dirname, join
+ from logilab.common.testlib import unittest_main
+ from cubicweb.devtools.testlib import CubicWebTC
+
+ class AcceptanceTC(CubicWebTC):
+
+ def test_scenario(self):
+ self.assertDocTestFile(join(dirname(__file__), 'scenario.txt'))
+
+ if __name__ == '__main__':
+ unittest_main()
+
+Skipping a scenario
+```````````````````
+
+If you want to set up initial conditions that you can't put in your unit test
+case, you have to use a :exc:`KeyboardInterrupt` exception only because of the
+way :mod:`doctest` module will catch all the exceptions internally.
+
+ >>> if condition_not_met:
+ ... raise KeyboardInterrupt('please, check your fixture.')
+
+Passing paramaters
+``````````````````
+Using extra arguments to parametrize your scenario is possible by prepending them
+by double dashes.
+
+Please refer to the `cubicweb-ctl shell --help` usage.
+
+.. important::
+ Your scenario file must be utf-8 encoded.
+
+Test APIS
+---------
+
+Using Pytest
+````````````
+
+The `pytest` utility (shipping with `logilab-common`_, which is a
+mandatory dependency of CubicWeb) extends the Python unittest
+functionality and is the preferred way to run the CubicWeb test
+suites. Bare unittests also work the usual way.
+
+.. _logilab-common: http://www.logilab.org/project/logilab-common
+
+To use it, you may:
+
+* just launch `pytest` in your cube to execute all tests (it will
+ discover them automatically)
+* launch `pytest unittest_foo.py` to execute one test file
+* launch `pytest unittest_foo.py bar` to execute all test methods and
+ all test cases whose name contains `bar`
+
+Additionally, the `-x` option tells pytest to exit at the first error
+or failure. The `-i` option tells pytest to drop into pdb whenever an
+exception occurs in a test.
+
+When the `-x` option has been used and the run stopped on a test, it
+is possible, after having fixed the test, to relaunch pytest with the
+`-R` option to tell it to start testing again from where it previously
+failed.
+
+Using the `TestCase` base class
+```````````````````````````````
+
+The base class of CubicWebTC is logilab.common.testlib.TestCase, which
+provides a lot of convenient assertion methods.
+
+.. autoclass:: logilab.common.testlib.TestCase
+ :members:
+
+CubicWebTC API
+``````````````
+.. autoclass:: cubicweb.devtools.testlib.CubicWebTC
+ :members:
+
+
+What you need to know about request and session
+-----------------------------------------------
+
+.. image:: ../images/request_session.png
+
+First, remember to think that some code run on a client side, some
+other on the repository side. More precisely:
+
+* client side: web interface, raw repoapi connection (cubicweb-ctl shell for
+ instance);
+
+* repository side: RQL query execution, that may trigger hooks and operation.
+
+The client interacts with the repository through a repoapi connection.
+
+
+.. note::
+
+ These distinctions are going to disappear in cubicweb 3.21 (if not
+ before).
+
+A repoapi connection is tied to a session in the repository. The connection and
+request objects are inaccessible from repository code / the session object is
+inaccessible from client code (theoretically at least).
+
+The web interface provides a request class. That `request` object provides
+access to all cubicweb resources, eg:
+
+* the registry (which itself provides access to the schema and the
+ configuration);
+
+* an underlying repoapi connection (when using req.execute, you actually call the
+ repoapi);
+
+* other specific resources depending on the client type (url generation according
+ to base url, form parameters, etc.).
+
+
+A `session` provides an api similar to a request regarding RQL execution and
+access to global resources (registry and all), but also has the following
+responsibilities:
+
+* handle transaction data, that will live during the time of a single
+ transaction. This includes the database connections that will be used to
+ execute RQL queries.
+
+* handle persistent data that may be used across different (web) requests
+
+* security and hooks control (not possible through a request)
+
+
+The `_cw` attribute
+```````````````````
+The `_cw` attribute available on every application object provides access to all
+cubicweb resources, i.e.:
+
+- For code running on the client side (eg web interface view), `_cw` is a request
+ instance.
+
+- For code running on the repository side (hooks and operation), `_cw` is a
+ Connection or Session instance.
+
+
+Beware some views may be called with a session (e.g. notifications) or with a
+request.
+
+
+Request, session and transaction
+````````````````````````````````
+
+In the web interface, an HTTP request is handled by a single request, which will
+be thrown away once the response is sent.
+
+The web publisher handles the transaction:
+
+* commit / rollback is done automatically
+
+* you should not commit / rollback explicitly, except if you really
+ need it
+
+Let's detail the process:
+
+1. an incoming RQL query comes from a client to the web stack
+
+2. the web stack opens an authenticated database connection for the
+ request, which is associated to a user session
+
+3. the query is executed (through the repository connection)
+
+4. this query may trigger hooks. Hooks and operations may execute some rql queries
+ through `cnx.execute`.
+
+5. the repository gets the result of the query in 1. If it was a RQL read query,
+ the database connection is released. If it was a write query, the connection
+ is then tied to the session until the transaction is commited or rolled back.
+
+6. results are sent back to the client
+
+This implies several things:
+
+* when using a request, or code executed in hooks, this database
+ connection handling is totally transparent
+
+* however, take care when writing tests: you are usually faking /
+ testing both the server and the client side, so you have to decide
+ when to use RepoAccess.client_cnx or RepoAccess.repo_cnx. Ask
+ yourself "where will the code I want to test be running, client or
+ repository side?". The response is usually: use a repo (since the
+ "client connection" concept is going away in a couple of releases).