--- a/doc/book/en/devrepo/testing.rst Mon Jul 06 17:39:35 2015 +0200
+++ /dev/null Thu Jan 01 00:00:00 1970 +0000
@@ -1,559 +0,0 @@
-.. -*- coding: utf-8 -*-
-
-Tests
-=====
-
-Unit tests
-----------
-
-The *CubicWeb* framework provides the
-:class:`cubicweb.devtools.testlib.CubicWebTC` test base class .
-
-Tests shall be put into the mycube/test directory. Additional test
-data shall go into mycube/test/data.
-
-It is much advised to write tests concerning entities methods,
-actions, hooks and operations, security. The
-:class:`~cubicweb.devtools.testlib.CubicWebTC` base class has
-convenience methods to help test all of this.
-
-In the realm of views, automatic tests check that views are valid
-XHTML. See :ref:`automatic_views_tests` for details.
-
-Most unit tests need a live database to work against. This is achieved
-by CubicWeb using automatically sqlite (bundled with Python, see
-http://docs.python.org/library/sqlite3.html) as a backend.
-
-The database is stored in the mycube/test/tmpdb,
-mycube/test/tmpdb-template files. If it does not (yet) exist, it will
-be built automatically when the test suite starts.
-
-.. warning::
-
- Whenever the schema changes (new entities, attributes, relations)
- one must delete these two files. Changes concerned only with entity
- or relation type properties (constraints, cardinalities,
- permissions) and generally dealt with using the
- `sync_schema_props_perms()` function of the migration environment do
- not need a database regeneration step.
-
-.. _hook_test:
-
-Unit test by example
-````````````````````
-
-We start with an example extracted from the keyword cube (available
-from http://www.cubicweb.org/project/cubicweb-keyword).
-
-.. sourcecode:: python
-
- from cubicweb.devtools.testlib import CubicWebTC
- from cubicweb import ValidationError
-
- class ClassificationHooksTC(CubicWebTC):
-
- def setup_database(self):
- with self.admin_access.repo_cnx() as cnx:
- group_etype = cnx.find('CWEType', name='CWGroup').one()
- c1 = cnx.create_entity('Classification', name=u'classif1',
- classifies=group_etype)
- user_etype = cnx.find('CWEType', name='CWUser').one()
- c2 = cnx.create_entity('Classification', name=u'classif2',
- classifies=user_etype)
- self.kw1eid = cnx.create_entity('Keyword', name=u'kwgroup', included_in=c1).eid
- cnx.commit()
-
- def test_cannot_create_cycles(self):
- with self.admin_access.repo_cnx() as cnx:
- kw1 = cnx.entity_from_eid(self.kw1eid)
- # direct obvious cycle
- with self.assertRaises(ValidationError):
- kw1.cw_set(subkeyword_of=kw1)
- cnx.rollback()
- # testing indirect cycles
- kw3 = cnx.execute('INSERT Keyword SK: SK name "kwgroup2", SK included_in C, '
- 'SK subkeyword_of K WHERE C name "classif1", K eid %(k)s'
- {'k': kw1}).get_entity(0,0)
- kw3.cw_set(reverse_subkeyword_of=kw1)
- self.assertRaises(ValidationError, cnx.commit)
-
-The test class defines a :meth:`setup_database` method which populates the
-database with initial data. Each test of the class runs with this
-pre-populated database.
-
-The test case itself checks that an Operation does its job of
-preventing cycles amongst Keyword entities.
-
-The `create_entity` method of connection (or request) objects allows
-to create an entity. You can link this entity to other entities, by
-specifying as argument, the relation name, and the entity to link, as
-value. In the above example, the `Classification` entity is linked to
-a `CWEtype` via the relation `classifies`. Conversely, if you are
-creating a `CWEtype` entity, you can link it to a `Classification`
-entity, by adding `reverse_classifies` as argument.
-
-.. note::
-
- the :meth:`commit` method is not called automatically. You have to
- call it explicitly if needed (notably to test operations). It is a
- good practice to regenerate entities with :meth:`entity_from_eid`
- after a commit to avoid request cache effects.
-
-You can see an example of security tests in the
-:ref:`adv_tuto_security`.
-
-It is possible to have these tests run continuously using `apycot`_.
-
-.. _apycot: http://www.cubicweb.org/project/apycot
-
-.. _securitytest:
-
-Managing connections or users
-+++++++++++++++++++++++++++++
-
-Since unit tests are done with the SQLITE backend and this does not
-support multiple connections at a time, you must be careful when
-simulating security, changing users.
-
-By default, tests run with a user with admin privileges. Connections
-using these credentials are accessible through the `admin_access` object
-of the test classes.
-
-The `repo_cnx()` method returns a connection object that can be used as a
-context manager:
-
-.. sourcecode:: python
-
- # admin_access is a pre-cooked session wrapping object
- # it is built with:
- # self.admin_access = self.new_access('admin')
- with self.admin_access.repo_cnx() as cnx:
- cnx.execute(...)
- self.create_user(cnx, login='user1')
- cnx.commit()
-
- user1access = self.new_access('user1')
- with user1access.web_request() as req:
- req.execute(...)
- req.cnx.commit()
-
-On exit of the context manager, a rollback is issued, which releases
-the connection. Don't forget to issue the `cnx.commit()` calls!
-
-.. warning::
-
- Do not use references kept to the entities created with a
- connection from another one!
-
-Email notifications tests
-`````````````````````````
-
-When running tests, potentially generated e-mails are not really sent
-but are found in the list `MAILBOX` of module
-:mod:`cubicweb.devtools.testlib`.
-
-You can test your notifications by analyzing the contents of this list, which
-contains objects with two attributes:
-
-* `recipients`, the list of recipients
-* `msg`, email.Message object
-
-Let us look at a simple example from the ``blog`` cube.
-
-.. sourcecode:: python
-
- from cubicweb.devtools.testlib import CubicWebTC, MAILBOX
-
- class BlogTestsCubicWebTC(CubicWebTC):
- """test blog specific behaviours"""
-
- def test_notifications(self):
- with self.admin_access.web_request() as req:
- cubicweb_blog = req.create_entity('Blog', title=u'cubicweb',
- description=u'cubicweb is beautiful')
- blog_entry_1 = req.create_entity('BlogEntry', title=u'hop',
- content=u'cubicweb hop')
- blog_entry_1.cw_set(entry_of=cubicweb_blog)
- blog_entry_2 = req.create_entity('BlogEntry', title=u'yes',
- content=u'cubicweb yes')
- blog_entry_2.cw_set(entry_of=cubicweb_blog)
- self.assertEqual(len(MAILBOX), 0)
- req.cnx.commit()
- self.assertEqual(len(MAILBOX), 2)
- mail = MAILBOX[0]
- self.assertEqual(mail.subject, '[data] hop')
- mail = MAILBOX[1]
- self.assertEqual(mail.subject, '[data] yes')
-
-Visible actions tests
-`````````````````````
-
-It is easy to write unit tests to test actions which are visible to
-a user or to a category of users. Let's take an example in the
-`conference cube`_.
-
-.. _`conference cube`: http://www.cubicweb.org/project/cubicweb-conference
-.. sourcecode:: python
-
- class ConferenceActionsTC(CubicWebTC):
-
- def setup_database(self):
- with self.admin_access.repo_cnx() as cnx:
- self.confeid = cnx.create_entity('Conference',
- title=u'my conf',
- url_id=u'conf',
- start_on=date(2010, 1, 27),
- end_on = date(2010, 1, 29),
- call_open=True,
- reverse_is_chair_at=chair,
- reverse_is_reviewer_at=reviewer).eid
-
- def test_admin(self):
- with self.admin_access.web_request() as req:
- rset = req.find('Conference').one()
- self.assertListEqual(self.pactions(req, rset),
- [('workflow', workflow.WorkflowActions),
- ('edit', confactions.ModifyAction),
- ('managepermission', actions.ManagePermissionsAction),
- ('addrelated', actions.AddRelatedActions),
- ('delete', actions.DeleteAction),
- ('generate_badge_action', badges.GenerateBadgeAction),
- ('addtalkinconf', confactions.AddTalkInConferenceAction)
- ])
- self.assertListEqual(self.action_submenu(req, rset, 'addrelated'),
- [(u'add Track in_conf Conference object',
- u'http://testing.fr/cubicweb/add/Track'
- u'?__linkto=in_conf%%3A%(conf)s%%3Asubject&'
- u'__redirectpath=conference%%2Fconf&'
- u'__redirectvid=' % {'conf': self.confeid}),
- ])
-
-You just have to execute a rql query corresponding to the view you want to test,
-and to compare the result of
-:meth:`~cubicweb.devtools.testlib.CubicWebTC.pactions` with the list of actions
-that must be visible in the interface. This is a list of tuples. The first
-element is the action's `__regid__`, the second the action's class.
-
-To test actions in a submenu, you just have to test the result of
-:meth:`~cubicweb.devtools.testlib.CubicWebTC.action_submenu` method. The last
-parameter of the method is the action's category. The result is a list of
-tuples. The first element is the action's title, and the second element the
-action's url.
-
-
-.. _automatic_views_tests:
-
-Automatic views testing
------------------------
-
-This is done automatically with the :class:`cubicweb.devtools.testlib.AutomaticWebTest`
-class. At cube creation time, the mycube/test/test_mycube.py file
-contains such a test. The code here has to be uncommented to be
-usable, without further modification.
-
-The ``auto_populate`` method uses a smart algorithm to create
-pseudo-random data in the database, thus enabling the views to be
-invoked and tested.
-
-Depending on the schema, hooks and operations constraints, it is not
-always possible for the automatic auto_populate to proceed.
-
-It is possible of course to completely redefine auto_populate. A
-lighter solution is to give hints (fill some class attributes) about
-what entities and relations have to be skipped by the auto_populate
-mechanism. These are:
-
-* `no_auto_populate`, may contain a list of entity types to skip
-* `ignored_relations`, may contain a list of relation types to skip
-* `application_rql`, may contain a list of rql expressions that
- auto_populate cannot guess by itself; these must yield resultsets
- against which views may be selected.
-
-.. warning::
-
- Take care to not let the imported `AutomaticWebTest` in your test module
- namespace, else both your subclass *and* this parent class will be run.
-
-Cache heavy database setup
--------------------------------
-
-Some test suites require a complex setup of the database that takes
-seconds (or even minutes) to complete. Doing the whole setup for each
-individual test makes the whole run very slow. The ``CubicWebTC``
-class offer a simple way to prepare a specific database once for
-multiple tests. The `test_db_id` class attribute of your
-``CubicWebTC`` subclass must be set to a unique identifier and the
-:meth:`pre_setup_database` class method must build the cached content. As
-the :meth:`pre_setup_database` method is not garanteed to be called
-every time a test method is run, you must not set any class attribute
-to be used during test *there*. Databases for each `test_db_id` are
-automatically created if not already in cache. Clearing the cache is
-up to the user. Cache files are found in the :file:`data/database`
-subdirectory of your test directory.
-
-.. warning::
-
- Take care to always have the same :meth:`pre_setup_database`
- function for all classes with a given `test_db_id` otherwise your
- tests will have unpredictable results depending on the first
- encountered one.
-
-
-Testing on a real-life database
--------------------------------
-
-The ``CubicWebTC`` class uses the `cubicweb.devtools.ApptestConfiguration`
-configuration class to setup its testing environment (database driver,
-user password, application home, and so on). The `cubicweb.devtools`
-module also provides a `RealDatabaseConfiguration`
-class that will read a regular cubicweb sources file to fetch all
-this information but will also prevent the database to be initalized
-and reset between tests.
-
-For a test class to use a specific configuration, you have to set
-the `_config` class attribute on the class as in:
-
-.. sourcecode:: python
-
- from cubicweb.devtools import RealDatabaseConfiguration
- from cubicweb.devtools.testlib import CubicWebTC
-
- class BlogRealDatabaseTC(CubicWebTC):
- _config = RealDatabaseConfiguration('blog',
- sourcefile='/path/to/realdb_sources')
-
- def test_blog_rss(self):
- with self.admin_access.web_request() as req:
- rset = req.execute('Any B ORDERBY D DESC WHERE B is BlogEntry, '
- 'B created_by U, U login "logilab", B creation_date D')
- self.view('rss', rset, req=req)
-
-
-Testing with other cubes
-------------------------
-
-Sometimes a small component cannot be tested all by itself, so one
-needs to specify other cubes to be used as part of the the unit test
-suite. This is handled by the ``bootstrap_cubes`` file located under
-``mycube/test/data``. One example from the `preview` cube::
-
- card, file, preview
-
-The format is:
-
-* possibly several empy lines or lines starting with ``#`` (comment lines)
-* one line containing a comma-separated list of cube names.
-
-It is also possible to add a ``schema.py`` file in
-``mycube/test/data``, which will be used by the testing framework,
-therefore making new entity types and relations available to the
-tests.
-
-Literate programming
---------------------
-
-CubicWeb provides some literate programming capabilities. The :ref:`cubicweb-ctl`
-`shell` command accepts different file formats. If your file ends with `.txt`
-or `.rst`, the file will be parsed by :mod:`doctest.testfile` with CubicWeb's
-:ref:`migration` API enabled in it.
-
-Create a `scenario.txt` file in the `test/` directory and fill with some content.
-Refer to the :mod:`doctest.testfile` `documentation`_.
-
-.. _documentation: http://docs.python.org/library/doctest.html
-
-Then, you can run it directly by::
-
- $ cubicweb-ctl shell <cube_instance> test/scenario.txt
-
-When your scenario file is ready, put it in a new test case to be able to run
-it automatically.
-
-.. sourcecode:: python
-
- from os.path import dirname, join
- from logilab.common.testlib import unittest_main
- from cubicweb.devtools.testlib import CubicWebTC
-
- class AcceptanceTC(CubicWebTC):
-
- def test_scenario(self):
- self.assertDocTestFile(join(dirname(__file__), 'scenario.txt'))
-
- if __name__ == '__main__':
- unittest_main()
-
-Skipping a scenario
-```````````````````
-
-If you want to set up initial conditions that you can't put in your unit test
-case, you have to use a :exc:`KeyboardInterrupt` exception only because of the
-way :mod:`doctest` module will catch all the exceptions internally.
-
- >>> if condition_not_met:
- ... raise KeyboardInterrupt('please, check your fixture.')
-
-Passing paramaters
-``````````````````
-Using extra arguments to parametrize your scenario is possible by prepending them
-by double dashes.
-
-Please refer to the `cubicweb-ctl shell --help` usage.
-
-.. important::
- Your scenario file must be utf-8 encoded.
-
-Test APIS
----------
-
-Using Pytest
-````````````
-
-The `pytest` utility (shipping with `logilab-common`_, which is a
-mandatory dependency of CubicWeb) extends the Python unittest
-functionality and is the preferred way to run the CubicWeb test
-suites. Bare unittests also work the usual way.
-
-.. _logilab-common: http://www.logilab.org/project/logilab-common
-
-To use it, you may:
-
-* just launch `pytest` in your cube to execute all tests (it will
- discover them automatically)
-* launch `pytest unittest_foo.py` to execute one test file
-* launch `pytest unittest_foo.py bar` to execute all test methods and
- all test cases whose name contains `bar`
-
-Additionally, the `-x` option tells pytest to exit at the first error
-or failure. The `-i` option tells pytest to drop into pdb whenever an
-exception occurs in a test.
-
-When the `-x` option has been used and the run stopped on a test, it
-is possible, after having fixed the test, to relaunch pytest with the
-`-R` option to tell it to start testing again from where it previously
-failed.
-
-Using the `TestCase` base class
-```````````````````````````````
-
-The base class of CubicWebTC is logilab.common.testlib.TestCase, which
-provides a lot of convenient assertion methods.
-
-.. autoclass:: logilab.common.testlib.TestCase
- :members:
-
-CubicWebTC API
-``````````````
-.. autoclass:: cubicweb.devtools.testlib.CubicWebTC
- :members:
-
-
-What you need to know about request and session
------------------------------------------------
-
-.. image:: ../images/request_session.png
-
-First, remember to think that some code run on a client side, some
-other on the repository side. More precisely:
-
-* client side: web interface, raw repoapi connection (cubicweb-ctl shell for
- instance);
-
-* repository side: RQL query execution, that may trigger hooks and operation.
-
-The client interacts with the repository through a repoapi connection.
-
-
-.. note::
-
- These distinctions are going to disappear in cubicweb 3.21 (if not
- before).
-
-A repoapi connection is tied to a session in the repository. The connection and
-request objects are inaccessible from repository code / the session object is
-inaccessible from client code (theoretically at least).
-
-The web interface provides a request class. That `request` object provides
-access to all cubicweb resources, eg:
-
-* the registry (which itself provides access to the schema and the
- configuration);
-
-* an underlying repoapi connection (when using req.execute, you actually call the
- repoapi);
-
-* other specific resources depending on the client type (url generation according
- to base url, form parameters, etc.).
-
-
-A `session` provides an api similar to a request regarding RQL execution and
-access to global resources (registry and all), but also has the following
-responsibilities:
-
-* handle transaction data, that will live during the time of a single
- transaction. This includes the database connections that will be used to
- execute RQL queries.
-
-* handle persistent data that may be used across different (web) requests
-
-* security and hooks control (not possible through a request)
-
-
-The `_cw` attribute
-```````````````````
-The `_cw` attribute available on every application object provides access to all
-cubicweb resources, i.e.:
-
-- For code running on the client side (eg web interface view), `_cw` is a request
- instance.
-
-- For code running on the repository side (hooks and operation), `_cw` is a
- Connection or Session instance.
-
-
-Beware some views may be called with a session (e.g. notifications) or with a
-request.
-
-
-Request, session and transaction
-````````````````````````````````
-
-In the web interface, an HTTP request is handled by a single request, which will
-be thrown away once the response is sent.
-
-The web publisher handles the transaction:
-
-* commit / rollback is done automatically
-
-* you should not commit / rollback explicitly, except if you really
- need it
-
-Let's detail the process:
-
-1. an incoming RQL query comes from a client to the web stack
-
-2. the web stack opens an authenticated database connection for the
- request, which is associated to a user session
-
-3. the query is executed (through the repository connection)
-
-4. this query may trigger hooks. Hooks and operations may execute some rql queries
- through `cnx.execute`.
-
-5. the repository gets the result of the query in 1. If it was a RQL read query,
- the database connection is released. If it was a write query, the connection
- is then tied to the session until the transaction is commited or rolled back.
-
-6. results are sent back to the client
-
-This implies several things:
-
-* when using a request, or code executed in hooks, this database
- connection handling is totally transparent
-
-* however, take care when writing tests: you are usually faking /
- testing both the server and the client side, so you have to decide
- when to use RepoAccess.client_cnx or RepoAccess.repo_cnx. Ask
- yourself "where will the code I want to test be running, client or
- repository side?". The response is usually: use a repo (since the
- "client connection" concept is going away in a couple of releases).