1 .. -*- coding: utf-8 -*-

     2 

     3 Tests

     4 =====

     5 

     6 Unit tests

     7 ----------

     8 

     9 The *CubicWeb* framework provides the

    10 :class:`cubicweb.devtools.testlib.CubicWebTC` test base class .

    11 

    12 Tests shall be put into the mycube/test directory. Additional test

    13 data shall go into mycube/test/data.

    14 

    15 It is much advised to write tests concerning entities methods,

    16 actions, hooks and operations, security. The

    17 :class:`~cubicweb.devtools.testlib.CubicWebTC` base class has

    18 convenience methods to help test all of this.

    19 

    20 In the realm of views, automatic tests check that views are valid

    21 XHTML. See :ref:`automatic_views_tests` for details.

    22 

    23 Most unit tests need a live database to work against. This is achieved

    24 by CubicWeb using automatically sqlite (bundled with Python, see

    25 http://docs.python.org/library/sqlite3.html) as a backend.

    26 

    27 The database is stored in the mycube/test/tmpdb,

    28 mycube/test/tmpdb-template files. If it does not (yet) exist, it will

    29 be built automatically when the test suite starts.

    30 

    31 .. warning::

    32 

    33   Whenever the schema changes (new entities, attributes, relations)

    34   one must delete these two files. Changes concerned only with entity

    35   or relation type properties (constraints, cardinalities,

    36   permissions) and generally dealt with using the

    37   `sync_schema_props_perms()` function of the migration environment do

    38   not need a database regeneration step.

    39 

    40 .. _hook_test:

    41 

    42 Unit test by example

    43 ````````````````````

    44 

    45 We start with an example extracted from the keyword cube (available

    46 from http://www.cubicweb.org/project/cubicweb-keyword).

    47 

    48 .. sourcecode:: python

    49 

    50     from cubicweb.devtools.testlib import CubicWebTC

    51     from cubicweb import ValidationError

    52 

    53     class ClassificationHooksTC(CubicWebTC):

    54 

    55         def setup_database(self):

    56             with self.admin_access.repo_cnx() as cnx:

    57                 group_etype = cnx.find('CWEType', name='CWGroup').one()

    58                 c1 = cnx.create_entity('Classification', name=u'classif1',

    59                                        classifies=group_etype)

    60                 user_etype = cnx.find('CWEType', name='CWUser').one()

    61                 c2 = cnx.create_entity('Classification', name=u'classif2',

    62                                        classifies=user_etype)

    63                 self.kw1eid = cnx.create_entity('Keyword', name=u'kwgroup', included_in=c1).eid

    64                 cnx.commit()

    65 

    66         def test_cannot_create_cycles(self):

    67             with self.admin_access.repo_cnx() as cnx:

    68                 kw1 = cnx.entity_from_eid(self.kw1eid)

    69                 # direct obvious cycle

    70                 with self.assertRaises(ValidationError):

    71                     kw1.cw_set(subkeyword_of=kw1)

    72                 cnx.rollback()

    73                 # testing indirect cycles

    74                 kw3 = cnx.execute('INSERT Keyword SK: SK name "kwgroup2", SK included_in C, '

    75                                   'SK subkeyword_of K WHERE C name "classif1", K eid %(k)s'

    76                                   {'k': kw1}).get_entity(0,0)

    77                 kw3.cw_set(reverse_subkeyword_of=kw1)

    78                 self.assertRaises(ValidationError, cnx.commit)

    79 

    80 The test class defines a :meth:`setup_database` method which populates the

    81 database with initial data. Each test of the class runs with this

    82 pre-populated database.

    83 

    84 The test case itself checks that an Operation does its job of

    85 preventing cycles amongst Keyword entities.

    86 

    87 The `create_entity` method of connection (or request) objects allows

    88 to create an entity. You can link this entity to other entities, by

    89 specifying as argument, the relation name, and the entity to link, as

    90 value. In the above example, the `Classification` entity is linked to

    91 a `CWEtype` via the relation `classifies`. Conversely, if you are

    92 creating a `CWEtype` entity, you can link it to a `Classification`

    93 entity, by adding `reverse_classifies` as argument.

    94 

    95 .. note::

    96 

    97    the :meth:`commit` method is not called automatically. You have to

    98    call it explicitly if needed (notably to test operations). It is a

    99    good practice to regenerate entities with :meth:`entity_from_eid`

   100    after a commit to avoid request cache effects.

   101 

   102 You can see an example of security tests in the

   103 :ref:`adv_tuto_security`.

   104 

   105 It is possible to have these tests run continuously using `apycot`_.

   106 

   107 .. _apycot: http://www.cubicweb.org/project/apycot

   108 

   109 .. _securitytest:

   110 

   111 Managing connections or users

   112 +++++++++++++++++++++++++++++

   113 

   114 Since unit tests are done with the SQLITE backend and this does not

   115 support multiple connections at a time, you must be careful when

   116 simulating security, changing users.

   117 

   118 By default, tests run with a user with admin privileges. Connections

   119 using these credentials are accessible through the `admin_access` object

   120 of the test classes.

   121 

   122 The `repo_cnx()` method returns a connection object that can be used as a

   123 context manager:

   124 

   125 .. sourcecode:: python

   126 

   127    # admin_access is a pre-cooked session wrapping object

   128    # it is built with:

   129    # self.admin_access = self.new_access('admin')

   130    with self.admin_access.repo_cnx() as cnx:

   131        cnx.execute(...)

   132        self.create_user(cnx, login='user1')

   133        cnx.commit()

   134 

   135    user1access = self.new_access('user1')

   136    with user1access.web_request() as req:

   137        req.execute(...)

   138        req.cnx.commit()

   139 

   140 On exit of the context manager, a rollback is issued, which releases

   141 the connection. Don't forget to issue the `cnx.commit()` calls!

   142 

   143 .. warning::

   144 

   145    Do not use references kept to the entities created with a

   146    connection from another one!

   147 

   148 Email notifications tests

   149 `````````````````````````

   150 

   151 When running tests, potentially generated e-mails are not really sent

   152 but are found in the list `MAILBOX` of module

   153 :mod:`cubicweb.devtools.testlib`.

   154 

   155 You can test your notifications by analyzing the contents of this list, which

   156 contains objects with two attributes:

   157 

   158 * `recipients`, the list of recipients

   159 * `msg`, email.Message object

   160 

   161 Let us look at a simple example from the ``blog`` cube.

   162 

   163 .. sourcecode:: python

   164 

   165     from cubicweb.devtools.testlib import CubicWebTC, MAILBOX

   166 

   167     class BlogTestsCubicWebTC(CubicWebTC):

   168         """test blog specific behaviours"""

   169 

   170         def test_notifications(self):

   171             with self.admin_access.web_request() as req:

   172                 cubicweb_blog = req.create_entity('Blog', title=u'cubicweb',

   173                                     description=u'cubicweb is beautiful')

   174                 blog_entry_1 = req.create_entity('BlogEntry', title=u'hop',

   175                                                  content=u'cubicweb hop')

   176                 blog_entry_1.cw_set(entry_of=cubicweb_blog)

   177                 blog_entry_2 = req.create_entity('BlogEntry', title=u'yes',

   178                                                  content=u'cubicweb yes')

   179                 blog_entry_2.cw_set(entry_of=cubicweb_blog)

   180                 self.assertEqual(len(MAILBOX), 0)

   181                 req.cnx.commit()

   182                 self.assertEqual(len(MAILBOX), 2)

   183                 mail = MAILBOX[0]

   184                 self.assertEqual(mail.subject, '[data] hop')

   185                 mail = MAILBOX[1]

   186                 self.assertEqual(mail.subject, '[data] yes')

   187 

   188 Visible actions tests

   189 `````````````````````

   190 

   191 It is easy to write unit tests to test actions which are visible to

   192 a user or to a category of users. Let's take an example in the

   193 `conference cube`_.

   194 

   195 .. _`conference cube`: http://www.cubicweb.org/project/cubicweb-conference

   196 .. sourcecode:: python

   197 

   198     class ConferenceActionsTC(CubicWebTC):

   199 

   200         def setup_database(self):

   201             with self.admin_access.repo_cnx() as cnx:

   202                 self.confeid = cnx.create_entity('Conference',

   203                                                  title=u'my conf',

   204                                                  url_id=u'conf',

   205                                                  start_on=date(2010, 1, 27),

   206                                                  end_on = date(2010, 1, 29),

   207                                                  call_open=True,

   208                                                  reverse_is_chair_at=chair,

   209                                                  reverse_is_reviewer_at=reviewer).eid

   210 

   211         def test_admin(self):

   212             with self.admin_access.web_request() as req:

   213                 rset = req.find('Conference').one()

   214                 self.assertListEqual(self.pactions(req, rset),

   215                                       [('workflow', workflow.WorkflowActions),

   216                                        ('edit', confactions.ModifyAction),

   217                                        ('managepermission', actions.ManagePermissionsAction),

   218                                        ('addrelated', actions.AddRelatedActions),

   219                                        ('delete', actions.DeleteAction),

   220                                        ('generate_badge_action', badges.GenerateBadgeAction),

   221                                        ('addtalkinconf', confactions.AddTalkInConferenceAction)

   222                                        ])

   223                 self.assertListEqual(self.action_submenu(req, rset, 'addrelated'),

   224                                       [(u'add Track in_conf Conference object',

   225                                         u'http://testing.fr/cubicweb/add/Track'

   226                                         u'?__linkto=in_conf%%3A%(conf)s%%3Asubject&'

   227                                         u'__redirectpath=conference%%2Fconf&'

   228                                         u'__redirectvid=' % {'conf': self.confeid}),

   229                                        ])

   230 

   231 You just have to execute a rql query corresponding to the view you want to test,

   232 and to compare the result of

   233 :meth:`~cubicweb.devtools.testlib.CubicWebTC.pactions` with the list of actions

   234 that must be visible in the interface. This is a list of tuples. The first

   235 element is the action's `__regid__`, the second the action's class.

   236 

   237 To test actions in a submenu, you just have to test the result of

   238 :meth:`~cubicweb.devtools.testlib.CubicWebTC.action_submenu` method. The last

   239 parameter of the method is the action's category. The result is a list of

   240 tuples. The first element is the action's title, and the second element the

   241 action's url.

   242 

   243 

   244 .. _automatic_views_tests:

   245 

   246 Automatic views testing

   247 -----------------------

   248 

   249 This is done automatically with the :class:`cubicweb.devtools.testlib.AutomaticWebTest`

   250 class. At cube creation time, the mycube/test/test_mycube.py file

   251 contains such a test. The code here has to be uncommented to be

   252 usable, without further modification.

   253 

   254 The ``auto_populate`` method uses a smart algorithm to create

   255 pseudo-random data in the database, thus enabling the views to be

   256 invoked and tested.

   257 

   258 Depending on the schema, hooks and operations constraints, it is not

   259 always possible for the automatic auto_populate to proceed.

   260 

   261 It is possible of course to completely redefine auto_populate. A

   262 lighter solution is to give hints (fill some class attributes) about

   263 what entities and relations have to be skipped by the auto_populate

   264 mechanism. These are:

   265 

   266 * `no_auto_populate`, may contain a list of entity types to skip

   267 * `ignored_relations`, may contain a list of relation types to skip

   268 * `application_rql`, may contain a list of rql expressions that

   269   auto_populate cannot guess by itself; these must yield resultsets

   270   against which views may be selected.

   271 

   272 .. warning::

   273 

   274   Take care to not let the imported `AutomaticWebTest` in your test module

   275   namespace, else both your subclass *and* this parent class will be run.

   276 

   277 Cache heavy database setup

   278 -------------------------------

   279 

   280 Some test suites require a complex setup of the database that takes

   281 seconds (or even minutes) to complete. Doing the whole setup for each

   282 individual test makes the whole run very slow. The ``CubicWebTC``

   283 class offer a simple way to prepare a specific database once for

   284 multiple tests. The `test_db_id` class attribute of your

   285 ``CubicWebTC`` subclass must be set to a unique identifier and the

   286 :meth:`pre_setup_database` class method must build the cached content. As

   287 the :meth:`pre_setup_database` method is not garanteed to be called

   288 every time a test method is run, you must not set any class attribute

   289 to be used during test *there*. Databases for each `test_db_id` are

   290 automatically created if not already in cache. Clearing the cache is

   291 up to the user. Cache files are found in the :file:`data/database`

   292 subdirectory of your test directory.

   293 

   294 .. warning::

   295 

   296   Take care to always have the same :meth:`pre_setup_database`

   297   function for all classes with a given `test_db_id` otherwise your

   298   tests will have unpredictable results depending on the first

   299   encountered one.

   300 

   301 

   302 Testing on a real-life database

   303 -------------------------------

   304 

   305 The ``CubicWebTC`` class uses the `cubicweb.devtools.ApptestConfiguration`

   306 configuration class to setup its testing environment (database driver,

   307 user password, application home, and so on). The `cubicweb.devtools`

   308 module also provides a `RealDatabaseConfiguration`

   309 class that will read a regular cubicweb sources file to fetch all

   310 this information but will also prevent the database to be initalized

   311 and reset between tests.

   312 

   313 For a test class to use a specific configuration, you have to set

   314 the `_config` class attribute on the class as in:

   315 

   316 .. sourcecode:: python

   317 

   318     from cubicweb.devtools import RealDatabaseConfiguration

   319     from cubicweb.devtools.testlib import CubicWebTC

   320 

   321     class BlogRealDatabaseTC(CubicWebTC):

   322         _config = RealDatabaseConfiguration('blog',

   323                                             sourcefile='/path/to/realdb_sources')

   324 

   325         def test_blog_rss(self):

   326             with self.admin_access.web_request() as req:

   327             rset = req.execute('Any B ORDERBY D DESC WHERE B is BlogEntry, '

   328                                'B created_by U, U login "logilab", B creation_date D')

   329             self.view('rss', rset, req=req)

   330 

   331 

   332 Testing with other cubes

   333 ------------------------

   334 

   335 Sometimes a small component cannot be tested all by itself, so one

   336 needs to specify other cubes to be used as part of the the unit test

   337 suite. This is handled by the ``bootstrap_cubes`` file located under

   338 ``mycube/test/data``. One example from the `preview` cube::

   339 

   340  card, file, preview

   341 

   342 The format is:

   343 

   344 * possibly several empy lines or lines starting with ``#`` (comment lines)

   345 * one line containing a comma-separated list of cube names.

   346 

   347 It is also possible to add a ``schema.py`` file in

   348 ``mycube/test/data``, which will be used by the testing framework,

   349 therefore making new entity types and relations available to the

   350 tests. 

   351 

   352 Literate programming

   353 --------------------

   354 

   355 CubicWeb provides some literate programming capabilities. The :ref:`cubicweb-ctl`

   356 `shell` command accepts different file formats. If your file ends with `.txt`

   357 or `.rst`, the file will be parsed by :mod:`doctest.testfile` with CubicWeb's

   358 :ref:`migration` API enabled in it.

   359 

   360 Create a `scenario.txt` file in the `test/` directory and fill with some content.

   361 Refer to the :mod:`doctest.testfile` `documentation`_.

   362 

   363 .. _documentation: http://docs.python.org/library/doctest.html

   364 

   365 Then, you can run it directly by::

   366 

   367     $ cubicweb-ctl shell <cube_instance> test/scenario.txt

   368 

   369 When your scenario file is ready, put it in a new test case to be able to run

   370 it automatically.

   371 

   372 .. sourcecode:: python

   373 

   374       from os.path import dirname, join

   375       from logilab.common.testlib import unittest_main

   376       from cubicweb.devtools.testlib import CubicWebTC

   377 

   378       class AcceptanceTC(CubicWebTC):

   379 

   380               def test_scenario(self):

   381                       self.assertDocTestFile(join(dirname(__file__), 'scenario.txt'))

   382 

   383       if __name__ == '__main__':

   384               unittest_main()

   385 

   386 Skipping a scenario

   387 ```````````````````

   388 

   389 If you want to set up initial conditions that you can't put in your unit test

   390 case, you have to use a :exc:`KeyboardInterrupt` exception only because of the

   391 way :mod:`doctest` module will catch all the exceptions internally.

   392 

   393     >>> if condition_not_met:

   394     ...     raise KeyboardInterrupt('please, check your fixture.')

   395 

   396 Passing paramaters

   397 ``````````````````

   398 Using extra arguments to parametrize your scenario is possible by prepending them

   399 by double dashes.

   400 

   401 Please refer to the `cubicweb-ctl shell --help` usage.

   402 

   403 .. important::

   404     Your scenario file must be utf-8 encoded.

   405 

   406 Test APIS

   407 ---------

   408 

   409 Using Pytest

   410 ````````````

   411 

   412 The `pytest` utility (shipping with `logilab-common`_, which is a

   413 mandatory dependency of CubicWeb) extends the Python unittest

   414 functionality and is the preferred way to run the CubicWeb test

   415 suites. Bare unittests also work the usual way.

   416 

   417 .. _logilab-common: http://www.logilab.org/project/logilab-common

   418 

   419 To use it, you may:

   420 

   421 * just launch `pytest` in your cube to execute all tests (it will

   422   discover them automatically)

   423 * launch `pytest unittest_foo.py` to execute one test file

   424 * launch `pytest unittest_foo.py bar` to execute all test methods and

   425   all test cases whose name contains `bar`

   426 

   427 Additionally, the `-x` option tells pytest to exit at the first error

   428 or failure. The `-i` option tells pytest to drop into pdb whenever an

   429 exception occurs in a test.

   430 

   431 When the `-x` option has been used and the run stopped on a test, it

   432 is possible, after having fixed the test, to relaunch pytest with the

   433 `-R` option to tell it to start testing again from where it previously

   434 failed.

   435 

   436 Using the `TestCase` base class

   437 ```````````````````````````````

   438 

   439 The base class of CubicWebTC is logilab.common.testlib.TestCase, which

   440 provides a lot of convenient assertion methods.

   441 

   442 .. autoclass:: logilab.common.testlib.TestCase

   443    :members:

   444 

   445 CubicWebTC API

   446 ``````````````

   447 .. autoclass:: cubicweb.devtools.testlib.CubicWebTC

   448    :members:

   449 

   450 

   451 What you need to know about request and session

   452 -----------------------------------------------

   453 

   454 .. image:: ../images/request_session.png

   455 

   456 First, remember to think that some code run on a client side, some

   457 other on the repository side. More precisely:

   458 

   459 * client side: web interface, raw repoapi connection (cubicweb-ctl shell for

   460   instance);

   461 

   462 * repository side: RQL query execution, that may trigger hooks and operation.

   463 

   464 The client interacts with the repository through a repoapi connection.

   465 

   466 

   467 .. note::

   468 

   469    These distinctions are going to disappear in cubicweb 3.21 (if not

   470    before).

   471 

   472 A repoapi connection is tied to a session in the repository. The connection and

   473 request objects are inaccessible from repository code / the session object is

   474 inaccessible from client code (theoretically at least).

   475 

   476 The web interface provides a request class.  That `request` object provides

   477 access to all cubicweb resources, eg:

   478 

   479 * the registry (which itself provides access to the schema and the

   480   configuration);

   481 

   482 * an underlying repoapi connection (when using req.execute, you actually call the

   483   repoapi);

   484 

   485 * other specific resources depending on the client type (url generation according

   486   to base url, form parameters, etc.).

   487 

   488 

   489 A `session` provides an api similar to a request regarding RQL execution and

   490 access to global resources (registry and all), but also has the following

   491 responsibilities:

   492 

   493 * handle transaction data, that will live during the time of a single

   494   transaction. This includes the database connections that will be used to

   495   execute RQL queries.

   496 

   497 * handle persistent data that may be used across different (web) requests

   498 

   499 * security and hooks control (not possible through a request)

   500 

   501 

   502 The `_cw` attribute

   503 ```````````````````

   504 The `_cw` attribute available on every application object provides access to all

   505 cubicweb resources, i.e.:

   506 

   507 - For code running on the client side (eg web interface view), `_cw` is a request

   508   instance.

   509 

   510 - For code running on the repository side (hooks and operation), `_cw` is a

   511   Connection or Session instance.

   512 

   513 

   514 Beware some views may be called with a session (e.g. notifications) or with a

   515 request.

   516 

   517 

   518 Request, session and transaction

   519 ````````````````````````````````

   520 

   521 In the web interface, an HTTP request is handled by a single request, which will

   522 be thrown away once the response is sent.

   523 

   524 The web publisher handles the transaction:

   525 

   526 * commit / rollback is done automatically

   527 

   528 * you should not commit / rollback explicitly, except if you really

   529   need it

   530 

   531 Let's detail the process:

   532 

   533 1. an incoming RQL query comes from a client to the web stack

   534 

   535 2. the web stack opens an authenticated database connection for the

   536    request, which is associated to a user session

   537 

   538 3. the query is executed (through the repository connection)

   539 

   540 4. this query may trigger hooks. Hooks and operations may execute some rql queries

   541    through `cnx.execute`.

   542 

   543 5. the repository gets the result of the query in 1. If it was a RQL read query,

   544    the database connection is released. If it was a write query, the connection

   545    is then tied to the session until the transaction is commited or rolled back.

   546 

   547 6. results are sent back to the client

   548 

   549 This implies several things:

   550 

   551 * when using a request, or code executed in hooks, this database

   552   connection handling is totally transparent

   553 

   554 * however, take care when writing tests: you are usually faking /

   555   testing both the server and the client side, so you have to decide

   556   when to use RepoAccess.client_cnx or RepoAccess.repo_cnx. Ask

   557   yourself "where will the code I want to test be running, client or

   558   repository side?". The response is usually: use a repo (since the

   559   "client connection" concept is going away in a couple of releases).