|
1 .. -*- coding: utf-8 -*- |
|
2 |
|
3 Tests |
|
4 ===== |
|
5 |
|
6 Unit tests |
|
7 ---------- |
|
8 |
|
9 The *CubicWeb* framework provides the |
|
10 :class:`cubicweb.devtools.testlib.CubicWebTC` test base class . |
|
11 |
|
12 Tests shall be put into the mycube/test directory. Additional test |
|
13 data shall go into mycube/test/data. |
|
14 |
|
15 It is much advised to write tests concerning entities methods, |
|
16 actions, hooks and operations, security. The |
|
17 :class:`~cubicweb.devtools.testlib.CubicWebTC` base class has |
|
18 convenience methods to help test all of this. |
|
19 |
|
20 In the realm of views, automatic tests check that views are valid |
|
21 XHTML. See :ref:`automatic_views_tests` for details. |
|
22 |
|
23 Most unit tests need a live database to work against. This is achieved |
|
24 by CubicWeb using automatically sqlite (bundled with Python, see |
|
25 http://docs.python.org/library/sqlite3.html) as a backend. |
|
26 |
|
27 The database is stored in the mycube/test/tmpdb, |
|
28 mycube/test/tmpdb-template files. If it does not (yet) exist, it will |
|
29 be built automatically when the test suite starts. |
|
30 |
|
31 .. warning:: |
|
32 |
|
33 Whenever the schema changes (new entities, attributes, relations) |
|
34 one must delete these two files. Changes concerned only with entity |
|
35 or relation type properties (constraints, cardinalities, |
|
36 permissions) and generally dealt with using the |
|
37 `sync_schema_props_perms()` function of the migration environment do |
|
38 not need a database regeneration step. |
|
39 |
|
40 .. _hook_test: |
|
41 |
|
42 Unit test by example |
|
43 ```````````````````` |
|
44 |
|
45 We start with an example extracted from the keyword cube (available |
|
46 from http://www.cubicweb.org/project/cubicweb-keyword). |
|
47 |
|
48 .. sourcecode:: python |
|
49 |
|
50 from cubicweb.devtools.testlib import CubicWebTC |
|
51 from cubicweb import ValidationError |
|
52 |
|
53 class ClassificationHooksTC(CubicWebTC): |
|
54 |
|
55 def setup_database(self): |
|
56 with self.admin_access.repo_cnx() as cnx: |
|
57 group_etype = cnx.find('CWEType', name='CWGroup').one() |
|
58 c1 = cnx.create_entity('Classification', name=u'classif1', |
|
59 classifies=group_etype) |
|
60 user_etype = cnx.find('CWEType', name='CWUser').one() |
|
61 c2 = cnx.create_entity('Classification', name=u'classif2', |
|
62 classifies=user_etype) |
|
63 self.kw1eid = cnx.create_entity('Keyword', name=u'kwgroup', included_in=c1).eid |
|
64 cnx.commit() |
|
65 |
|
66 def test_cannot_create_cycles(self): |
|
67 with self.admin_access.repo_cnx() as cnx: |
|
68 kw1 = cnx.entity_from_eid(self.kw1eid) |
|
69 # direct obvious cycle |
|
70 with self.assertRaises(ValidationError): |
|
71 kw1.cw_set(subkeyword_of=kw1) |
|
72 cnx.rollback() |
|
73 # testing indirect cycles |
|
74 kw3 = cnx.execute('INSERT Keyword SK: SK name "kwgroup2", SK included_in C, ' |
|
75 'SK subkeyword_of K WHERE C name "classif1", K eid %(k)s' |
|
76 {'k': kw1}).get_entity(0,0) |
|
77 kw3.cw_set(reverse_subkeyword_of=kw1) |
|
78 self.assertRaises(ValidationError, cnx.commit) |
|
79 |
|
80 The test class defines a :meth:`setup_database` method which populates the |
|
81 database with initial data. Each test of the class runs with this |
|
82 pre-populated database. |
|
83 |
|
84 The test case itself checks that an Operation does its job of |
|
85 preventing cycles amongst Keyword entities. |
|
86 |
|
87 The `create_entity` method of connection (or request) objects allows |
|
88 to create an entity. You can link this entity to other entities, by |
|
89 specifying as argument, the relation name, and the entity to link, as |
|
90 value. In the above example, the `Classification` entity is linked to |
|
91 a `CWEtype` via the relation `classifies`. Conversely, if you are |
|
92 creating a `CWEtype` entity, you can link it to a `Classification` |
|
93 entity, by adding `reverse_classifies` as argument. |
|
94 |
|
95 .. note:: |
|
96 |
|
97 the :meth:`commit` method is not called automatically. You have to |
|
98 call it explicitly if needed (notably to test operations). It is a |
|
99 good practice to regenerate entities with :meth:`entity_from_eid` |
|
100 after a commit to avoid request cache effects. |
|
101 |
|
102 You can see an example of security tests in the |
|
103 :ref:`adv_tuto_security`. |
|
104 |
|
105 It is possible to have these tests run continuously using `apycot`_. |
|
106 |
|
107 .. _apycot: http://www.cubicweb.org/project/apycot |
|
108 |
|
109 .. _securitytest: |
|
110 |
|
111 Managing connections or users |
|
112 +++++++++++++++++++++++++++++ |
|
113 |
|
114 Since unit tests are done with the SQLITE backend and this does not |
|
115 support multiple connections at a time, you must be careful when |
|
116 simulating security, changing users. |
|
117 |
|
118 By default, tests run with a user with admin privileges. Connections |
|
119 using these credentials are accessible through the `admin_access` object |
|
120 of the test classes. |
|
121 |
|
122 The `repo_cnx()` method returns a connection object that can be used as a |
|
123 context manager: |
|
124 |
|
125 .. sourcecode:: python |
|
126 |
|
127 # admin_access is a pre-cooked session wrapping object |
|
128 # it is built with: |
|
129 # self.admin_access = self.new_access('admin') |
|
130 with self.admin_access.repo_cnx() as cnx: |
|
131 cnx.execute(...) |
|
132 self.create_user(cnx, login='user1') |
|
133 cnx.commit() |
|
134 |
|
135 user1access = self.new_access('user1') |
|
136 with user1access.web_request() as req: |
|
137 req.execute(...) |
|
138 req.cnx.commit() |
|
139 |
|
140 On exit of the context manager, a rollback is issued, which releases |
|
141 the connection. Don't forget to issue the `cnx.commit()` calls! |
|
142 |
|
143 .. warning:: |
|
144 |
|
145 Do not use references kept to the entities created with a |
|
146 connection from another one! |
|
147 |
|
148 Email notifications tests |
|
149 ````````````````````````` |
|
150 |
|
151 When running tests, potentially generated e-mails are not really sent |
|
152 but are found in the list `MAILBOX` of module |
|
153 :mod:`cubicweb.devtools.testlib`. |
|
154 |
|
155 You can test your notifications by analyzing the contents of this list, which |
|
156 contains objects with two attributes: |
|
157 |
|
158 * `recipients`, the list of recipients |
|
159 * `msg`, email.Message object |
|
160 |
|
161 Let us look at a simple example from the ``blog`` cube. |
|
162 |
|
163 .. sourcecode:: python |
|
164 |
|
165 from cubicweb.devtools.testlib import CubicWebTC, MAILBOX |
|
166 |
|
167 class BlogTestsCubicWebTC(CubicWebTC): |
|
168 """test blog specific behaviours""" |
|
169 |
|
170 def test_notifications(self): |
|
171 with self.admin_access.web_request() as req: |
|
172 cubicweb_blog = req.create_entity('Blog', title=u'cubicweb', |
|
173 description=u'cubicweb is beautiful') |
|
174 blog_entry_1 = req.create_entity('BlogEntry', title=u'hop', |
|
175 content=u'cubicweb hop') |
|
176 blog_entry_1.cw_set(entry_of=cubicweb_blog) |
|
177 blog_entry_2 = req.create_entity('BlogEntry', title=u'yes', |
|
178 content=u'cubicweb yes') |
|
179 blog_entry_2.cw_set(entry_of=cubicweb_blog) |
|
180 self.assertEqual(len(MAILBOX), 0) |
|
181 req.cnx.commit() |
|
182 self.assertEqual(len(MAILBOX), 2) |
|
183 mail = MAILBOX[0] |
|
184 self.assertEqual(mail.subject, '[data] hop') |
|
185 mail = MAILBOX[1] |
|
186 self.assertEqual(mail.subject, '[data] yes') |
|
187 |
|
188 Visible actions tests |
|
189 ````````````````````` |
|
190 |
|
191 It is easy to write unit tests to test actions which are visible to |
|
192 a user or to a category of users. Let's take an example in the |
|
193 `conference cube`_. |
|
194 |
|
195 .. _`conference cube`: http://www.cubicweb.org/project/cubicweb-conference |
|
196 .. sourcecode:: python |
|
197 |
|
198 class ConferenceActionsTC(CubicWebTC): |
|
199 |
|
200 def setup_database(self): |
|
201 with self.admin_access.repo_cnx() as cnx: |
|
202 self.confeid = cnx.create_entity('Conference', |
|
203 title=u'my conf', |
|
204 url_id=u'conf', |
|
205 start_on=date(2010, 1, 27), |
|
206 end_on = date(2010, 1, 29), |
|
207 call_open=True, |
|
208 reverse_is_chair_at=chair, |
|
209 reverse_is_reviewer_at=reviewer).eid |
|
210 |
|
211 def test_admin(self): |
|
212 with self.admin_access.web_request() as req: |
|
213 rset = req.find('Conference').one() |
|
214 self.assertListEqual(self.pactions(req, rset), |
|
215 [('workflow', workflow.WorkflowActions), |
|
216 ('edit', confactions.ModifyAction), |
|
217 ('managepermission', actions.ManagePermissionsAction), |
|
218 ('addrelated', actions.AddRelatedActions), |
|
219 ('delete', actions.DeleteAction), |
|
220 ('generate_badge_action', badges.GenerateBadgeAction), |
|
221 ('addtalkinconf', confactions.AddTalkInConferenceAction) |
|
222 ]) |
|
223 self.assertListEqual(self.action_submenu(req, rset, 'addrelated'), |
|
224 [(u'add Track in_conf Conference object', |
|
225 u'http://testing.fr/cubicweb/add/Track' |
|
226 u'?__linkto=in_conf%%3A%(conf)s%%3Asubject&' |
|
227 u'__redirectpath=conference%%2Fconf&' |
|
228 u'__redirectvid=' % {'conf': self.confeid}), |
|
229 ]) |
|
230 |
|
231 You just have to execute a rql query corresponding to the view you want to test, |
|
232 and to compare the result of |
|
233 :meth:`~cubicweb.devtools.testlib.CubicWebTC.pactions` with the list of actions |
|
234 that must be visible in the interface. This is a list of tuples. The first |
|
235 element is the action's `__regid__`, the second the action's class. |
|
236 |
|
237 To test actions in a submenu, you just have to test the result of |
|
238 :meth:`~cubicweb.devtools.testlib.CubicWebTC.action_submenu` method. The last |
|
239 parameter of the method is the action's category. The result is a list of |
|
240 tuples. The first element is the action's title, and the second element the |
|
241 action's url. |
|
242 |
|
243 |
|
244 .. _automatic_views_tests: |
|
245 |
|
246 Automatic views testing |
|
247 ----------------------- |
|
248 |
|
249 This is done automatically with the :class:`cubicweb.devtools.testlib.AutomaticWebTest` |
|
250 class. At cube creation time, the mycube/test/test_mycube.py file |
|
251 contains such a test. The code here has to be uncommented to be |
|
252 usable, without further modification. |
|
253 |
|
254 The ``auto_populate`` method uses a smart algorithm to create |
|
255 pseudo-random data in the database, thus enabling the views to be |
|
256 invoked and tested. |
|
257 |
|
258 Depending on the schema, hooks and operations constraints, it is not |
|
259 always possible for the automatic auto_populate to proceed. |
|
260 |
|
261 It is possible of course to completely redefine auto_populate. A |
|
262 lighter solution is to give hints (fill some class attributes) about |
|
263 what entities and relations have to be skipped by the auto_populate |
|
264 mechanism. These are: |
|
265 |
|
266 * `no_auto_populate`, may contain a list of entity types to skip |
|
267 * `ignored_relations`, may contain a list of relation types to skip |
|
268 * `application_rql`, may contain a list of rql expressions that |
|
269 auto_populate cannot guess by itself; these must yield resultsets |
|
270 against which views may be selected. |
|
271 |
|
272 .. warning:: |
|
273 |
|
274 Take care to not let the imported `AutomaticWebTest` in your test module |
|
275 namespace, else both your subclass *and* this parent class will be run. |
|
276 |
|
277 Cache heavy database setup |
|
278 ------------------------------- |
|
279 |
|
280 Some test suites require a complex setup of the database that takes |
|
281 seconds (or even minutes) to complete. Doing the whole setup for each |
|
282 individual test makes the whole run very slow. The ``CubicWebTC`` |
|
283 class offer a simple way to prepare a specific database once for |
|
284 multiple tests. The `test_db_id` class attribute of your |
|
285 ``CubicWebTC`` subclass must be set to a unique identifier and the |
|
286 :meth:`pre_setup_database` class method must build the cached content. As |
|
287 the :meth:`pre_setup_database` method is not garanteed to be called |
|
288 every time a test method is run, you must not set any class attribute |
|
289 to be used during test *there*. Databases for each `test_db_id` are |
|
290 automatically created if not already in cache. Clearing the cache is |
|
291 up to the user. Cache files are found in the :file:`data/database` |
|
292 subdirectory of your test directory. |
|
293 |
|
294 .. warning:: |
|
295 |
|
296 Take care to always have the same :meth:`pre_setup_database` |
|
297 function for all classes with a given `test_db_id` otherwise your |
|
298 tests will have unpredictable results depending on the first |
|
299 encountered one. |
|
300 |
|
301 |
|
302 Testing on a real-life database |
|
303 ------------------------------- |
|
304 |
|
305 The ``CubicWebTC`` class uses the `cubicweb.devtools.ApptestConfiguration` |
|
306 configuration class to setup its testing environment (database driver, |
|
307 user password, application home, and so on). The `cubicweb.devtools` |
|
308 module also provides a `RealDatabaseConfiguration` |
|
309 class that will read a regular cubicweb sources file to fetch all |
|
310 this information but will also prevent the database to be initalized |
|
311 and reset between tests. |
|
312 |
|
313 For a test class to use a specific configuration, you have to set |
|
314 the `_config` class attribute on the class as in: |
|
315 |
|
316 .. sourcecode:: python |
|
317 |
|
318 from cubicweb.devtools import RealDatabaseConfiguration |
|
319 from cubicweb.devtools.testlib import CubicWebTC |
|
320 |
|
321 class BlogRealDatabaseTC(CubicWebTC): |
|
322 _config = RealDatabaseConfiguration('blog', |
|
323 sourcefile='/path/to/realdb_sources') |
|
324 |
|
325 def test_blog_rss(self): |
|
326 with self.admin_access.web_request() as req: |
|
327 rset = req.execute('Any B ORDERBY D DESC WHERE B is BlogEntry, ' |
|
328 'B created_by U, U login "logilab", B creation_date D') |
|
329 self.view('rss', rset, req=req) |
|
330 |
|
331 |
|
332 Testing with other cubes |
|
333 ------------------------ |
|
334 |
|
335 Sometimes a small component cannot be tested all by itself, so one |
|
336 needs to specify other cubes to be used as part of the the unit test |
|
337 suite. This is handled by the ``bootstrap_cubes`` file located under |
|
338 ``mycube/test/data``. One example from the `preview` cube:: |
|
339 |
|
340 card, file, preview |
|
341 |
|
342 The format is: |
|
343 |
|
344 * possibly several empy lines or lines starting with ``#`` (comment lines) |
|
345 * one line containing a comma-separated list of cube names. |
|
346 |
|
347 It is also possible to add a ``schema.py`` file in |
|
348 ``mycube/test/data``, which will be used by the testing framework, |
|
349 therefore making new entity types and relations available to the |
|
350 tests. |
|
351 |
|
352 Literate programming |
|
353 -------------------- |
|
354 |
|
355 CubicWeb provides some literate programming capabilities. The :ref:`cubicweb-ctl` |
|
356 `shell` command accepts different file formats. If your file ends with `.txt` |
|
357 or `.rst`, the file will be parsed by :mod:`doctest.testfile` with CubicWeb's |
|
358 :ref:`migration` API enabled in it. |
|
359 |
|
360 Create a `scenario.txt` file in the `test/` directory and fill with some content. |
|
361 Refer to the :mod:`doctest.testfile` `documentation`_. |
|
362 |
|
363 .. _documentation: http://docs.python.org/library/doctest.html |
|
364 |
|
365 Then, you can run it directly by:: |
|
366 |
|
367 $ cubicweb-ctl shell <cube_instance> test/scenario.txt |
|
368 |
|
369 When your scenario file is ready, put it in a new test case to be able to run |
|
370 it automatically. |
|
371 |
|
372 .. sourcecode:: python |
|
373 |
|
374 from os.path import dirname, join |
|
375 from logilab.common.testlib import unittest_main |
|
376 from cubicweb.devtools.testlib import CubicWebTC |
|
377 |
|
378 class AcceptanceTC(CubicWebTC): |
|
379 |
|
380 def test_scenario(self): |
|
381 self.assertDocTestFile(join(dirname(__file__), 'scenario.txt')) |
|
382 |
|
383 if __name__ == '__main__': |
|
384 unittest_main() |
|
385 |
|
386 Skipping a scenario |
|
387 ``````````````````` |
|
388 |
|
389 If you want to set up initial conditions that you can't put in your unit test |
|
390 case, you have to use a :exc:`KeyboardInterrupt` exception only because of the |
|
391 way :mod:`doctest` module will catch all the exceptions internally. |
|
392 |
|
393 >>> if condition_not_met: |
|
394 ... raise KeyboardInterrupt('please, check your fixture.') |
|
395 |
|
396 Passing paramaters |
|
397 `````````````````` |
|
398 Using extra arguments to parametrize your scenario is possible by prepending them |
|
399 by double dashes. |
|
400 |
|
401 Please refer to the `cubicweb-ctl shell --help` usage. |
|
402 |
|
403 .. important:: |
|
404 Your scenario file must be utf-8 encoded. |
|
405 |
|
406 Test APIS |
|
407 --------- |
|
408 |
|
409 Using Pytest |
|
410 ```````````` |
|
411 |
|
412 The `pytest` utility (shipping with `logilab-common`_, which is a |
|
413 mandatory dependency of CubicWeb) extends the Python unittest |
|
414 functionality and is the preferred way to run the CubicWeb test |
|
415 suites. Bare unittests also work the usual way. |
|
416 |
|
417 .. _logilab-common: http://www.logilab.org/project/logilab-common |
|
418 |
|
419 To use it, you may: |
|
420 |
|
421 * just launch `pytest` in your cube to execute all tests (it will |
|
422 discover them automatically) |
|
423 * launch `pytest unittest_foo.py` to execute one test file |
|
424 * launch `pytest unittest_foo.py bar` to execute all test methods and |
|
425 all test cases whose name contains `bar` |
|
426 |
|
427 Additionally, the `-x` option tells pytest to exit at the first error |
|
428 or failure. The `-i` option tells pytest to drop into pdb whenever an |
|
429 exception occurs in a test. |
|
430 |
|
431 When the `-x` option has been used and the run stopped on a test, it |
|
432 is possible, after having fixed the test, to relaunch pytest with the |
|
433 `-R` option to tell it to start testing again from where it previously |
|
434 failed. |
|
435 |
|
436 Using the `TestCase` base class |
|
437 ``````````````````````````````` |
|
438 |
|
439 The base class of CubicWebTC is logilab.common.testlib.TestCase, which |
|
440 provides a lot of convenient assertion methods. |
|
441 |
|
442 .. autoclass:: logilab.common.testlib.TestCase |
|
443 :members: |
|
444 |
|
445 CubicWebTC API |
|
446 `````````````` |
|
447 .. autoclass:: cubicweb.devtools.testlib.CubicWebTC |
|
448 :members: |
|
449 |
|
450 |
|
451 What you need to know about request and session |
|
452 ----------------------------------------------- |
|
453 |
|
454 .. image:: ../images/request_session.png |
|
455 |
|
456 First, remember to think that some code run on a client side, some |
|
457 other on the repository side. More precisely: |
|
458 |
|
459 * client side: web interface, raw repoapi connection (cubicweb-ctl shell for |
|
460 instance); |
|
461 |
|
462 * repository side: RQL query execution, that may trigger hooks and operation. |
|
463 |
|
464 The client interacts with the repository through a repoapi connection. |
|
465 |
|
466 |
|
467 .. note:: |
|
468 |
|
469 These distinctions are going to disappear in cubicweb 3.21 (if not |
|
470 before). |
|
471 |
|
472 A repoapi connection is tied to a session in the repository. The connection and |
|
473 request objects are inaccessible from repository code / the session object is |
|
474 inaccessible from client code (theoretically at least). |
|
475 |
|
476 The web interface provides a request class. That `request` object provides |
|
477 access to all cubicweb resources, eg: |
|
478 |
|
479 * the registry (which itself provides access to the schema and the |
|
480 configuration); |
|
481 |
|
482 * an underlying repoapi connection (when using req.execute, you actually call the |
|
483 repoapi); |
|
484 |
|
485 * other specific resources depending on the client type (url generation according |
|
486 to base url, form parameters, etc.). |
|
487 |
|
488 |
|
489 A `session` provides an api similar to a request regarding RQL execution and |
|
490 access to global resources (registry and all), but also has the following |
|
491 responsibilities: |
|
492 |
|
493 * handle transaction data, that will live during the time of a single |
|
494 transaction. This includes the database connections that will be used to |
|
495 execute RQL queries. |
|
496 |
|
497 * handle persistent data that may be used across different (web) requests |
|
498 |
|
499 * security and hooks control (not possible through a request) |
|
500 |
|
501 |
|
502 The `_cw` attribute |
|
503 ``````````````````` |
|
504 The `_cw` attribute available on every application object provides access to all |
|
505 cubicweb resources, i.e.: |
|
506 |
|
507 - For code running on the client side (eg web interface view), `_cw` is a request |
|
508 instance. |
|
509 |
|
510 - For code running on the repository side (hooks and operation), `_cw` is a |
|
511 Connection or Session instance. |
|
512 |
|
513 |
|
514 Beware some views may be called with a session (e.g. notifications) or with a |
|
515 request. |
|
516 |
|
517 |
|
518 Request, session and transaction |
|
519 ```````````````````````````````` |
|
520 |
|
521 In the web interface, an HTTP request is handled by a single request, which will |
|
522 be thrown away once the response is sent. |
|
523 |
|
524 The web publisher handles the transaction: |
|
525 |
|
526 * commit / rollback is done automatically |
|
527 |
|
528 * you should not commit / rollback explicitly, except if you really |
|
529 need it |
|
530 |
|
531 Let's detail the process: |
|
532 |
|
533 1. an incoming RQL query comes from a client to the web stack |
|
534 |
|
535 2. the web stack opens an authenticated database connection for the |
|
536 request, which is associated to a user session |
|
537 |
|
538 3. the query is executed (through the repository connection) |
|
539 |
|
540 4. this query may trigger hooks. Hooks and operations may execute some rql queries |
|
541 through `cnx.execute`. |
|
542 |
|
543 5. the repository gets the result of the query in 1. If it was a RQL read query, |
|
544 the database connection is released. If it was a write query, the connection |
|
545 is then tied to the session until the transaction is commited or rolled back. |
|
546 |
|
547 6. results are sent back to the client |
|
548 |
|
549 This implies several things: |
|
550 |
|
551 * when using a request, or code executed in hooks, this database |
|
552 connection handling is totally transparent |
|
553 |
|
554 * however, take care when writing tests: you are usually faking / |
|
555 testing both the server and the client side, so you have to decide |
|
556 when to use RepoAccess.client_cnx or RepoAccess.repo_cnx. Ask |
|
557 yourself "where will the code I want to test be running, client or |
|
558 repository side?". The response is usually: use a repo (since the |
|
559 "client connection" concept is going away in a couple of releases). |