|
1 . -*- coding: utf-8 -*- |
|
2 |
|
3 .. _dataimport: |
|
4 |
|
5 Dataimport |
|
6 ========== |
|
7 |
|
8 *CubicWeb* is designed to manipulate huge of amount of data, and provides helper functions to do so. |
|
9 These functions insert data within different levels of the *CubicWeb* API, |
|
10 allowing different speed/security tradeoffs. Those keeping all the *CubicWeb* hooks |
|
11 and security will be slower but the possible errors in insertion |
|
12 (bad data types, integrity error, ...) will be raised. |
|
13 |
|
14 These dataimport function are provided in the file `dataimport.py`. |
|
15 |
|
16 All the stores have the following API:: |
|
17 |
|
18 >>> store = ObjectStore() |
|
19 >>> user = store.create_entity('CWUser', login=u'johndoe') |
|
20 >>> group = store.create_entity('CWUser', name=u'unknown') |
|
21 >>> store.relate(user.eid, 'in_group', group.eid) |
|
22 |
|
23 |
|
24 ObjectStore |
|
25 ----------- |
|
26 |
|
27 This store keeps objects in memory for *faster* validation. It may be useful |
|
28 in development mode. However, as it will not enforce the constraints of the schema, |
|
29 it may miss some problems. |
|
30 |
|
31 |
|
32 |
|
33 RQLObjectStore |
|
34 -------------- |
|
35 |
|
36 This store works with an actual RQL repository, and it may be used in production mode. |
|
37 |
|
38 |
|
39 NoHookRQLObjectStore |
|
40 -------------------- |
|
41 |
|
42 This store works similarly to the *RQLObjectStore* but bypasses some *CubicWeb* hooks to be faster. |
|
43 |
|
44 |
|
45 SQLGenObjectStore |
|
46 ----------------- |
|
47 |
|
48 This store relies on *COPY FROM*/execute many sql commands to directly push data using SQL commands |
|
49 rather than using the whole *CubicWeb* API. For now, **it only works with PostgresSQL** as it requires |
|
50 the *COPY FROM* command. |
|
51 |
|
52 The API is similar to the other stores, but **it requires a flush** after some imports to copy data |
|
53 in the database (these flushes may be multiples through the processes, or be done only once at the |
|
54 end if there is no memory issue):: |
|
55 |
|
56 >>> store = SQLGenObjectStore(session) |
|
57 >>> store.create_entity('Person', ...) |
|
58 >>> store.flush() |