+
−.. -*- coding: utf-8 -*-
+
−
+
−.. _dataimport:
+
−
+
−Dataimport
+
−==========
+
−
+
−*CubicWeb* is designed to manipulate huge of amount of data, and provides utilities to do so.
+
−
+
−The main entry point is :mod:`cubicweb.dataimport.importer` which defines an
+
−:class:`ExtEntitiesImporter` class responsible for importing data from an external source in the
+
−form :class:`ExtEntity` objects. An :class:`ExtEntity` is a transitional representation of an
+
−entity to be imported in the CubicWeb instance; building this representation is usually
+
−domain-specific -- e.g. dependent of the kind of data source (RDF, CSV, etc.) -- and is thus the
+
−responsibility of the end-user.
+
−
+
−Along with the importer, a *store* must be selected, which is responsible for insertion of data into
+
−the database. There exists different kind of stores_, allowing to insert data within different
+
−levels of the *CubicWeb* API and with different speed/security tradeoffs. Those keeping all the
+
−*CubicWeb* hooks and security will be slower but the possible errors in insertion (bad data types,
+
−integrity error, ...) will be handled.
+
−
+
−
+
−Example
+
−-------
+
−
+
−Consider the following schema snippet.
+
−
+
−.. code-block:: python
+
−
+
−    class Person(EntityType):
+
−        name = String(required=True)
+
−
+
−    class knows(RelationDefinition):
+
−        subject = 'Person'
+
−        object = 'Person'
+
−
+
−along with some data in a ``people.csv`` file::
+
−
+
−    # uri,name,knows
+
−    http://www.example.org/alice,Alice,
+
−    http://www.example.org/bob,Bob,http://www.example.org/alice
+
−
+
−The following code (using a shell context) defines a function `extentities_from_csv` to read
+
−`Person` external entities coming from a CSV file and calls the :class:`ExtEntitiesImporter` to
+
−insert corresponding entities and relations into the CubicWeb instance.
+
−
+
−.. code-block:: python
+
−
+
−    from cubicweb.dataimport import ucsvreader, RQLObjectStore
+
−    from cubicweb.dataimport.importer import ExtEntity, ExtEntitiesImporter
+
−
+
−    def extentities_from_csv(fpath):
+
−        """Yield Person ExtEntities read from `fpath` CSV file."""
+
−        with open(fpath) as f:
+
−            for uri, name, knows in ucsvreader(f, skipfirst=True, skip_empty=False):
+
−                yield ExtEntity('Personne', uri,
+
−                                {'nom': set([name]), 'connait': set([knows])})
+
−
+
−    extenties = extentities_from_csv('people.csv')
+
−    store = RQLObjectStore(cnx)
+
−    importer = ExtEntitiesImporter(schema, store)
+
−    importer.import_entities(extenties)
+
−    commit()
+
−    rset = cnx.execute('String N WHERE X nom N, X connait Y, Y nom "Alice"')
+
−    assert rset[0][0] == u'Bob', rset
+
−
+
−Importer API
+
−------------
+
−
+
−.. automodule:: cubicweb.dataimport.importer
+
−
+
−
+
−Stores
+
−~~~~~~
+
−
+
−Stores are responsible to insert properly formatted entities and relations into the database. They
+
−have the following API::
+
−
+
−    >>> user_eid = store.prepare_insert_entity('CWUser', login=u'johndoe')
+
−    >>> group_eid = store.prepare_insert_entity('CWUser', name=u'unknown')
+
−    >>> store.relate(user_eid, 'in_group', group_eid)
+
−    >>> store.flush()
+
−    >>> store.commit()
+
−    >>> store.finish()
+
−
+
−Some stores **require a flush** to copy data in the database, so if you want to have store
+
−independent code you should explicitly call it. (There may be multiple flushes during the
+
−process, or only one at the end if there is no memory issue). This is different from the
+
−commit which validates the database transaction. At last, the `finish()` method should be called in
+
−case the store requires additional work once everything is done.
+
−
+
−* ``prepare_insert_entity(<entity type>, **kwargs) -> eid``: given an entity
+
−  type, attributes and inlined relations, return the eid of the entity to be
+
−  inserted, *with no guarantee that anything has been inserted in database*.
+
−
+
−* ``prepare_update_entity(<entity type>, eid, **kwargs) -> None``: given an
+
−  entity type and eid, promise for update given attributes and inlined
+
−  relations *with no guarantee that anything has been inserted in database*.
+
−
+
−* ``prepare_insert_relation(eid_from, rtype, eid_to) -> None``: indicate that a
+
−  relation ``rtype`` should be added between entities with eids ``eid_from``
+
−  and ``eid_to``. Similar to ``prepare_insert_entity()``, *there is no
+
−  guarantee that the relation has been inserted in database*.
+
−
+
−* ``flush() -> None``: flush any temporary data to database. May be called
+
−  several times during an import.
+
−
+
−* ``commit() -> None``: commit the database transaction.
+
−
+
−* ``finish() -> None``: additional stuff to do after import is terminated.
+
−
+
−ObjectStore
+
−-----------
+
−
+
−This store keeps objects in memory for *faster* validation. It may be useful in development
+
−mode. However, as it will not enforce the constraints of the schema nor insert anything in the
+
−database, so it may miss some problems.
+
−
+
−
+
−RQLObjectStore
+
−--------------
+
−
+
−This store works with an actual RQL repository, and it may be used in production mode.
+
−
+
−
+
−NoHookRQLObjectStore
+
−--------------------
+
−
+
−This store works similarly to the *RQLObjectStore* but bypasses some *CubicWeb* hooks to be faster.
+
−
+
−
+
−SQLGenObjectStore
+
−-----------------
+
−
+
−This store relies on *COPY FROM*/execute many sql commands to directly push data using SQL commands
+
−rather than using the whole *CubicWeb* API. For now, **it only works with PostgresSQL** as it requires
+
−the *COPY FROM* command.