Import

The Import Tool loads objects from a CSV, YAML or JSON file. It is reached from the sidebar (Import).

Requires the p_data_import permission, plus the create/update permission of each object’s own schema - a row you are not allowed to write is refused on its own, the rest of the file still goes through.

File

Pick a .csv, .yml, .yaml or .json file. After Check or Import you have to select the file again: no browser lets a page prefill a file input. Every other field keeps the value you set.

Each object needs a schema and a keyname. In YAML and JSON that is the classname and keyname keys:

- classname: site
  keyname: GAVLE
  displayname: Gavle
  country: SE

In CSV, the schema column is named _schema or classname, and one row is one object:

_schema,keyname,displayname,country
site,GAVLE,Gavle,SE
site,MALMO,Malmo,SE

A column named schema, without the underscore, is not the schema column: it is read as an ordinary field, and the row is refused for having no schema. _schema is what the Exporter writes, so an exported file re-imports as it is.

Common options

Touch last_sync - ticked by default. Stamps last_sync on every object the file writes, which is how you tell “this object was confirmed by its source” from “nobody touched it”. Untick it for a one-off correction you don’t want counted as a sync.

Pipeline - optional. Runs each object through a named Pipeline before writing it, so the file can be transformed (renamed fields, computed values, discarded rows) on the way in.

First row / Last row - optional row range, both bounds included. Row 1 is the first data row; a CSV header does not count. Leave both blank to import the whole file. Useful to load a big file in slices, or to try the first few rows before committing to all of them:

CSV specific options

Only used when the file is a .csv.

Encoding - utf-8 by default; iso-8859-1, cp1252 and a few others are offered for files out of a spreadsheet. A leading byte-order mark is removed either way.

Delimiter - Auto-detect by default, or pick ,, ; or | yourself.

Auto-detect reads the header and the next few rows, and keeps the first of |, ;, , that separates the same number of columns on every one of them. A character that really separates columns gives an identical count on each row; one that merely occurs in the data almost never does - so town holding Gavle, SE in a ; file does not make it a , file. Delimiters inside "quotes" are not counted.

It never guesses: if no character lines up, the import is refused with an error on the Delimiter field and you pick one. The same happens for a tab-delimited file, which this tool cannot read whichever delimiter you choose. A file with a single column and no delimiter at all imports fine - there is nothing to separate, so every delimiter is equivalent.

Split multi-value fields (^) - ticked by default. A cell holding a^b^c becomes three values. Untick it to keep the ^ as plain text.

Schema (p_data_admin only) - applies one schema to every row, overriding whatever the row says. Needed when the file carries no schema column at all.

Advanced usage

Force action (p_data_admin only) - applies one action to every row, overriding each row’s own _action: create, init, update, delete, enable, disable, append, unappend, noop. See Action for what each one does.

Check, then Import

Check parses the file and reports how many rows it found, and which delimiter was detected when the delimiter was left on Auto-detect. It writes nothing, and it does not validate schemas, permissions or field values - those are only checked while importing. Use it to confirm the encoding and delimiter are right before writing anything: a row count that looks plausible on the wrong delimiter is exactly what Check is for.

Import writes the objects.

Progress window

Every import runs as a background task, whatever its size, and a progress window opens on submit showing rows, done, success and error counts, elapsed and remaining time, the rate in rows per second, and the first errors.

While it runs:

• Abort stops the remaining rows. It is not a rollback: rows already written stay written.
• Hide closes the window and leaves the import running. Pick it up again from the Tasks menu in the top bar, and stop it from there if you need to: following and stopping your own tasks needs no permission.

Once it is finished (or aborted, or failed), the window offers Close and a link to the full task report, which lists every error rather than the first few.

When at least one row failed, a Download the N error(s) link below the error list gives the full list as a text file, one message per line - handy for fixing the source file away from the screen.

An import stops by itself if more than CAVALIBA_MAX_IMPORT_ERROR rows (default 20) fail - the window says so, and the rows written up to that point stay written.

A task that never starts, or that fails before reading the file, shows Import failed with the reason.