Transformation Primitives
Transformation tasks modify the fields of each data entry.
Syntax:
- TASKNAME: ["[!]CONDITION", "opt1", "opt2", ...]
Use '' as condition to always execute. Use !COND to execute only when condition is False.
Quick reference
- field_noop: [COND] : do nothing
- field_toint: [COND, field1, ...] : convert fields to int
- field_tofloat: [COND, field1, ...] : convert fields to float
- field_tostring: [COND, field1, ...] : convert to string
- field_nospace: [COND, field1, ...] : remove all whitespaces from fields
- field_regexp_sub: [COND, field, 'pattern', 'replace'] : apply regexp/replace
- field_set: [COND, field, 'value'] : create field with value
- field_copy: [COND, field1, field2] : field1 to field2
- field_rename: [COND, field1, field2] : field1 to field2
- field_merge: [COND, field1, field2, field3] : field1 + field2 > field3
- field_delete: [COND, field1, field2, ...] : remove fields
- field_keep: [COND, field1, field2, ...] : keep only these fields (and classname/keyname)
- field_lower: [COND, field1, ...] : lowercase
- field_upper: [COND, field1, ...] : uppercase
- field_date_now: [COND, field1, ...] : set field(s) to YYYY-MM-DD
- field_time_now: [COND, field1, ...] : set field(s) to HH:MM:SS
- field_datetime_now: [COND, field1, ...] : set field(s) to YYYY-MM-DD HH:MM:SS
- field_uuid: [COND, field1, ...] : set field(s) to random UUID string (distinct)
- field_append: [COND, field, 'suffix'] : append a suffix to field
- field_prepend: [COND, field, 'prefix'] : prefix field with prefix string
- field_md5: [COND, dst, src1, src2, ...] : dst = md5(concat(src1, src2, ...))
- field_join: [COND, sep, dst_field, src1, src2, ...]: join fields with separator
- field_datetime: [COND, target, source, delta] : compute datetime into target field
- field_truncate: [COND, field, length] : cut field content to length chars
- field_lookup: [COND, dstfield, schemaname, remote_field, srcfield]: resolve foreign key, set dstfield to matched keyname or ""
- align_subnet4: [COND, field] : align IPv4/mask to subnet boundaries
- discard: [COND] : eliminate full entry
- exit: [COND, 'status'] : stop processing tasks, keep entryfield_noop
- field_noop: [CONDITION]
Performs nothing. Useful as a placeholder or for testing conditions.
field_set
- field_set: [CONDITION, "fieldname", "AnyValue"]
Creates or overwrites fieldname and sets its value.
field_copy
- field_copy: [CONDITION, "field1", "field2"]
Creates/overwrites field2 with the value of field1.
field_rename
- field_rename: [CONDITION, "field1", "field2"]
Renames field1 to field2. Creates/overwrites field2 with field1’s value and removes field1.
field_delete
- field_delete: [CONDITION, "field1", "field2", ...]
Removes all specified fields.
field_keep
- field_keep: [CONDITION, "field1", "field2", ...]
Keeps provided fields only. Removes all other fields (except classname and keyname).
field_lower
- field_lower: [CONDITION, "field", "field", ...]
Converts field value(s) to lowercase.
field_upper
- field_upper: [CONDITION, "field", "field", ...]
Converts field value(s) to uppercase.
field_toint
- field_toint: [CONDITION, "field", "field", ...]
Converts a field’s value to integer.
field_tofloat
- field_tofloat: [CONDITION, "field", "field", ...]
Converts a field’s value to floating point value.
field_tostring
- field_tostring: [CONDITION, "field", "field", ...]
Converts a field’s value to a string.
field_nospace
- field_nospace: [CONDITION, "field", "field", ...]
Removes all whitespace and tab characters from field values. Faster than a general regexp task.
field_regexp_sub
- field_regexp_sub: [CONDITION, "fieldname", "pattern", "replace"]
Alters fieldname’s value by replacing a regexp pattern with the provided replacement. Accepts any Python standard regexp.
Example:
- field_regexp_sub: ['', 'field_a', 'test', 'QWERTY']
# Before : {'field_a': 'This is a test from unittest !'}
# After : {'field_a': 'This is a QWERTY from unitQWERTY !'}
field_merge
- field_merge: [CONDITION, 'field1', 'field2', 'field3']
Concatenates field1 and field2 into new field3 (overwrites if already exists).
If field values are numerical, a mathematical addition is performed. If they are strings, string concatenation is performed.
Use field_toint / field_tofloat / field_tostring to handle type conversions beforehand.
field_prepend
- field_prepend: [CONDITION, 'field1', 'prefix']
Puts the prefix string in front of field1’s value.
field_append
- field_append: [CONDITION, 'field1', 'suffix']
Puts the suffix string at the end of field1’s value.
field_date_now
- field_date_now: [CONDITION, "field", "field", ...]
Sets field(s) to the current date in YYYY-MM-DD format.
Useful for automated or periodic imports to track in-sync/out-of-sync objects.
field_time_now
- field_time_now: [CONDITION, "field", "field", ...]
Sets field(s) to the current time in HH:MM:SS format.
field_datetime_now
- field_datetime_now: [CONDITION, "field", "field", ...]
Sets field(s) to the current datetime in YYYY-MM-DD HH:MM:SS format.
field_uuid
- field_uuid: [CONDITION, "field", "field", ...]
Creates/overwrites field(s) with a random UUID string. Each field receives a distinct UUID.
Use field_copy if the same UUID is needed in multiple fields.
See field_md5 for a deterministic primary key.
field_md5
new v3.24
- field_md5: [CONDITION, dst, src1, src2, ...]
Computes md5 of the concatenated values of src* fields and stores the result in dst.
If any source field is missing, no md5 is computed and dst is not created.
Useful for creating a deterministic primary key by combining multiple fields.
field_join
new v3.24
- field_join: [CONDITION, separator, dst_field, src1, src2, src3, ...]
Joins multiple source fields into a destination field using a separator (similar to Python’s str.join()). All values are converted to strings.
Example:
- field_join: ['', '-', 'result', 'field1', 'field2']
# Before : {'field1': 'hello', 'field2': 'world'}
# After : {'field1': 'hello', 'field2': 'world', 'result': 'hello-world'}
- field_join: ['', ' ', 'fullname', 'firstname', 'lastname']
# Before : {'firstname': 'John', 'lastname': 'Doe'}
# After : {'firstname': 'John', 'lastname': 'Doe', 'fullname': 'John Doe'}
field_datetime
new v3.33
- field_datetime: [CONDITION, target_field, source, delta]
Computes a datetime value and stores it in target_field.
source (required): one of:
now()— current local datetime- a field name present in the entry — its value is parsed as a datetime
- a datetime string in ISO format
YYYY-MM-DD HH:MM:SS
delta (optional): a duration to add or subtract, in the form [+-]<int><unit>.
Supported units: second, minute, hour, day, week, month, year (plural form also accepted).
delta can also be the name of a field whose value contains the delta string.
Output format: YYYY-MM-DD HH:MM:SS
If source cannot be resolved or parsed, the target field is not set.
Examples:
# set result to current datetime
- field_datetime: ['', result, 'now()']
# set expiry to start + 30 days
- field_datetime: ['', expiry, start, '+30day']
# subtract 3 months from a fixed date
- field_datetime: ['', result, '2025-06-01 00:00:00', '-3month']
# use a field as both source and delta
- field_datetime: ['', result, last_sync, retention_delay]
field_truncate
- field_truncate: [CONDITION, field, length]
Cuts field content to length characters. The field value is converted to string first. If length is greater than the field length, the value is unchanged.
Example:
- field_truncate: ['', description, 100]
# Before : {'description': 'A very long description that exceeds the limit...'}
# After : {'description': 'A very long description that exceeds the limit...'}[0:100]
field_lookup
- field_lookup: [CONDITION, dstfield, schema_name, remote_field, srcfield]
Foreign-key resolution. Finds the instance in schema_name whose remote_field value equals srcfield’s value, and sets dstfield to that instance’s keyname (the first match is kept if several instances share the same value). Sets dstfield to "" if no match is found.
Use remote_field: keyname to match by keyname directly (e.g. to validate that a reference exists). Any other schema field name looks up entries by that field’s stored value instead.
schema_name is any schema keyname (user, group, role, or a custom schema).
srcfield must hold a single scalar value — a bare string/number, or a list containing exactly one item (as produced by a monovalued multi-cardinality field). Anything else (missing field, empty list, list with more than one item, non-scalar) sets dstfield to "".
This lookup is served from the DataEAV cache, which may lag slightly behind very recent writes (see cavaliba cache --eav_refresh).
Examples:
# Check that a username from an import exists as a Cavaliba user
- field_lookup: ['', resolved_user, user, keyname, raw_login]
# Before : {'raw_login': 'alice'}
# After : {'raw_login': 'alice', 'resolved_user': 'alice'} (if alice exists)
# After : {'raw_login': 'nobody', 'resolved_user': ''} (if not found)
# Resolve a server's keyname from an external dataset, matching on its external_code field
- field_lookup: ['', server, myserver_schema, external_code, raw_server_code]
align_subnet4
new v3.24
- align_subnet4: [CONDITION, fieldname]
Aligns an IPv4 address/mask to subnet boundaries. Transforms an IP address with prefix length to the network address with the same prefix length.
Example:
- align_subnet4: ['', subnet_field]
# Before : {'subnet_field': '10.1.1.1/24'}
# After : {'subnet_field': '10.1.1.0/24'}
# Before : {'subnet_field': '192.168.50.100/16'}
# After : {'subnet_field': '192.168.0.0/16'}
discard
- discard: [CONDITION]
Eliminates the full entry. The entry is not written to the database.
exit
new v3.33
- exit: [CONDITION, status]
Stops processing remaining tasks for the current entry. The entry is kept (not discarded).
The optional status parameter sets the return value for other processing steps. Defaults to 'exit' if omitted.
Example:
- set_condition: [IS_DONE, field_match, status, '^done$']
- exit: [IS_DONE, "all-done"]
- field_set: ['', needs_processing, 'yes']
# entries with status='done' skip remaining tasks and are saved as-is