This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Data Management

Data Models

1: DATA - Class/Schema
2: DATA - Pipeline
3: DATA - DataViews
4: DATA - Enumerate
5: DATA - Authorizations
6: DATA - Handle

1 - DATA - Class/Schema

DataClass and Schema

DataClass are top-level asset families : Devices, Applications, Laptops, Projects, Customers, Books, Facilities, Cooking Receipes, Invoices, etc.

Schemas describes fields/attributes for each DataClass. Schema can be modified at any time during the life of the system: add or remove fields, change constraints.

Field types can be basic : string, int, float, boolean, date, IP addresses, … or more complex structure : other objects, enumetate (static) lists, users and groups, computed fields, etc. Fields can be single or multi-valued.

Instances are individual assets.

Instances have relationships : a geographical sites is located in a region or country. A virtual machine is hosted on a physical server, etc.

JSON/YAML - Datamodels (Class and Schema) are best configured with JSON/YAML files uploaded to Cavaliba from UI, API or CLI command, with appropriate permissions.

Cavaliba Data Management dynamicaly provides :

a Responsive Web UI for humans : create, view, edit assets
a REST API for machines: same operations, single instance/field level, or massive bulk transfer
a CLI command to import/export in batch from/to external systems
a relationship framework, including inheritance / propagation of fields and objects between Instances
a Role/Permission model to handle authorization at Class/Instance/Field level
a Data storage at scale on standard relationnal databases (PostgreSQL, MariaDB, MySQL, …) with an Entity-Attribute-Value (EAV) dynamic datamodel.

Global Fields for all DataClass

All DataClass share a small number of mandatory fields.

keyname - primarykey - unique name of the DataClass ; no special chars ; must be unique
displayname - a short string for humans
page - string, page name in the Web UI displaying the list of available DataClasses
order - int, display order in the Web UI
icon - string, name of a FontAwsome Icon to display in UI
handle_method - method to create a unique HTML friendly primary key for instances
permissions… - a set of permissions to control authorizations on instances

Schema Fields

Each DataClass Schema can have zero or more custom fields. These fields can be single-valued or multi-valued. They have a type from one of the available field types below.

Field name must be a valid slug string (unique, no special chararcter) within the DataClass schema.

Available field types:

string
int
float
date
datetime
time
boolean
ipv4
text
enumerate (static lists)
schema (other DataClass)
user group
user

YAML Example

This simple example create a DataClass to manage user Laptops.

# Asset = User Laptops

- classname: _schema
  keyname: laptop
  displayname: User Laptops
  is_enabled: yes
  icon: fa-laptop
  order: 200
  page: Data
  handle_method: md5

  serial:
        displayname: serial_number
        page: General
        order: 100
        dataformat: string
               
  name:
        displayname: hostname
        page: General
        order: 110
        dataformat: string

  os:
        displayname: Operating System
        page: General
        order: 120
        dataformat: enumerate
        dataformat_ext: enumerate_os

  site:      
        displayname: Site
        dataformat: schema
        dataformat_ext: site
        order: 130
        page: General
        cardinal_min: 0
        cardinal_max: 1 

  owner:      
        displayname: User
        dataformat: user
        order: 140
        page: General
        cardinal_min: 0
        cardinal_max: 0

  purchase_date:
        displayname: Purchase date
        dataformat: date

  waranty_end_date:
        displayname: Waranty end date
        dataformat: date

  bitlocker:
        displayname: Bitlocker
        page: Security
        order: 500
        dataformat: boolean

YAML Reference


- classname: _schema
  _action: create/delete/update/enable/disable
  keyname: string
  displayname: string
  is_enabled: boolean
  handle_method: [keyname|uuid|external|md5]
  icon: icon_name
  order: int
  page: string
  p_admin:  PERMISSION_NAME
  p_read:   PERMISSION_NAME
  p_create: PERMISSION_NAME
  p_update: PERMISSION_NAME
  p_delete: PERMISSION_NAME
  p_import: PERMISSION_NAME
  p_export: PERMISSION_NAME

  field_name:
       _action: create/delete/update/enable/disable
       displayname: string
       dataformat: fieldtype
       dataformat_ext: extended_fieldtype
       cardinal_min: int
       cardinal_max: int
       default : string
   (...)

2 - DATA - Pipeline

Pipelines

Pipelines are small transformations applied on incoming data before writing Instances to the Database.

Pipelines are managed as a special built-in _pipeline DataClass.

Pipelines contain tasks to perform atomic operations on each field of each incoming data instance.

Pipeline Example

# A small pipeline to adapt user.csv to bulk load users from external systems

- classname: _pipeline
  keyname: user_import_pipeline
  displayname: user_import_csv
  is_enabled: true
  description: Use this pipeline to import users as a csv file from system X/Y/Z
  content: |
      csv_delimiter: ';'
      classname: user
      keyfield: login
      tasks: 
          - field_lower: email
          - field_lower: login
          - field_upper: external_id
          - field_uuid: uuid_auto
          - field_datetime_now: last_sync

Pipeline usecase

$ docker exec -it cavaliba_app python manage.py \
         cavaliba_load /files/user.csv --pipeline user_import_pipeline

Pipeline tasks reference


        - field_add: new_field
        - field_copy: ['firstname','firstname2']
        - field_rename: ['lastname','LastName']
        - field_delete : 'external_id'
        - field_lower: 'displayname'
        - field_upper: 'displayname'
        - field_date_now: fieldname              : YYYY-MM-DD
        - field_time_now: fieldname              : HH:MM:SS
        - field_datetime_now: fieldname          : YYYY-MM-DD HH:MM:SS
        - field_keep: ["field1", "field2", ...]  : keep only these fields (and classname/keyname)
        - field_regexp_sub: ["fieldname", "pattern","replace"]
        - field_uuid: "fieldname"

3 - DATA - DataViews

Reference

A DataView describes a subset of a DataClass to be presented to the users in the Web Interface.

You can define multiple DataViews per DataClass for different type of users.

DataView objects

DataViews are implemented as regular DataClass objects and can thus be managed from the Web UI, REST API, console CLI, import/export, etc.

DataViews

# optional Role definitions to manage DataView authorizations
_role:role_data_view_ro:
    displayname: Role data:data_view:ro
    subgroups:
      - admin_cavaliba

_role:role_data_view_rw:
    displayname: Role data:data_view:rw
    subgroups:
      - admin_cavaliba

_role:role_data_view_admin:
    displayname: Role data:data_view:admin
    subgroups:
      - admin_cavaliba

# DataView definition as a specific Data Schema structure

_schema:data_view:
    _displayname: Data Views
    _is_enabled: yes
    _icon: fa-table
    _order: 810
    _page: Internal
    _role_show: role_data_view_ro
    _role_access: role_data_view_ro
    _role_read: role_data_view_ro
    _role_create: role_data_view_rw
    _role_update: role_data_view_rw
    _role_delete: role_data_view_rw
    _role_onoff: role_data_view_rw
    _role_import: role_data_view_admin
    _role_export: role_data_view_admin
    description:
        #_action: create_or_update          
        displayname: Description
        order: 100
        dataformat: string
        dataformat_ext: ""
        cardinal_min: 0
        cardinal_max: 1
        default : ""
    # DataClass on which a DataView operates
    classname:
        displayname: classname
        cardinal_min: 0
        cardinal_max: 1
        order: 110
        dataformat: string
    # nested YAML structure with columns and operations on columns
    content:
        displayname: content
        cardinal_min: 0
        cardinal_max: 1
        order: 120
        dataformat: text
        dataformat_ext: yaml

Common Dataview attributes

The classname attribute defines the DataClass on which the DataView will operate.

keyname, displayname and last_update are common attributes to all DataClass. They can be omitted from the DataViews columns. If none is requested , Cavaliba adds keyname by default.

content attribute

The content attribute in each DataView object is a list of columns to be displayed and operators to apply on each of these columns. It is a YAML columns array which preserves order. Each column name starts with a -.

columns:
    - columnA
    - columnB
    (...)
    - columnX:
        from: my_original_long_column_name

The from option defines a source field which provides content for that column. An invalid from value will create an empty column. The intended use of the column/from pattern is to provide a way to provide nicer (custome, translated, compact, …) column names to users.

Some new operators may be added in the future, like aggregate values or computed values from other columns.

Example:

columns:
   - ZipAgain:
       from: zipcode

Full DataView Example

The follwing YAML snippet create a DataView on the site objects to present some relevant geographical informations.

It says:

create a user available DataView for the DataClass site objects.
show a table with the following columns: address, region, …
some of the columns will have a different name than the underlying DataClass attribute

data_view:site_postal:
  #_action: create_or_update
  classname: site
  is_enabled: true
  displayname: MySiteView_postal
  description: This View displays Geographical information about sites
  content: |
    columns:
      - site name:
          from: keyname
      - address
      - zipcode
      - city:
      - region
           from: state_or_region
      - country
      - code
           from: iso_code

4 - DATA - Enumerate

Reference

Enumerate are predefined lists of values which can be used as Field attributes when defining DataClass models. Enumerate values have various additional options : display widget, description or intended use, etc.

When editing DataClass objects, user will be presented with a list of available values only.

When displaying a DataClass objets, nice widgets can be presented to users.

Enumerate usecase examples:

status : OK, KO, N/A
maturity_level : Good, Medium, Average, Bad, Terrible, NotAvailable, Unkown, ToBeChecked, …
grade : A,B,C,D,E
fruits : apple, pear, mango, …
publish_status : draft, published, retracted
color : red, blue, yellow

Enumerate objects

Enumerate are implemented as regular DataClass objects and can thus be managed from the Web UI, REST API, console CLI, import/export, etc.

# optional Role definitions to manage DataView authorizations
_role:role_data_enumerate_ro:
    displayname: Role data:data_enumerate:ro
    subgroups:
      - admin_meteosi


_role:role_data_enumerate_rw:
    displayname: Role data:data_enumerate:rw
    subgroups:
      - admin_meteosi

_role:role_data_enumerate_admin:
    displayname: Role data:data_enumerate:admin
    subgroups:
      - admin_meteosi

# Enumerate definition as a specific DataClass structure
_schema:data_enumerate:
    _displayname: Data Enumerates
    _is_enabled: yes
    _icon: fa-table
    _order: 810
    _page: Internal
    _role_show: role_data_enumerate_ro
    _role_access: role_data_enumerate_ro
    _role_read: role_data_enumerate_ro
    _role_create: role_data_enumerate_rw
    _role_update: role_data_enumerate_rw
    _role_delete: role_data_enumerate_rw
    _role_onoff: role_data_enumerate_rw
    _role_import: role_data_enumerate_admin
    _role_export: role_data_enumerate_admin
    description:
        #_action: create_or_update          
        displayname: Description
        order: 100
        dataformat: string
        dataformat_ext: ""
        cardinal_min: 0
        cardinal_max: 1
        default : ""
    # nested YAML content, with values, widget, see below
    content:
        displayname: content
        cardinal_min: 0
        cardinal_max: 1
        page: input
        order: 120
        dataformat: text
        dataformat_ext: yaml

Enumerate Content format

The content attribute defines available values and their options:

value : short value to display
value_long : longer value to display
value_num : used in aggregate function accross fields (min, max, avg, …)
is_enabled : available for selection
description : explain what/when this value can be selected

Example:

- value: "A"
  value_long: "A - good"
  value_num: 1
  is_enabled: True
  widget: "green_circle"
  description: "A value to use when eveything is good"

- value: "Z"
  is_enabled: False     
  description: "deprecated; not available"

        "red_circle":    "&#x1f534;",
        "orange_circle": "&#x1F7E0;",
        "yellow_circle": "&#x1F7E1;",
        "green_circle":  "&#x1F7E2;",
        "purple_circle": "&#1F7E3;",
        "brown_circle":  "&#1F7E4;",
        "blue_circle":   "&#x1F535;",
        "white_circle":  "&#x25EF;",
        "black_circle":  "&#x2B24;;",
        "default" : ""

Using Enumerate in Data Schemas


_schema:my_test_class:
    #_action: create
    _displayname: TestClass

    # (...)

    my_enumerate_abc:      
        displayname: MyEnumerate_ABC
        dataformat: enumerate
        dataformat_ext: enum_ABC
        description: Choose an enum_ABC value
        order: 520
        page: Static-Enum
        cardinal_min: 0
        cardinal_max: 1

    my_enumerate_abc_multi:      
        displayname: MyEnumerate_ABC_Multi
        dataformat: enumerate
        dataformat_ext: enum_ABC
        description: Choose enum_ABC values
        order: 525
        page: Static-Enum
        cardinal_min: 0
        cardinal_max: 0

    # (...)

Enumerate Examples

# ---------------------
# standard enumerates
# ---------------------

data_enumerate:enum_ABC:
  # ACTION
  is_enabled: True
  displayname: "ABC Scale"
  description: "Neutral ABC Scale"
  content: |
    - value: "A"
      widget: "green_circle"
      description: "A"
      value_num: 1

    - value: "B"
      widget: "orange_circle"
      value_num: 2

    - value: "C"
      widget: "red_circle"
      value_num: 3    


data_enumerate:maturite_ABCDE:
  # ACTION
  is_enabled: True
  displayname: "ABCDE Maturity scale"
  description: "Use for product assessment"
  content: |
    - value: "A"
      value_long: "A - Perfect"
      #is_enabled: True
      widget: "green_circle"
      description: "Use if perfect"
      value_num: 1

    - value: "B"
      value_long: "B - Correct"
      widget: "yellow_circle"
      value_num: 2

    - value: "C"
      value_long: "C - Average"
      widget: "orange_circle"
      value_num: 3

    - value: "D"
      value_long: "D - Bad"
      widget: "red_circle"
      value_num: 4

    - value: "E"
      value_long: "E - Alert"
      widget: "black_circle"
      value_num: 5

    - value: "n/a"
      value_long: "n/a"
      widget: "white_circle"

    - value: "?"
      value_long: "to be checked"
      widget: "blue_circle"    



data_enumerate:enum_OK_KO:
  # ACTION
  is_enabled: True
  displayname: "OK/KO Scale"
  content: |
    - value: "OK"
      widget: "green_circle"
      description: "OK"
      value_num: 1

    - value: "KO"
      widget: "red_circle"
      value_num: 2    


data_enumerate:enum_OK_KO_NA:
  # ACTION
  is_enabled: True
  displayname: "OK/KO/NA Scale"
  content: |
    - value: "OK"
      widget: "green_circle"
      description: "OK"
      value_num: 1

    - value: "KO"
      widget: "red_circle"
      value_num: 2

    - value: "n/a"
      widget: "white_circle"

    - value: "?"

5 - DATA - Authorizations

Data Reference

Data Authorization model

User access to Data Ressources requires a combination of two authorizations:

Global - Operations Authorization on All Data Ressources (access/read/write/…) ; given by Permissions.
Class/Schema - Operations Authorizations on specific Class/Schema ; given by Roles attached to Class/Schema

Global permissions

Global operations on all Data objects is controlled by a small set Permissions.

     ("p_data_access", "Access UI", "", False),
     ("p_data_class_ro", "Access RO on classes", "", False),
     ("p_data_class_rw", "Access RWD on classes", "", False),
     ("p_data_schema_ro", "Schema RO", "", False),
     ("p_data_schema_rw", "Schema RWD", "", False),
     ("p_data_instance_ro", "Global RO on instances", "", False),
     ("p_data_instance_rw", "Global RWD on instances", "", False),
     ("p_data_import", "Use file or YAML import tool", "", False),
     ("p_data_admin", "Other sensitive actions on data app", "", False),

you may provide thes Permissions to users through standard Role/Group. You can use Built-in Roles.

Built-in Roles

Give the following Roles to Users and Groups to provide Global Authorization.


ROLES_BUILTIN = {
    'role_data_ro': [
        "p_data_access",
        "p_data_class_ro",
        "p_data_schema_ro",
        "p_data_instance_ro",

    ],
    # class: create/update/delete/onoff
    'role_data_rw': [
        "p_data_access", 
        "p_data_class_rw",
        "p_data_schema_rw",
        "p_data_instance_rw",
    ],
    # class: import/export
    'role_data_admin': [
        "p_data_access", 
        "p_data_class_rw",
        "p_data_schema_rw",
        "p_data_instance_rw",
        "p_data_import",
        "p_data_admin",
    ],
}

Per Class/Schema Authorizations

Per Class/Schema Authorization is provided by Roles and is mandatory in addition to Global Authorization.

Each class has a set of _role attributes to define which Role controls allowed operation on the Class and its Instances.

# role definitions ...

_role:role_data_test_ro:
    displayname: Role DATA test RO
    subgroups:
      - usergroup1


_role:role_data_test_rw:
    displayname: Role DATA test RW
    subgroups:
      - usergroup2

_role:role_data_test_admin:
    displayname: Role DATA test ADMIN
    subgroups:
      - usergroup3

# .. can be used in specific Class/Schema definition:

_schema:test_class:
    
    _role_show: role_data_test_ro
    _role_access: role_data_test_ro
    _role_read: role_data_test_ro
    _role_create: role_data_test_rw
    _role_update: role_data_test_rw
    _role_delete: role_data_test_rw
    _role_onoff: role_data_test_rw
    _role_import: role_data_test_admin
    _role_export: role_data_test_admin

Per Instance Authorization

Not implemented.

Per Field Authorization

Not implemented.

6 - DATA - Handle

Reference

Handle is a built-in field for all Instances as a companion field to keyname primary keys. Handles are automatically computed fields to derive an URI friendly primary key for Instances

keyname must be a primary key, that is, must have a unique value accross all instances of a DataClass.

Sometimes, you may want to use an external primary key from an other system to maintain synchronization ; or may wan’t a nice human friendly keyname for display reason. If your keyname - although unique - does have special characters (accent, white spaces, comma, …) it’is not suitable for Web URIs and REST APIs.

When you define a DataClass, you have to choose the method to compoute the handle field. By default, the keyname field is used.

Handle methods

handle_method
    # keyname
    # uuid
    # md5
    # external

Data Management

1 - DATA - Class/Schema

Global Fields for all DataClass

Schema Fields

YAML Example

YAML Reference

2 - DATA - Pipeline

Pipeline Example

Pipeline usecase

Pipeline tasks reference

3 - DATA - DataViews

DataView objects

Common Dataview attributes

content attribute

Full DataView Example

4 - DATA - Enumerate

Enumerate objects

Enumerate Content format

Available Widget values

Using Enumerate in Data Schemas

Enumerate Examples

5 - DATA - Authorizations

Data Authorization model

Global permissions

Built-in Roles

Per Class/Schema Authorizations

Per Instance Authorization

Per Field Authorization

6 - DATA - Handle

Handle methods