SRE / Administration

• 1: CLI tools
• 2: Backup and restore
• 3: Periodic tasks

1 - CLI tools

Cavaliba comes with various command-line tools.

First, create these 2 aliases for easy access to command line commands (CLI):

$ alias cavaliba="docker exec -it cavaliba_app python manage.py cavaliba"
$ alias cavmanage="docker exec -it cavaliba_app python manage.py"

Add them to your ~/.basrc for persistence.

The cavmanagecommand will launch the integrated management framework.

You can now type :

$ cavmanage
$ cavaliba --help

You’ll see all available commands.

version

$ cavaliba version
3.17.1

Load schema and data

$ cavaliba load /files/demo/
$ cavaliba load /files/demo/100_site.yaml --pass 2

Note: absolute folder for /files, inside containers ; local folder is files/ on host.

Load CSV data file

$ cavaliba load /files/demo/user/user.csv –pipeline user_import_csv

A pipeline is needed to map columns to Schema fields and usually apply some transformations.

multipass

--pass 2

The --pass option is recommanded for schema, because cross-references between schemas requires some schemas to be created before others. Multiple pass resolve dependencies.

progress

Display progression for large files :

--progress 

File: /files/import/user.csv ...
Found: 55456 objects
Pipeline: user_import_csv
Loading to DB...
100 done
200 done
(...)

first/last

Choose quantity (first/last lines to process) ; 0 for last is unlimited.

--first 1 
--last 0

log view

$ cavaliba log
$ cavaliba log --first 0 --last 1000

log purge

apply automatic log purge:

$ cavaliba log --log_purge
log entries: 203718
log deleted (old) : 48 entries
log done

purge all logs (!):

$ cavaliba log --log_purge_all
log entries: 203676
log deleted (all) : 203676 entries
log done

conf view

$ cavaliba conf
$ cavaliba conf --text
$ cavaliba conf --json
$ cavaliba conf --yaml
$ cavaliba conf --yaml --key LOG_KEEP_DAYS_ERROR

conf sync

Creates default entries / purge orphan. Run automatically during startup.

$ cavaliba conf --confsync --verbose

conf backup/restore

Backup conf:

$ cavaliba conf > files/export/conf_backup.json

Reload conf (from container accessible folder)

$ cavaliba conf --conf_file /files/export/conf_backup.json --verbose

data query/export

Get available schemas:

$ cavaliba schema

Query a schema:

$ cavaliba schema --schema user 
- login: admin
  displayname: Built-in Global Admin user
  is_enabled: true
  want_notifications: true
  want_24: true
  want_email: true
  classname: _user

- login: i.trento
  displayname: Izongua Trento
  is_enabled: true
  want_notifications: true
  want_24: true
  want_email: true
  classname: _user

  (...)

Query a schema and a specific object:

$ cavaliba cavaliba_export --schema user --key testuser01
- login: testuser01
  email: testuser01@test.com
  mobile: 0123456789
  displayname: Test User 01
  description: Test User 01
  is_enabled: true
  want_notifications: true
  want_24: true
  want_sms: true
  want_email: true
  classname: _user

Specify an output format:

$ cavaliba cavaliba_export --schema site --key testsite01 --json
[
    {
        "classname": "site",
        "keyname": "testsite01",
        "handle": "testsite01",
        "displayname": "Test Site 01",
        "p_read": null,
        "p_update": null,
        "p_delete": null,
        "is_enabled": true,
        "sirene_group": [
            "testgroup01"
        ]
    }
]

Run the integrated test suite

First, either create an empty test database in your database server or use the root DµB user in your docker .env file for variable CAVALIBA_DB_USER. Alternatively you may prefer to grant create database privileges to your current CAVALIBA_DB_USER.

Then,

$ cavmanage test

Options exists such as:

$ cavmanage test --keepdb
$ cavmanage test -v 2
$ cavmanage test --keepdb app_home.tests.test_home.ConfTest

You may need to create a test database manually in MariaDB. Example:

docker exec -it cavaliba_db mysql -u root -p
create database test_cavaliba;
show grants for 'cavaliba';
GRANT ALL PRIVILEGES ON `test_cavaliba`.* TO `cavaliba`@`%`;
flush privileges;

2 - Backup and restore

System .env file

You must backup and protect the docker .env file in the cavaliba app directory.

$ cd appdir/
$ cp .env /my/backup/dir/

It contains sensitive informations needed at startup time. It is created / modified during system setup and reconfig/

Most needed data are encryption keys, and various credentials : DB, admin account, OIDC client secrets, etc.

Restoration is a simple as copying back the file to the restored application folder.

Application configuration

Most configuration items are managed from the application itself, and are stored in the database.

You must backup this configuration:

$ cd appdir/                                              # or whatever your setup root dir is
$ cavaliba conf                                           # show current configuration
$ cavaliba conf --yaml > files/export/conf_backup.yaml    # load and replace in-DB configuration

Remember to setup cavaliba as an alias to management commands as explained in the command-line (CLI) doc page.

To restore the configuration on a new deployement :

$ cd appdir
$ cavaliba conf 
$ cavaliba conf --conf_file /files/export/conf_backup.json

Notice the / before files/ to ensure access from inside the containers.

Full database backup

Cavaliba provides script/db_backup.sh to backup the full MySQL/MariaDB automatically.

Setup:

1. create your backup dir
2. install standard zstd compression tool if missing
3. edit the script and set your choosen DB password (.env), BACKUPDIR
4. launch manually ./script/db_backup and check produced backup file in your Backupdir
3. add to crontab with provided example in the script

   25 */6 * * * /opt/cavaliba/script/db_backup.sh > /tmp/backup_db.log 2>&1

It creates a periodically rolling backup for 7 days.

Restore:

You can restore as a standard MariaDB / MySQL dump

1. stop the application
2. drop / create cavaliba_db in your mysql server
3. zstdcat mybackup.zstd | mysql cavaliba_db

Logicial Data backup

Cavaliba provides a simple data_export script:

# (c) cavaliba.com - DATA Export backup script
#
# crontab example
# 45 */6 * * * /opt/cavaliba/script/data_backup.sh > /tmp/backup_data.log 2>&1


BACKUPDIR="/opt/cavaliba/backup"
USER="root"
PASSWORD="xxxxxxxxxxxxxxxxxxx"

cd $BACKUPDIR

DAY=$(date "+%u")
HOUR=$(date "+%H")


DATE=$(date +%Y%m%d_%H:%M:%S)
FILE="$BACKUPDIR/${DAY}_${HOUR}_schema.zst"
echo "$DATE start dumping SCHEMA to $FILE"
docker exec cavaliba_app python manage.py dumpdata --format json --indent 2 app_data.dataschema --natural-primary --natural-foreign | zstd >$FILE

DATE=$(date +%Y%m%d_%H:%M:%S)
FILE="$BACKUPDIR/${DAY}_${HOUR}_classname.zst"
echo "$DATE start dumping CLASS to $FILE"
docker exec cavaliba_app python manage.py dumpdata --format json --indent 2 app_data.dataclass --natural-primary --natural-foreign | zstd >$FILE

DATE=$(date +%Y%m%d_%H:%M:%S)
FILE="$BACKUPDIR/${DAY}_${HOUR}_instance.zst"
echo "$DATE start dumping INSTANCE to $FILE"
docker exec cavaliba_app python manage.py dumpdata --format json --indent 2 app_data.datainstance --natural-primary --natural-foreign | zstd >$FILE

DATE=$(date +%Y%m%d_%H:%M:%S)
FILE="$BACKUPDIR/${DAY}_${HOUR}_conf.zst"
echo "$DATE start dumping CONF to $FILE"
docker exec cavaliba_app python manage.py dumpdata --format json --indent 2 app_home.cavalibaconfiguration --natural-primary --natural-foreign | zstd >$FILE


DATE=$(date +%Y%m%d_%H:%M:%S)
FILE="$BACKUPDIR/${DAY}_${HOUR}_iam.zst"
echo "$DATE start dumping IAM (USER GROUPS PERM ROLES) to $FILE"
docker exec cavaliba_app python manage.py dumpdata --format json --indent 2 app_user --natural-primary --natural-foreign | zstd >$FILE

CLI data export

You can export specific object to any file:

$ cavaliba schema --schema mybookstore --json > /my/backup/dir/schema_mybookstore.json

And reload with

$ cavaliba load schema_mybookstore.json

Cavaliba Software distribution

You should also keep the downloaded software distribution for later reinstall :

$ wget https://cavaliba.com/download/cavaliba/latest.tar.gz
$ cp latest.tar.gz /my/backup/dir

3 - Periodic tasks

Cavaliba has an internal scheduler to run various periodic tasks. You have nothing to configure.

The only tasks not handled internally are backup tasks which you should configure on your own (see dedicated doc page).

Current internal Tasks

"log_purge": {
    "task": "app_home.tasks.log_purge",
    "schedule": crontab(hour=20, minute=42),
},
"clear_expired_sessions": {
    "task": "app_home.tasks.clear_expired_sessions",
    "schedule": crontab(hour=19, minute=12),
},
"archive_expired_sirene": {
    "task": "app_sirene.notify.archive_expired_sirene",
    "schedule": 60.0,
},
"status_run_monitor": {
    "task": "app_status.tasks.task_run_monitor",
    "schedule": 15.0,
},
"status_sample_hour": {
    "task": "app_status.tasks.task_sample_hour",
    "schedule": 60.0,
},
"status_sample_day": {
    "task": "app_status.tasks.task_sample_day",
    "schedule": 900.0,
},
"status_cleanup_raw": {
    "task": "app_status.tasks.task_cleanup_raw",
    "schedule": 900.0,
},
"status_cleanup_hour": {
    "task": "app_status.tasks.task_cleanup_hour",
    "schedule": 18000.0,
},