diff --git a/README.md b/README.md index 3fdae3df22ffa34da705b677e52288c2d3ce9225..e35da06e0311893563f40edcbff5f2bf485787c5 100644 --- a/README.md +++ b/README.md @@ -109,19 +109,20 @@ pip3 install kwollect Kwollect needs a [PostgreSQL](https://www.postgresql.org/) database with [TimescaleDB](https://www.timescale.com/) extension to store metrics. -For example, use these commands to install them on Debian Buster: +For example, use these commands to install them on Debian Bullseye: ``` -sudo sh -c "echo 'deb https://packagecloud.io/timescale/timescaledb/debian/ `lsb_release -c -s` main' > /etc/apt/sources.list.d/timescaledb.list" -wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | sudo apt-key add - -sudo apt update -sudo apt-get install -y postgresql postgresql-client timescaledb-postgresql-11 postgresql-plpython3-11 +wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | apt-key add - +echo 'deb https://packagecloud.io/timescale/timescaledb/debian/ bullseye main' > /etc/apt/sources.list.d/timescaledb.list +apt update +apt-get install -y --no-install-recommends postgresql postgresql-client libpq-dev timescaledb-2-postgresql-13 timescaledb-tools postgresql-plpython3-13 -# TimescaleDB comes with a script to tune Postgres configuration that you might want to use: -sudo cp /etc/postgresql/11/main/postgresql.conf /etc/postgresql/11/main/postgresql.conf-timescaledb_tune.backup -sudo timescaledb-tune -yes -quiet -echo 'timescaledb.telemetry_level=off' | sudo tee -a /etc/postgresql/11/main/postgresql.conf -sudo systemctl restart postgresql +## TimescaleDB comes with a script to tune Postgres configuration that you might want to use: +cp /etc/postgresql/13/main/postgresql.conf /etc/postgresql/13/main/postgresql.conf-timescaledb_tune.backup +timescaledb-tune -yes -quiet +echo 'timescaledb.telemetry_level=off' >> /etc/postgresql/13/main/postgresql.conf + +systemctl restart postgresql ``` Then, you can setup Kwollect database using the `kwollect-setup-db` tool. It is @@ -166,7 +167,7 @@ jwt-secret = "changemechangemechangemechangemechangeme" *kwollector-setup-db* also outputs an *API token* that is needed to perform write access to the database. -Finally, don't forget to start Postgres with `postgrest +Finally, don't forget to start Postgrest with `postgrest <path_to_configuration_file>` @@ -319,8 +320,6 @@ connected to your Kwollect's database and [import](https://grafana.com/docs/grafana/latest/dashboards/export-import/#importing-a-dashboard) our dashboard. -Some more specialized visualization dashboard [have also been developed](https://gitlab.inria.fr/grid5000/kwollect-dashboard). - # Advanced topics @@ -360,12 +359,28 @@ requests on `metrics_by_job`, e.g.: SELECT timestamp, device_id, metric_id, value FROM metrics_by_job WHERE job_id = 1234; ``` -It is also possible to provide the "job_id" argument when calling API: +It is also possible to provide the "job_id" argument when calling API's `get_metrics`: ``` curl http://kwollect.host:3000/rpc/get_metrics?job_id=1234 ``` +Once job scheduler integration configured, an additional `get_job_metrics` API function is available to provide "job-wide" metrics: + +``` +curl http://kwollect.host:3000/rpc/get_job_metrics?job_id=1234 +``` + +The list of SQL requests to be performed to obtain "job-wide" metrics must be defined in a configuration file (one request per-line, performed on `jobmetrics` table), which must be provided to `--jobmetrics_requests_path` option of `kwollect-setup-db`. Here is an example of such file: +``` +SELECT SUM(jobnode_energy_watthour) AS job_energy_watthour FROM (SELECT AVG(value) * EXTRACT(EPOCH FROM AGE(MAX(timestamp), MIN(timestamp)))/3600 AS jobnode_energy_watthour FROM jobmetrics WHERE metric_id = 'bmc_node_power_watt' GROUP BY device_id) s +SELECT MAX(value) AS jobnode_maxpower_watt FROM jobmetrics WHERE metric_id = 'bmc_node_power_watt' +SELECT AVG(1-value) AS job_cpu_avgusage_percent FROM jobmetrics WHERE metric_id = 'prom_node_cpu_seconds_total' AND labels->>'mode' = 'idle' +SELECT MAX(1-value) AS job_cpu_maxusage_percent FROM jobmetrics WHERE metric_id = 'prom_node_cpu_seconds_total' AND labels->>'mode' = 'idle' +SELECT AVG(value)*8/1024/1024 AS job_network_avginput_mbps FROM jobmetrics WHERE metric_id = 'network_ifaceout_bytes_total' +SELECT MAX(value)*8/1024/1024 AS jobnode_network_maxinput_mbps FROM jobmetrics WHERE metric_id = 'network_ifaceout_bytes_total' +``` + ## Optional metrics @@ -390,6 +405,16 @@ currently existing jobs. It is called by OAR at the beginning and at the end of the jobs. +## Metrics summary + +Metrics stored in the database are automatically averaged over a 5 minutes +period. These summarized metrics are used by Grafana dashboard when the +timerange to be displayed is greater than 30 minutes, to avoid overload it. + +If required, summarized metrics can be accessed using the `summary=1` +parameter of the `get_metrics` API call. + + ## Wattmetre A specific kwollector, called *kwollector-wattmetre*, is available to read and @@ -426,3 +451,15 @@ wattmetre, may be provided to associate metrics collected from a wattmetre port to the corresponding device. The file should contain one `device_id: [wattmetre_id-portX, wattmetre_id-portY]` line per device, containing the device identifier followed by the list of wattmetres and ports which power it. + + +## Other features + +- Prometheus metrics with name "kwollect_custom" are managed specially: if they + include a `_metric_id` and/or `_timestamp` in their label, they will be + inserted with specified metric_id and/or timestamp in the Kwollect database +- Automatic archiving of oldest metrics in a different filesystem is available + using `--archive_path` option of `kwollector-setup-db` +- Increase rate of counter-type metrics can be obtained by adding to API call + `get_metrics` the parameter `as_rate=1` (or `as_rate=auto` to only process + metrics ending with `_total`) .