# Goal Facilitate obtaining metrics from [Kwollect](https://www.grid5000.fr/w/Monitoring_Using_Kwollect) using EnOSlib. Example: obtain the power consumption of physical nodes during an experiment, as measured by physical wattmeters or the BMC. # Scope - support all Kwollect metrics: energy, network traffic, BMC, node metrics - possibly add a more specialized interface for energy monitoring (automatic selection of the most appropriate energy metric, automatic computation of total energy usage) - allow both interactive processing of the metrics (e.g. to display them graphically in a notebook) and dump them to a file for archival purpose # Existing support python-grid5000 already supports querying kwollect: see "Metrics API" in https://gitlab.inria.fr/msimonin/python-grid5000 # User interface Implement the "Service" interface, similarly to Docker or K3S. ## Envisioned usage ```python # Instantiate the service with a list of nodes (through a role) k = en.Kwollect(roles=roles["mycomputenodes"]) # Validates that nodes are all from Grid'5000 and are known to kwollect k.deploy() # Register timestamp k.start() # the main part of your experiment goes here # Register timestamp k.stop() ``` ## Obtaining metrics directly ```python # Obtain metrics measured during start and stop. # By default, all metrics for all nodes would be returned. # We allow to filter by metrics and by nodes to only get a subset of the data. data = k.get_metrics(metrics=["metric1", "metric2"], roles=roles["subrole"]) # Output data is a dictionary: keys are nodes, values are list of data points # All metrics are returned in the same list, in chronological order. data["nova-1.lyon.grid5000.fr"][0] {'timestamp': '2024-11-26T15:44:04+01:00', 'device_id': 'nova-1', 'metric_id': 'wattmetre_power_watt', 'value': 6.914999999999997, 'labels': {'_device_orig': ['wattmetre1-port7']}} data["nova-1.lyon.grid5000.fr"][1] {'timestamp': '2024-11-26T15:44:05.460909+01:00', 'device_id': 'nova-1', 'metric_id': 'network_ifacein_bytes_total', 'value': 1157021682632, 'labels': {'interface': 'eth0', '_device_orig': ['force10-port-25']}} ``` ## Dumping metrics to a file ```python k.backup(backup_dir="my_experiment") ``` TODO: evaluate if CSV is suitable, or if we need a more complex format. The labels will be hard to represent as CSV. ## Advanced usage: several instances It should be possible to instantiate the service several times with different nodes and/or different start and stop time. Instances can be arbitrarily nested (start kwollect1, start kwollect2, stop kwollect1, stop kwollect2). ## Advanced usage: custom timestamps For projects like E2CLAB, it would be easier to collect timestamps first (e.g. at each step of an experiment), and then decide later which period(s) are interesting and for which we need metrics. It can be done by overriding timestamps for start and stop: ```python k.start(timestamp=t1) k.stop(timestamp=t2) k.get_metrics() ``` # Documentation The service class should be documented. An example with a custom job type (to collect energy metrics at a higher frequency) should be provided. Some examples of how to filter/merge/analyze the data should be provided. python-grid5000 has such an example, it could be adapted. # Implementation - deploy() performs sanity checks - start() records a timestamp (no API call) - stop() records a timestamp (no API call) - get_metrics() and backup() perform the actual API call to kwollect to fetch data - one API call for each Grid'5000 site (ideally in parallel) - then split the resulting data on each node - for large datasets, the kwollect API can timeout: we may need to perform smaller requests (e.g. one API call for subgroups of nodes) - the dumping process in backup() should be efficient and avoid loading all data in memory (to allow dumping data for large experiments) # TODO - [x] implement the skeleton of the code - [x] implement `get_available_metrics` method - [ ] add `info` methods - [ ] add a Jupyter example - [ ] determine a suitable export format (CSV, JSONP, prometheus...) - [ ] think about a more specialized service for energy monitoring - [ ] add another service for "after-the-fact" analysis, based on the job ID (e.g. to compute average CPU or energy usage for a previous job)