From 08bdb5fb91fc55aa7a61b4d5fe93af7d3b0f5e11 Mon Sep 17 00:00:00 2001 From: Francesco Cremonesi <francesco.cremonesi@inria.fr> Date: Mon, 3 Jul 2023 16:53:51 +0200 Subject: [PATCH] Heart disease notebooks --- _toc.yml | 4 + .../tutorial-sklearn-problem.ipynb | 395 +++ .../tutorial-sklearn-solutions.ipynb | 2208 +++++++++++++++++ 3 files changed, 2607 insertions(+) create mode 100644 fedbiomed-tutorial/tutorial-sklearn-problem.ipynb create mode 100644 fedbiomed-tutorial/tutorial-sklearn-solutions.ipynb diff --git a/_toc.yml b/_toc.yml index 9b1284e..87003f1 100644 --- a/_toc.yml +++ b/_toc.yml @@ -12,6 +12,10 @@ parts: chapters: - file: fedbiomed-tutorial/intro-tutorial-mednist.ipynb title: Intro tutorial (MedNIST) + - file: fedbiomed-tutorial/tutorial-sklearn-problem.ipynb + title: Heart disease detection + - file: fedbiomed-tutorial/tutorial-sklearn-solutions.ipynb + title: Heart disease detection - file: fedbiomed-tutorial/brain-segmentation-exercise.ipynb title: Brain segmentation - file: fedbiomed-tutorial/brain-segmentation-solution.ipynb diff --git a/fedbiomed-tutorial/tutorial-sklearn-problem.ipynb b/fedbiomed-tutorial/tutorial-sklearn-problem.ipynb new file mode 100644 index 0000000..d3bb4db --- /dev/null +++ b/fedbiomed-tutorial/tutorial-sklearn-problem.ipynb @@ -0,0 +1,395 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "64e87007", + "metadata": { + "pycharm": { + "name": "#%% md\n" + } + }, + "source": [ + "# Heart disease detection" + ] + }, + { + "cell_type": "markdown", + "id": "9e351015", + "metadata": {}, + "source": [ + "In this tutorial, we will focus on applying federated learning techniques to a classification problem using Scikit-Learn, a popular machine learning library in Python. We will walk you through the process step by step, from setting up the federated learning environment to evaluating the model's performance.\n", + "\n", + "\n", + "Scikit-Learn, also known as sklearn, is a popular machine learning library in Python. It provides a wide range of tools and algorithms for tasks such as data preprocessing, feature selection, model training, and evaluation. Sklearn is widely used for tasks such as classification, regression, clustering, and dimensionality reduction. It offers a user-friendly interface and integrates well with other libraries in the Python ecosystem, making it a go-to choice for many machine learning practitioners and researchers." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ade4cbea", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "%load_ext autoreload\n", + "%autoreload 2" + ] + }, + { + "cell_type": "markdown", + "id": "5827f560", + "metadata": {}, + "source": [ + "# Table of content\n", + "\n", + "1. [The dataset](#dataset)\n", + "2. [Task 1: training plan](#task1)\n", + "3. [Task 2: the experment](#task2)\n", + "4. [Task 3: model validation](#task3)" + ] + }, + { + "cell_type": "markdown", + "id": "59d00ae5", + "metadata": {}, + "source": [ + "# Tutorial" + ] + }, + { + "cell_type": "markdown", + "id": "b9036f7d", + "metadata": {}, + "source": [ + "## The dataset <a name=\"dataset\"></a>" + ] + }, + { + "cell_type": "markdown", + "id": "e4f03a34", + "metadata": {}, + "source": [ + "The Heart Disease dataset available at https://archive.ics.uci.edu/dataset/45/heart+disease is a widely used dataset in the field of cardiovascular research and machine learning. It contains a collection of medical attributes from patients suspected of having heart disease, along with their corresponding diagnosis (presence or absence of heart disease). The dataset includes information such as age, sex, blood pressure, cholesterol levels, and various other clinical measurements.\n", + "\n", + "It was collected in 4 hospitals in the USA, Switzerland and Hungary. This dataset contains tabular information about 740 patients distributed among these four clients. \n", + "\n", + "A federated version of this dataset has been proposed in [Flamby](https://arxiv.org/pdf/2210.04620.pdf). Following thier actions, we preprocess the dataset by removing missing values and encoding non-binary categorical variables as dummy variables. We finally obtain the following centers:\n", + "\n", + "| Number | Client | Dataset size |\n", + "|--------|----------------------|--------------|\n", + "| 0 | Cleveland’s Hospital | 303 |\n", + "| 1 | Hungarian Hospital | 261 |\n", + "| 2 | Switzerland Hospital | 46 |\n", + "| 3 | Long Beach Hospital | 130 | \n" + ] + }, + { + "cell_type": "markdown", + "id": "6d8554fa", + "metadata": {}, + "source": [ + "For teaching purposes, we decided to merge: client0 with client3 and client1 with client2.\n", + "The final federated scenario, in this way, is the following:\n", + "- **client1**, with 349 elements\n", + "- **client2**, with 391 elements" + ] + }, + { + "cell_type": "markdown", + "id": "e860a74e", + "metadata": {}, + "source": [ + "## Task 1: Defining the training plan <a name=\"task1\"></a>" + ] + }, + { + "cell_type": "markdown", + "id": "fd2daf01", + "metadata": {}, + "source": [ + "A training plan is a class that defines the four main components of federated model training: the data, the model, he loss and the optimizer. It is responsible for providing custom methods allowing every node to perform the training.\n", + "\n", + "\n", + "In the case of scikit-learn, Fed-BioMed already does a lot of the heavy lifting for you by providing the FedPerceptron, FedSGDClassifier and FedSGDRegressor classes as training plans. These classes already take care of the model, optimizer, loss function and related dependencies for you, so you only need to define how the data will be loaded.\n", + "\n", + "\n", + "In this tutorial we are going to use an [SGDClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html), so the related FedSGDClassifier training plan." + ] + }, + { + "cell_type": "markdown", + "id": "11ec284b", + "metadata": {}, + "source": [ + "### Model arguments\n", + "*model_args* is a dictionary with the arguments related to the model, that will be passed to the Perceptron constructor.\n", + "\n", + "**IMPORTANT** For classification tasks, you are required to specify the following two fields:\n", + "\n", + "- n_features: the number of features in each input sample (in our case, the number of pixels in the images)\n", + "- n_classes: the number of classes in the target data\n", + "\n", + "Other model arguments depend on the specific model you are using, and are defined in the model definition. Refer to the model documentation \n", + "\n", + "\n", + "### Training arguments\n", + "*training_args* is a dictionary containing the arguments for the training routine (e.g. batch size, learning rate, epochs, etc.). This will be passed to the routine on the node side.\n", + "\n", + "**IMPORTANT** To set the training arguments we may either pass them to the Experiment constructor, or set them on an instance with the setter method:\n", + "\n", + " 'exp.set_training_arguments(training_args=training_args)'\n", + "The setters are available also for single training arguments, like: \n", + "\n", + " 'exp.set_aggregator(aggregator=FedAverage)'" + ] + }, + { + "cell_type": "markdown", + "id": "7571ab66", + "metadata": {}, + "source": [ + "**TO_DO:**\n", + "\n", + "- Apply the scaler to your data\n", + "- Define training args: num_updates, batch_size.\n", + "- Define model args as explained above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "998bb584", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "from fedbiomed.common.training_plans import FedSGDRegressor, FedPerceptron, FedSGDClassifier\n", + "from fedbiomed.common.data import DataManager\n", + "from sklearn.preprocessing import MinMaxScaler\n", + "class SkLearnClassifierTrainingPlan(FedSGDClassifier):\n", + " def init_dependencies(self):\n", + " \"\"\"Define additional dependencies.\n", + " return [\"from torchvision import datasets, transforms\",\n", + " \"from torch.utils.data import DataLoader\"]\n", + "\n", + " def training_data(self, batch_size):\n", + " \n", + " In this case, we rely on torchvision functions for preprocessing the images.\n", + " \"\"\"\n", + " return [\"from sklearn.preprocessing import MinMaxScaler\"]\n", + " \n", + " def training_data(self, batch_size):\n", + " df = pd.read_csv(self.dataset_path, delimiter=';', header=None)\n", + "\n", + " X = df.iloc[:, :-1]\n", + " y = df.iloc[:, -1]\n", + " self.scaler = MinMaxScaler()\n", + " # X = [...] TODO: apply the transformer to the data.\n", + " return DataManager(dataset=X,target=y, batch_size=batch_size, shuffle=True)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "653e8aea", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "n_features = 18\n", + "n_classes = 2\n", + "\n", + "model_args = { 'max_iter':100, \n", + " 'tol': 1e-1 , \n", + " 'loss': 'huber',\n", + " # [...] TODO: Insert the missing model arguments.\n", + " }\n", + "\n", + "training_args = { \n", + " # [...] TODO: Insert the training arguments as elements in the dic.\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "b3020aa6", + "metadata": {}, + "source": [ + "## Task 2: the Experiment <a name=\"task2\"></a>" + ] + }, + { + "cell_type": "markdown", + "id": "9ae58ccf", + "metadata": {}, + "source": [ + "The experiment enables Federated Learning by orchestrating the training process across multiple nodes. It searches for datasets based on specific tags, uploads the training plan file, sends model and training arguments, tracks and checks training progress, and downloads and aggregates model parameters for the next round." + ] + }, + { + "cell_type": "markdown", + "id": "c17ef104", + "metadata": {}, + "source": [ + "**TO_DO:**\n", + "\n", + "- Define the used training plan.\n", + "- Pass model and training arguments " + ] + }, + { + "cell_type": "markdown", + "id": "28f602da", + "metadata": {}, + "source": [ + "<div class=\"alert alert-block alert-info\"> <b>TAGS:</b> Replace %%%% in the tags with your username </div>" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4b1a1341", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "from fedbiomed.researcher.experiment import Experiment\n", + "from fedbiomed.researcher.aggregators.fedavg import FedAverage\n", + "\n", + "tags = ['heart-jupyter-%%%%']\n", + "rounds = 10\n", + "\n", + "# search for corresponding datasets across nodes datasets\n", + "exp = Experiment(tags=tags,\n", + " model_args=None, #TODO: insert the correct value\n", + " training_plan_class=None, #TODO: insert the correct value\n", + " training_args=None, #TODO: insert the correct value\n", + " round_limit=rounds,\n", + " aggregator=FedAverage(),\n", + " node_selection_strategy=None)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6ff55da", + "metadata": { + "pycharm": { + "name": "#%%\n" + }, + "scrolled": false + }, + "outputs": [], + "source": [ + "exp.run()" + ] + }, + { + "cell_type": "markdown", + "id": "88e2a782", + "metadata": { + "pycharm": { + "name": "#%% md\n" + } + }, + "source": [ + "## Task 3: Model Validation <a name=\"task3\"></a>" + ] + }, + { + "cell_type": "markdown", + "id": "84f2ad10", + "metadata": {}, + "source": [ + "During federated training, model validation plays a crucial role in assessing performance without a dedicated holdout dataset. Fed-BioMed enables separate model validation on each node after parameter updates, allowing comparison of model performances. Two types of validation can be performed:\n", + "- one on globally updated parameters before training a round, \n", + "- another on locally updated parameters after local training is completed on a node. \n", + "\n", + "This helps users evaluate the impact of node-specific training on model improvement." + ] + }, + { + "cell_type": "markdown", + "id": "1cbc4d1a", + "metadata": {}, + "source": [ + "Here is the list of validation arguments that can be configured.\n", + "\n", + "- *test_ratio*: Ratio of the validation partition of the dataset. The remaining samples will be used for training. By default, it is 0.0.\n", + "- *test_on_global_updates*: Boolean value that indicates whether validation will be applied to globally updated (aggregated) parameters (see Figure 1). Default is False\n", + "- *test_on_local_updates*: Boolean value that indicates whether validation will be applied to locally updated (trained) parameters (see Figure 1). Default is False\n", + "- *test_metric*: One of MetricTypes that indicates which metric will be used for validation. It can be str or an instance of MetricTypes (e.g. MetricTypes.RECALL or RECALL ). If it is None and there isn't testing_step defined in the training plan (see section: Define Custom Validation Step) default metric will be ACCURACY.\n", + "- *test_metric_args*: A dictionary that contains the arguments that will be used for the metric function." + ] + }, + { + "cell_type": "markdown", + "id": "700d8da1", + "metadata": {}, + "source": [ + "**TO_DO:**\n", + "\n", + "- Initialize a new experiements.\n", + "- Use the setters to define the validation arguments.\n", + "- Launch the training and check the validation performances." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9972f57d", + "metadata": {}, + "outputs": [], + "source": [ + "exp = Experiment(\n", + " # [...]\n", + " training_args=training_args\n", + ")\n", + "#TODO: set the parameters using the setters. Example: exp.set_test_ratio(test_ratio=0.1)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "827b9920", + "metadata": {}, + "outputs": [], + "source": [ + "exp.run()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.7" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/fedbiomed-tutorial/tutorial-sklearn-solutions.ipynb b/fedbiomed-tutorial/tutorial-sklearn-solutions.ipynb new file mode 100644 index 0000000..3c5b35a --- /dev/null +++ b/fedbiomed-tutorial/tutorial-sklearn-solutions.ipynb @@ -0,0 +1,2208 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "64e87007", + "metadata": { + "pycharm": { + "name": "#%% md\n" + } + }, + "source": [ + "# Heart disease detection - Solution" + ] + }, + { + "cell_type": "markdown", + "id": "0faf6999", + "metadata": {}, + "source": [ + "In this tutorial, we will focus on applying federated learning techniques to a classification problem using Scikit-Learn, a popular machine learning library in Python. We will walk you through the process step by step, from setting up the federated learning environment to evaluating the model's performance.\n", + "\n", + "\n", + "Scikit-Learn, also known as sklearn, is a popular machine learning library in Python. It provides a wide range of tools and algorithms for tasks such as data preprocessing, feature selection, model training, and evaluation. Sklearn is widely used for tasks such as classification, regression, clustering, and dimensionality reduction. It offers a user-friendly interface and integrates well with other libraries in the Python ecosystem, making it a go-to choice for many machine learning practitioners and researchers." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "ade4cbea", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "%load_ext autoreload\n", + "%autoreload 2" + ] + }, + { + "cell_type": "markdown", + "id": "e6e1f0b1", + "metadata": {}, + "source": [ + "# Table of content\n", + "\n", + "1. [The dataset](#dataset)\n", + "2. [Task 1: training plan](#task1)\n", + "3. [Task 2: the experment](#task2)\n", + "4. [Task 3: model validation](#task3)" + ] + }, + { + "cell_type": "markdown", + "id": "a758a4f3", + "metadata": {}, + "source": [ + "# Tutorial" + ] + }, + { + "cell_type": "markdown", + "id": "3715cb94", + "metadata": {}, + "source": [ + "## The dataset <a name=\"dataset\"></a>" + ] + }, + { + "cell_type": "markdown", + "id": "ad6ec555", + "metadata": {}, + "source": [ + "The Heart Disease dataset available at https://archive.ics.uci.edu/dataset/45/heart+disease is a widely used dataset in the field of cardiovascular research and machine learning. It contains a collection of medical attributes from patients suspected of having heart disease, along with their corresponding diagnosis (presence or absence of heart disease). The dataset includes information such as age, sex, blood pressure, cholesterol levels, and various other clinical measurements.\n", + "\n", + "It was collected in 4 hospitals in the USA, Switzerland and Hungary. This dataset contains tabular information about 740 patients distributed among these four clients. \n", + "\n", + "A federated version of this dataset has been proposed in [Flamby](https://arxiv.org/pdf/2210.04620.pdf). Following thier actions, we preprocess the dataset by removing missing values and encoding non-binary categorical variables as dummy variables. We finally obtain the following centers:\n", + "\n", + "| Number | Client | Dataset size |\n", + "|--------|----------------------|--------------|\n", + "| 0 | Cleveland’s Hospital | 303 |\n", + "| 1 | Hungarian Hospital | 261 |\n", + "| 2 | Switzerland Hospital | 46 |\n", + "| 3 | Long Beach Hospital | 130 | \n" + ] + }, + { + "cell_type": "markdown", + "id": "7c57a1cd", + "metadata": {}, + "source": [ + "For teaching purposes, we decided to merge: client0 with client3 and client1 with client2.\n", + "The final federated scenario, in this way, is the following:\n", + "- **client1**, with 349 elements\n", + "- **client2**, with 391 elements" + ] + }, + { + "cell_type": "markdown", + "id": "c3c4430a", + "metadata": {}, + "source": [ + "## Task 1: Defining the training plan <a name=\"task1\"></a>" + ] + }, + { + "cell_type": "markdown", + "id": "37e16e35", + "metadata": {}, + "source": [ + "A training plan is a class that defines the four main components of federated model training: the data, the model, he loss and the optimizer. It is responsible for providing custom methods allowing every node to perform the training.\n", + "\n", + "\n", + "In the case of scikit-learn, Fed-BioMed already does a lot of the heavy lifting for you by providing the FedPerceptron, FedSGDClassifier and FedSGDRegressor classes as training plans. These classes already take care of the model, optimizer, loss function and related dependencies for you, so you only need to define how the data will be loaded.\n", + "\n", + "\n", + "In this tutorial we are going to use an [SGDClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html), so the related FedSGDClassifier training plan." + ] + }, + { + "cell_type": "markdown", + "id": "b0098dcb", + "metadata": {}, + "source": [ + "### Model arguments\n", + "*model_args* is a dictionary with the arguments related to the model, that will be passed to the Perceptron constructor.\n", + "\n", + "**IMPORTANT** For classification tasks, you are required to specify the following two fields:\n", + "\n", + "- n_features: the number of features in each input sample (in our case, the number of pixels in the images)\n", + "- n_classes: the number of classes in the target data\n", + "\n", + "Other model arguments depend on the specific model you are using, and are defined in the model definition. Refer to the model documentation \n", + "\n", + "\n", + "### Training arguments\n", + "*training_args* is a dictionary containing the arguments for the training routine (e.g. batch size, learning rate, epochs, etc.). This will be passed to the routine on the node side.\n", + "\n", + "**IMPORTANT** To set the training arguments we may either pass them to the Experiment constructor, or set them on an instance with the setter method:\n", + "\n", + " 'exp.set_training_arguments(training_args=training_args)'\n", + "The setters are available also for single training arguments, like: \n", + "\n", + " 'exp.set_aggregator(aggregator=FedAverage)'" + ] + }, + { + "cell_type": "markdown", + "id": "9f58d0a9", + "metadata": {}, + "source": [ + "**TO_DO:**\n", + "\n", + "- Apply the scaler to your data\n", + "- Define training args: num_updates, batch_size.\n", + "- Define model args." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "998bb584", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "from fedbiomed.common.training_plans import FedSGDRegressor, FedPerceptron, FedSGDClassifier\n", + "from fedbiomed.common.data import DataManager\n", + "from sklearn.preprocessing import MinMaxScaler\n", + "class SkLearnClassifierTrainingPlan(FedSGDClassifier):\n", + " def init_dependencies(self):\n", + " \"\"\"Define additional dependencies.\n", + " return [\"from torchvision import datasets, transforms\",\n", + " \"from torch.utils.data import DataLoader\"]\n", + "\n", + " def training_data(self, batch_size):\n", + " \n", + " In this case, we rely on torchvision functions for preprocessing the images.\n", + " \"\"\"\n", + " return [\"from sklearn.preprocessing import MinMaxScaler\"]\n", + " \n", + " def training_data(self, batch_size):\n", + " df = pd.read_csv(self.dataset_path, delimiter=';')\n", + "\n", + " X = df.iloc[:, :-1]\n", + " y = df.iloc[:, -1]\n", + " self.scaler = MinMaxScaler()\n", + " X = self.scaler.fit_transform(X)\n", + " return DataManager(dataset=X, target=y, batch_size=batch_size, shuffle=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "653e8aea", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "n_features = 18\n", + "n_classes = 2\n", + "\n", + "model_args = { 'max_iter':1000, 'tol': 1e-1 , \n", + " 'n_features' : n_features, 'n_classes' : n_classes, 'loss': 'hinge'}\n", + "\n", + "training_args = { \n", + " 'num_updates': 5,\n", + " 'batch_size': 128,\n", + " 'dry_run': False,\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "9ce04246", + "metadata": {}, + "source": [ + "## Task 2: the Experiment <a name=\"task2\"></a>" + ] + }, + { + "cell_type": "markdown", + "id": "338566d8", + "metadata": {}, + "source": [ + "The experiment enables Federated Learning by orchestrating the training process across multiple nodes. It searches for datasets based on specific tags, uploads the training plan file, sends model and training arguments, tracks and checks training progress, and downloads and aggregates model parameters for the next round." + ] + }, + { + "cell_type": "markdown", + "id": "c342b718", + "metadata": {}, + "source": [ + "**TO_DO:**\n", + "\n", + "- Define the tags to select the nodes\n", + "- Define the used training plan.\n", + "- Define the number of rounds for the federated process." + ] + }, + { + "cell_type": "markdown", + "id": "4a1185cc", + "metadata": {}, + "source": [ + "<div class=\"alert alert-block alert-info\"> <b>TAGS:</b> Replace %%%% in the tags with your username </div>" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "4b1a1341", + "metadata": { + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 14:38:00,927 fedbiomed INFO - Messaging researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87 successfully connected to the message broker, object = <fedbiomed.common.messaging.Messaging object at 0x7fce404c4b20>\n", + "2023-07-03 14:38:00,959 fedbiomed INFO - Searching dataset with data tags: ['heart-jupyter-sharkovsky'] for all nodes\n", + "2023-07-03 14:38:10,973 fedbiomed INFO - Node selected for training -> node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1\n", + "2023-07-03 14:38:10,974 fedbiomed INFO - Node selected for training -> node_10797f2f-2524-4595-a1c6-f3c67e03add1\n", + "2023-07-03 14:38:10,976 fedbiomed INFO - Checking data quality of federated datasets...\n", + "2023-07-03 14:38:10,977 fedbiomed DEBUG - Using native Sklearn Optimizer\n", + "2023-07-03 14:38:10,978 fedbiomed DEBUG - Model file has been saved: /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/my_model_8b48c219-cb97-4963-b035-4745618fca23.py\n", + "2023-07-03 14:38:11,071 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/my_model_8b48c219-cb97-4963-b035-4745618fca23.py successful, with status code 201\n", + "2023-07-03 14:38:11,080 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_b7de0161-57ad-4857-ac2b-bd5a43674bb6.mpk successful, with status code 201\n", + "2023-07-03 14:38:11,081 fedbiomed INFO - Removing tensorboard logs from previous experiment\n" + ] + } + ], + "source": [ + "from fedbiomed.researcher.experiment import Experiment\n", + "from fedbiomed.researcher.aggregators.fedavg import FedAverage\n", + "\n", + "tags = ['heart-jupyter-sharkovsky']\n", + "rounds = 10\n", + "\n", + "# search for corresponding datasets across nodes datasets\n", + "exp = Experiment(tags=tags,\n", + " model_args=model_args,\n", + " training_plan_class=SkLearnClassifierTrainingPlan,\n", + " training_args=training_args,\n", + " round_limit=rounds,\n", + " aggregator=FedAverage(),\n", + " node_selection_strategy=None)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d6ff55da", + "metadata": { + "pycharm": { + "name": "#%%\n" + }, + "scrolled": true + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 14:39:41,787 fedbiomed INFO - Sampled nodes in round 0 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:39:41,788 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 0, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_b7de0161-57ad-4857-ac2b-bd5a43674bb6.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:39:41,788 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:39:41,789 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 0, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_b7de0161-57ad-4857-ac2b-bd5a43674bb6.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:39:41,790 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:39:42,030 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 1 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.000000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:42,031 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 1 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.000000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:42,053 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 1 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.006065\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:42,057 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 1 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.852417\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:42,076 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 1 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.580541\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:42,078 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 1 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m4.077033\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:42,093 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 1 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.068150\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:51,806 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_d42587cb-dd29-460f-887a-910642918d60.mpk\n", + "2023-07-03 14:39:51,811 fedbiomed DEBUG - download of file node_params_79d8a21d-99a3-40cf-af83-11ed8868ec9a.mpk successful, with status code 200\n", + "2023-07-03 14:39:51,811 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_d84fbfc3-a23c-4195-b5bd-638143b4abac.mpk\n", + "2023-07-03 14:39:51,815 fedbiomed DEBUG - download of file node_params_182ef421-d214-40a3-ae8b-d2d4d3e9ceeb.mpk successful, with status code 200\n", + "2023-07-03 14:39:51,819 fedbiomed INFO - Nodes that successfully reply in round 0 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:39:51,828 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_c1adb784-b870-4b6d-bf3e-c6cec5578306.mpk successful, with status code 201\n", + "2023-07-03 14:39:51,828 fedbiomed INFO - Saved aggregated params for round 0 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_c1adb784-b870-4b6d-bf3e-c6cec5578306.mpk\n", + "2023-07-03 14:39:51,829 fedbiomed INFO - Sampled nodes in round 1 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:39:51,829 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 1, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_c1adb784-b870-4b6d-bf3e-c6cec5578306.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:39:51,829 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:39:51,830 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 1, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_c1adb784-b870-4b6d-bf3e-c6cec5578306.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 14:39:51,830 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:39:51,864 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 2 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.049416\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:51,865 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 2 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.027009\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:51,899 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 2 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.063915\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:51,904 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 2 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.911342\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:51,923 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 2 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.797854\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:51,924 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 2 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.870021\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:39:51,939 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 2 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.679158\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:01,843 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_18867bc3-13a0-43ec-9397-ba4d0b5b05b8.mpk\n", + "2023-07-03 14:40:01,848 fedbiomed DEBUG - download of file node_params_a6124a9a-5bfb-4549-b27c-cbf72f5e0d6c.mpk successful, with status code 200\n", + "2023-07-03 14:40:01,848 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_227d2e9f-8b09-45ba-bf96-b342bb22c144.mpk\n", + "2023-07-03 14:40:01,851 fedbiomed DEBUG - download of file node_params_7cba8c4d-bc05-4b41-acd7-f409f5b94d3a.mpk successful, with status code 200\n", + "2023-07-03 14:40:01,853 fedbiomed INFO - Nodes that successfully reply in round 1 ['node_10797f2f-2524-4595-a1c6-f3c67e03add1', 'node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1']\n", + "2023-07-03 14:40:01,863 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_c66687a0-6b7e-41ce-b9fc-7d05339bdc56.mpk successful, with status code 201\n", + "2023-07-03 14:40:01,863 fedbiomed INFO - Saved aggregated params for round 1 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_c66687a0-6b7e-41ce-b9fc-7d05339bdc56.mpk\n", + "2023-07-03 14:40:01,864 fedbiomed INFO - Sampled nodes in round 2 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:01,864 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 2, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_c66687a0-6b7e-41ce-b9fc-7d05339bdc56.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:01,865 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:01,866 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 2, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_c66687a0-6b7e-41ce-b9fc-7d05339bdc56.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:01,867 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:01,899 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 3 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.172746\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:01,942 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 3 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.089842\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:01,943 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 3 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.969353\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:01,946 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 3 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.823574\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:01,956 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 3 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m5.964549\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:01,965 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 3 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.420680\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:01,985 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 3 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.777982\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:11,876 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_8fbbaf35-845f-4a2f-af90-8ad0a3275401.mpk\n", + "2023-07-03 14:40:11,880 fedbiomed DEBUG - download of file node_params_e0f7c6dd-6c13-4eef-9ce4-981e6287212a.mpk successful, with status code 200\n", + "2023-07-03 14:40:11,881 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_6f5bfd2e-4e6d-4283-97dc-02b3b543e12e.mpk\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 14:40:11,884 fedbiomed DEBUG - download of file node_params_f8b8c152-6cb0-439b-b141-d605b6a499e9.mpk successful, with status code 200\n", + "2023-07-03 14:40:11,885 fedbiomed INFO - Nodes that successfully reply in round 2 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:11,894 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_e02fc9ff-20e8-4065-8670-5a67fe6b7914.mpk successful, with status code 201\n", + "2023-07-03 14:40:11,894 fedbiomed INFO - Saved aggregated params for round 2 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_e02fc9ff-20e8-4065-8670-5a67fe6b7914.mpk\n", + "2023-07-03 14:40:11,895 fedbiomed INFO - Sampled nodes in round 3 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:11,895 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 3, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_e02fc9ff-20e8-4065-8670-5a67fe6b7914.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:11,895 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:11,896 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 3, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_e02fc9ff-20e8-4065-8670-5a67fe6b7914.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:11,897 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:11,938 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 4 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.874855\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:11,939 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 4 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.016869\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:11,982 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 4 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.953992\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:11,983 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 4 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.983414\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:11,987 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 4 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.187833\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:11,991 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 4 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.484909\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:12,010 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 4 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.560063\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:21,909 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_86313eb7-16fe-427e-b5d2-a2dca7d7520b.mpk\n", + "2023-07-03 14:40:21,913 fedbiomed DEBUG - download of file node_params_34662ed3-632e-42ef-9c22-fc909e7f7dc1.mpk successful, with status code 200\n", + "2023-07-03 14:40:21,913 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_9f2e0e27-43e6-4378-b983-48ccc61ebf53.mpk\n", + "2023-07-03 14:40:21,916 fedbiomed DEBUG - download of file node_params_ff111a95-edfe-4efa-91d4-d563f3149d7a.mpk successful, with status code 200\n", + "2023-07-03 14:40:21,918 fedbiomed INFO - Nodes that successfully reply in round 3 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:21,926 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_e2f557b0-ba8c-4190-a3fd-0d9a5400ee49.mpk successful, with status code 201\n", + "2023-07-03 14:40:21,927 fedbiomed INFO - Saved aggregated params for round 3 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_e2f557b0-ba8c-4190-a3fd-0d9a5400ee49.mpk\n", + "2023-07-03 14:40:21,927 fedbiomed INFO - Sampled nodes in round 4 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:21,927 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 4, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_e2f557b0-ba8c-4190-a3fd-0d9a5400ee49.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 14:40:21,928 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:21,929 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 4, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_e2f557b0-ba8c-4190-a3fd-0d9a5400ee49.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:21,929 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:21,968 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 5 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.421098\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:21,969 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 5 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.677845\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:22,010 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 5 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.560698\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:22,010 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 5 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.570311\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:22,020 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 5 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.906239\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:22,023 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 5 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.144794\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:22,042 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 5 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.139720\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:31,941 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_dd9973c0-9fa0-4d01-adc2-f19aadcaa5fd.mpk\n", + "2023-07-03 14:40:31,945 fedbiomed DEBUG - download of file node_params_1ce2ab96-2a7e-43e7-9277-1c3bf1680518.mpk successful, with status code 200\n", + "2023-07-03 14:40:31,945 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_0f7907e3-e56b-4d9b-b6f7-e034cd6d351b.mpk\n", + "2023-07-03 14:40:31,948 fedbiomed DEBUG - download of file node_params_4326d6f7-895f-4cf6-af05-2b45473b6f6f.mpk successful, with status code 200\n", + "2023-07-03 14:40:31,950 fedbiomed INFO - Nodes that successfully reply in round 4 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:31,959 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_66b3d1f3-52f8-41be-bccc-be0240743d63.mpk successful, with status code 201\n", + "2023-07-03 14:40:31,959 fedbiomed INFO - Saved aggregated params for round 4 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_66b3d1f3-52f8-41be-bccc-be0240743d63.mpk\n", + "2023-07-03 14:40:31,960 fedbiomed INFO - Sampled nodes in round 5 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:31,960 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 5, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_66b3d1f3-52f8-41be-bccc-be0240743d63.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:31,960 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:31,961 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 5, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_66b3d1f3-52f8-41be-bccc-be0240743d63.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:31,961 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:32,014 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 6 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.960151\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:32,018 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 6 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.176058\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:32,051 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 6 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.156323\u001b[0m \n", + "\t\t\t\t\t ---------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 14:40:32,052 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 6 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.865018\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:32,053 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 6 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.246942\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:32,079 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 6 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.201610\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:32,080 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 6 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.450477\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:41,974 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_5328fe90-7ce9-455e-9cb1-d7018c8424df.mpk\n", + "2023-07-03 14:40:41,978 fedbiomed DEBUG - download of file node_params_101ae7c7-06fc-4c5f-8484-c1d8578e8e3d.mpk successful, with status code 200\n", + "2023-07-03 14:40:41,978 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_d2c4c801-1ce1-4fee-b9fd-7025d130932b.mpk\n", + "2023-07-03 14:40:41,981 fedbiomed DEBUG - download of file node_params_abdfd281-d477-4f70-b4eb-8eda46fcd2ef.mpk successful, with status code 200\n", + "2023-07-03 14:40:41,982 fedbiomed INFO - Nodes that successfully reply in round 5 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:41,991 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_08cc9313-21f1-4c35-b919-bcd0ec20a017.mpk successful, with status code 201\n", + "2023-07-03 14:40:41,992 fedbiomed INFO - Saved aggregated params for round 5 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_08cc9313-21f1-4c35-b919-bcd0ec20a017.mpk\n", + "2023-07-03 14:40:41,992 fedbiomed INFO - Sampled nodes in round 6 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:41,992 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 6, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_08cc9313-21f1-4c35-b919-bcd0ec20a017.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:41,993 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:41,994 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 6, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_08cc9313-21f1-4c35-b919-bcd0ec20a017.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:41,994 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:42,041 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 7 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.332799\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:42,042 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 7 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.287343\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:42,082 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 7 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.000000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:42,082 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 7 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.003612\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:42,085 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 7 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.350821\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:42,112 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 7 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.574242\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:42,112 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 7 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.177146\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:52,006 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_7c4193d5-0545-440f-ba95-a3ca6f54b567.mpk\n", + "2023-07-03 14:40:52,010 fedbiomed DEBUG - download of file node_params_8152e0d4-d70c-4949-a46d-cae5184a58af.mpk successful, with status code 200\n", + "2023-07-03 14:40:52,011 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_38b37e72-dec3-4df1-9771-6a1954f6016d.mpk\n", + "2023-07-03 14:40:52,013 fedbiomed DEBUG - download of file node_params_e79562dc-2049-4038-bb28-ff96158484eb.mpk successful, with status code 200\n", + "2023-07-03 14:40:52,015 fedbiomed INFO - Nodes that successfully reply in round 6 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:52,023 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_9adf34fb-59ca-4107-b602-49e5c9fbc8f1.mpk successful, with status code 201\n", + "2023-07-03 14:40:52,024 fedbiomed INFO - Saved aggregated params for round 6 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_9adf34fb-59ca-4107-b602-49e5c9fbc8f1.mpk\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 14:40:52,024 fedbiomed INFO - Sampled nodes in round 7 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:40:52,025 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 7, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_9adf34fb-59ca-4107-b602-49e5c9fbc8f1.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:52,025 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:52,026 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 7, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_9adf34fb-59ca-4107-b602-49e5c9fbc8f1.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:40:52,026 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:40:52,069 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 8 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.074448\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:52,070 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 8 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.741083\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:52,110 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 8 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.501979\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:52,111 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 8 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.574092\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:52,117 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 8 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.381271\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:52,122 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 8 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.232035\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:40:52,141 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 8 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.495739\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:02,038 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_809ca19c-99a4-4bda-803a-7d6741b1e7a1.mpk\n", + "2023-07-03 14:41:02,042 fedbiomed DEBUG - download of file node_params_8c22eda7-5d1c-4181-8657-afc804b15335.mpk successful, with status code 200\n", + "2023-07-03 14:41:02,043 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_3176c29b-8233-432f-beb5-5bc00389e4d2.mpk\n", + "2023-07-03 14:41:02,045 fedbiomed DEBUG - download of file node_params_4f79e57f-8f88-4e81-a39d-f40d7d255f8a.mpk successful, with status code 200\n", + "2023-07-03 14:41:02,047 fedbiomed INFO - Nodes that successfully reply in round 7 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:41:02,056 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_2b46701a-eb29-461a-bfd1-416223570a76.mpk successful, with status code 201\n", + "2023-07-03 14:41:02,056 fedbiomed INFO - Saved aggregated params for round 7 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_2b46701a-eb29-461a-bfd1-416223570a76.mpk\n", + "2023-07-03 14:41:02,056 fedbiomed INFO - Sampled nodes in round 8 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:41:02,057 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 8, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_2b46701a-eb29-461a-bfd1-416223570a76.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:41:02,057 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:41:02,058 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 8, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_2b46701a-eb29-461a-bfd1-416223570a76.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 14:41:02,058 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:41:02,091 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 9 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.330070\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:02,137 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 9 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.796861\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:02,138 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 9 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.255121\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:02,139 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 9 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.549095\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:02,158 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 9 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.943776\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:02,158 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 9 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.770924\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:02,198 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 9 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.308329\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:12,070 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_b2c70cae-7bf3-407e-979b-5cee3291fc62.mpk\n", + "2023-07-03 14:41:12,074 fedbiomed DEBUG - download of file node_params_c248198b-60f0-4f7c-b333-33c5c8de43fa.mpk successful, with status code 200\n", + "2023-07-03 14:41:12,075 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_05161a57-768b-42c7-b76f-cc63666e5052.mpk\n", + "2023-07-03 14:41:12,077 fedbiomed DEBUG - download of file node_params_5c6b2079-5bf5-4260-9977-f1abbd3b4139.mpk successful, with status code 200\n", + "2023-07-03 14:41:12,079 fedbiomed INFO - Nodes that successfully reply in round 8 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:41:12,087 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_9b3da84a-606d-4689-8384-911c4656123d.mpk successful, with status code 201\n", + "2023-07-03 14:41:12,088 fedbiomed INFO - Saved aggregated params for round 8 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_9b3da84a-606d-4689-8384-911c4656123d.mpk\n", + "2023-07-03 14:41:12,088 fedbiomed INFO - Sampled nodes in round 9 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:41:12,088 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 9, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_9b3da84a-606d-4689-8384-911c4656123d.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_6dc88f60-68fb-4bc7-aa01-2f339ea0c08d'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:41:12,089 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:41:12,090 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87', 'job_id': 'fbcc50ad-d4a1-4528-b491-4d1a0bbd5b2f', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.0, 'test_on_local_updates': False, 'test_on_global_updates': False, 'test_metric': None, 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 9, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_8b48c219-cb97-4963-b035-4745618fca23.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_9b3da84a-606d-4689-8384-911c4656123d.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_3fffa907-7569-46f0-a2b1-fa7245e42499', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 14:41:12,090 fedbiomed DEBUG - researcher_dba292ff-efe1-40ad-a1c6-1a4d5f6bbd87\n", + "2023-07-03 14:41:12,130 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 10 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.886490\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:12,131 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 10 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.704667\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:12,174 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 10 | Iteration: 4/5 (80%) | Samples: 391/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.202284\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:12,175 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 10 | Iteration: 3/5 (60%) | Samples: 349/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.827798\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:12,181 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 \n", + "\t\t\t\t\t Round 10 | Iteration: 5/5 (100%) | Samples: 519/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m10.244533\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:12,183 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 10 | Iteration: 4/5 (80%) | Samples: 477/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.449496\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:12,202 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_10797f2f-2524-4595-a1c6-f3c67e03add1 \n", + "\t\t\t\t\t Round 10 | Iteration: 5/5 (100%) | Samples: 605/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.531648\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 14:41:22,102 fedbiomed INFO - Downloading model params after training on node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_5417d5e1-5850-4473-81a3-5c42fdc8c7b7.mpk\n", + "2023-07-03 14:41:22,106 fedbiomed DEBUG - download of file node_params_1c0634dc-8f5e-4082-acea-bc297ab46452.mpk successful, with status code 200\n", + "2023-07-03 14:41:22,107 fedbiomed INFO - Downloading model params after training on node_10797f2f-2524-4595-a1c6-f3c67e03add1 - from http://localhost:8844/media/uploads/2023/07/03/node_params_cbf71f9b-ecd0-4922-bc78-a7fb7f2eee81.mpk\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 14:41:22,110 fedbiomed DEBUG - download of file node_params_bd17ee53-2501-4e5e-a572-115d610dc49d.mpk successful, with status code 200\n", + "2023-07-03 14:41:22,111 fedbiomed INFO - Nodes that successfully reply in round 9 ['node_e27ab041-6133-4e2f-b0ce-afa0a0c59fa1', 'node_10797f2f-2524-4595-a1c6-f3c67e03add1']\n", + "2023-07-03 14:41:22,123 fedbiomed DEBUG - HTTP POST request of file /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_2ff74778-3640-470d-8657-185b0dd76a3e.mpk successful, with status code 201\n", + "2023-07-03 14:41:22,123 fedbiomed INFO - Saved aggregated params for round 9 in /home/jupyter-sharkovsky/fedbiomed/var/experiments/Experiment_0032/aggregated_params_2ff74778-3640-470d-8657-185b0dd76a3e.mpk\n" + ] + }, + { + "data": { + "text/plain": [ + "10" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# train experiments\n", + "exp.run()" + ] + }, + { + "cell_type": "markdown", + "id": "88e2a782", + "metadata": { + "pycharm": { + "name": "#%% md\n" + } + }, + "source": [ + "## Task 3: Model Validation <a name=\"task3\"></a>" + ] + }, + { + "cell_type": "markdown", + "id": "57472006", + "metadata": {}, + "source": [ + "During federated training, model validation plays a crucial role in assessing performance without a dedicated holdout dataset. Fed-BioMed enables separate model validation on each node after parameter updates, allowing comparison of model performances. Two types of validation can be performed:\n", + "- one on globally updated parameters before training a round, \n", + "- another on locally updated parameters after local training is completed on a node. \n", + "\n", + "This helps users evaluate the impact of node-specific training on model improvement." + ] + }, + { + "cell_type": "markdown", + "id": "c8f0f2d5", + "metadata": {}, + "source": [ + "Here is the list of validation arguments that can be configured.\n", + "\n", + "- *test_ratio*: Ratio of the validation partition of the dataset. The remaining samples will be used for training. By default, it is 0.0.\n", + "- *test_on_global_updates*: Boolean value that indicates whether validation will be applied to globally updated (aggregated) parameters (see Figure 1). Default is False\n", + "- *test_on_local_updates*: Boolean value that indicates whether validation will be applied to locally updated (trained) parameters (see Figure 1). Default is False\n", + "- *test_metric*: One of MetricTypes that indicates which metric will be used for validation. It can be str or an instance of MetricTypes (e.g. MetricTypes.RECALL or RECALL ). If it is None and there isn't testing_step defined in the training plan (see section: Define Custom Validation Step) default metric will be ACCURACY.\n", + "- *test_metric_args*: A dictionary that contains the arguments that will be used for the metric function." + ] + }, + { + "cell_type": "markdown", + "id": "da90c331", + "metadata": {}, + "source": [ + "**TO_DO:**\n", + "\n", + "- Initialize a new experiements.\n", + "- Use the setters to define the validation arguments.\n", + "- Launch the training and check the validation performances." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "9972f57d", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:09:00,198 fedbiomed INFO - Searching dataset with data tags: ['heart'] for all nodes\n", + "2023-07-03 13:09:10,211 fedbiomed INFO - Node selected for training -> node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "2023-07-03 13:09:10,212 fedbiomed INFO - Node selected for training -> node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "2023-07-03 13:09:10,214 fedbiomed INFO - Checking data quality of federated datasets...\n", + "2023-07-03 13:09:10,215 fedbiomed DEBUG - Using native Sklearn Optimizer\n", + "2023-07-03 13:09:10,217 fedbiomed DEBUG - Model file has been saved: /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py\n", + "2023-07-03 13:09:10,233 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py successful, with status code 201\n", + "2023-07-03 13:09:10,253 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_c63bc355-b053-443c-95cd-25eb2ab66237.mpk successful, with status code 201\n", + "2023-07-03 13:09:10,256 fedbiomed DEBUG - Experimentation training_args updated for `job`\n", + "2023-07-03 13:09:10,257 fedbiomed DEBUG - Experimentation training_args updated for `job`\n", + "2023-07-03 13:09:10,259 fedbiomed DEBUG - Experimentation training_args updated for `job`\n", + "2023-07-03 13:09:10,260 fedbiomed DEBUG - Experimentation training_args updated for `job`\n" + ] + }, + { + "data": { + "text/plain": [ + "('F1_SCORE', {})" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "exp = Experiment(tags=tags,\n", + " model_args=model_args,\n", + " training_plan_class=SkLearnClassifierTrainingPlan,\n", + " training_args=training_args,\n", + " round_limit=rounds,\n", + " aggregator=FedAverage(),\n", + " node_selection_strategy=None)\n", + "exp.set_test_ratio(0.10)\n", + "exp.set_test_on_local_updates(True)\n", + "exp.set_test_on_global_updates(True)\n", + "exp.set_test_metric('F1_SCORE')" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "6167ae04", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'num_updates': 5,\n", + " 'batch_size': 128,\n", + " 'dry_run': False,\n", + " 'optimizer_args': {},\n", + " 'epochs': None,\n", + " 'batch_maxnum': None,\n", + " 'test_ratio': 0.1,\n", + " 'test_on_local_updates': True,\n", + " 'test_on_global_updates': True,\n", + " 'test_metric': 'F1_SCORE',\n", + " 'test_metric_args': {},\n", + " 'log_interval': 10,\n", + " 'fedprox_mu': None,\n", + " 'use_gpu': False,\n", + " 'dp_args': None,\n", + " 'share_persistent_buffers': True}" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "exp.training_args()" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "88320ee0", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:10:47,594 fedbiomed INFO - Sampled nodes in round 0 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:10:47,595 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 0, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_c63bc355-b053-443c-95cd-25eb2ab66237.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:10:47,597 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:10:47,599 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 0, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_c63bc355-b053-443c-95cd-25eb2ab66237.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:10:47,600 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:10:47,628 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:47,660 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:47,701 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:47,703 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 1 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.620690\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,705 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 1 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.000000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,709 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:47,717 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 1 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.541667\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,763 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 1 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.000000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,820 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 1 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.646654\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,848 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 1 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.874489\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,873 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 1 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.070804\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,899 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 1 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.961074\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,940 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 1 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.185373\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,942 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 1 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.654545\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,962 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:47,965 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 1 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.339485\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,967 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 1 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.000000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:47,991 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:57,619 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_a851d235-e20b-485f-b5f3-ef256fce6e94.mpk\n", + "2023-07-03 13:10:57,624 fedbiomed DEBUG - download of file node_params_087d00eb-43dc-4f0a-acf2-06773c51b403.mpk successful, with status code 200\n", + "2023-07-03 13:10:57,626 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_bfc19c75-168f-4f5f-b694-a2241d595e72.mpk\n", + "2023-07-03 13:10:57,633 fedbiomed DEBUG - download of file node_params_549a8218-017f-4785-92ac-d5bbeda56bf3.mpk successful, with status code 200\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:10:57,638 fedbiomed INFO - Nodes that successfully reply in round 0 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:10:57,666 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_da8e3b9e-a27a-499e-ad3d-d8a2bd60f2e7.mpk successful, with status code 201\n", + "2023-07-03 13:10:57,668 fedbiomed INFO - Saved aggregated params for round 0 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_da8e3b9e-a27a-499e-ad3d-d8a2bd60f2e7.mpk\n", + "2023-07-03 13:10:57,669 fedbiomed INFO - Sampled nodes in round 1 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:10:57,671 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 1, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_da8e3b9e-a27a-499e-ad3d-d8a2bd60f2e7.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:10:57,672 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:10:57,674 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 1, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_da8e3b9e-a27a-499e-ad3d-d8a2bd60f2e7.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:10:57,676 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:10:57,711 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:57,751 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:57,756 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:57,758 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 2 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.650000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:57,760 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:57,761 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 2 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.814815\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:57,799 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 2 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.571441\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:57,813 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 2 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.826935\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:57,891 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 2 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.080854\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:57,908 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 2 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m5.621247\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:57,948 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 2 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.503852\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:57,971 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 2 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.111656\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:58,000 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 2 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.101825\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:58,006 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 2 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.666667\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:58,036 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 2 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.136677\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:58,046 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 2 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.769231\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:10:58,056 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:10:58,077 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:07,696 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_01298da2-5f69-44bd-a3fb-a79d1c3a3b8b.mpk\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:11:07,703 fedbiomed DEBUG - download of file node_params_f9d00178-e37a-4711-91c1-452364324219.mpk successful, with status code 200\n", + "2023-07-03 13:11:07,705 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_4fbd15b3-06d2-4f99-834b-71b026f910be.mpk\n", + "2023-07-03 13:11:07,714 fedbiomed DEBUG - download of file node_params_0764d423-ba75-4eec-8fac-e01490bad39c.mpk successful, with status code 200\n", + "2023-07-03 13:11:07,718 fedbiomed INFO - Nodes that successfully reply in round 1 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:07,745 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_f2749202-bb4a-4289-8dca-c370b43795ce.mpk successful, with status code 201\n", + "2023-07-03 13:11:07,746 fedbiomed INFO - Saved aggregated params for round 1 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_f2749202-bb4a-4289-8dca-c370b43795ce.mpk\n", + "2023-07-03 13:11:07,747 fedbiomed INFO - Sampled nodes in round 2 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:07,747 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 2, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_f2749202-bb4a-4289-8dca-c370b43795ce.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:07,748 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:07,750 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 2, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_f2749202-bb4a-4289-8dca-c370b43795ce.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:07,751 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:07,790 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:07,792 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:07,831 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 3 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.723404\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:07,833 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:07,834 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:07,837 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 3 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.750000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:07,854 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 3 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.847779\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:07,893 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 3 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.976905\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:07,952 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 3 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.424982\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:07,975 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 3 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.831711\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:08,010 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 3 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.862112\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:08,042 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 3 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.656498\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:08,082 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 3 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.762910\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:08,087 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 3 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.756757\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:08,116 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:08,120 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 3 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.686199\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:08,125 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 3 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.666667\u001b[0m \n", + "\t\t\t\t\t ---------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:11:08,151 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:17,770 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_e2048bb7-4666-4b16-82e2-6b61a4dc9c9b.mpk\n", + "2023-07-03 13:11:17,779 fedbiomed DEBUG - download of file node_params_467cd333-c31b-486d-be19-7a330f059e66.mpk successful, with status code 200\n", + "2023-07-03 13:11:17,780 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_50e22a77-5eee-473e-a1ff-afc72d6c5317.mpk\n", + "2023-07-03 13:11:17,788 fedbiomed DEBUG - download of file node_params_1a214c29-9dab-4e99-a94a-c601964d06c8.mpk successful, with status code 200\n", + "2023-07-03 13:11:17,790 fedbiomed INFO - Nodes that successfully reply in round 2 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:17,816 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_b325e05d-1e8d-4806-bb12-b3c95be8fb56.mpk successful, with status code 201\n", + "2023-07-03 13:11:17,817 fedbiomed INFO - Saved aggregated params for round 2 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_b325e05d-1e8d-4806-bb12-b3c95be8fb56.mpk\n", + "2023-07-03 13:11:17,818 fedbiomed INFO - Sampled nodes in round 3 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:17,818 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 3, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_b325e05d-1e8d-4806-bb12-b3c95be8fb56.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:17,819 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:17,821 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 3, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_b325e05d-1e8d-4806-bb12-b3c95be8fb56.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:17,822 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:17,851 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:17,853 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:17,892 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 4 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.880000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:17,894 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:17,928 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 4 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.110141\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:17,930 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:17,932 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 4 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.722222\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:17,958 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 4 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.788128\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:18,035 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 4 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.704669\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:18,045 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 4 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.378920\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:18,097 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 4 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.010502\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:18,102 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 4 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.556234\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:18,156 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 4 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.040663\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:18,160 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 4 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.900000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:18,162 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 4 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.326774\u001b[0m \n", + "\t\t\t\t\t ---------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:11:18,166 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 4 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.755556\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:18,189 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:18,205 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:27,839 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_3f0dfc90-78b6-44b6-80ff-02e4ec5c1893.mpk\n", + "2023-07-03 13:11:27,845 fedbiomed DEBUG - download of file node_params_7b7a3d7f-f590-463c-8254-e41e0d714867.mpk successful, with status code 200\n", + "2023-07-03 13:11:27,847 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_fd0a41e2-e711-41a6-816e-aa1cba5257b6.mpk\n", + "2023-07-03 13:11:27,855 fedbiomed DEBUG - download of file node_params_c1b91ec8-b0a6-4df4-9298-855defddc08a.mpk successful, with status code 200\n", + "2023-07-03 13:11:27,858 fedbiomed INFO - Nodes that successfully reply in round 3 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:27,888 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_184fa830-c29d-411c-a289-0ade92732955.mpk successful, with status code 201\n", + "2023-07-03 13:11:27,890 fedbiomed INFO - Saved aggregated params for round 3 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_184fa830-c29d-411c-a289-0ade92732955.mpk\n", + "2023-07-03 13:11:27,892 fedbiomed INFO - Sampled nodes in round 4 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:27,893 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 4, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_184fa830-c29d-411c-a289-0ade92732955.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:27,894 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:27,896 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 4, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_184fa830-c29d-411c-a289-0ade92732955.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:27,897 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:27,932 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:27,934 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:27,975 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 5 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.840000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:27,977 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:27,979 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:27,981 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 5 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.717949\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,017 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 5 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.220582\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,051 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 5 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.791132\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,139 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 5 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.935866\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,148 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 5 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.739728\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,203 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 5 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.499985\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,210 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 5 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.843874\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,256 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 5 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.393408\u001b[0m \n", + "\t\t\t\t\t ---------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:11:28,257 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 5 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.647059\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,259 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 5 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.451019\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,265 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 5 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.750000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:28,282 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:28,292 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:37,917 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_5abd9e05-6407-4216-8ac4-e0c14b226afa.mpk\n", + "2023-07-03 13:11:37,925 fedbiomed DEBUG - download of file node_params_911e8f78-bc17-46bd-9d4e-f5a072d9db2c.mpk successful, with status code 200\n", + "2023-07-03 13:11:37,927 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_107c9cca-8051-4a9d-8c83-095453773d1b.mpk\n", + "2023-07-03 13:11:37,936 fedbiomed DEBUG - download of file node_params_d935363b-2f26-4775-ad88-3a76c0997939.mpk successful, with status code 200\n", + "2023-07-03 13:11:37,941 fedbiomed INFO - Nodes that successfully reply in round 4 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:37,972 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_149700c9-f09e-4f2b-b59f-3cf8ca9dda9f.mpk successful, with status code 201\n", + "2023-07-03 13:11:37,973 fedbiomed INFO - Saved aggregated params for round 4 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_149700c9-f09e-4f2b-b59f-3cf8ca9dda9f.mpk\n", + "2023-07-03 13:11:37,974 fedbiomed INFO - Sampled nodes in round 5 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:37,975 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 5, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_149700c9-f09e-4f2b-b59f-3cf8ca9dda9f.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:37,976 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:37,977 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 5, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_149700c9-f09e-4f2b-b59f-3cf8ca9dda9f.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:37,978 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:38,016 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:38,019 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:38,059 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 6 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.740741\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,060 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:38,062 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:38,064 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 6 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.774194\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,083 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 6 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.168365\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,122 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 6 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.097516\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,187 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 6 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.427672\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,214 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 6 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.031782\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,251 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 6 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.022977\u001b[0m \n", + "\t\t\t\t\t ---------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:11:38,270 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 6 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.552519\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,309 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 6 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.463381\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,314 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 6 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.756757\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,325 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 6 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.149643\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,327 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 6 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.545455\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:38,337 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:38,347 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:47,996 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_8051ef50-58f1-431b-ba61-fa2a7de4425e.mpk\n", + "2023-07-03 13:11:48,003 fedbiomed DEBUG - download of file node_params_eb0d8cf1-796c-441f-be19-3efb7ec9a348.mpk successful, with status code 200\n", + "2023-07-03 13:11:48,005 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_23b6ff12-0114-4d39-b611-5407b97a298c.mpk\n", + "2023-07-03 13:11:48,014 fedbiomed DEBUG - download of file node_params_96e1f4da-2feb-43d2-b739-aec6f9470a64.mpk successful, with status code 200\n", + "2023-07-03 13:11:48,018 fedbiomed INFO - Nodes that successfully reply in round 5 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:48,044 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_1e4efd08-3675-4c18-a8d8-71ee656cf2e4.mpk successful, with status code 201\n", + "2023-07-03 13:11:48,045 fedbiomed INFO - Saved aggregated params for round 5 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_1e4efd08-3675-4c18-a8d8-71ee656cf2e4.mpk\n", + "2023-07-03 13:11:48,046 fedbiomed INFO - Sampled nodes in round 6 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:48,047 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 6, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_1e4efd08-3675-4c18-a8d8-71ee656cf2e4.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:48,047 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:48,048 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 6, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_1e4efd08-3675-4c18-a8d8-71ee656cf2e4.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:48,049 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:48,075 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:48,077 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:48,112 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 7 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.685714\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,114 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:48,115 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:48,118 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 7 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.722222\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,157 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 7 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.142721\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,193 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 7 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.086165\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,289 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 7 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.863901\u001b[0m \n", + "\t\t\t\t\t ---------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:11:48,304 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 7 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.980729\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,361 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 7 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.270667\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,375 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 7 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.046473\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,436 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 7 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.967612\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,438 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 7 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.652174\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,448 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 7 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.025577\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,449 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 7 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.818182\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:48,463 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:48,473 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:58,062 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_91cb60c3-262f-4462-b47c-46b168b222ad.mpk\n", + "2023-07-03 13:11:58,069 fedbiomed DEBUG - download of file node_params_6578107f-179c-4a8e-805f-0b6577afcf02.mpk successful, with status code 200\n", + "2023-07-03 13:11:58,071 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_a0330ed0-5d73-4282-890d-f718327c26cf.mpk\n", + "2023-07-03 13:11:58,079 fedbiomed DEBUG - download of file node_params_e3b1a95e-5bf4-4db4-9682-32a05dc624d6.mpk successful, with status code 200\n", + "2023-07-03 13:11:58,081 fedbiomed INFO - Nodes that successfully reply in round 6 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:58,112 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_ff50f66b-d95f-4c80-b9e2-b9d02395e61c.mpk successful, with status code 201\n", + "2023-07-03 13:11:58,113 fedbiomed INFO - Saved aggregated params for round 6 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_ff50f66b-d95f-4c80-b9e2-b9d02395e61c.mpk\n", + "2023-07-03 13:11:58,113 fedbiomed INFO - Sampled nodes in round 7 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:11:58,114 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 7, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_ff50f66b-d95f-4c80-b9e2-b9d02395e61c.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:58,115 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:58,116 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 7, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_ff50f66b-d95f-4c80-b9e2-b9d02395e61c.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:11:58,117 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:11:58,151 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:58,192 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:58,198 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:58,199 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:58,201 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 8 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.867925\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,205 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 8 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.869565\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,239 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 8 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.966867\u001b[0m \n", + "\t\t\t\t\t ---------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:11:58,264 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 8 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.986017\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,341 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 8 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.760947\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,349 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 8 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.215534\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,404 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 8 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.841887\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,419 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 8 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m2.925228\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,459 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 8 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.613500\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,461 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 8 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.666667\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,469 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 8 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.259462\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,470 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 8 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.837209\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:11:58,486 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:11:58,497 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:08,137 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_dea4b03c-c2bd-440b-9c50-bc8e0ebedd99.mpk\n", + "2023-07-03 13:12:08,145 fedbiomed DEBUG - download of file node_params_8a4dcb11-f6c5-4492-ac8f-e2246f3cb971.mpk successful, with status code 200\n", + "2023-07-03 13:12:08,147 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_fc32ffcd-2ab3-452b-a6fd-bb8abb558737.mpk\n", + "2023-07-03 13:12:08,153 fedbiomed DEBUG - download of file node_params_655c3559-2d88-447a-94d7-f56020623245.mpk successful, with status code 200\n", + "2023-07-03 13:12:08,155 fedbiomed INFO - Nodes that successfully reply in round 7 ['node_8641a287-262f-40cb-a01d-0b17f5c76e2e', 'node_1c6bdcb7-99e7-4278-95c9-98caeda6971b']\n", + "2023-07-03 13:12:08,178 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_2ed84242-6722-45ef-b979-3e0453616abe.mpk successful, with status code 201\n", + "2023-07-03 13:12:08,179 fedbiomed INFO - Saved aggregated params for round 7 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_2ed84242-6722-45ef-b979-3e0453616abe.mpk\n", + "2023-07-03 13:12:08,179 fedbiomed INFO - Sampled nodes in round 8 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:12:08,180 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 8, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_2ed84242-6722-45ef-b979-3e0453616abe.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:12:08,180 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:12:08,181 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 8, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_2ed84242-6722-45ef-b979-3e0453616abe.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:12:08,182 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:12:08,221 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:08,224 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:08,264 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 9 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.727273\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,266 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:08,268 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:12:08,270 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 9 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.850000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,307 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 9 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.944156\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,347 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 9 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.995382\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,430 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 9 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.384562\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,446 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 9 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.229974\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,487 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 9 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.127323\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,498 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 9 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.205052\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,543 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 9 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.900732\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,546 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 9 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.666667\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,563 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 9 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.179523\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,568 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 9 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.777778\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:08,582 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:08,588 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:18,201 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_b6c491dc-c946-4799-aaec-584e33a040fa.mpk\n", + "2023-07-03 13:12:18,206 fedbiomed DEBUG - download of file node_params_fc646d1c-eaf2-41dc-b155-3413e8ada296.mpk successful, with status code 200\n", + "2023-07-03 13:12:18,207 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_b4447156-3406-4cba-8842-bfb2fd27208b.mpk\n", + "2023-07-03 13:12:18,213 fedbiomed DEBUG - download of file node_params_23e098aa-aa5a-41c0-8dd6-d72f9a2cdeb2.mpk successful, with status code 200\n", + "2023-07-03 13:12:18,215 fedbiomed INFO - Nodes that successfully reply in round 8 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:12:18,239 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_5f0677f1-a85a-4bc3-b213-c607a2030e8d.mpk successful, with status code 201\n", + "2023-07-03 13:12:18,240 fedbiomed INFO - Saved aggregated params for round 8 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_5f0677f1-a85a-4bc3-b213-c607a2030e8d.mpk\n", + "2023-07-03 13:12:18,241 fedbiomed INFO - Sampled nodes in round 9 ['node_1c6bdcb7-99e7-4278-95c9-98caeda6971b', 'node_8641a287-262f-40cb-a01d-0b17f5c76e2e']\n", + "2023-07-03 13:12:18,242 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 9, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_5f0677f1-a85a-4bc3-b213-c607a2030e8d.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_73474707-cfad-4e6a-b585-9a3520fdd776'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:12:18,243 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:12:18,246 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m: Perform training with the arguments: {'researcher_id': 'researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e', 'job_id': '598d23e9-d91e-4526-b1b4-aa8ab8b2e30a', 'training_args': {'num_updates': 5, 'batch_size': 128, 'dry_run': False, 'optimizer_args': {}, 'epochs': None, 'batch_maxnum': None, 'test_ratio': 0.1, 'test_on_local_updates': True, 'test_on_global_updates': True, 'test_metric': 'F1_SCORE', 'test_metric_args': {}, 'log_interval': 10, 'fedprox_mu': None, 'use_gpu': False, 'dp_args': None, 'share_persistent_buffers': True}, 'training': True, 'model_args': {'max_iter': 1000, 'tol': 0.1, 'n_features': 18, 'n_classes': 2, 'loss': 'hinge', 'verbose': 1}, 'round': 9, 'secagg_servkey_id': None, 'secagg_biprime_id': None, 'secagg_random': None, 'secagg_clipping_range': None, 'command': 'train', 'training_plan_url': 'http://localhost:8844/media/uploads/2023/07/03/my_model_b511f38a-c219-4bcd-9a43-27bcd5a25797.py', 'params_url': 'http://localhost:8844/media/uploads/2023/07/03/aggregated_params_5f0677f1-a85a-4bc3-b213-c607a2030e8d.mpk', 'training_plan_class': 'SkLearnClassifierTrainingPlan', 'dataset_id': 'dataset_d45526f2-d699-4b64-8934-90d06d77967a', 'protocol_version': '1'} \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:12:18,247 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:12:18,281 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:18,282 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:18,323 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 10 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.863636\u001b[0m \n", + "\t\t\t\t\t ---------\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2023-07-03 13:12:18,326 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:18,357 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:18,359 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 10 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.620690\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,363 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 10 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.196435\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,395 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 10 | Iteration: 1/5 (20%) | Samples: 128/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m0.714835\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,466 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 10 | Iteration: 3/5 (60%) | Samples: 314/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.649812\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,467 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 10 | Iteration: 3/5 (60%) | Samples: 351/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m4.314792\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,512 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 10 | Iteration: 4/5 (80%) | Samples: 442/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m3.994241\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,516 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 10 | Iteration: 4/5 (80%) | Samples: 479/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.290517\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,560 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 10 | Iteration: 5/5 (100%) | Samples: 570/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.708250\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,563 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 10 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.750000\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,572 fedbiomed INFO - \u001b[1mTRAINING\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 10 | Iteration: 5/5 (100%) | Samples: 607/640\n", + " \t\t\t\t\t Loss hinge: \u001b[1m1.282847\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,575 fedbiomed INFO - \u001b[1mVALIDATION ON LOCAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 10 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.851064\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:18,595 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:18,604 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m results uploaded successfully \u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:28,259 fedbiomed INFO - Downloading model params after training on node_8641a287-262f-40cb-a01d-0b17f5c76e2e - from http://localhost:8844/media/uploads/2023/07/03/node_params_deca3fdb-b149-4302-bd2b-d16ddbbadab8.mpk\n", + "2023-07-03 13:12:28,263 fedbiomed DEBUG - download of file node_params_1dd53dca-818d-400e-9975-6ffe7eb6170f.mpk successful, with status code 200\n", + "2023-07-03 13:12:28,264 fedbiomed INFO - Downloading model params after training on node_1c6bdcb7-99e7-4278-95c9-98caeda6971b - from http://localhost:8844/media/uploads/2023/07/03/node_params_212cd78e-6078-4b5d-a21e-eef8154f0a37.mpk\n", + "2023-07-03 13:12:28,269 fedbiomed DEBUG - download of file node_params_2f5c982f-cc6e-4fc5-94e0-77f159bf3aa5.mpk successful, with status code 200\n", + "2023-07-03 13:12:28,271 fedbiomed INFO - Nodes that successfully reply in round 9 ['node_8641a287-262f-40cb-a01d-0b17f5c76e2e', 'node_1c6bdcb7-99e7-4278-95c9-98caeda6971b']\n", + "2023-07-03 13:12:28,291 fedbiomed DEBUG - HTTP POST request of file /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_112c956e-b7d7-4fb6-9238-d450adb9770f.mpk successful, with status code 201\n", + "2023-07-03 13:12:28,292 fedbiomed INFO - Saved aggregated params for round 9 in /user/linnocen/home/fedbiomed/var/experiments/Experiment_0001/aggregated_params_112c956e-b7d7-4fb6-9238-d450adb9770f.mpk\n", + "2023-07-03 13:12:28,293 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m:Perform final validation on aggregated parameters \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:12:28,294 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:12:28,295 fedbiomed INFO - \u001b[1mSending request\u001b[0m \n", + "\t\t\t\t\t\u001b[1m To\u001b[0m: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t\u001b[1m Request: \u001b[0m:Perform final validation on aggregated parameters \n", + " -----------------------------------------------------------------\n", + "2023-07-03 13:12:28,296 fedbiomed DEBUG - researcher_5d3b393d-e248-4e3f-a028-b87fe5fbbe6e\n", + "2023-07-03 13:12:28,326 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:28,328 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_1c6bdcb7-99e7-4278-95c9-98caeda6971b\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:28,358 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_1c6bdcb7-99e7-4278-95c9-98caeda6971b \n", + "\t\t\t\t\t Round 11 | Iteration: 1/1 (100%) | Samples: 40/40\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.812500\u001b[0m \n", + "\t\t\t\t\t ---------\n", + "2023-07-03 13:12:28,361 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:28,362 fedbiomed INFO - \u001b[1mINFO\u001b[0m\n", + "\t\t\t\t\t\u001b[1m NODE\u001b[0m node_8641a287-262f-40cb-a01d-0b17f5c76e2e\n", + "\t\t\t\t\t\u001b[1m MESSAGE:\u001b[0m NPDataLoader expanding 1-dimensional target to become 2-dimensional.\u001b[0m\n", + "-----------------------------------------------------------------\n", + "2023-07-03 13:12:28,398 fedbiomed INFO - \u001b[1mVALIDATION ON GLOBAL UPDATES\u001b[0m \n", + "\t\t\t\t\t NODE_ID: node_8641a287-262f-40cb-a01d-0b17f5c76e2e \n", + "\t\t\t\t\t Round 11 | Iteration: 1/1 (100%) | Samples: 35/35\n", + " \t\t\t\t\t F1_SCORE: \u001b[1m0.750000\u001b[0m \n", + "\t\t\t\t\t ---------\n" + ] + }, + { + "data": { + "text/plain": [ + "10" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "exp.run()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "fedbiomed-researcher", + "language": "python", + "name": "fedbiomed-researcher" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.16" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} -- GitLab