Mentions légales du service

Skip to content
Snippets Groups Projects
Commit cb3800f4 authored by sidi mohammed kaddour's avatar sidi mohammed kaddour
Browse files

fixed bugs

parent 92b99403
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
deploy/G5k/K3S/script.ipynb
+ 11
− 4
View file @ cb3800f4
%% Cell type:markdown id:a6b84c8e-fae7-402b-a5d7-9436d741c228 tags:
# Enoslib and Kubernetes Deployment Workflow
## Step 0: Update Paths and Credentials
1. **Modify Paths:** Replace absolute paths with a placeholder for the project root.
```python
project_root = "/path/to/your/project"
local_directory = f"{project_root}/deploy/G5k/K3S"
```
2. **Update Credentials:** Replace placeholders for sensitive data such as:
- Your Grid'5000 username.
- Docker credentials (to be used in Step 7).
## Step 1: Initialize Logging and Job Name
This step initializes logging for Enoslib and generates a unique job name using the current timestamp.
%% Cell type:code id:b81e774d-009d-4b59-ae48-73850055e69d tags:
``` python
import enoslib as en
from datetime import datetime
# Enable rich logging
_ = en.init_logging()
# Generate a timestamp for the job name
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
job_name = f"usecase_{timestamp}"
```
%% Cell type:markdown id:45707018-8271-4572-9492-511b6859ea7e tags:
## Step 2: Resource Allocation
Defines the resource allocation for the cluster, specifying roles, cluster type, and machine configurations.
%% Cell type:code id:f9b0d202-d119-461f-8896-ba164a00eb39 tags:
``` python
# claim the resources
cluster="ecotype"
conf = (
en.VMonG5kConf
.from_settings(job_name=job_name, walltime="04:00:00")
.add_machine(
roles=["master"],
cluster=cluster,
number=1,
flavour="large",
)
.add_machine(
roles=["agent","object_recognizer"],
cluster=cluster,
number=1,
flavour_desc={"core": 8, "mem": 8192}
)
.add_machine(
roles=["agent","motion_detector"],
cluster=cluster,
number=3,
flavour_desc={"core": 4, "mem": 4096})
.add_machine(
roles=["agent","camera"],
cluster=cluster,
number=3,
flavour_desc={"core": 1, "mem": 1024}
)
.finalize()
)
```
%% Cell type:markdown id:6e2b8ee9-0e3d-4104-aba1-e9bc8571ce09 tags:
## Step 3: Initialize the Provider
Initializes the virtual machine provider with the defined configuration and waits for the resources to be ready.
%% Cell type:code id:6d183604-09f8-41e4-a96f-4e4b16818ed7 tags:
``` python
provider = en.VMonG5k(conf)
roles, networks = provider.init()
roles
en.wait_for(roles)
```
%% Cell type:markdown id:bea70b45-72cb-4c1c-9b8a-faaa2891f5cb tags:
## Step 4: Deploy K3s
Deploys a lightweight Kubernetes distribution (K3s) on the allocated master and agent nodes.
%% Cell type:code id:169fbc50-a57c-46c2-993e-292011defca3 tags:
``` python
k3s = en.K3s(master=roles["master"], agent=roles["agent"])
k3s.deploy()
```
%% Cell type:markdown id:652dbf5c-9433-4e1d-b31e-9b2b9415fa42 tags:
## Step 5: SSH Tunnel Setup
Displays the command to establish an SSH tunnel from your local machine to the Kubernetes master node.
%% Cell type:code id:128b9653-2c2c-4647-a8ea-6cc7de0ee9e8 tags:
``` python
print("Create a tunnel from your local machine to the head node:")
print(f"ssh -NL 8001:{roles['master'][0].address}:8001 your_username@access.grid5000.fr")
```
%% Cell type:markdown id:5ffb7c25-d6df-47c9-be25-956424d56e1f tags:
## Step 6: Label Kubernetes Nodes
Labels the Kubernetes nodes with roles such as `cloud`, `edge`, and `camera` for better management and deployment.
%% Cell type:code id:95df0232-3017-4ac6-87f8-2d5b642a41bf tags:
``` python
def label_nodes(node_names, labels):
"""
Labels the specified Kubernetes nodes with the provided labels.
:param node_names: List of node names to label.
:param labels: Dictionary of label key-value pairs.
"""
# Construct the label string from the dictionary
label_str = ",".join(f"{key}={value}" for key, value in labels.items())
for node in node_names:
# Construct the kubectl command
command = "kubectl label nodes "+ node +" "+ label_str+ " --overwrite"
try:
# Execute the kubectl command
result = en.run_command(command, roles=roles['master'])
print(f"Successfully labeled {node} with {label_str}")
except subprocess.CalledProcessError as e:
print(f"Failed to label {node}. Error: {e.stderr.decode('utf-8')}")
label_nodes([str(node.alias) for node in roles["object_recognizer"]],{"pos":"cloud"})
label_nodes([str(node.alias) for node in roles["motion_detector"]],{"pos":"edge"})
label_nodes([str(node.alias) for node in roles["camera"]],{"pos":"camera"})
```
%% Cell type:markdown id:5db18e59-e5e5-4814-a024-e1a0c6c29ace tags:
## Step 7: Create Docker Registry Secret
Creates a Docker registry secret in Kubernetes to securely access private Docker images.
%% Cell type:code id:d3fd6791-a3dc-4735-a8f3-111ca24edd6e tags:
``` python
en.run_command("kubectl create secret docker-registry dockerhub-secret \
--docker-username=your_docker_username \
--docker-password=your_docker_password \
--docker-email=your_email", roles=roles['master'])
```
%% Cell type:markdown id:6eaed1fe-f891-4945-b8ea-f520072bc3c1 tags:
## Step 8: Transfer Deployment Files
Transfers deployment files to the Kubernetes master node and applies them using `kubectl`.
%% Cell type:code id:257fa070-f648-4f63-ac18-3d3089872d12 tags:
``` python
import os
import subprocess
def send_file(file_path,file_name):
scp_command = f"scp {file_path} root@{roles['master'][0].address}:{file_name}"
os.system( f"ssh-keygen -f ~/.ssh/known_hosts -R {roles['master'][0].address}")
os.system(scp_command)
return scp_command
local_directory = "/path/to/your/deployment/files/"
for root, dirs, files in os.walk(local_directory):
for file_name in files:
print(local_directory +file_name)
send_file(local_directory +file_name,file_name)
en.run_command("kubectl apply -f deployement.yaml", roles=roles['master'])
en.run_command("kubectl apply -f configmap.yaml", roles=roles['master'])
```
%% Cell type:markdown id:8d584885-cea9-4475-ae2c-919a8e0e856b tags:
## Step 9: Install Helm and Prometheus
Installs Helm on the Kubernetes master node and uses it to deploy Prometheus for monitoring.
%% Cell type:code id:b448903b-6209-4620-a495-8ec3af61edca tags:
``` python
with en.actions(roles=roles["master"]) as a:
# Helm
a.raw("curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3")
a.raw("chmod 700 get_helm.sh")
a.raw("./get_helm.sh")
# Prometheus
a.raw("helm --kubeconfig /etc/rancher/k3s/k3s.yaml repo add prometheus-community https://prometheus-community.github.io/helm-charts")
a.raw("helm --kubeconfig /etc/rancher/k3s/k3s.yaml repo update")
#a.raw("helm --kubeconfig /etc/rancher/k3s/k3s.yaml uninstall prometheus -n default")
a.raw("helm --kubeconfig /etc/rancher/k3s/k3s.yaml install prometheus prometheus-community/prometheus -n default --set grafana.enabled=True -f costum-values.yaml")
#a.raw("helm upgrade --kubeconfig /etc/rancher/k3s/k3s.yaml prometheus prometheus-community/prometheus --namespace default -f costum-values.yaml")
a.raw("helm upgrade --kubeconfig /etc/rancher/k3s/k3s.yaml prometheus prometheus-community/prometheus --namespace default --set prometheus-node-exporter.service.port=9500")
```
%% Cell type:code id:80cc2eba-3a04-4a5a-9712-f4c61283173b tags:
``` python
en.run_command("helm upgrade --kubeconfig /etc/rancher/k3s/k3s.yaml prometheus prometheus-community/prometheus --namespace default -f costum-values.yaml", roles=roles['master'])
```
%% Cell type:markdown id:11ba3c0b-dbff-4790-8a3d-015ae2dc2f1a tags:
## Step 10: Host Extraction
Extracts the internal IP address of a Kubernetes service host by parsing `kubectl` output.
%% Cell type:code id:b8b5c6eb-f554-49a3-a601-458ba79fb215 tags:
``` python
def get_host(service):
results = en.run_command("kubectl get pods -n kube-system -l app="+service, roles=roles['master'])
#print(results)
import re
# Define a regex pattern to match the host value
pattern = r"host='([^']*)'"
# Search for the pattern in the text
match = re.search(pattern, str(results))
if match:
host = match.group(1)
print("Extracted host:", host)
else:
print("Host not found")
results = en.run_command("kubectl describe node "+host, roles=roles['master'])
#print(results)
pattern = r"InternalIP:\s*([\d.]+)"
# Search for the pattern in the text
match = re.search(pattern, str(results))
if match:
host = match.group(1)
return host
else:
print("Host not found")
```
%% Cell type:markdown id:f429ac09-2488-41e3-9261-fff34c445d83 tags:
## Step 11: Chaos Mesh Installation
Deploys Chaos Mesh using Helm to simulate failures and test Kubernetes workloads.
%% Cell type:code id:e78c3f76-b77b-40d3-9958-c926e2af81cf tags:
``` python
with en.actions(roles=roles["master"]) as a:
# Install Chaos Mesh
#a.raw("kubectl apply -f https://charts.chaos-mesh.org/chaos-mesh-v2.1.0.tgz")
# Install Helm chart for Chaos Mesh
a.raw("helm --kubeconfig /etc/rancher/k3s/k3s.yaml repo add chaos-mesh https://charts.chaos-mesh.org")
a.raw("helm --kubeconfig /etc/rancher/k3s/k3s.yaml repo update")
a.raw("helm --kubeconfig /etc/rancher/k3s/k3s.yaml install chaos-mesh chaos-mesh/chaos-mesh --set chaosDaemon.runtime=containerd --set chaosDaemon.containerdSocket=/run/containerd/containerd.sock -n default")
a.raw("helm --kubeconfig /etc/rancher/k3s/k3s.yaml install chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh --create-namespace\
--set chaosDaemon.runtime=containerd \
--set chaosDaemon.socketPath=/run/k3s/containerd/containerd.sock \
--set chaosDaemon.extraVolumes[0].name=fuse-device \
--set chaosDaemon.extraVolumes[0].hostPath.path=/dev/fuse \
--set chaosDaemon.extraVolumes[0].hostPath.type=CharDevice \
--set chaosDaemon.extraVolumeMounts[0].name=fuse-device \
--set chaosDaemon.extraVolumeMounts[0].mountPath=/dev/fuse \
--set chaosDaemon.securityContext.privileged=true")
a.raw("kubectl apply -f rbac.yaml")
a.raw("kubectl create token account-default-admin-eechh")
a.raw('kubectl patch svc chaos-dashboard -n default -p \'{"spec": {"ports": [{"name": "http", "protocol": "TCP", "port": 2333, "targetPort": 2333, "nodePort": 30312}]}}\'')
a.raw('kubectl patch svc chaos-dashboard -n chaos-mesh -p \'{"spec": {"ports": [{"name": "http", "protocol": "TCP", "port": 2333, "targetPort": 2333, "nodePort": 30312}]}}\'')
```
%% Cell type:markdown id:c029bc8e-c9a3-46bc-8c66-283d6d994d6d tags:
## Step 12: Expose Prometheus Service
Exposes the Prometheus service using a NodePort for external access.
%% Cell type:code id:36a27fcb-80dd-4daa-a8d4-3981fa4a88e7 tags:
``` python
with en.actions(roles=roles["master"]) as a:
# Helm
a.raw('kubectl patch svc prometheus-server -n default -p \'{"spec": {"type": "NodePort", "ports": [{"name": "http", "port": 80, "targetPort": 9090, "nodePort": 30090}]}}\'')
```
%% Cell type:markdown id:a968ef4f-2ab9-4cab-b057-cc71501fe32b tags:
## Step 13: Display Service URLs
Generates and prints URLs for the feedback mechanism, Prometheus, and Chaos Mesh dashboards.
%% Cell type:code id:a46efa7f-4c7e-4f24-8f96-55fcee00686c tags:
``` python
# Assuming get_host is a function that returns the host string
urlfb = f"http://{get_host('feedbackmechanism')}:30095"
print(f"feedbackmechanisme web page host: {urlfb}")
url = f"http://{get_host('prometheus')}:30090"
print(f"prometheus web page host: {url}")
url = f"http://{get_host('chaos-dashboard')}:30312"
print(f"Chaos-mesh web page host: {url}")
```
%% Cell type:markdown id:c81c734e-07f4-4889-b189-3f881dd9a385 tags:
## Step 14: Retrieve Admin Token
Retrieves the admin token for Kubernetes using a `kubectl` command.
%% Cell type:code id:b5130637-d891-4506-bc0e-47f0c4ea24d7 tags:
``` python
results=en.run_command("kubectl create token account-default-admin-eechh",roles=roles["master"])
for res in results:
print(res.payload['stdout'])
```
%% Cell type:markdown id:6e33d61f-6182-44e4-8111-a34a47782ce4 tags:
## Step 15: Destroy Resources
Cleans up and releases the allocated resources after the workflow is complete.
%% Cell type:code id:b7198604-e8c6-47b0-8f85-e7afcc505d04 tags:
``` python
provider.destroy()
```
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment