Mentions légales du service

Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • batsim/pybatsim
  • x-QGuill/pybatsim
  • adfaure/pybatsim
  • gaupy/pybatsim
4 results
Show changes
Commits on Source (106)
Showing with 182 additions and 931 deletions
# git specific files
.gitattributes export-ignore
.gitignore export-ignore
.gitkeep export-ignore
.mailmap export-ignore
......@@ -54,3 +54,6 @@ docs/_build/
# emacs and vim tmp files
*~
*.swp
# Nix
result*
Raphaël Bleuse <raphael.bleuse@inria.fr>
Henri Casanova <henric@hawaii.edu> henricasanova <henric@dirt00.ics.hawaii.edu>
Pierre-François Dutot <pfdutot@imag.fr>
David Glesser <david.glesser@imag.fr>
Steffen Lackner <lackner@cs.tu-darmstadt.de>
Michael Mercier <michael.mercier@inria.fr>
Clément Mommessin <clement.mommessin@inria.fr>
Millian Poquet <millian.poquet@inria.fr>
Olivier Richard <olivier.richard@imag.fr>
......@@ -11,22 +11,76 @@ The format is based on `Keep a Changelog`_.
Unreleased
----------
- `Commits since v3.0.0 <https://gitlab.inria.fr/batsim/pybatsim/compare/v3.0.0...master>`_
- ``nix-env -f https://github.com/oar-team/kapack/archive/master.tar.gz -iA pybatsim_dev``
- `Commits since v3.2.1 <https://gitlab.inria.fr/batsim/pybatsim/compare/v3.2.1...master>`_
- ``nix-env -f https://github.com/oar-team/nur-kapack/archive/master.tar.gz -iA pybatsim-master``
........................................................................................................................
v3.2.1
------
- Release date: 2021-12-15
- `Commits since v3.2.0 <https://gitlab.inria.fr/batsim/pybatsim/compare/v3.2.0...v3.2.1>`_
Miscellaneous
~~~~~~~~~~~~~
- Clean packaging:
- moved to poetry, a PEP517-compliant tool
- `pandas` dependency is now optional, use ``pip install pybatsim[tools]`` if you wish to keep previous behavior of ``pip install``
........................................................................................................................
v3.0.0
v3.2.0
------
- Release date: 2019-01-16
- `Commits since v2.1.1 <https://gitlab.inria.fr/batsim/pybatsim/compare/2.1.1...v3.0.0>`_
- ``nix-env -f https://github.com/oar-team/kapack/archive/master.tar.gz -iA pybatsim300``
- Release date: 2020-07-30
- `Commits since v3.1.0 <https://gitlab.inria.fr/batsim/pybatsim/compare/v3.1.0...v3.2.0>`_
- ``nix-env -f https://github.com/oar-team/nur-kapack/archive/master.tar.gz -iA pybatsim320``
- Recommended Batsim version: `v4.0.0 <https://gitlab.inria.fr/batsim/batsim/tags/v4.0.0>`_
This version is synchronized with Batsim v4.0.0.
See `Batsim changelog <https://batsim.readthedocs.io/en/latest/changelog.html#v4-0-0>`_ for more details.
Additions
~~~~~~~~~
- Added the handling of `machine_available` and `machine_unavailable` events, as well as the unknown external events.
- The `storage_mapping` of a job is now attached to an `EXECUTE_JOB` event.
- The `register_job` function now returns the created Job object.
- Added the `StorageController` permitting to easily manage `storage` resources and manipulate data between them with `data_staging` jobs.
To be used, the StorageController object must be instanciated by the scheduler after the `SIMULATION_BEGINS` event.
- Added the `reject_jobs_by_ids` function to reject jobs by giving a list of job ids, as opposed to `reject_jobs` which expects a list of Job objects.
Miscellaneous
~~~~~~~~~~~~~
- Example schedulers in the `schedulers` folder are now up-to-date, except from those in the `unMaintained` sub-folder.
........................................................................................................................
v3.1.0
------
- Release date: 2019-01-18
- `Commits since v2.1.1 <https://gitlab.inria.fr/batsim/pybatsim/compare/2.1.1...v3.1.0>`_
- ``nix-env -f https://github.com/oar-team/kapack/archive/master.tar.gz -iA pybatsim3``
- Recommended Batsim version: `v3.0.0 <https://gitlab.inria.fr/batsim/batsim/tags/v3.0.0>`_
This version is synchronized with Batsim v3.0.0.
See `Batsim changelog <https://batsim.readthedocs.io/en/latest/changelog.html#v3-0-0>`_ for more details.
Changes in API
~~~~~~~~~~~~~~
- Mark `start_jobs` as DEPRECATED, please now use `execute_jobs`.
- `set_resource_state`, `notify_resources_added` and `notify_resources_removed` functions now expect a ProcSet for the `resources` argument.
- `onAddResources` and `onRemoveResources` now sends a ProcSet for the `to_add` and `to_remove` arguments, respectively.
........................................................................................................................
v2.1.1
......@@ -48,4 +102,4 @@ v2.0.0
.. _Keep a Changelog: http://keepachangelog.com/en/1.0.0/
\ No newline at end of file
.. _Keep a Changelog: http://keepachangelog.com/en/1.0.0/
from batsim._version import __version__
__version__ = "3.0.0"
"""
batsim.cmds
~~~~~~~~~~~
Additional tools installed in the path.
"""
'''
Run PyBatsim Experiments.
Usage:
pybatsim-experiment <experiment> [options]
Options:
--version Print the version of pybatsim and exit
-h --help Show this help message and exit.
-q --quiet Silent experiment output.
-d --debug Print additional debug messages.
'''
import sys
import json
from docopt import docopt
from batsim.tools.experiments import launch_experiment
from batsim import __version__
def main():
arguments = docopt(__doc__, version=__version__)
verbose = not bool(arguments["--quiet"])
debug = bool(arguments["--debug"])
options_file = arguments["<experiment>"]
try:
with open(options_file) as f:
options = json.loads(f.read())
except FileNotFoundError:
if debug:
raise
print("Experiment file does not exist: {}".format(
options_file), file=sys.stderr)
sys.exit(1)
except Exception:
if debug:
raise
print("Error in json file: {}".format(options_file), file=sys.stderr)
sys.exit(1)
options["options-file"] = options_file
if verbose:
print("Running experiment: {}".format(options_file))
return launch_experiment(options, verbose=verbose)
if __name__ == "__main__":
sys.exit(main())
'''
Run PyBatsim Schedulers.
Usage:
pybatsim <scheduler> [-o <options_string>] [options]
Options:
--version Print the version of pybatsim and exit
-h --help Show this help message and exit.
-v --verbosity=<verbosity-level> Sets the verbosity level. Available
values are {debug, info, warning, error, critical}
Default: info
-p --protect Protect the scheduler using a validating machine.
-s --socket-endpoint=<endpoint> Batsim socket endpoint to use [default: tcp://*:28000]
-e --event-socket-endpoint=<endpoint> Socket endpoint to use to publish scheduler events
-o --options=<options_string> A Json string to pass to the scheduler [default: {}]
-O --options-file=<options_file> A file containing the json options
-t --timeout=<timeout> How long to wait for responses from Batsim [default: 2000]
'''
import sys
import json
import logging
from docopt import docopt
from batsim.tools.launcher import launch_scheduler, instanciate_scheduler
from batsim import __version__
def main():
arguments = docopt(__doc__, version=__version__)
loglevel = logging.WARNING
if not arguments['--verbosity']:
loglevel = logging.INFO
else:
loglevel = logging.getLevelName(arguments['--verbosity'].upper())
FORMAT = '[pybatsim - %(asctime)s - %(name)s - %(levelname)s] %(message)s'
logging.basicConfig(format=FORMAT, level=loglevel)
timeout = int(arguments['--timeout'] or float("inf"))
protect = bool(arguments['--protect'])
if arguments["--options-file"]:
with open(arguments["--options-file"]) as options_file:
options = json.load(options_file)
elif arguments["--options"]:
options = json.loads(arguments['--options'])
else:
options = {}
scheduler_filename = arguments['<scheduler>']
socket_endpoint = arguments['--socket-endpoint']
event_socket_endpoint = arguments['--event-socket-endpoint']
scheduler = instanciate_scheduler(scheduler_filename, options=options)
return launch_scheduler(scheduler,
socket_endpoint,
event_socket_endpoint,
options,
timeout,
protect)
if __name__ == "__main__":
sys.exit(main())
"""
batsim.sched
~~~~~~~~~~~~
An advanced scheduler API based on Pybatsim.
"""
from .scheduler import *
from .job import *
from .resource import *
from .profiles import *
from .alloc import *
from .workloads import *
__all__ = [
"Scheduler",
"as_scheduler",
"Job",
"Jobs",
"ComputeResource",
"Resource",
"Resources",
"ResourceRequirement",
"Profiles",
"Profile",
"Allocation",
"JobDescription",
"WorkloadDescription",
"generate_workload",
]
"""
batsim.tools
~~~~~~~~~~~~
Experimental tools useful only for data from Pybatsim schedulers.
"""
"""
batsim.tools.experiments
~~~~~~~~~~~~~~~~~~~~~~~~
Tools to launch experiments.
"""
import subprocess
import os
import os.path
import sys
import json
import time
import signal
import functools
from batsim.network import NetworkHandler
def is_executable(file):
return os.access(file, os.X_OK)
def get_value(options, keys, fallback_keys=None, default=None):
original_options = options
if not isinstance(keys, list):
keys = [keys]
try:
for k in keys:
options = options[k]
except KeyError:
if fallback_keys:
return get_value(original_options, fallback_keys, default=default)
elif default is not None:
return default
else:
print(
"Option is missing in experiment settings ({})".format(
".".join(keys)),
file=sys.stderr)
sys.exit(1)
return options
def delete_key(options, keys):
if not isinstance(keys, list):
keys = [keys]
try:
for k in keys[:-1]:
options = options[k]
del options[keys[-1]]
except KeyError:
pass
def tail(f, n):
return subprocess.check_output(["tail", "-n", str(n), f]).decode("utf-8")
def truncate_string(s, len_s=30):
if len(s) > len_s:
return "..." + s[len(s) - len_s:]
return s
def get_terminal_size():
try:
rows, columns = subprocess.check_output(["stty", "size"]).split()
return int(rows), int(columns)
except Exception:
return None, None
def print_separator(header=None):
rows, columns = get_terminal_size()
columns = columns or 30
if header:
header = truncate_string(header, columns // 3)
header = " " + header + " "
rem_line_len_part = (columns - len(header)) // 2
line = ("".ljust(rem_line_len_part, "=") +
header +
"".ljust(rem_line_len_part, "="))
else:
line = "".ljust(columns, "=")
print(line)
def check_print(header, s):
if s:
print_separator(header)
print(s)
def execute_cl(
name,
cl,
stdout=None,
stderr=None,
on_failure=None,
verbose=False):
if verbose:
print("Starting: {}".format(" ".join(cl)), end="")
if stdout:
print(" 1>{}".format(stdout.name), end="")
if stderr:
print(" 2>{}".format(stderr.name), end="")
print()
exec = subprocess.Popen(
cl, stdout=stdout, stderr=stderr)
exec.name = name
return exec
def terminate_cl(p, terminate=False):
try:
print("Terminating {}".format(p.name), file=sys.stderr)
except AttributeError:
print("Terminating subprocess", file=sys.stderr)
os.killpg(os.getpgid(p.pid), signal.SIGTERM)
p.wait()
if terminate:
sys.exit(3)
def run_workload_script(options, verbose):
script = options["batsim"]["workload-script"]["path"]
interpreter = options["batsim"]["workload-script"].get("interpreter", None)
args = [str(s) for s in options["batsim"]
["workload-script"].get("args", [])]
def do_run_script(cmds):
out_workload_file_path = os.path.join(
options["output-dir"], "workload.json")
with open(out_workload_file_path, "w") as f:
script_exec = execute_cl(script, cmds, stdout=f, verbose=verbose)
ret = script_exec.wait()
if ret != 0:
raise ValueError(
"Workload script {} failed with return code {}".format(
script, ret))
return out_workload_file_path
if not os.path.exists(script):
raise ValueError("Workload script {} does not exist".format(script))
if interpreter:
return do_run_script([interpreter, script] + args)
else:
if is_executable(script):
return do_run_script(["./" + script] + args)
elif script.endswith(".py"):
return do_run_script(["python3", script] + args)
else:
raise ValueError(
"Workload script {} is not executable but also does not seem to be a python script.".format(script))
def generate_config(options):
out_config_file_path = os.path.join(
options["output-dir"], "config.json")
with open(out_config_file_path, "w") as f:
f.write(json.dumps(options["batsim"]["config"], indent=4))
return out_config_file_path
def prepare_batsim_cl(options, verbose):
if "batsim" not in options:
print("batsim section is missing in experiment settings", file=sys.stderr)
sys.exit(1)
batsim_cl = [
get_value(
options, [
"batsim", "executable", "path"], [
"batsim", "bin"], default="batsim")]
batsim_cl += get_value(options, ["batsim",
"executable", "args"], default=[])
delete_key(options, ["batsim", "executable"])
delete_key(options, ["batsim", "bin"])
batsim_cl.append(
'--export=' +
os.path.join(
options["output-dir"],
options.get("export", "out")))
if "workload-script" in options["batsim"]:
options["batsim"]["workload"] = run_workload_script(options, verbose)
delete_key(options, ["batsim", "workload-script"])
if "config" in options["batsim"]:
options["batsim"]["config-file"] = generate_config(options)
delete_key(options, ["batsim", "config"])
for key, val in options.get("batsim", {}).items():
if not key.startswith("_"):
if isinstance(val, bool):
if not val:
continue
val = ""
else:
val = "=" + str(val)
batsim_cl.append("--" + key + val)
return batsim_cl
def prepare_scheduler_cl(options, verbose):
if "scheduler" not in options:
print(
"scheduler section is missing in experiment settings",
file=sys.stderr)
sys.exit(1)
sched_cl = []
if 'interpreter' in options["scheduler"]:
if options["scheduler"]["interpreter"] == "coverage":
interpreter = ["python3", "-m", "coverage", "run", "-a"]
elif options["scheduler"]["interpreter"] == "pypy":
interpreter = ["pypy", "-OOO"]
elif options["scheduler"]["interpreter"] == "profile":
interpreter = ["python3", "-m", "cProfile", "-o", "simul.cprof"]
else:
assert False, "Unknown interpreter"
sched_cl += interpreter
launcher_path = "launcher.py"
if 'srcdir' in options["scheduler"]:
launcher_path = os.path.join(
options["scheduler"].get(
"srcdir", "."), launcher_path)
sched_cl.append(launcher_path)
else:
sched_cl.append("pybatsim")
sched_cl.append(options["scheduler"]["name"])
try:
sched_options = options["scheduler"]["options"]
except KeyError:
sched_options = {}
sched_options["export-prefix"] = os.path.join(
options["output-dir"], options.get("export", "out"))
sched_cl.append("-o")
sched_cl.append(json.dumps(sched_options))
if options["scheduler"].get("verbose", False):
sched_cl.append('-v')
if options["scheduler"].get("protection", False):
sched_cl.append('-p')
if "socket-endpoint" in options["scheduler"]:
sched_cl.append('-s')
sched_cl.append(options["scheduler"]["socket-endpoint"])
if "event-socket-endpoint" in options["scheduler"]:
sched_cl.append('-e')
sched_cl.append(options["scheduler"]["event-socket-endpoint"])
if "timeout" in options["scheduler"]:
sched_cl.append('-t')
sched_cl.append(options["scheduler"]["timeout"])
return sched_cl
def prepare_exec_cl(options, name):
path = options[name].get("path")
args = options[name].get("args", [])
if path:
return [path] + args
def launch_experiment(options, verbose=True):
if options.get("output-dir", "SELF") == "SELF":
options["output-dir"] = os.path.dirname("./" + options["options-file"])
if not os.path.exists(options["output-dir"]):
os.makedirs(options["output-dir"])
batsim_cl = prepare_batsim_cl(options, verbose)
sched_cl = prepare_scheduler_cl(options, verbose)
if "pre" in options:
pre_cl = prepare_exec_cl(options, "pre")
if verbose:
print()
pre_exec = execute_cl("pre", pre_cl, verbose=verbose)
pre_exec.wait()
if pre_exec.returncode != 0:
sys.exit(pre_exec.returncode)
with open(os.path.join(options["output-dir"], "batsim.stdout"), "w") as batsim_stdout_file, \
open(os.path.join(options["output-dir"], "batsim.stderr"), "w") as batsim_stderr_file, \
open(os.path.join(options["output-dir"], "sched.stdout"), "w") as sched_stdout_file, \
open(os.path.join(options["output-dir"], "sched.stderr"), "w") as sched_stderr_file:
if verbose:
print()
batsim_exec = execute_cl(
batsim_cl[0],
batsim_cl,
stdout=batsim_stdout_file,
stderr=batsim_stderr_file,
verbose=verbose)
if verbose:
print()
sched_exec = execute_cl(
sched_cl[0],
sched_cl,
stdout=sched_stdout_file,
stderr=sched_stderr_file,
on_failure=functools.partial(
terminate_cl,
batsim_exec,
terminate=True),
verbose=verbose)
event_socket_connect = options.get(
"event-socket-connect", "tcp://localhost:28001")
network_client = NetworkHandler(
socket_endpoint=event_socket_connect, timeout=100)
network_client.subscribe()
try:
if verbose:
print("\nSimulation is in progress:")
while True:
if batsim_exec.poll() is not None:
if verbose:
print()
break
elif sched_exec.poll() is not None:
if verbose:
print()
break
time.sleep(0.5)
if verbose:
while True:
status = network_client.recv_string()
if status is None:
break
rows, columns = get_terminal_size()
if columns:
if len(status) > columns:
status = status[:columns - 3] + "..."
status = status.ljust(columns)
print(status, end='\r', flush=True)
except KeyboardInterrupt:
print("\nSimulation was aborted => Terminating batsim and the scheduler")
terminate_cl(batsim_exec)
terminate_cl(sched_exec)
if sched_exec.poll() is not None and sched_exec.returncode != 0 and batsim_exec.poll() is None:
print("Scheduler has died => Terminating batsim")
terminate_cl(batsim_exec)
if batsim_exec.poll() is not None and batsim_exec.returncode != 0 and sched_exec.poll() is None:
print("Batsim has died => Terminating the scheduler")
terminate_cl(sched_exec)
sched_exec.wait()
batsim_exec.wait()
ret_code = abs(batsim_exec.returncode) + abs(sched_exec.returncode)
if verbose or ret_code != 0:
print()
check_print(
"Excerpt of log: " +
batsim_stdout_file.name,
tail(
batsim_stdout_file.name,
5))
check_print(
"Excerpt of log: " +
batsim_stderr_file.name,
tail(
batsim_stderr_file.name,
5))
check_print(
"Excerpt of log: " +
sched_stdout_file.name,
tail(
sched_stdout_file.name,
5))
check_print(
"Excerpt of log: " +
sched_stderr_file.name,
tail(
sched_stderr_file.name,
5))
print("Scheduler return code: " + str(sched_exec.returncode))
print("Batsim return code: " + str(batsim_exec.returncode))
if ret_code == 0:
if "post" in options:
post_cl = prepare_exec_cl(options, "post")
if verbose:
print()
post_exec = execute_cl("post", post_cl, verbose=verbose)
post_exec.wait()
if post_exec.returncode != 0:
sys.exit(post_exec.returncode)
return ret_code
"""
batsim.tools.launcher
~~~~~~~~~~~~~~~~~~~~~
Tools to launch pybatsim schedulers.
"""
import json
import sys
import time
import types
from datetime import timedelta
import importlib.util
import os.path
from batsim.batsim import Batsim, BatsimScheduler, NetworkHandler
from batsim.validatingmachine import ValidatingMachine
from docopt import docopt
import zmq
def module_to_class(module):
"""
transform fooBar to FooBar
"""
return (module[0]).upper() + module[1:]
def filename_to_module(fn):
return str(fn).split(".")[0]
def instanciate_scheduler(name, options):
# A scheduler module in the package "schedulers" is expected.
if "." not in name and "/" not in name:
my_module = name # filename_to_module(my_filename)
my_class = module_to_class(my_module)
# load module(or file)
package = __import__('schedulers', fromlist=[my_module])
if my_module not in package.__dict__:
print("No such scheduler (module file not found).")
sys.exit(1)
if my_class not in package.__dict__[my_module].__dict__:
print("No such scheduler (class within the module file not found).")
sys.exit(1)
# load the class
scheduler_non_instancied = package.__dict__[
my_module].__dict__[my_class]
# A full file path to the scheduler is expected
else:
# Add path to allow relative imports in the scheduler implementation
sys.path.insert(0, os.path.abspath(os.path.dirname(name)))
sys.path.insert(
0, os.path.abspath(
os.path.dirname(
os.path.dirname(name))))
package_path = os.path.split(os.path.dirname(name))[1]
if package_path:
package_path = [package_path]
else:
package_path = []
module_name = os.path.basename(name).split(".")[0]
module_path = ".".join(package_path + [module_name])
my_class = module_to_class(module_name)
# Try to load the module with the outer package
try:
mod = importlib.import_module(module_path)
# Try to load only the module as fallback
except ModuleNotFoundError:
spec = importlib.util.spec_from_file_location(
"schedulers." + module_name, name)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
del sys.path[1]
del sys.path[0]
try:
scheduler_non_instancied = mod.__dict__[my_class]
except KeyError:
print("Module {} contains no scheduler named {}".format(
mod, my_class))
sys.exit(1)
if isinstance(scheduler_non_instancied, types.FunctionType):
from batsim.sched import as_scheduler
scheduler = as_scheduler()(scheduler_non_instancied)
scheduler = scheduler(options)
else:
scheduler = scheduler_non_instancied(options)
if not isinstance(scheduler, BatsimScheduler):
scheduler = scheduler()
return scheduler
def launch_scheduler(scheduler,
socket_endpoint,
event_socket_endpoint,
options,
timeout,
protect):
if protect:
vm = ValidatingMachine
else:
vm = None
print("Scheduler: {} ({})".format(scheduler.__class__.__name__, options))
time_start = time.time()
#try:
bs = Batsim(scheduler,
socket_endpoint,
timeout,
event_socket_endpoint,
validatingmachine=vm)
aborted = False
# try:
bs.start()
# except KeyboardInterrupt:
# print("Aborted...")
# aborted = True
time_ran = str(timedelta(seconds=time.time() - time_start))
print("Simulation ran for: " + time_ran)
print("Job submitted:", bs.nb_jobs_submitted,
", scheduled:", bs.nb_jobs_scheduled,
", rejected:", bs.nb_jobs_rejected,
", killed:", bs.nb_jobs_killed,
", changed:", len(bs.jobs_manually_changed),
", timeout:", bs.nb_jobs_timeout,
", success", bs.nb_jobs_successful,
", complete:", bs.nb_jobs_completed)
if bs.nb_jobs_submitted != (
bs.nb_jobs_scheduled + bs.nb_jobs_rejected +
len(bs.jobs_manually_changed)):
return 1
return 1 if aborted else 0
#except KeyboardInterrupt:
# print("Aborted...")
# return 1
return 0
def launch_scheduler_main(
scheduler_class,
argv=None,
standalone=True,
**kwargs):
for arg in argv or sys.argv[1:]:
if arg == "--protect":
kwargs["protect"] = True
elif arg == "--verbose":
kwargs["verbose"] = 999
elif arg.startswith("--options="):
kwargs["options"] = json.loads(arg[arg.index("=") + 1:])
elif arg.startswith("--options-file="):
with open(arg) as options_file:
kwargs["options"] = json.load(options_file)
elif arg.startswith("--timeout="):
kwargs["timeout"] = int(arg[arg.index("=") + 1:])
elif arg.startswith("--socket-endpoint="):
kwargs["socket_endpoint"] = int(arg[arg.index("=") + 1:])
elif arg.startswith("--event-socket-endpoint="):
kwargs["event_socket_endpoint"] = int(arg[arg.index("=") + 1:])
else:
print("Invalid argument: {}".format(arg))
scheduler = scheduler_class(options)
ret = launch_scheduler(scheduler, **kwargs)
if standalone:
sys.exit(ret)
else:
if ret != 0:
raise ValueError(
"Scheduler exited with return code: {}".format(ret))
from batsim.batsim import BatsimScheduler
from sortedcontainers import SortedSet
from procset import ProcSet
class ValidatingMachine(BatsimScheduler):
"""
This class tries to do a lot of checks to prevent from stupid and invisible errors.
You should use this when you are developping and testing a scheduler.
It checks that:
- not 2 jobs use the same resource as the same time
- a job is only started once
- a job is launched after his submit time
"""
def __init__(self, scheduler):
super().__init__()
self.scheduler = scheduler
def onAfterBatsimInit(self):
self.scheduler.onAfterBatsimInit()
def onSimulationBegins(self):
self.nb_res = self.bs.nb_compute_resources
self.availableResources = SortedSet(range(self.nb_res))
self.jobs_waiting = []
self.previousAllocations = dict()
# save real job start function
self.real_start_jobs = self.bs.start_jobs
self.real_execute_jobs = self.bs.execute_jobs
# intercept job start
self.scheduler.bs = self.bs
self.scheduler.bs.start_jobs = self.start_jobs_valid
self.scheduler.bs.execute_jobs = self.execute_jobs_valid
self.scheduler.onSimulationBegins()
def onSimulationEnds(self):
self.scheduler.onSimulationEnds()
def onDeadlock(self):
self.scheduler.onDeadlock()
def onJobSubmission(self, job):
self.jobs_waiting.append(job)
self.scheduler.onJobSubmission(job)
def onJobCompletion(self, job):
for res in self.previousAllocations[job.id]:
self.availableResources.add(res)
self.previousAllocations.pop(job.id)
self.scheduler.onJobCompletion(job)
def onJobMessage(self, timestamp, job, message):
self.scheduler.onJobMessage(timestamp, job, message)
def onJobsKilled(self, jobs):
self.scheduler.onJobsKilled(jobs)
def onMachinePStateChanged(self, nodeid, pstate):
self.scheduler.onMachinePStateChanged(nodeid, pstate)
def onReportEnergyConsumed(self, consumed_energy):
self.scheduler.onReportEnergyConsumed(consumed_energy)
def onRequestedCall(self):
self.scheduler.onRequestedCall()
def start_jobs_valid(self, jobs, res):
for j in jobs:
try:
self.jobs_waiting.remove(j)
except KeyError:
raise ValueError(
"Job {} was not waiting (waiting: {})".format(
j, [j2.id for j2 in self.jobs_waiting]))
self.previousAllocations[j.id] = res[j.id]
for r in res[j.id]:
try:
self.availableResources.remove(r)
except KeyError:
raise ValueError(
"Resource {} was not available (available: {})".format(
r, list(
self.availableResources)))
j.allocation = ProcSet(*res[j.id])
self.real_execute_jobs(jobs)
def execute_jobs_valid(self, jobs, io_jobs=None):
for j in jobs:
try:
self.jobs_waiting.remove(j)
except KeyError:
raise ValueError(
"Job {} was not waiting (waiting: {})".format(
j, [j2.id for j2 in self.jobs_waiting]))
self.previousAllocations[j.id] = j.allocation
for r in j.allocation:
try:
self.availableResources.remove(r)
except KeyError:
raise ValueError(
"Resource {} was not available (available: {})".format(
r, list(
self.availableResources)))
self.real_execute_jobs(jobs, io_jobs)
\ No newline at end of file
Copyright:
2016 , David Glesser <david.glesser@imag.fr>
2016-2017, Millian Poquet <millian.poquet@inria.fr>
2016 , Henri Casanova <henric@hawaii.edu>
2016 , Pierre-François Dutot <pfdutot@imag.fr>
2017 , Olivier Richard <olivier.richard@imag.fr>
2017 , Michael Mercier <michael.mercier@inria.fr>
2017 , Steffen Lackner <lackner@cs.tu-darmstadt.de>
License: LGPL-3.0
2016 , Henri Casanova <henric@hawaii.edu>
2016 , Pierre-François Dutot <pfdutot@imag.fr>
2016 , David Glesser <david.glesser@imag.fr>
2017-2018 , Steffen Lackner <lackner@cs.tu-darmstadt.de>
2017-2019 , Michael Mercier <michael.mercier@inria.fr>
2018-2020 , Clément Mommessin <clement.mommessin@inria.fr>
2016-2017,2021 , Millian Poquet <millian.poquet@inria.fr>
2017-2018 , Olivier Richard <olivier.richard@imag.fr>
License: LGPL-3.0-only
{ kapack ? import (fetchTarball {
url = "https://github.com/oar-team/nur-kapack/archive/7fa57b4170962b5c88d077d6f625628e7763c81c.tar.gz";
sha256 = "sha256:13irywc4lm30xj4z722whk5fnrqkd3a71vh0fc2c4sqfd1rhcfl6";
}) {}
, pybatsim-core-base ? kapack.pybatsim-core
, pybatsim-functional-base ? kapack.pybatsim-functional
}:
let
self = rec {
pkgs = kapack.pkgs;
lib = pkgs.lib;
python3Packages = pkgs.python3Packages;
pybatsim-core = pybatsim-core-base.overrideAttrs (attrs: rec {
name = "${attrs.name}-local";
src = lib.sourceByRegex ./pybatsim-core [
"^pyproject\.toml$"
"^poetry\.lock$"
"^README\.rst$"
"^src$"
"^src/pybatsim$"
"^src/pybatsim/.\+\.py$"
"^src/pybatsim/batsim$"
"^src/pybatsim/batsim/.\+\.py$"
"^src/pybatsim/schedulers$"
"^src/pybatsim/schedulers/.\+\.py$"
"^src/pybatsim/schedulers/unMaintained$"
"^src/pybatsim/schedulers/unMaintained/.\+\.py$"
];
});
pybatsim-functional = pybatsim-functional-base.overrideAttrs (attrs: rec {
name = "${attrs.name}-local";
src = lib.sourceByRegex ./pybatsim-functional [
"^pyproject\.toml$"
"^poetry\.lock$"
"^src$"
"^src/pybatsim_functional$"
"^src/pybatsim_functional/.\+\.py$"
"^src/pybatsim_functional/algorithms$"
"^src/pybatsim_functional/algorithms/.\+\.py$"
"^src/pybatsim_functional/schedulers$"
"^src/pybatsim_functional/schedulers/.\+\.py$"
"^src/pybatsim_functional/schedulers/unmaintained$"
"^src/pybatsim_functional/schedulers/unmaintained/.\+\.py$"
"^src/pybatsim_functional/tools$"
"^src/pybatsim_functional/tools/.\+\.py$"
"^src/pybatsim_functional/workloads$"
"^src/pybatsim_functional/workloads/.\+\.py$"
"^src/pybatsim_functional/workloads/models$"
"^src/pybatsim_functional/workloads/models/.\+\.py$"
];
# change the pybatsim-core to use (local one, not base one)
propagatedBuildInputs = lib.remove pybatsim-core-base attrs.propagatedBuildInputs ++
[ pybatsim-core ];
});
# entry point for external scheduler example
pybatsim-example = python3Packages.buildPythonPackage rec {
pname = "pybatsim-example-entry-point";
version = "local";
format = "pyproject";
src = lib.sourceByRegex ./pybatsim-example [
"^pyproject\.toml$"
"^poetry\.lock$"
"^.*\.py$"
];
buildInputs = with python3Packages; [
poetry
];
propagatedBuildInputs = [
pybatsim-core
];
};
# example shell that enables to run the example scheduler (run `pybatsim rejector` in the shell)
example-shell = pkgs.mkShell rec {
buildInputs = [
pybatsim-example
];
};
# small shell to dev and test schedulers
dev-shell = pkgs.mkShell rec {
buildInputs = [
pkgs.stdenv.cc.cc.lib
python3Packages.poetry
];
LD_LIBRARY_PATH="${pkgs.stdenv.cc.cc.lib}/lib";
};
};
in
self
# Use this environment to publish this package on pypi:
#
# # Enter the environment
# nix-shell do_release.nix
#
# # create the package
# python setup.py sdist
#
# # register to pypi (if not registered yet)
# twine register dist/project_name-x.y.z.tar.gz
#
# # upload you package
# twine upload dist/project_name-x.y.z.tar.gz
with import <nixpkgs> {};
(pkgs.python36.withPackages (ps: with ps; [twine setuptools])).env
#!/usr/bin/env python3
'''
Run PyBatsim experiments.
'''
import sys
from batsim.cmds.experiments import main
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
'''
Run PyBatsim Schedulers.
'''
import sys
from batsim.cmds.launcher import main
if __name__ == "__main__":
sys.exit(main())
File moved