Verified Commit f244c9fe authored by Raphaël Bleuse's avatar Raphaël Bleuse
Browse files

Merge branch 'packaging-refactor'

See merge request !14
parents e7bcd4e7 94fe4357
# git specific files
.gitattributes export-ignore
.gitignore export-ignore
.gitkeep export-ignore
.mailmap export-ignore
Raphaël Bleuse <raphael.bleuse@inria.fr>
Henri Casanova <henric@hawaii.edu> henricasanova <henric@dirt00.ics.hawaii.edu>
Pierre-François Dutot <pfdutot@imag.fr>
David Glesser <david.glesser@imag.fr>
Steffen Lackner <lackner@cs.tu-darmstadt.de>
Michael Mercier <michael.mercier@inria.fr>
Clément Mommessin <clement.mommessin@inria.fr>
Millian Poquet <millian.poquet@inria.fr>
Olivier Richard <olivier.richard@imag.fr>
......@@ -11,10 +11,26 @@ The format is based on `Keep a Changelog`_.
Unreleased
----------
- `Commits since v3.2.0 <https://gitlab.inria.fr/batsim/pybatsim/compare/v3.2.0...master>`_
- `Commits since v3.2.1 <https://gitlab.inria.fr/batsim/pybatsim/compare/v3.2.1...master>`_
- ``nix-env -f https://github.com/oar-team/nur-kapack/archive/master.tar.gz -iA pybatsim-master``
........................................................................................................................
v3.2.1
------
- Release date: 2021-12-15
- `Commits since v3.2.0 <https://gitlab.inria.fr/batsim/pybatsim/compare/v3.2.0...v3.2.1>`_
Miscellaneous
~~~~~~~~~~~~~
- Clean packaging:
- moved to poetry, a PEP517-compliant tool
- `pandas` dependency is now optional, use ``pip install pybatsim[tools]`` if you wish to keep previous behavior of ``pip install``
........................................................................................................................
v3.2.0
......@@ -41,7 +57,7 @@ Additions
Miscellaneous
~~~~~~~~~~~~~
- Example schedulers in the `schedulers` folder are now up-to-date, except from those in the `unMaintained` sub-folder.
- Example schedulers in the `schedulers` folder are now up-to-date, except from those in the `unMaintained` sub-folder.
........................................................................................................................
......
.PHONY: test
test:
make --directory=tests/
.PHONY: doc
doc: clean
sphinx-apidoc --force --output-dir=doc/apidoc/batsim/ batsim/
sphinx-apidoc --force --output-dir=doc/apidoc/schedulers/ schedulers/
make --directory=doc/ html
.PHONY: clean
clean:
make --directory=doc/ clean
rm --recursive --force -- doc/apidoc/
......@@ -22,7 +22,9 @@ The following commands are provided:
To launch schedulers
*pybatsim-experiment*
To launch experiments
To launch experiments.
See `sample.expe.json` for an example configuration file expected by this launcher.
The launcher will start Batsim and the scheduler with the correct options.
*pybatsim-postprocess-jobs*
To manipulate the `out_jobs.csv` file based on data only available in the
......@@ -49,7 +51,7 @@ See the *schedulers* directory for the available built-in schedulers.
A simple built-in scheduler instance can be executed by calling::
pybatsim fillerSched
This command, however, requires an already running Batsim instance.
The parameter to `pybatsim` can also be a file outside of the project directory
......@@ -60,7 +62,7 @@ like::
Schedulers of the higher level API (`batsim.sched`) can be executed in the same way::
pybatsim delayProfilesAsTasks
This example scheduler will make use of dynamic jobs and convert delay jobs into tasks.
Note that dynamic job submissions have to be enabled in your running Batsim instance to be able to use this scheduler.
......@@ -70,23 +72,16 @@ To see all available starting options see also::
Run an experiment
~~~~~~~~~~~~~~~~~
To run a complete experiment the experiment launcher can be used::
pybatsim-experiment --verbose sample.expe.json
Please note that Batsim has to be installed and the environment has to be set-up for this command to succeed.
Files
-----
*launcher.py*
This is the main entry point to launch things
*launch_expe.py*
Provide an easy way to launch a complete simulation.
Just provide a file that looks like `sample.expe.json` to this script and it will launch Batsim and the scheduler with the right options.
*sample.expe.json*
See `launch_expe.json`
......@@ -95,7 +90,7 @@ Files
*batsim/sched/*
High level scheduler API
*batsim/tools/*
Tools to start the schedulers or for working with the generated data
......
from batsim._version import __version__
import sys
if sys.version_info >= (3, 8):
from importlib import metadata
else:
import importlib_metadata as metadata
__version__ = metadata.version('pybatsim')
__all__ = [__version__]
__version__ = "3.2.0"
"""
batsim.sched.events
~~~~~~~~~~~~~~~~~~~
batsim.sched.events
~~~~~~~~~~~~~~~~~~~
This module provides handling of scheduling events.
This module provides handling of scheduling events.
"""
import logging
......@@ -67,11 +67,11 @@ class LoggingEvent:
return output.getvalue().strip()
def to_csv_line(self):
'''def conv_obj(o):
try:
return o.__dict__
except (AttributeError, ValueError):
return str(o)'''
# def conv_obj(o):
# try:
# return o.__dict__
# except (AttributeError, ValueError):
# return str(o)
data = {}
for k, v in self.data.items():
......@@ -87,12 +87,12 @@ class LoggingEvent:
new_v.append(e)
v = new_v
data[k] = v
'''try:
data = json.dumps(data, default=lambda o: conv_obj(o))
except Exception as e:
raise ValueError(
"Error while dumping json data: {}"
.format(data))'''
# try:
# data = json.dumps(data, default=lambda o: conv_obj(o))
# except Exception as e:
# raise ValueError(
# "Error while dumping json data: {}"
# .format(data))
output = io.StringIO()
csvdata = [self.time, self.level, self.processed_jobs, self.open_jobs,
......
......@@ -304,7 +304,7 @@ class ComputeResource(Resource):
:param time: the starting time after which a time slot is needed
:param future_reservation: if future_reservation is set to True, it must not be
guaranteed that the resource is already freed by Batsim at the given time.
guaranteed that the resource is already freed by Batsim at the given time.
"""
present_time = self._scheduler.time
if time is None:
......
Copyright:
2016 , David Glesser <david.glesser@imag.fr>
2016-2017, Millian Poquet <millian.poquet@inria.fr>
2016 , Henri Casanova <henric@hawaii.edu>
2016 , Pierre-François Dutot <pfdutot@imag.fr>
2017 , Olivier Richard <olivier.richard@imag.fr>
2017 , Michael Mercier <michael.mercier@inria.fr>
2017 , Steffen Lackner <lackner@cs.tu-darmstadt.de>
License: LGPL-3.0
2016 , Henri Casanova <henric@hawaii.edu>
2016 , Pierre-François Dutot <pfdutot@imag.fr>
2016 , David Glesser <david.glesser@imag.fr>
2017-2018 , Steffen Lackner <lackner@cs.tu-darmstadt.de>
2017-2019 , Michael Mercier <michael.mercier@inria.fr>
2018-2020 , Clément Mommessin <clement.mommessin@inria.fr>
2016-2017,2021 , Millian Poquet <millian.poquet@inria.fr>
2017-2018 , Olivier Richard <olivier.richard@imag.fr>
License: LGPL-3.0-only
{ kapack ? import
(fetchTarball "https://github.com/oar-team/nur-kapack/archive/master.tar.gz")
{}
}:
let
self = rec {
pybatsim = kapack.pkgs.python3Packages.buildPythonPackage rec {
pname = "pybatsim";
version = "local";
format = "pyproject";
src = kapack.pkgs.lib.sourceByRegex ./. [
"^pyproject\.toml"
"^poetry\.lock"
"^README\.rst"
"^batsim"
"^batsim/.*\.py"
"^batsim/cmds"
"^batsim/cmds/.*\.py"
"^batsim/sched"
"^batsim/sched/.*\.py"
"^batsim/sched/algorithms"
"^batsim/sched/algorithms/.*\.py"
"^batsim/sched/workloads"
"^batsim/sched/workloads/.*\.py"
"^batsim/sched/workloads/models"
"^batsim/sched/workloads/models/.*\.py"
"^batsim/tools"
"^batsim/tools/.*\.py"
"^schedulers"
"^schedulers/.*\.py"
"^schedulers/unMaintained"
"^schedulers/unMaintained/.*\.py"
];
buildInputs = with kapack.pkgs.python3Packages; [
poetry
];
propagatedBuildInputs = with kapack.pkgs.python3Packages; [
sortedcontainers
pyzmq
redis
click
docopt
kapack.procset
];
doCheck = false;
meta = with kapack.pkgs.lib; {
description = "Python API and schedulers for Batsim";
homepage = "https://gitlab.inria.fr/batsim/pybatsim";
platforms = platforms.all;
license = licenses.lgpl3;
broken = false;
longDescription = "PyBatsim is the Python API for Batsim.";
};
};
};
in
self
......@@ -14,10 +14,10 @@ Library reference
.. toctree::
:maxdepth: 1
apidoc/modules
apidoc/schedulers
apidoc/batsim.tools
apidoc/batsim.sched.algorithms
apidoc/batsim/modules
apidoc/schedulers/modules
apidoc/batsim/batsim.tools
apidoc/batsim/batsim.sched.algorithms
Indices and tables
==================
......
#!/usr/bin/env python3
'''
Run PyBatsim experiments.
'''
import sys
from batsim.cmds.experiments import main
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
'''
Run PyBatsim Schedulers.
'''
import sys
from batsim.cmds.launcher import main
if __name__ == "__main__":
sys.exit(main())
This diff is collapsed.
[tool.poetry]
name = "pybatsim"
version = "3.2.1"
description = "Python API and schedulers for Batsim"
readme = "README.rst"
keywords = [
"scheduler",
"simulation",
]
repository = "https://gitlab.inria.fr/batsim/pybatsim"
authors = [
"Henri Casanova <henric@hawaii.edu>",
"Pierre-François Dutot <pfdutot@imag.fr>",
"David Glesser <david.glesser@imag.fr>",
"Steffen Lackner <lackner@cs.tu-darmstadt.de>",
"Michael Mercier <michael.mercier@inria.fr>",
"Clément Mommessin <clement.mommessin@inria.fr>",
"Millian Poquet <millian.poquet@inria.fr>",
"Olivier Richard <olivier.richard@imag.fr>",
]
maintainers = [
"Raphaël Bleuse <raphael.bleuse@inria.fr>",
]
license = "LGPL-3.0-only"
classifiers = [
"Development Status :: 4 - Beta",
"Environment :: Console",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"Topic :: Scientific/Engineering",
"Topic :: System :: Distributed Computing",
]
packages = [
{include = "batsim"},
{include = "schedulers"},
]
[tool.poetry.dependencies]
python = "^3.7.1"
procset = "^1.0"
sortedcontainers = "^2.3.0"
pyzmq = "^22.0.3"
docopt = "^0.6.2"
importlib-metadata = {version = ">=1.0", python = "<3.8"}
pandas = {version = "^1.3.4", optional = true}
[tool.poetry.dev-dependencies]
coverage = "^6.2"
Sphinx = "^4.3.1"
[tool.poetry.extras]
tools = ["pandas"]
[tool.poetry.scripts]
pybatsim = "batsim.cmds.launcher:main"
pybatsim-experiment = "batsim.cmds.experiments:main"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
import sys
if sys.version_info >= (3, 8):
from importlib import metadata
else:
import importlib_metadata as metadata
__version__ = metadata.version('pybatsim')
__all__ = [__version__]
"""
schedBebida
~~~~~~~~~
~~~~~~~~~~~
This scheduler is the implementation of the BigData scheduler for the
Bebida on batsim project.
......@@ -151,7 +151,6 @@ def generate_dfs_io_profile(
if nb_blocks_to_read == 0:
real_locality = None
else:
# import ipdb; ipdb.set_trace()
real_locality = (
nb_wanted_local_read - nb_blocks_to_read_local
) / nb_blocks_to_read
......@@ -180,7 +179,6 @@ def generate_dfs_io_profile(
# NOTE: We can also manage write location here (under HPC node or
# not)
row = index_of(io_alloc, random.choice(list(io_alloc)))
# import ipdb; ipdb.set_trace()
comm_matrix[(row * len(io_alloc)) + col] += block_size_in_Bytes
# Round robin trough the hosts
......@@ -233,7 +231,6 @@ class SchedBebida(BatsimScheduler):
for curr_interval in iter_intervals:
if len(allocation) >= job.requested_resources:
break
# import ipdb; ipdb.set_trace()
interval_size = len(curr_interval)
self.logger.debug("Interval lookup: {}".format(curr_interval))
......
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import sys
import subprocess
import os
import os.path
from setuptools import setup, find_packages, Command
requirements = [
"sortedcontainers",
"pyzmq",
"pandas",
"docopt==0.6.2",
"procset",
]
setup_requirements = [
"coverage",
"autopep8",
"ipdb",
"docopt"
]
class UserCommand(Command):
def initialize_options(self):
pass
def finalize_options(self):
pass
def run_external_command(self, command, *args, cwd=None):
p = subprocess.Popen([command] + list(args), cwd=cwd)
p.communicate()
if p.returncode != 0:
print(
'Command failed with exit code',
p.returncode,
file=sys.stderr)
sys.exit(p.returncode)
class TestCommand(UserCommand):
description = 'Run tests'
user_options = [
('batsim-bin=', None, 'Path/to/batsim/binary')
]
def initialize_options(self):
self.batsim_bin = None
def finalize_options(self):
self.args = []
if self.batsim_bin is not None:
self.args.append('BATSIMBIN=--batsim-bin=' + str(self.batsim_bin))
else:
raise Exception("Command line option '--batsim-bin' was not set.")
def run(self):
if len(self.args) > 0:
self.run_external_command("make", *self.args ,cwd="tests")
else:
self.run_external_command("make", cwd="tests")
class DocCommand(UserCommand):
description = 'Generate documentation'
user_options = []
def run(self):
self.run_external_command("make", "clean", cwd="doc")
self.run_external_command(
"sphinx-apidoc", "-o", "doc/apidoc", "batsim")
self.run_external_command(
"sphinx-apidoc",
"-o",
"doc/apidoc",
"schedulers")
self.run_external_command("make", "html", cwd="doc")
import webbrowser
new = 2 # open in a new tab
dir_path = os.path.dirname(os.path.realpath(__file__))
webbrowser.open(
"file:///" +
os.path.join(
dir_path,
"doc/_build/html/index.html"),
new=new)
class FormatCommand(UserCommand):
description = 'Format the source code'
user_options = [
('path=', 'p', 'start directory or file'),
]
def initialize_options(self):
self.path = "."
def finalize_options(self):
if self.path is None:
raise Exception("Parameter --path is missing")
elif not os.path.exists(self.path):
raise Exception("Path {} does not exist".format(self.path))
def run(self):
self.run_external_command(
"autopep8",
"-i",
"-r",
"-j",
"0",
"-aaaaaa",
"--experimental",
self.path)
f = open("./README.rst")
read_me = f.read().strip()
f.close()
# Get the version
f = open('batsim/_version.py')
version = f.read()
exec(version)
f.close()
setup(
name='pybatsim',
author="Michael Mercier",
author_email="michael.mercier@inria.fr",
version=__version__,
url='https://gitlab.inria.fr/batsim/pybatsim',
packages=find_packages(),
install_requires=requirements,
setup_requires=setup_requirements,
include_package_data=True,
zip_safe=False,
description="Python scheduler for Batsim",
long_description=read_me,
keywords='Scheduler',
license='LGPLv3',
classifiers=[
'Development Status :: 4 - Beta',
'Programming Language :: Python :: 3',
'Topic :: System :: Clustering',
],
entry_points={
"console_scripts": [
"pybatsim=batsim.cmds.launcher:main",
"pybatsim-experiment=batsim.cmds.experiments:main"
]
},
cmdclass={
'test': TestCommand,
'format': FormatCommand,
'doc': DocCommand,
},
)
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment