Attention une mise à jour du serveur va être effectuée le lundi 17 mai entre 13h et 13h30. Cette mise à jour va générer une interruption du service de quelques minutes.

Commit d5fe58a2 authored by Cyril's avatar Cyril Committed by Cyril Rohr

[admin] Added code for OAR synchronization. See rake -D. Updated README.

parent e02cd5e6
......@@ -4,7 +4,7 @@
{{Portal|Admin}}
== Synopsis ==
The reference data is stored in a Git repository as JSON files, organized into hierarchical folders. These files can be manually written but the Git repository comes with a "/generators" folder which contains a script to ease their generation, based on high-level description files written in Ruby. Given one or more input files that describe the data you want to add, it will generate the required JSON files, directories and sym links.
The reference data is stored in a Git repository as JSON files, organized into hierarchical folders. These files can be manually written but the Git repository comes with a "/generators" folder which contains a script to ease their generation, based on high-level description files written in Ruby. Given one or more input files that describe the data you want to add, it will generate the required JSON files, directories and symlinks.
== Requirements ==
* Ruby >= 1.8.6
......@@ -61,6 +61,10 @@ For more information about the available options and usage of the <code>grid5000
g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 --help
You may also use the rake task available (run <code>$ rake -D</code> to see the list of available tasks):
g5kadmin@host:/somewhere/reference-repository$ rake g5k:generate
Your changes won't be applied but you'll see what would have been changed. Thus, the simulation mode is useful to review your changes before committing and check the ruby syntax of the input files.
When you are happy with your changes, you can then run the command without the -s flag:
......@@ -73,7 +77,7 @@ Please be aware that config files (YAML format) may be passed on the command lin
{{Note|text=Note that the extension of config files MUST be <code>.yaml</code> or <code>.yml</code>, otherwise they won't be included.}}
Finally, commit your changes with a meaningful message (you SHOULD first review the changes that will be committed by running the <code>git diff</code> command) and push them immediately to the MASTER REPOSITORY:
Finally, commit your changes with a meaningful message in ENGLISH (you SHOULD first review the changes that will be committed by running the <code>git diff</code> command) and push them immediately to the MASTER REPOSITORY:
g5kadmin@host:/somewhere/reference-repository$ git commit -a -m "[<code class="replace">TAGS</code>] message"
......@@ -81,12 +85,58 @@ Finally, commit your changes with a meaningful message (you SHOULD first review
g5kadmin@host:/somewhere/reference-repository$ git push
== Synchronizing the OAR database ==
As of 2010/09/08, a synchronization task has been added that allows you to generate the diff between 2 commits (not necessarily consecutive).
Once you've committed your changes, run the <code>oar:generate</code> rake task to generate the corresponding oaradmin lines (run <code>$ rake -D</code> to see the list of available tasks):
g5kadmin@host:/somewhere/reference-repository$ rake oar:generate -s FROM=<PREVIOUS-COMMIT-ID> TO=<LATEST-COMMIT-ID>
By default, <code>TO</code> is set to <code>HEAD</code>.
The oaradmin lines are sent to STDOUT, the logging data to STDERR.
== Filling the reference - Guidelines ==
=== <code>network_adapters</code> ===
Many machines have several network interfaces, which are not always all configured. We have identified 4 cases in G5K clusters:
# The interface is not connected to any cable.
# The interface is the admin interface (e.g. IPMI).
# The interface is not mounted in the production environment, but users may use it in their own deployed environment.
# The interface is mounted in the production environment.
After several discussions inside the PS team, we have fixed some attributes but some still require a decision (those with a leading *). We cannot change the names of the fixed attributes for backward compatibility reasons, only their role if necessary. <u>'''Underlined attributes'''</u> would always have to be defined.
* <u>'''interface'''</u>: the type of network interface, &isin; {"Ethernet", "Myrinet", "InfiniBand"}
:: NB: It is useless to define "Myrinet 10G" or "Myri-2000" values, because the '''rate''' will differentiate them.
* <u>'''rate'''</u>: speed of the interface in b/s
* * <u>'''mac'''</u>: the MAC address of this interface
:: Alternate propositions: '''mac_address'''
* '''vendor''': the company which made the device
* '''version''': its version according to the company nomenclatura
* '''driver''': name of the driver for the device in the linux kernel
* <u>'''enabled'''</u>: <code>true</code> if there is any cable connected to this interface
* * <u>'''management'''</u>: <code>true</code> if this interface is on the administration network (IPMI,...)
* * <u>'''mountable'''</u>: <code>true</code> if it is usable by any user (even if it possibly requires a customized environment)
:: NB: ('''enabled'''==false) => ('''mountable'''==false)
:: Alternate propositions: '''usable''', '''available'''
* * <u>'''mounted'''</u>: <code>true</code> if the production environment mounts, configures this interface
:: NB: ('''mountable'''==false) => ('''mounted'''==false)
:: Alternate propositions: '''used''', '''production'''
* * <u>'''device'''</u>: name of this interface in the production environment (would be <u>'''mandatory'''</u> if '''mounted'''==true)
* '''network_address''': the DNS entry of the machine by this interface (<u>'''mandatory'''</u> if '''mounted'''==true)
* '''ip''': the IP of this interface (<u>'''mandatory'''</u> if '''enabled'''==true)
* * '''ip6''': the IPv6 of this interface, for future use...
:: Alternate propositions: '''ipv6_address'''
There is also a proposition where '''mounted''' is renamed as '''enabled''', which preserves the backward compatibility on the attribute name, but the role of '''enabled''' would be given to an attribute '''connected''', or '''wired'''.
== Resources ==
* [http://cheat.errtheblog.com/s/git Git Cheat Sheet]
== FAQ ==
=== Resolv::ResolvError occurred when I tried to generate ===
At runtime, the generator resolves the hostname of each node to obtain its IP address. As a consequence it shoud be run from within Grid5000 so that it can query the Grid5000 DNS.
At runtime, the generator resolves the hostname of each node to obtain its IP address. As a consequence it should be run from within Grid5000 so that it can query the Grid5000 DNS.
=== What kind of commit message ? ===
After each modification to the repository, you should immediately commit your changes with a meaningful message, so that people can easily understand what has changed (latest changes will be displayed in a syndication feed). Your commits should also be site-specific, or even cluster-specific to avoid merge conflicts. Try to avoid putting a lot of changes in only one commit.
......
require 'fileutils'
require 'json'
require 'logger'
ROOT_DIR = File.expand_path File.dirname(__FILE__)
LIB_DIR = File.join(ROOT_DIR, "generators", "lib")
$LOAD_PATH.unshift(LIB_DIR) unless $LOAD_PATH.include?(LIB_DIR)
require 'grid5000'
task :environment do
Dir.chdir(ROOT_DIR)
@logger = Logger.new(STDERR)
@logger.level = Logger.const_get((ENV['DEBUG'] || 'INFO').upcase)
end
namespace :g5k do
desc "Generates the JSON files based on the generators, for all sites.\nUse SITE=<SITE-NAME> if you wish to restrict the generation to a specific site.\nUse DRY=1 to simulate the execution."
task :generate => :environment do
site = if ENV['SITE']
ENV['SITE']
else
"*"
end
command = "#{File.join(ROOT_DIR, "generators", "grid5000")} #{File.join(ROOT_DIR, "generators", "input", "#{site}.rb")} #{File.join(ROOT_DIR, "generators", "input", "#{site}.yaml")}"
command << " -s" if ENV['DRY'] && ENV['DRY'] != "0"
@logger.info "Executing #{command.inspect}..."
system command
end
end
# TESTS
# Deletion:
# rake -s oar:generate FROM=4cfebf92e9cce05315782b51e05eded4ab4f0e7e TO=7d2648eaad7dbbc6f1fdb9c0279f73d374ccd47a
#
# Update:
# rake -s oar:generate FROM=7d2648eaad7dbbc6f1fdb9c0279f73d374ccd47a TO=bb528643003757942521942eaeab74b15aaa976d
#
# Add:
# rake -s oar:generate FROM=be9f7338b9750ce675447c13d172157992041ec1 TO=7dc3a4101a657230b7ad0534025a7ca93c905411
#
# All:
# rake -s oar:generate FROM=be9f7338b9750ce675447c13d172157992041ec1 TO=7d2648eaad7dbbc6f1fdb9c0279f73d374ccd47a
#
namespace :oar do
desc "Generates the oaradmin lines to update the OAR database after a change in the reference repository.\nUse FROM=<SHA-ID> and TO=<SHA-ID> to specify the starting and ending commits.\nUse -s to suppress the 'in directory' announcement."
task :generate => :environment do
if ENV['FROM'].nil? || ENV['FROM'].empty?
@logger.fatal "You MUST specify a commit id from where to start using the FROM=<SHA-ID> argument. Ex: rake -s oar:generate FROM=be9f7338b9750ce675447c13d172157992041ec1 TO=7dc3a4101a657230b7ad0534025a7ca93c905411 2> /dev/null"
exit(1)
end
ENV['TO'] ||= 'HEAD'
@logger.info "Analysing changes between #{ENV['FROM']}..#{ENV['TO']}..."
commands = []
diff = `git diff --name-status #{ENV['FROM']}..#{ENV['TO']}`
diff.split("\n").each do |line|
action, filename = line.split("\t")
next unless filename =~ %r{.+/nodes/.+}
node_uid, site_uid, grid_uid = filename.gsub(/\.json/,'').split("/").values_at(-1, -5, -7)
cluster_uid = node_uid.split("-")[0]
host = [node_uid, site_uid, grid_uid, "fr"].flatten.join(".")
if ENV['SITE'] && !ENV['SITE'].split(",").include?(site_uid)
@logger.info "Skipping #{host} since you only want changes that occured on #{ENV['SITE'].inspect}"
next
end
command = "oaradmin resources"
case action
when "A", "C", "M"
node_properties = JSON.parse(File.read(filename))
cluster_properties = JSON.parse(File.read(filename.gsub(%r{/nodes.*}, "/#{cluster_uid}.json")))
cluster = Grid5000::Cluster.new(cluster_properties)
node = Grid5000::Node.new(cluster, node_properties)
begin
export = node.export("oar-2.4")
rescue Grid5000::MissingProperty => e
@logger.warn "Error when exporting #{host}: #{e.message}. Skipped."
next
end
if action == "M" # modification of a file
command.concat(" -s node=#{host} ")
else # new file
command.concat(" -a node=#{host}/cpu={#{node.properties['architecture']['smp_size']}}/core={#{export['cpucore']}} ")
end
command.concat(" -p ").concat( export.to_a.map{|(k,v)|
if v.nil?
nil
elsif v.kind_of?(String) && v =~ / / # to remove after oaradmin correctly parses options
[k, v.inspect.inspect].join("=")
else
[k, v.inspect].join("=")
end
}.compact.join(" -p ") )
when "D" # deletion of a file
command.concat(" -d node=#{host}")
else
@logger.warn "Don't know what to do with #{line.inspect}. Ignoring."
next
end
commands << command
end
commands.each do |command|
puts command
end
end
end
require 'grid5000/cluster'
require 'grid5000/node'
module Grid5000
class Error < StandardError; end
class MissingProperty < Grid5000::Error; end
end
module Grid5000
class Cluster
attr_reader :properties
def initialize(properties)
@properties = properties
end
end
end
\ No newline at end of file
module Grid5000
class Node
attr_reader :properties, :cluster
MAPPINGS = {
"oar-2.4" => Proc.new{ |cluster, properties|
# see https://www.grid5000.fr/mediawiki/index.php/OAR2_properties for list of properties
h = {}
main_network_adapter = properties["network_adapters"].find{|na| na['enabled'] == true}
h['host'] = main_network_adapter['network_address']
raise MissingProperty, "Node has no network_address" unless h['host']
h['ip'] = main_network_adapter['ip']
raise MissingProperty, "Node has no IP" unless h['ip']
h['cluster'] = cluster.properties['uid']
h['nodemodel'] = cluster.properties['model']
h['switch'] = main_network_adapter['switch']
h['besteffort'] = properties['supported_job_types']['besteffort'] ? "YES" : "NO"
h['deploy'] = properties['supported_job_types']['deploy'] ? "YES" : "NO"
h['ip_virtual'] = properties['supported_job_types']['virtual'] ? "YES" : "NO"
h['virtual'] = properties['supported_job_types']['virtual']
h['cpuarch'] = properties['architecture']['platform_type']
h['cpucore'] = properties['architecture']['smt_size']/properties['architecture']['smp_size']
h['cputype'] = [properties['processor']['model'], properties['processor']['version']].join(" ")
h['cpufreq'] = properties['processor']['clock_speed']/1_000_000_000
h['disktype'] = (properties['storage_devices'].first || {})['interface']
h['ethnb'] = properties["network_adapters"].select{|na| na['interface'] =~ /ethernet/i}.length
ib10g = properties['network_adapters'].detect{|na| na['interface'] =~ /infiniband/i && na['rate'] == 10_000_000_000}
h['ib10g'] = ib10g.nil? ? "NO" : "YES"
h['ib10gmodel'] = ib10g.nil? ? "none" : ib10g['version']
myri10g = properties['network_adapters'].detect{|na| na['interface'] =~ /myri/i && na['rate'] == 10_000_000_000}
h['myri10g'] = myri10g.nil? ? "NO" : "YES"
h['myri10gmodel'] = myri10g.nil? ? "none" : myri10g['version']
myri2g = properties['network_adapters'].detect{|na| na['interface'] =~ /myri/i && na['rate'] == 2_000_000_000}
h['myri2g'] = myri2g.nil? ? "NO" : "YES"
h['myri2gmodel'] = myri2g.nil? ? "none" : myri2g['version']
h['memcore'] = properties['main_memory']['ram_size']/properties['architecture']['smt_size']
h['memcpu'] = properties['main_memory']['ram_size']/properties['architecture']['smp_size']
h['memnode'] = properties['main_memory']['ram_size']
h
}
}
def initialize(cluster, properties)
@cluster = cluster
@properties = properties
end
def export(destination = "oar-2.4")
if MAPPINGS.has_key?(destination)
MAPPINGS[destination].call(cluster, properties)
else
raise ArgumentError, "Unsupported destination for export: #{destination.inspect}"
end
end
end
end
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment