Commit 4fe96b25 authored by Cyril Rohr's avatar Cyril Rohr
Browse files

[admin] updated documentation and generator code, so that it checks that the...

[admin] updated documentation and generator code, so that it checks that the required config files are passed as arguments.
parent b260203d
{{Maintainer|Cyril Rohr}}
{{Author|Cyril Rohr}}
{{Status|Draft}}
{{Portal|Admin}}
== Synopsis ==
The reference data is stored in a Git repository as JSON files, organized into hierarchical folders. These files can be manually written but the Git repository comes with a "/generators" folder which contains a script to ease their generation, based on high-level description files written in Ruby. Given one or more input files that describe the data you want to add, it will generate the required JSON files, directories and hard links.
The reference data is stored in a Git repository as JSON files, organized into hierarchical folders. These files can be manually written but the Git repository comes with a "/generators" folder which contains a script to ease their generation, based on high-level description files written in Ruby. Given one or more input files that describe the data you want to add, it will generate the required JSON files, directories and sym links.
== Requirements ==
* Ruby >= 1.8.6
......@@ -23,10 +24,15 @@ The general overview of the workflow between git repositories is as follows:
API REPOSITORY SITE X <--| |--> API REPOSITORY SITE Y
Each site administrator must first clone the remote MASTER REPOSITORY located on the git.grid5000.fr server and store it on a local machine (this is what I call the ADMIN REPOSITORY and has to be done once):
g5kadmin@host:/somewhere$ git clone ssh://g5kadmin@git.grid5000.fr/srv/git/repos/reference-repository.git
When there is a need for change, the site administrator PULLs from the MASTER REPOSITORY to get the latest changes:
g5kadmin@host:/somewhere/reference-repository$ git pull
Then she manually adds/edits/removes the raw JSON files or uses the generator (more on that later). When she's done, she COMMITs her changes and PUSHes them to the MASTER REPOSITORY:
... editing ...
g5kadmin@host:/somewhere/reference-repository$ git commit -a -m "list of modifications"
g5kadmin@host:/somewhere/reference-repository$ git push
......@@ -37,18 +43,35 @@ Finally, these changes are automatically replicated every minute to each API REP
== Getting started ==
First, clone the remote MASTER REPOSITORY if it is not already done:
g5kadmin@host:/somewhere/reference-repository$ git clone ssh://g5kadmin@git.grid5000.fr/srv/git/repos/reference-repository.git
Right now, the easiest way to get started is to look at some existing input files in the "generators/input" directory. There you can see how you can define sites, clusters, nodes and environments programmatically. Then you may create a new input file or change an existing one and run it in simulation mode:
g5kadmin@host:/somewhere/reference-repository$ ruby generators/grid5000.rb generators/input/*.rb -s
Right now, the easiest way to get started is to look at some existing input files in the "generators/input" directory. There you can see how you can define sites, clusters, nodes and environments programmatically.
Then you may create a new input file or change an existing one and run it in simulation mode:
g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/*.rb -s
or, if you want to explicitly specify the input files:
g5kadmin@host:/somewhere/reference-repository$ ruby generators/grid5000.rb generators/input/input-file1.rb generators/input/input-file2.rb -s
g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/input-file1.rb generators/input/input-file2.rb -s
For more information about the available options and usage of the <code>grid5000</code> generator, run:
g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 --help
Your changes won't be applied but you'll see what would have been changed. Thus, the simulation mode is useful to review your changes before committing and check the ruby syntax of the input files.
When you are happy with your changes, you can then run the command without the -s flag:
g5kadmin@host:/somewhere/reference-repository$ ruby generators/grid5000.rb generators/input/*.rb
Finally, commit your changes with a meaningful message and push them immediately to the MASTER REPOSITORY.
g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/*.rb
Please be aware that config files (YAML format) may be passed on the command line, so that the values can be used in the input files via the <code>lookup(config_filename, key)</code> function. To tell the generator to include one or more config files, you must pass them in your command arguments:
g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/*.rb generators/input/*.yaml
Note that the extension of config files MUST be <code>.yaml</code> or <code>.yml</code>, otherwise they won't be included.
Finally, commit your changes with a meaningful message (you SHOULD first review the changes that will be committed by running the <code>git diff</code> command) and push them immediately to the MASTER REPOSITORY.
== Resources ==
* [http://cheat.errtheblog.com/s/git Git Cheat Sheet]
......@@ -61,9 +84,12 @@ At runtime, the generator resolves the hostname of each node to obtain its IP ad
After each modification to the repository, you should immediately commit your changes with a meaningful message, so that people can easily understand what has changed (latest changes will be displayed in a syndication feed). Your commits should also be site-specific, or even cluster-specific to avoid merge conflicts. Try to avoid putting a lot of changes in only one commit.
You should also check that your name and email are correctly configured in your Git configuration:
$ git config --get user.name
$ git config --get user.email
Otherwise you can set them by issuing:
$ git config --global user.name 'John Doe'
$ git config --global user.email johndoe@example.com
......
#!/usr/bin/env ruby
require 'pp'
require 'rubygems'
require 'fileutils'
require 'json/pure'
require 'yaml'
require 'time'
require 'optparse'
require File.dirname(__FILE__)+'/lib/core_extensions'
require File.dirname(__FILE__)+'/lib/g5k_generator'
usage = %{
options = {:simulate => false}
option_parser = OptionParser.new do |opts|
opts.banner = %{
A tool to generate the Grid5000 reference data.
Usage:
ruby grid5000.rb [input files] [config files] [options]
Options:
-s : simulation mode
./grid5000 input_files [config_files] [options]
Notes:
input files must be RUBY files and end with a .rb extension.
config files mst be YAML files and end with a .yaml extension.
}
# true if we want to simulate:
simulation_mode = !$*.delete("-s").nil?
* input_files MUST be .rb files.
* config_files MUST be .yml or .yaml files.
Examples:
./grid5000 input/*.rb input/*.yaml -s
./grid5000 input/nancy.rb input/nancy.yaml
./grid5000 input/rennes.rb input/lille.rb -s
Options:}
opts.on("-s", "--simulate", "Run in simulation mode (do not write changes). Default to false.") do |v|
options[:simulate] = v
end
opts.on_tail("-h", "--help", "Show this message") do
puts opts
exit
end
end
if $*.empty?
puts usage
option_parser.parse!
files = ARGV
if files.empty?
$stderr.puts option_parser.help
exit 1
elsif ($*.map{|file| File.exists?(file)}.include? false)
puts "Error: one of your input file do not exist."
elsif (files.map{|file| File.exists?(file)}.include? false)
$stderr.puts "Error: one of your input or config file does not exist."
exit 2
else
description_files = $*
input = {}
config = {}
description_files.each do |filename|
case File.extname(filename)
when ".rb" then input[File.basename(filename, ".rb")] = File.read(filename)
when ".yaml" then config[File.basename(filename, ".yaml")] = YAML.load_file(filename)
files.each do |filename|
case (ext = File.extname(filename))
when ".rb"
input[File.basename(filename, ext)] = File.read(filename)
when ".yml", ".yaml"
config[File.basename(filename, ext)] = YAML.load_file(filename)
else
raise ArgumentError, "The file #{filename} is neither an input file nor a config file."
end
end
puts "[Input files:\t\t #{input.keys.join(", ")}]"
puts "[Config files:\t\t #{config.keys.join(", ")}]"
puts "[Simulation mode:\t #{simulation_mode}]"
puts "[Simulation mode:\t #{options[:simulate]}]"
generator = G5K::ReferenceGenerator.new({:uid => "grid5000", :type => "grid"}, :input => input, :config => config)
data = generator.generate
directory_to_write = File.expand_path File.join(File.dirname(__FILE__), "../data")
generator.write(directory_to_write, :simulate => simulation_mode)
generator.write(directory_to_write, options)
exit 0
end
......@@ -23,7 +23,7 @@ module G5K
def valid?
true
# we could do sthg like: Load(type.class).new(hash_values).valid?
# we could do sthg like: Object.const_get(type.class.name).new(hash_values).valid?
end
end
class Folder < Array
......@@ -51,163 +51,162 @@ module G5K
end
end
class ReferenceGenerator
attr_reader :data
attr_reader :config
attr_reader :input
class ReferenceGenerator
attr_reader :data
attr_reader :config
attr_reader :input
def method_missing(method, *args)
@context.recursive_merge!(method.to_sym => args.first)
end
def method_missing(method, *args)
@context.recursive_merge!(method.to_sym => args.first)
end
def dns_lookup(network_address)
Resolv.getaddress(network_address)
end
def dns_lookup(network_address)
Resolv.getaddress(network_address)
end
# Lookup a key in one of the configuration files passed to the generator
#
# Usage:
# lookup('nancy', 'nodes', 'paramount-1', 'property_name')
# or
# lookup('nancy', 'nodes') { |result| result['paramount-1']['property_name'] }
# or
# lookup('nancy') { |result| result['nodes']['paramount-1']['property_name'] }
#
# assuming you passed a <tt>nancy.yaml</tt> file to the generator
# Be careful with null values!
#
def lookup(filename, *keys, &block)
if config.has_key?(filename)
result = config[filename]
if !keys.empty?
while !keys.empty? do
result = result[keys.shift]
break if result.nil?
# Lookup a key in one of the configuration files passed to the generator
#
# Usage:
# lookup('nancy', 'nodes', 'paramount-1', 'property_name')
# or
# lookup('nancy', 'nodes') { |result| result['paramount-1']['property_name'] }
# or
# lookup('nancy') { |result| result['nodes']['paramount-1']['property_name'] }
#
# assuming you passed a <tt>nancy.yaml</tt> file to the generator
#
def lookup(filename, *keys, &block)
if config.has_key?(filename)
result = config[filename]
if !keys.empty?
while !keys.empty? do
result = result[keys.shift]
break if result.nil?
end
end
if block
block.call(result)
else
result
end
end
if block
block.call(result)
else
result
raise ArgumentError, "Cannot fetch the values for '#{keys.inspect}' in the input file '#{filename}'. The config files you gave to me are: '#{config.keys.inspect}'."
end
else
nil
end
end
# This doesn't work with Ruby < 1.8.7. Replaced by a call to build_context (see below).
#
# %w{site cluster environment node service}.each do |method|
# define_method(method) do |uid, *options, &block|
# key = method.pluralize.to_sym
# uid = uid.to_s
# options = options.first || Hash.new
# old_context = @context
# @context[key] ||= G5K::Folder.new
# if options.has_key? :refer_to
# @context[key] << G5K::Link.new(uid, options[:refer_to])
# else
# # if the same object already exists, we return it for completion/modification
# if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
# @context = same_trees.first
# else
# @context[key] << G5K::Tree.new.replace({:uid => uid, :type => method})
# @context = @context[key].last
# end
# block.call(uid) if block
# end
# @context = old_context
# end
# end
# This doesn't work with Ruby < 1.8.7. Replaced by a call to build_context (see below).
#
# %w{site cluster environment node service}.each do |method|
# define_method(method) do |uid, *options, &block|
# key = method.pluralize.to_sym
# uid = uid.to_s
# options = options.first || Hash.new
# old_context = @context
# @context[key] ||= G5K::Folder.new
# if options.has_key? :refer_to
# @context[key] << G5K::Link.new(uid, options[:refer_to])
# else
# # if the same object already exists, we return it for completion/modification
# if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
# @context = same_trees.first
# else
# @context[key] << G5K::Tree.new.replace({:uid => uid, :type => method})
# @context = @context[key].last
# end
# block.call(uid) if block
# end
# @context = old_context
# end
# end
def site(uid, *options, &block)
build_context(:sites, uid, *options, &block)
end
def cluster(uid, *options, &block)
build_context(:clusters, uid, *options, &block)
end
def server(uid, *options, &block)
build_context(:servers, uid, *options, &block)
end
def environment(uid, *options, &block)
build_context(:environments, uid, *options, &block)
end
def node(uid, *options, &block)
build_context(:nodes, uid, *options, &block)
end
def service(uid, *options, &block)
build_context(:services, uid, *options, &block)
end
def build_context(key, uid, *options, &block)
type = key.to_s.chop
uid = uid.to_s
options = options.first || Hash.new
old_context = @context
@context[key] ||= G5K::Folder.new
if options.has_key? :refer_to
@context[key] << G5K::Link.new(uid, options[:refer_to])
else
# if the same object already exists, we return it for completion/modification
if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
@context = same_trees.first
else
@context[key] << G5K::Tree.new.replace({:uid => uid, :type => type})
@context = @context[key].last
def site(uid, *options, &block)
build_context(:sites, uid, *options, &block)
end
def cluster(uid, *options, &block)
build_context(:clusters, uid, *options, &block)
end
def server(uid, *options, &block)
build_context(:servers, uid, *options, &block)
end
def environment(uid, *options, &block)
build_context(:environments, uid, *options, &block)
end
def node(uid, *options, &block)
build_context(:nodes, uid, *options, &block)
end
def service(uid, *options, &block)
build_context(:services, uid, *options, &block)
end
def build_context(key, uid, *options, &block)
type = key.to_s.chop
uid = uid.to_s
options = options.first || Hash.new
old_context = @context
@context[key] ||= G5K::Folder.new
if options.has_key? :refer_to
@context[key] << G5K::Link.new(uid, options[:refer_to])
else
# if the same object already exists, we return it for completion/modification
if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
@context = same_trees.first
else
@context[key] << G5K::Tree.new.replace({:uid => uid, :type => type})
@context = @context[key].last
end
block.call(uid) if block
end
block.call(uid) if block
@context = old_context
end
@context = old_context
end
# Initializes a new generator that will generate data files in a hierachical way.
# The root of the tree will be named with the value of <tt>data_description[:uid]</tt>.
def initialize(data_description = {:uid => ""}, options = {:input => {}, :config => {}})
@input = options[:input]
raise(ArgumentError, "INPUT cannot be null or empty.") if input.nil? || input.empty?
@config = options[:config] || {}
@data = G5K::Tree.new.replace(data_description)
@context = @data
end
def generate
input.each do |filename, content|
eval(content)
# Initializes a new generator that will generate data files in a hierachical way.
# The root of the tree will be named with the value of <tt>data_description[:uid]</tt>.
def initialize(data_description = {:uid => ""}, options = {:input => {}, :config => {}})
@input = options[:input]
raise(ArgumentError, "INPUT cannot be null or empty.") if input.nil? || input.empty?
@config = options[:config] || {}
@data = G5K::Tree.new.replace(data_description)
@context = @data
end
@data
end
def write(repository, options = {:simulate => false})
things_to_create = []
@data.write("/") do |thing_to_create|
things_to_create << thing_to_create if thing_to_create.valid?
end
groups = things_to_create.group_by{|thing| thing.class}
groups.has_key?(G5K::Folder) and groups[G5K::Folder].each do |folder|
full_path = File.join(repository, folder.path)
unless File.exists?(full_path)
puts "Directory to be written = \t#{full_path}"
FileUtils.mkdir_p(full_path) unless options[:simulate]
def generate
input.each do |filename, content|
eval(content)
end
@data
end
groups.has_key?(G5K::Tree) and groups[G5K::Tree].each do |file|
full_path = File.join(repository, file.path)
new_content = JSON.pretty_generate(file.contents)
existing_content = File.exists?(full_path) ? File.open(full_path, "r").read : ""
if new_content.hash != existing_content.hash
puts "File to be written = \t#{full_path}"
File.open(full_path, "w+"){ |f| f << new_content } unless options[:simulate]
def write(repository, options = {:simulate => false})
things_to_create = []
@data.write("/") do |thing_to_create|
things_to_create << thing_to_create if thing_to_create.valid?
end
end
groups.has_key?(G5K::Link) and groups[G5K::Link].each do |link|
FileUtils.cd(repository) do |dir|
to = File.join(".", link.path)
from = File.join([".."]*(to.count(File::SEPARATOR)-1), "#{link.from}.json")
unless File.exists?(to)
puts "Symbolic link to be written = \t#{to} -> #{from}"
FileUtils.ln_s(from, to, :force => true) unless options[:simulate]
groups = things_to_create.group_by{|thing| thing.class}
groups.has_key?(G5K::Folder) and groups[G5K::Folder].each do |folder|
full_path = File.join(repository, folder.path)
unless File.exists?(full_path)
puts "Directory to be written = \t#{full_path}"
FileUtils.mkdir_p(full_path) unless options[:simulate]
end
end
end
end
groups.has_key?(G5K::Tree) and groups[G5K::Tree].each do |file|
full_path = File.join(repository, file.path)
new_content = JSON.pretty_generate(file.contents)
existing_content = File.exists?(full_path) ? File.open(full_path, "r").read : ""
if new_content.hash != existing_content.hash
puts "File to be written = \t#{full_path}"
File.open(full_path, "w+"){ |f| f << new_content } unless options[:simulate]
end
end
groups.has_key?(G5K::Link) and groups[G5K::Link].each do |link|
FileUtils.cd(repository) do |dir|
to = File.join(".", link.path)
from = File.join([".."]*(to.count(File::SEPARATOR)-1), "#{link.from}.json")
unless File.exists?(to)
puts "Symbolic link to be written = \t#{to} -> #{from}"
FileUtils.ln_s(from, to, :force => true) unless options[:simulate]
end
end
end
end
end
end
end
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment