From 4fe96b25d2cbfee16abe5a4fb999c82dbafc2ee8 Mon Sep 17 00:00:00 2001
From: Cyril Rohr <cyril.rohr@gmail.com>
Date: Thu, 22 Oct 2009 13:10:11 +0200
Subject: [PATCH] [admin] updated documentation and generator code, so that it
checks that the required config files are passed as arguments.
---
README => README.wiki | 38 ++++-
generators/grid5000 | 70 ++++++++
generators/grid5000.rb | 49 ------
generators/lib/g5k_generator.rb | 281 ++++++++++++++++----------------
4 files changed, 242 insertions(+), 196 deletions(-)
rename README => README.wiki (75%)
create mode 100755 generators/grid5000
delete mode 100755 generators/grid5000.rb
diff --git a/README b/README.wiki
similarity index 75%
rename from README
rename to README.wiki
index fa01979a0c0..1574b98f98d 100755
--- a/README
+++ b/README.wiki
@@ -1,8 +1,9 @@
{{Maintainer|Cyril Rohr}}
+{{Author|Cyril Rohr}}
{{Status|Draft}}
{{Portal|Admin}}
== Synopsis ==
-The reference data is stored in a Git repository as JSON files, organized into hierarchical folders. These files can be manually written but the Git repository comes with a "/generators" folder which contains a script to ease their generation, based on high-level description files written in Ruby. Given one or more input files that describe the data you want to add, it will generate the required JSON files, directories and hard links.
+The reference data is stored in a Git repository as JSON files, organized into hierarchical folders. These files can be manually written but the Git repository comes with a "/generators" folder which contains a script to ease their generation, based on high-level description files written in Ruby. Given one or more input files that describe the data you want to add, it will generate the required JSON files, directories and sym links.
== Requirements ==
* Ruby >= 1.8.6
@@ -23,10 +24,15 @@ The general overview of the workflow between git repositories is as follows:
API REPOSITORY SITE X <--| |--> API REPOSITORY SITE Y
Each site administrator must first clone the remote MASTER REPOSITORY located on the git.grid5000.fr server and store it on a local machine (this is what I call the ADMIN REPOSITORY and has to be done once):
+
g5kadmin@host:/somewhere$ git clone ssh://g5kadmin@git.grid5000.fr/srv/git/repos/reference-repository.git
+
When there is a need for change, the site administrator PULLs from the MASTER REPOSITORY to get the latest changes:
+
g5kadmin@host:/somewhere/reference-repository$ git pull
+
Then she manually adds/edits/removes the raw JSON files or uses the generator (more on that later). When she's done, she COMMITs her changes and PUSHes them to the MASTER REPOSITORY:
+
... editing ...
g5kadmin@host:/somewhere/reference-repository$ git commit -a -m "list of modifications"
g5kadmin@host:/somewhere/reference-repository$ git push
@@ -37,18 +43,35 @@ Finally, these changes are automatically replicated every minute to each API REP
== Getting started ==
First, clone the remote MASTER REPOSITORY if it is not already done:
+
g5kadmin@host:/somewhere/reference-repository$ git clone ssh://g5kadmin@git.grid5000.fr/srv/git/repos/reference-repository.git
-Right now, the easiest way to get started is to look at some existing input files in the "generators/input" directory. There you can see how you can define sites, clusters, nodes and environments programmatically. Then you may create a new input file or change an existing one and run it in simulation mode:
- g5kadmin@host:/somewhere/reference-repository$ ruby generators/grid5000.rb generators/input/*.rb -s
+Right now, the easiest way to get started is to look at some existing input files in the "generators/input" directory. There you can see how you can define sites, clusters, nodes and environments programmatically.
+Then you may create a new input file or change an existing one and run it in simulation mode:
+
+ g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/*.rb -s
+
or, if you want to explicitly specify the input files:
- g5kadmin@host:/somewhere/reference-repository$ ruby generators/grid5000.rb generators/input/input-file1.rb generators/input/input-file2.rb -s
+
+ g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/input-file1.rb generators/input/input-file2.rb -s
+
+For more information about the available options and usage of the <code>grid5000</code> generator, run:
+
+ g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 --help
+
Your changes won't be applied but you'll see what would have been changed. Thus, the simulation mode is useful to review your changes before committing and check the ruby syntax of the input files.
When you are happy with your changes, you can then run the command without the -s flag:
- g5kadmin@host:/somewhere/reference-repository$ ruby generators/grid5000.rb generators/input/*.rb
-Finally, commit your changes with a meaningful message and push them immediately to the MASTER REPOSITORY.
+ g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/*.rb
+
+Please be aware that config files (YAML format) may be passed on the command line, so that the values can be used in the input files via the <code>lookup(config_filename, key)</code> function. To tell the generator to include one or more config files, you must pass them in your command arguments:
+
+ g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/*.rb generators/input/*.yaml
+
+Note that the extension of config files MUST be <code>.yaml</code> or <code>.yml</code>, otherwise they won't be included.
+
+Finally, commit your changes with a meaningful message (you SHOULD first review the changes that will be committed by running the <code>git diff</code> command) and push them immediately to the MASTER REPOSITORY.
== Resources ==
* [http://cheat.errtheblog.com/s/git Git Cheat Sheet]
@@ -61,9 +84,12 @@ At runtime, the generator resolves the hostname of each node to obtain its IP ad
After each modification to the repository, you should immediately commit your changes with a meaningful message, so that people can easily understand what has changed (latest changes will be displayed in a syndication feed). Your commits should also be site-specific, or even cluster-specific to avoid merge conflicts. Try to avoid putting a lot of changes in only one commit.
You should also check that your name and email are correctly configured in your Git configuration:
+
$ git config --get user.name
$ git config --get user.email
+
Otherwise you can set them by issuing:
+
$ git config --global user.name 'John Doe'
$ git config --global user.email johndoe@example.com
diff --git a/generators/grid5000 b/generators/grid5000
new file mode 100755
index 00000000000..67caef0ff01
--- /dev/null
+++ b/generators/grid5000
@@ -0,0 +1,70 @@
+#!/usr/bin/env ruby
+
+require 'pp'
+require 'rubygems'
+require 'fileutils'
+require 'json/pure'
+require 'yaml'
+require 'time'
+require 'optparse'
+require File.dirname(__FILE__)+'/lib/core_extensions'
+require File.dirname(__FILE__)+'/lib/g5k_generator'
+
+options = {:simulate => false}
+option_parser = OptionParser.new do |opts|
+ opts.banner = %{
+ A tool to generate the Grid5000 reference data.
+
+ Usage:
+ ./grid5000 input_files [config_files] [options]
+
+ Notes:
+ * input_files MUST be .rb files.
+ * config_files MUST be .yml or .yaml files.
+
+ Examples:
+ ./grid5000 input/*.rb input/*.yaml -s
+ ./grid5000 input/nancy.rb input/nancy.yaml
+ ./grid5000 input/rennes.rb input/lille.rb -s
+
+ Options:}
+ opts.on("-s", "--simulate", "Run in simulation mode (do not write changes). Default to false.") do |v|
+ options[:simulate] = v
+ end
+ opts.on_tail("-h", "--help", "Show this message") do
+ puts opts
+ exit
+ end
+
+end
+
+option_parser.parse!
+files = ARGV
+if files.empty?
+ $stderr.puts option_parser.help
+ exit 1
+elsif (files.map{|file| File.exists?(file)}.include? false)
+ $stderr.puts "Error: one of your input or config file does not exist."
+ exit 2
+else
+ input = {}
+ config = {}
+ files.each do |filename|
+ case (ext = File.extname(filename))
+ when ".rb"
+ input[File.basename(filename, ext)] = File.read(filename)
+ when ".yml", ".yaml"
+ config[File.basename(filename, ext)] = YAML.load_file(filename)
+ else
+ raise ArgumentError, "The file #{filename} is neither an input file nor a config file."
+ end
+ end
+ puts "[Input files:\t\t #{input.keys.join(", ")}]"
+ puts "[Config files:\t\t #{config.keys.join(", ")}]"
+ puts "[Simulation mode:\t #{options[:simulate]}]"
+ generator = G5K::ReferenceGenerator.new({:uid => "grid5000", :type => "grid"}, :input => input, :config => config)
+ data = generator.generate
+ directory_to_write = File.expand_path File.join(File.dirname(__FILE__), "../data")
+ generator.write(directory_to_write, options)
+ exit 0
+end
diff --git a/generators/grid5000.rb b/generators/grid5000.rb
deleted file mode 100755
index 27567a721ec..00000000000
--- a/generators/grid5000.rb
+++ /dev/null
@@ -1,49 +0,0 @@
-require 'pp'
-require 'rubygems'
-require 'fileutils'
-require 'json/pure'
-require 'yaml'
-require 'time'
-require File.dirname(__FILE__)+'/lib/core_extensions'
-require File.dirname(__FILE__)+'/lib/g5k_generator'
-
-usage = %{
- A tool to generate the Grid5000 reference data.
-
- Usage:
- ruby grid5000.rb [input files] [config files] [options]
- Options:
- -s : simulation mode
- Notes:
- input files must be RUBY files and end with a .rb extension.
- config files mst be YAML files and end with a .yaml extension.
-}
-
-# true if we want to simulate:
-simulation_mode = !$*.delete("-s").nil?
-
-if $*.empty?
- puts usage
- exit 1
-elsif ($*.map{|file| File.exists?(file)}.include? false)
- puts "Error: one of your input file do not exist."
- exit 2
-else
- description_files = $*
- input = {}
- config = {}
- description_files.each do |filename|
- case File.extname(filename)
- when ".rb" then input[File.basename(filename, ".rb")] = File.read(filename)
- when ".yaml" then config[File.basename(filename, ".yaml")] = YAML.load_file(filename)
- end
- end
- puts "[Input files:\t\t #{input.keys.join(", ")}]"
- puts "[Config files:\t\t #{config.keys.join(", ")}]"
- puts "[Simulation mode:\t #{simulation_mode}]"
- generator = G5K::ReferenceGenerator.new({:uid => "grid5000", :type => "grid"}, :input => input, :config => config)
- data = generator.generate
- directory_to_write = File.expand_path File.join(File.dirname(__FILE__), "../data")
- generator.write(directory_to_write, :simulate => simulation_mode)
- exit 0
-end
diff --git a/generators/lib/g5k_generator.rb b/generators/lib/g5k_generator.rb
index b20e47d7f6e..835b9f02209 100755
--- a/generators/lib/g5k_generator.rb
+++ b/generators/lib/g5k_generator.rb
@@ -23,7 +23,7 @@ module G5K
def valid?
true
- # we could do sthg like: Load(type.class).new(hash_values).valid?
+ # we could do sthg like: Object.const_get(type.class.name).new(hash_values).valid?
end
end
class Folder < Array
@@ -51,163 +51,162 @@ module G5K
end
end
-class ReferenceGenerator
- attr_reader :data
- attr_reader :config
- attr_reader :input
+ class ReferenceGenerator
+ attr_reader :data
+ attr_reader :config
+ attr_reader :input
- def method_missing(method, *args)
- @context.recursive_merge!(method.to_sym => args.first)
- end
+ def method_missing(method, *args)
+ @context.recursive_merge!(method.to_sym => args.first)
+ end
- def dns_lookup(network_address)
- Resolv.getaddress(network_address)
- end
+ def dns_lookup(network_address)
+ Resolv.getaddress(network_address)
+ end
- # Lookup a key in one of the configuration files passed to the generator
- #
- # Usage:
- # lookup('nancy', 'nodes', 'paramount-1', 'property_name')
- # or
- # lookup('nancy', 'nodes') { |result| result['paramount-1']['property_name'] }
- # or
- # lookup('nancy') { |result| result['nodes']['paramount-1']['property_name'] }
- #
- # assuming you passed a <tt>nancy.yaml</tt> file to the generator
- # Be careful with null values!
- #
- def lookup(filename, *keys, &block)
- if config.has_key?(filename)
- result = config[filename]
- if !keys.empty?
- while !keys.empty? do
- result = result[keys.shift]
- break if result.nil?
+ # Lookup a key in one of the configuration files passed to the generator
+ #
+ # Usage:
+ # lookup('nancy', 'nodes', 'paramount-1', 'property_name')
+ # or
+ # lookup('nancy', 'nodes') { |result| result['paramount-1']['property_name'] }
+ # or
+ # lookup('nancy') { |result| result['nodes']['paramount-1']['property_name'] }
+ #
+ # assuming you passed a <tt>nancy.yaml</tt> file to the generator
+ #
+ def lookup(filename, *keys, &block)
+ if config.has_key?(filename)
+ result = config[filename]
+ if !keys.empty?
+ while !keys.empty? do
+ result = result[keys.shift]
+ break if result.nil?
+ end
+ end
+ if block
+ block.call(result)
+ else
+ result
end
- end
- if block
- block.call(result)
else
- result
+ raise ArgumentError, "Cannot fetch the values for '#{keys.inspect}' in the input file '#{filename}'. The config files you gave to me are: '#{config.keys.inspect}'."
end
- else
- nil
end
- end
- # This doesn't work with Ruby < 1.8.7. Replaced by a call to build_context (see below).
- #
- # %w{site cluster environment node service}.each do |method|
- # define_method(method) do |uid, *options, &block|
- # key = method.pluralize.to_sym
- # uid = uid.to_s
- # options = options.first || Hash.new
- # old_context = @context
- # @context[key] ||= G5K::Folder.new
- # if options.has_key? :refer_to
- # @context[key] << G5K::Link.new(uid, options[:refer_to])
- # else
- # # if the same object already exists, we return it for completion/modification
- # if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
- # @context = same_trees.first
- # else
- # @context[key] << G5K::Tree.new.replace({:uid => uid, :type => method})
- # @context = @context[key].last
- # end
- # block.call(uid) if block
- # end
- # @context = old_context
- # end
- # end
+ # This doesn't work with Ruby < 1.8.7. Replaced by a call to build_context (see below).
+ #
+ # %w{site cluster environment node service}.each do |method|
+ # define_method(method) do |uid, *options, &block|
+ # key = method.pluralize.to_sym
+ # uid = uid.to_s
+ # options = options.first || Hash.new
+ # old_context = @context
+ # @context[key] ||= G5K::Folder.new
+ # if options.has_key? :refer_to
+ # @context[key] << G5K::Link.new(uid, options[:refer_to])
+ # else
+ # # if the same object already exists, we return it for completion/modification
+ # if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
+ # @context = same_trees.first
+ # else
+ # @context[key] << G5K::Tree.new.replace({:uid => uid, :type => method})
+ # @context = @context[key].last
+ # end
+ # block.call(uid) if block
+ # end
+ # @context = old_context
+ # end
+ # end
- def site(uid, *options, &block)
- build_context(:sites, uid, *options, &block)
- end
- def cluster(uid, *options, &block)
- build_context(:clusters, uid, *options, &block)
- end
- def server(uid, *options, &block)
- build_context(:servers, uid, *options, &block)
- end
- def environment(uid, *options, &block)
- build_context(:environments, uid, *options, &block)
- end
- def node(uid, *options, &block)
- build_context(:nodes, uid, *options, &block)
- end
- def service(uid, *options, &block)
- build_context(:services, uid, *options, &block)
- end
- def build_context(key, uid, *options, &block)
- type = key.to_s.chop
- uid = uid.to_s
- options = options.first || Hash.new
- old_context = @context
- @context[key] ||= G5K::Folder.new
- if options.has_key? :refer_to
- @context[key] << G5K::Link.new(uid, options[:refer_to])
- else
- # if the same object already exists, we return it for completion/modification
- if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
- @context = same_trees.first
- else
- @context[key] << G5K::Tree.new.replace({:uid => uid, :type => type})
- @context = @context[key].last
+ def site(uid, *options, &block)
+ build_context(:sites, uid, *options, &block)
+ end
+ def cluster(uid, *options, &block)
+ build_context(:clusters, uid, *options, &block)
+ end
+ def server(uid, *options, &block)
+ build_context(:servers, uid, *options, &block)
+ end
+ def environment(uid, *options, &block)
+ build_context(:environments, uid, *options, &block)
+ end
+ def node(uid, *options, &block)
+ build_context(:nodes, uid, *options, &block)
+ end
+ def service(uid, *options, &block)
+ build_context(:services, uid, *options, &block)
+ end
+ def build_context(key, uid, *options, &block)
+ type = key.to_s.chop
+ uid = uid.to_s
+ options = options.first || Hash.new
+ old_context = @context
+ @context[key] ||= G5K::Folder.new
+ if options.has_key? :refer_to
+ @context[key] << G5K::Link.new(uid, options[:refer_to])
+ else
+ # if the same object already exists, we return it for completion/modification
+ if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
+ @context = same_trees.first
+ else
+ @context[key] << G5K::Tree.new.replace({:uid => uid, :type => type})
+ @context = @context[key].last
+ end
+ block.call(uid) if block
end
- block.call(uid) if block
+ @context = old_context
end
- @context = old_context
- end
- # Initializes a new generator that will generate data files in a hierachical way.
- # The root of the tree will be named with the value of <tt>data_description[:uid]</tt>.
- def initialize(data_description = {:uid => ""}, options = {:input => {}, :config => {}})
- @input = options[:input]
- raise(ArgumentError, "INPUT cannot be null or empty.") if input.nil? || input.empty?
- @config = options[:config] || {}
- @data = G5K::Tree.new.replace(data_description)
- @context = @data
- end
-
- def generate
- input.each do |filename, content|
- eval(content)
+ # Initializes a new generator that will generate data files in a hierachical way.
+ # The root of the tree will be named with the value of <tt>data_description[:uid]</tt>.
+ def initialize(data_description = {:uid => ""}, options = {:input => {}, :config => {}})
+ @input = options[:input]
+ raise(ArgumentError, "INPUT cannot be null or empty.") if input.nil? || input.empty?
+ @config = options[:config] || {}
+ @data = G5K::Tree.new.replace(data_description)
+ @context = @data
end
- @data
- end
- def write(repository, options = {:simulate => false})
- things_to_create = []
- @data.write("/") do |thing_to_create|
- things_to_create << thing_to_create if thing_to_create.valid?
- end
- groups = things_to_create.group_by{|thing| thing.class}
- groups.has_key?(G5K::Folder) and groups[G5K::Folder].each do |folder|
- full_path = File.join(repository, folder.path)
- unless File.exists?(full_path)
- puts "Directory to be written = \t#{full_path}"
- FileUtils.mkdir_p(full_path) unless options[:simulate]
+ def generate
+ input.each do |filename, content|
+ eval(content)
end
+ @data
end
- groups.has_key?(G5K::Tree) and groups[G5K::Tree].each do |file|
- full_path = File.join(repository, file.path)
- new_content = JSON.pretty_generate(file.contents)
- existing_content = File.exists?(full_path) ? File.open(full_path, "r").read : ""
- if new_content.hash != existing_content.hash
- puts "File to be written = \t#{full_path}"
- File.open(full_path, "w+"){ |f| f << new_content } unless options[:simulate]
+
+ def write(repository, options = {:simulate => false})
+ things_to_create = []
+ @data.write("/") do |thing_to_create|
+ things_to_create << thing_to_create if thing_to_create.valid?
end
- end
- groups.has_key?(G5K::Link) and groups[G5K::Link].each do |link|
- FileUtils.cd(repository) do |dir|
- to = File.join(".", link.path)
- from = File.join([".."]*(to.count(File::SEPARATOR)-1), "#{link.from}.json")
- unless File.exists?(to)
- puts "Symbolic link to be written = \t#{to} -> #{from}"
- FileUtils.ln_s(from, to, :force => true) unless options[:simulate]
+ groups = things_to_create.group_by{|thing| thing.class}
+ groups.has_key?(G5K::Folder) and groups[G5K::Folder].each do |folder|
+ full_path = File.join(repository, folder.path)
+ unless File.exists?(full_path)
+ puts "Directory to be written = \t#{full_path}"
+ FileUtils.mkdir_p(full_path) unless options[:simulate]
end
end
- end
- end
+ groups.has_key?(G5K::Tree) and groups[G5K::Tree].each do |file|
+ full_path = File.join(repository, file.path)
+ new_content = JSON.pretty_generate(file.contents)
+ existing_content = File.exists?(full_path) ? File.open(full_path, "r").read : ""
+ if new_content.hash != existing_content.hash
+ puts "File to be written = \t#{full_path}"
+ File.open(full_path, "w+"){ |f| f << new_content } unless options[:simulate]
+ end
+ end
+ groups.has_key?(G5K::Link) and groups[G5K::Link].each do |link|
+ FileUtils.cd(repository) do |dir|
+ to = File.join(".", link.path)
+ from = File.join([".."]*(to.count(File::SEPARATOR)-1), "#{link.from}.json")
+ unless File.exists?(to)
+ puts "Symbolic link to be written = \t#{to} -> #{from}"
+ FileUtils.ln_s(from, to, :force => true) unless options[:simulate]
+ end
+ end
+ end
+ end
-end
+ end
end
\ No newline at end of file
--
GitLab