[admin] updated documentation and generator code, so that it checks that the...

[admin] updated documentation and generator code, so that it checks that the required config files are passed as arguments.

README → README.wiki

 {{Maintainer|Cyril Rohr}}
+{{Author|Cyril Rohr}}
 {{Status|Draft}}
 {{Portal|Admin}}
 == Synopsis ==
-The reference data is stored in a Git repository as JSON files, organized into hierarchical folders. These files can be manually written but the Git repository comes with a "/generators" folder which contains a script to ease their generation, based on high-level description files written in Ruby. Given one or more input files that describe the data you want to add, it will generate the required JSON files, directories and hard links.
+The reference data is stored in a Git repository as JSON files, organized into hierarchical folders. These files can be manually written but the Git repository comes with a "/generators" folder which contains a script to ease their generation, based on high-level description files written in Ruby. Given one or more input files that describe the data you want to add, it will generate the required JSON files, directories and sym links.
 
 == Requirements ==
 * Ruby >= 1.8.6
@@ -23,10 +24,15 @@ The general overview of the workflow between git repositories is as follows:
                    API REPOSITORY SITE X <--|     |--> API REPOSITORY SITE Y
 
 Each site administrator must first clone the remote MASTER REPOSITORY located on the git.grid5000.fr server and store it on a local machine (this is what I call the ADMIN REPOSITORY and has to be done once):
+
   g5kadmin@host:/somewhere$ git clone ssh://g5kadmin@git.grid5000.fr/srv/git/repos/reference-repository.git
+
 When there is a need for change, the site administrator PULLs from the MASTER REPOSITORY to get the latest changes:
+
   g5kadmin@host:/somewhere/reference-repository$ git pull
+
 Then she manually adds/edits/removes the raw JSON files or uses the generator (more on that later). When she's done, she COMMITs her changes and PUSHes them to the MASTER REPOSITORY:
+
   ... editing ...
   g5kadmin@host:/somewhere/reference-repository$ git commit -a -m "list of modifications"
   g5kadmin@host:/somewhere/reference-repository$ git push
@@ -37,18 +43,35 @@ Finally, these changes are automatically replicated every minute to each API REP
 
 == Getting started ==
 First, clone the remote MASTER REPOSITORY if it is not already done:
+
   g5kadmin@host:/somewhere/reference-repository$ git clone ssh://g5kadmin@git.grid5000.fr/srv/git/repos/reference-repository.git
 
-Right now, the easiest way to get started is to look at some existing input files in the "generators/input" directory. There you can see how you can define sites, clusters, nodes and environments programmatically. Then you may create a new input file or change an existing one and run it in simulation mode:
-  g5kadmin@host:/somewhere/reference-repository$ ruby generators/grid5000.rb generators/input/*.rb -s
+Right now, the easiest way to get started is to look at some existing input files in the "generators/input" directory. There you can see how you can define sites, clusters, nodes and environments programmatically. 
+Then you may create a new input file or change an existing one and run it in simulation mode:
+
+  g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/*.rb -s
+
 or, if you want to explicitly specify the input files:
-  g5kadmin@host:/somewhere/reference-repository$ ruby generators/grid5000.rb generators/input/input-file1.rb generators/input/input-file2.rb -s
+
+  g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/input-file1.rb generators/input/input-file2.rb -s
+
+For more information about the available options and usage of the <code>grid5000</code> generator, run:
+
+  g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 --help
+
 Your changes won't be applied but you'll see what would have been changed. Thus, the simulation mode is useful to review your changes before committing and check the ruby syntax of the input files.
 
 When you are happy with your changes, you can then run the command without the -s flag:
-  g5kadmin@host:/somewhere/reference-repository$ ruby generators/grid5000.rb generators/input/*.rb
 
-Finally, commit your changes with a meaningful message and push them immediately to the MASTER REPOSITORY.
+  g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/*.rb
+
+Please be aware that config files (YAML format) may be passed on the command line, so that the values can be used in the input files via the <code>lookup(config_filename, key)</code> function. To tell the generator to include one or more config files, you must pass them in your command arguments:
+
+  g5kadmin@host:/somewhere/reference-repository$ ./generators/grid5000 generators/input/*.rb generators/input/*.yaml
+
+Note that the extension of config files MUST be <code>.yaml</code> or <code>.yml</code>, otherwise they won't be included.
+
+Finally, commit your changes with a meaningful message (you SHOULD first review the changes that will be committed by running the <code>git diff</code> command) and push them immediately to the MASTER REPOSITORY.
 
 == Resources ==
 * [http://cheat.errtheblog.com/s/git Git Cheat Sheet]
@@ -61,9 +84,12 @@ At runtime, the generator resolves the hostname of each node to obtain its IP ad
 After each modification to the repository, you should immediately commit your changes with a meaningful message, so that people can easily understand what has changed (latest changes will be displayed in a syndication feed). Your commits should also be site-specific, or even cluster-specific to avoid merge conflicts. Try to avoid putting a lot of changes in only one commit.
 
 You should also check that your name and email are correctly configured in your Git configuration:
+
   $ git config --get user.name
   $ git config --get user.email
+
 Otherwise you can set them by issuing:
+
   $ git config --global user.name 'John Doe'
   $ git config --global user.email johndoe@example.com
 

generators/grid5000.rb → generators/grid5000

+#!/usr/bin/env ruby
+
 require 'pp'
 require 'rubygems'
 require 'fileutils'
 require 'json/pure'
 require 'yaml'
 require 'time'
+require 'optparse'
 require File.dirname(__FILE__)+'/lib/core_extensions'
 require File.dirname(__FILE__)+'/lib/g5k_generator'
 
-usage = %{ 
+options = {:simulate => false}
+option_parser = OptionParser.new do |opts|
+  opts.banner = %{
   A tool to generate the Grid5000 reference data.
   
   Usage:
-    ruby grid5000.rb [input files] [config files] [options]
-  Options:
-    -s : simulation mode
+    ./grid5000 input_files [config_files] [options]
+  
   Notes:
-    input files must be RUBY files and end with a .rb extension.
-    config files mst be YAML files and end with a .yaml extension.
-}
-
-# true if we want to simulate:
-simulation_mode = !$*.delete("-s").nil?
+    * input_files   MUST be .rb files.
+    * config_files  MUST be .yml or .yaml files.
+  
+  Examples:
+    ./grid5000 input/*.rb input/*.yaml -s
+    ./grid5000 input/nancy.rb input/nancy.yaml
+    ./grid5000 input/rennes.rb input/lille.rb -s
+    
+  Options:}
+  opts.on("-s", "--simulate", "Run in simulation mode (do not write changes). Default to false.") do |v|
+    options[:simulate] = v
+  end
+  opts.on_tail("-h", "--help", "Show this message") do
+    puts opts
+    exit
+  end
+  
+end
 
-if $*.empty?
-  puts usage
+option_parser.parse!
+files = ARGV
+if files.empty?
+  $stderr.puts option_parser.help
   exit 1
-elsif ($*.map{|file| File.exists?(file)}.include? false)
-  puts "Error: one of your input file do not exist."
+elsif (files.map{|file| File.exists?(file)}.include? false)
+  $stderr.puts "Error: one of your input or config file does not exist."
   exit 2
 else
-  description_files = $*
   input = {}
   config = {}
-  description_files.each do |filename|
-    case File.extname(filename)
-    when ".rb"    then input[File.basename(filename, ".rb")] = File.read(filename)
-    when ".yaml"  then config[File.basename(filename, ".yaml")] = YAML.load_file(filename)
+  files.each do |filename|
+    case (ext = File.extname(filename))
+    when ".rb"
+      input[File.basename(filename, ext)] = File.read(filename)
+    when ".yml", ".yaml"
+      config[File.basename(filename, ext)] = YAML.load_file(filename)
+    else
+      raise ArgumentError, "The file #{filename} is neither an input file nor a config file."
     end
   end
   puts "[Input files:\t\t #{input.keys.join(", ")}]"
   puts "[Config files:\t\t #{config.keys.join(", ")}]"
-  puts "[Simulation mode:\t #{simulation_mode}]"
+  puts "[Simulation mode:\t #{options[:simulate]}]"
   generator = G5K::ReferenceGenerator.new({:uid => "grid5000", :type => "grid"}, :input => input, :config => config)
   data = generator.generate
   directory_to_write = File.expand_path File.join(File.dirname(__FILE__), "../data")
-  generator.write(directory_to_write, :simulate => simulation_mode)
+  generator.write(directory_to_write, options)
   exit 0
 end

generators/lib/g5k_generator.rb

@@ -23,7 +23,7 @@ module G5K
     
     def valid?
       true
-      # we could do sthg like: Load(type.class).new(hash_values).valid?
+      # we could do sthg like: Object.const_get(type.class.name).new(hash_values).valid?
     end
   end
   class Folder < Array
@@ -51,163 +51,162 @@ module G5K
     end
   end
 
-class ReferenceGenerator  
-  attr_reader :data
-  attr_reader :config
-  attr_reader :input
+  class ReferenceGenerator  
+    attr_reader :data
+    attr_reader :config
+    attr_reader :input
   
-  def method_missing(method, *args)
-    @context.recursive_merge!(method.to_sym => args.first)
-  end
+    def method_missing(method, *args)
+      @context.recursive_merge!(method.to_sym => args.first)
+    end
   
-  def dns_lookup(network_address)
-    Resolv.getaddress(network_address)
-  end
+    def dns_lookup(network_address)
+      Resolv.getaddress(network_address)
+    end
   
-  # Lookup a key in one of the configuration files passed to the generator
-  #
-  # Usage:
-  #   lookup('nancy', 'nodes', 'paramount-1', 'property_name')
-  # or
-  #   lookup('nancy', 'nodes') { |result| result['paramount-1']['property_name'] }
-  # or
-  #   lookup('nancy') { |result| result['nodes']['paramount-1']['property_name'] }
-  # 
-  # assuming you passed a <tt>nancy.yaml</tt> file to the generator
-  # Be careful with null values!
-  #
-  def lookup(filename, *keys, &block)
-    if config.has_key?(filename)
-      result = config[filename]
-      if !keys.empty?
-        while !keys.empty? do
-          result = result[keys.shift]
-          break if result.nil?
+    # Lookup a key in one of the configuration files passed to the generator
+    #
+    # Usage:
+    #   lookup('nancy', 'nodes', 'paramount-1', 'property_name')
+    # or
+    #   lookup('nancy', 'nodes') { |result| result['paramount-1']['property_name'] }
+    # or
+    #   lookup('nancy') { |result| result['nodes']['paramount-1']['property_name'] }
+    # 
+    # assuming you passed a <tt>nancy.yaml</tt> file to the generator
+    #
+    def lookup(filename, *keys, &block)
+      if config.has_key?(filename)
+        result = config[filename]
+        if !keys.empty?
+          while !keys.empty? do
+            result = result[keys.shift]
+            break if result.nil?
+          end
+        end  
+        if block
+          block.call(result) 
+        else
+          result
         end
-      end  
-      if block
-        block.call(result) 
       else
-        result
+        raise ArgumentError, "Cannot fetch the values for '#{keys.inspect}' in the input file '#{filename}'. The config files you gave to me are: '#{config.keys.inspect}'."
       end
-    else
-      nil
     end
-  end
-  # This doesn't work with Ruby < 1.8.7. Replaced by a call to build_context (see below).
-  #
-  # %w{site cluster environment node service}.each do |method|
-  #   define_method(method) do |uid, *options, &block|
-  #     key = method.pluralize.to_sym
-  #     uid = uid.to_s
-  #     options = options.first || Hash.new
-  #     old_context = @context
-  #     @context[key] ||= G5K::Folder.new
-  #     if options.has_key? :refer_to
-  #       @context[key] << G5K::Link.new(uid, options[:refer_to])
-  #     else    
-  #       # if the same object already exists, we return it for completion/modification
-  #       if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
-  #         @context = same_trees.first
-  #       else
-  #         @context[key] << G5K::Tree.new.replace({:uid => uid, :type => method})
-  #         @context = @context[key].last
-  #       end
-  #       block.call(uid) if block
-  #     end
-  #     @context = old_context
-  #   end
-  # end
+    # This doesn't work with Ruby < 1.8.7. Replaced by a call to build_context (see below).
+    #
+    # %w{site cluster environment node service}.each do |method|
+    #   define_method(method) do |uid, *options, &block|
+    #     key = method.pluralize.to_sym
+    #     uid = uid.to_s
+    #     options = options.first || Hash.new
+    #     old_context = @context
+    #     @context[key] ||= G5K::Folder.new
+    #     if options.has_key? :refer_to
+    #       @context[key] << G5K::Link.new(uid, options[:refer_to])
+    #     else    
+    #       # if the same object already exists, we return it for completion/modification
+    #       if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
+    #         @context = same_trees.first
+    #       else
+    #         @context[key] << G5K::Tree.new.replace({:uid => uid, :type => method})
+    #         @context = @context[key].last
+    #       end
+    #       block.call(uid) if block
+    #     end
+    #     @context = old_context
+    #   end
+    # end
   
-  def site(uid, *options, &block)
-    build_context(:sites, uid, *options, &block)
-  end
-  def cluster(uid, *options, &block)
-    build_context(:clusters, uid, *options, &block)
-  end
-  def server(uid, *options, &block)
-    build_context(:servers, uid, *options, &block)
-  end
-  def environment(uid, *options, &block)
-    build_context(:environments, uid, *options, &block)
-  end
-  def node(uid, *options, &block)
-    build_context(:nodes, uid, *options, &block)
-  end
-  def service(uid, *options, &block)
-    build_context(:services, uid, *options, &block)
-  end
-  def build_context(key, uid, *options, &block)
-    type = key.to_s.chop
-    uid = uid.to_s
-    options = options.first || Hash.new
-    old_context = @context
-    @context[key] ||= G5K::Folder.new
-    if options.has_key? :refer_to
-      @context[key] << G5K::Link.new(uid, options[:refer_to])
-    else    
-      # if the same object already exists, we return it for completion/modification
-      if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
-        @context = same_trees.first
-      else
-        @context[key] << G5K::Tree.new.replace({:uid => uid, :type => type})
-        @context = @context[key].last
+    def site(uid, *options, &block)
+      build_context(:sites, uid, *options, &block)
+    end
+    def cluster(uid, *options, &block)
+      build_context(:clusters, uid, *options, &block)
+    end
+    def server(uid, *options, &block)
+      build_context(:servers, uid, *options, &block)
+    end
+    def environment(uid, *options, &block)
+      build_context(:environments, uid, *options, &block)
+    end
+    def node(uid, *options, &block)
+      build_context(:nodes, uid, *options, &block)
+    end
+    def service(uid, *options, &block)
+      build_context(:services, uid, *options, &block)
+    end
+    def build_context(key, uid, *options, &block)
+      type = key.to_s.chop
+      uid = uid.to_s
+      options = options.first || Hash.new
+      old_context = @context
+      @context[key] ||= G5K::Folder.new
+      if options.has_key? :refer_to
+        @context[key] << G5K::Link.new(uid, options[:refer_to])
+      else    
+        # if the same object already exists, we return it for completion/modification
+        if (same_trees = @context[key].select{|tree| tree[:uid] == uid}).size > 0
+          @context = same_trees.first
+        else
+          @context[key] << G5K::Tree.new.replace({:uid => uid, :type => type})
+          @context = @context[key].last
+        end
+        block.call(uid) if block
       end
-      block.call(uid) if block
+      @context = old_context
     end
-    @context = old_context
-  end
   
-  # Initializes a new generator that will generate data files in a hierachical way. 
-  # The root of the tree will be named with the value of <tt>data_description[:uid]</tt>.
-  def initialize(data_description = {:uid => ""}, options = {:input => {}, :config => {}})
-    @input = options[:input]
-    raise(ArgumentError, "INPUT cannot be null or empty.") if input.nil? || input.empty?
-    @config = options[:config] || {}
-    @data = G5K::Tree.new.replace(data_description)
-    @context = @data
-  end
-  
-  def generate
-    input.each do |filename, content|
-      eval(content)
+    # Initializes a new generator that will generate data files in a hierachical way. 
+    # The root of the tree will be named with the value of <tt>data_description[:uid]</tt>.
+    def initialize(data_description = {:uid => ""}, options = {:input => {}, :config => {}})
+      @input = options[:input]
+      raise(ArgumentError, "INPUT cannot be null or empty.") if input.nil? || input.empty?
+      @config = options[:config] || {}
+      @data = G5K::Tree.new.replace(data_description)
+      @context = @data
     end
-    @data
-  end
   
-  def write(repository, options = {:simulate => false})
-    things_to_create = []
-    @data.write("/") do |thing_to_create|
-      things_to_create << thing_to_create if thing_to_create.valid?
-    end
-    groups = things_to_create.group_by{|thing| thing.class}
-    groups.has_key?(G5K::Folder) and groups[G5K::Folder].each do |folder|
-      full_path = File.join(repository, folder.path)
-      unless File.exists?(full_path)
-        puts "Directory to be written = \t#{full_path}"
-        FileUtils.mkdir_p(full_path) unless options[:simulate]
+    def generate
+      input.each do |filename, content|
+        eval(content)
       end
+      @data
     end
-    groups.has_key?(G5K::Tree) and groups[G5K::Tree].each do |file|
-      full_path = File.join(repository, file.path)
-      new_content = JSON.pretty_generate(file.contents)
-      existing_content = File.exists?(full_path) ? File.open(full_path, "r").read : ""
-      if new_content.hash != existing_content.hash
-        puts "File to be written      = \t#{full_path}"
-        File.open(full_path, "w+"){ |f| f << new_content  } unless options[:simulate]
+  
+    def write(repository, options = {:simulate => false})
+      things_to_create = []
+      @data.write("/") do |thing_to_create|
+        things_to_create << thing_to_create if thing_to_create.valid?
       end
-    end
-    groups.has_key?(G5K::Link) and groups[G5K::Link].each do |link|      
-      FileUtils.cd(repository) do |dir|
-        to = File.join(".", link.path)
-        from = File.join([".."]*(to.count(File::SEPARATOR)-1), "#{link.from}.json")
-        unless File.exists?(to)
-          puts "Symbolic link to be written = \t#{to} -> #{from}"
-          FileUtils.ln_s(from, to, :force => true) unless options[:simulate]
+      groups = things_to_create.group_by{|thing| thing.class}
+      groups.has_key?(G5K::Folder) and groups[G5K::Folder].each do |folder|
+        full_path = File.join(repository, folder.path)
+        unless File.exists?(full_path)
+          puts "Directory to be written = \t#{full_path}"
+          FileUtils.mkdir_p(full_path) unless options[:simulate]
         end
       end
-    end  
-  end
+      groups.has_key?(G5K::Tree) and groups[G5K::Tree].each do |file|
+        full_path = File.join(repository, file.path)
+        new_content = JSON.pretty_generate(file.contents)
+        existing_content = File.exists?(full_path) ? File.open(full_path, "r").read : ""
+        if new_content.hash != existing_content.hash
+          puts "File to be written      = \t#{full_path}"
+          File.open(full_path, "w+"){ |f| f << new_content  } unless options[:simulate]
+        end
+      end
+      groups.has_key?(G5K::Link) and groups[G5K::Link].each do |link|      
+        FileUtils.cd(repository) do |dir|
+          to = File.join(".", link.path)
+          from = File.join([".."]*(to.count(File::SEPARATOR)-1), "#{link.from}.json")
+          unless File.exists?(to)
+            puts "Symbolic link to be written = \t#{to} -> #{from}"
+            FileUtils.ln_s(from, to, :force => true) unless options[:simulate]
+          end
+        end
+      end  
+    end
   
-end
+  end
 end
\ No newline at end of file