Update packaging to converge with users-api-ror

.gitlab-ci.yml

@@ -77,7 +77,7 @@ deb-for-stretch:
 
 .push-packages-script: &push-packages-script |
   echo "Request to push for debian version ${TARGET_DEBIAN_VERSION} for branch ${TARGET_G5K_API_VARIANT}"
-  g5k-deploy-files --only-if-tag '^\d+\.\d+\.\d+' --files 'pkg/g5k-api_*.deb' --directory deb/g5k-api/${TARGET_DEBIAN_VERSION}/${TARGET_G5K_API_VARIANT}
+  g5k-deploy-files --only-if-tag '^v\d+\.\d+\.\d+' --files 'pkg/g5k-api_*.deb' --directory deb/g5k-api/${TARGET_DEBIAN_VERSION}/${TARGET_G5K_API_VARIANT}
 
 push-jessie-package-v3: &push-jessie-package-v3
   stage: deploy

README.md

@@ -11,10 +11,23 @@ but if you read this, it's normally good…
 
 The app is packaged for Debian jessie and Debian stretch. Therefore installation is as follows:
 
-    sudo apt-get update
-    sudo apt-get install g5k-api
+        $ sudo apt-get update
+        $ sudo apt-get install g5k-api
 
-In particular, runtime dependencies of the app include `ruby2.1.5` and `git-core`.
+Most configuration file for the application are those in `/opt/g5k-api/config`. They are linked from `/etc/g5k-api`, but should the link be broken, the application will use the version `/opt/g5k-api/config`.
+
+If the database is not configured when the package is installed, you'll need to run migration after it has been setup
+
+       $ sudo g5k-api rake db:migrate
+
+The service's execution environment is built by the `/usr/bin/g5k-api` script, that will source any `.sh` file in `/etc/g5k-api/conf.d/` before calling `/opt/ug5k-api/bin/g5k-api`. This is how bundled gems are setup before execution, and `SECRET_KEY_BASE` is set for production environment.
+
+You can check the installation by running a few commands to gather information about the application's execution environment
+
+        $ sudo g5k-api rake db:migrate:status
+        $ sudo g5k-api rake about
+        $ sudo g5k-api console #this will drop you in the rails console
+        $ sudo g5k-api rake dbconsole #this will drop you in the database's console
 
 ## Development
 
@@ -129,13 +142,13 @@ In particular, runtime dependencies of the app include `ruby2.1.5` and `git-core
 
 * If you want to develop on the UI, using the apache proxy, run your browser on 
         
-				$ firefox http://127.0.0.1:8080/ui
+        $ firefox http://127.0.0.1:8080/ui
 
 * If you want to develop on the UI, interacting directly with the server, run your browser on 
         
-				$ firefox http://127.0.0.1:8000/ui
+        $ firefox http://127.0.0.1:8000/ui
 
-That's it. If you're not too familiar with `rails` 3, have a look at
+That's it. If you're not too familiar with `rails` 4, have a look at
 <http://guides.rubyonrails.org/>.
 
 You can also list the available rake tasks and capistrano tasks to see what's
@@ -194,73 +207,61 @@ Updating the OAR2 test db therefore requires either
 
 ## Packaging
 
-* Bumping the version number is done as follows (a new changelog entry is
-  automatically generated in `debian/changelog`):
+### Use the build infrastructure
+The debian package build is done automatically as a stage in gitlab-ci. See `.gitlab-ci.yaml` and https://gitlab.inria.fr/grid5000/g5k-api/pipelines , but only tagged commits get pushed to the packages.grid5000.fr repository.
 
-        $ rake package:bump:patch # or: package:bump:minor, package:bump:major
+Tasks described in `lib/tasks/packaging.rake` are available to automatically manage version bumping, changelog generation and package building. If you use these tasks, a tag will be created each time version is bumped. Therefore, the `lib/grid5000/version.rb` file should only be changed using these tasks, at the end of a development cycle (if production has version X.Y.Z running, the file will keep that version during the next development cycle and it will only change at the end of the development cycle).
 
-* Building the `.deb` package is easy, since we kindly provide a rake task 
-  to run in the vagrant box, that will copy the latest committed code 
-  (`HEAD`) in /tmp/g5k-api, generate a `.deb` package, and move the generated
-  package to the `pkg/` directory.
+For this to work properly, you need a working .gitconfig.
 
-  Just execute from the host machine:
+- You can copy your main .gitconfig into the vagrant box
 
-        $ vagrant ssh -- "cd /vagrant ; rake package:build:debian"
+        $ cat ~/.gitconfig | vagrant ssh -- 'cat - > ~/.gitconfig'
 
-## Releasing and Installing and new version
+- Or you can configure the vagrant box to your needs
 
-* Once you've packaged the new version, you must release it to the APT
-  repository hosted on apt.grid5000.fr. There is Capistrano task for this:
-  By default it will release to g5k-api-devel repository hosted by apt.grid5000.fr
+        vagrant@g5k-local: git config --global user.name "Your Name"
+        vagrant@g5k-local: git config --global user.email you@example.com
 
-        $ REMOTE_USER=g5kadmin cap release
+- You can now name the version you are about to package
 
-  The system does not support different keys for the gateway and the remote node.
-  In this case, the following might work for you
+        vagrant@g5k-local: rake package:bump:patch #replace patch by minor or major when appropriate)
 
-        $ SSH_KEY=~/.ssh/id_rsa_grid5000_g5k NOPROXY=true REMOTE_USER=g5kadmin cap release
+- And then build the debian package
 
-  Note that you can use the same task to release on a different host (for
-  testing purposes for example), by setting the HOST environment variable to
-  another server (a Grid'5000 node for instance).
+        vagrant@g5k-local: rake package:build:debian
 
-  To release to stable API, you will need to set PROD_REPO before running the task
 
-	$ REMOTE_USER=g5kadmin PROD_REPO=true cap release
+The `package:build:debian` rake task has several arguments:
 
-* If you released on apt.grid5000.fr, then you are now able to install the new
-  version on any server by launching the following commands:
-  	  
-	puppet-repo $ cap cmd HOST="api-server-devel.[sites]" CMD='sudo apt-get update && sudo apt-get install -y -o Dpkg::Options::="--force-confold" g5k-api'
-        
-  A more flexible mechanism can be used base on the script/puppet-repo-custom.rb file
-
-	puppet-repo $ bundle exec cap shell ROLES=devel
-        cap> sudo apt-get update && sudo apt-get install g5k-api -y && sudo puppetd -t
-
-  For this command to work, you should have a look in the
-  `script/puppet-repo-custom.rb` file, which in my case was dropped into the
-  `config/` directory of the `puppet-repo` repository.
-
-  Then you can check that every server has the correct version running by
-  grepping through the processlist to check the version numbers:
-
-        puppet-repo $ bundle exec cap shell ROLES=devel
-        cap> ps aux | grep g5k-api | grep -v grep
-        [establishing connection(s) to api-server-devel.bordeaux.grid5000.fr, api-server-devel.grenoble.grid5000.fr, api-server-devel.lille.grid5000.fr, api-server-devel.lyon.grid5000.fr, api-server-devel.luxembourg.grid5000.fr, api-server-devel.nancy.grid5000.fr, api-server-devel.reims.grid5000.fr, api-server-devel.rennes.grid5000.fr, api-server-devel.orsay.grid5000.fr, api-server-devel.sophia.grid5000.fr, api-server-devel.toulouse.grid5000.fr]
-         ** [out :: api-server-devel.bordeaux.grid5000.fr] g5k-api  26110  0.3 13.9 246432 72840 ?        Sl   Nov20  14:43 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.grenoble.grid5000.fr] g5k-api  18510  0.3  7.5 249072 78620 ?        Sl   Nov18  22:48 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.luxembourg.grid5000.fr] g5k-api  13384  0.1 13.6 203396 71184 ?        Sl   Nov22   1:29 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.nancy.grid5000.fr] g5k-api  12702  0.3 26.9 312236 69704 ?        Sl   Nov20  17:27 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.sophia.grid5000.fr] g5k-api  19366  0.2 16.2 325972 84864 ?        Sl   Nov18  21:03 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.rennes.grid5000.fr] g5k-api  19353  0.9  7.2 246492 76220 ?        Sl   05:53   4:38 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.lille.grid5000.fr] g5k-api  18563  0.2 35.3 262788 92760 ?        Sl   Nov18  19:43 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.orsay.grid5000.fr] g5k-api  30348  0.3 27.4 244552 71028 ?        Sl   Nov22   4:57 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.toulouse.grid5000.fr] g5k-api   3825  0.3 32.3 259840 83660 ?        Sl   Nov18  23:54 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.reims.grid5000.fr] g5k-api   1749  0.8 31.9 324736 82696 ?        Sl   Nov18  65:08 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
-         ** [out :: api-server-devel.lyon.grid5000.fr] g5k-api   9940  0.5 33.9 329864 87796 ?        Sl   Nov18  40:32 thin server (0.0.0.0:8000) [g5k-api-3.0.16]
+* NO_COMMIT: when bumping version number, do not commit the version file
+* NO_TAG: do not tag the current git commit with the built version. Default is to tag. Has no effect with NO_COMMIT
+
+If everything went ok you should have a package like: `pkg/g5k-api_X.Y.Z-<date of last commit>_amd64.deb`
+
+See the `.gitlab-ci.yml` file for the use of the rake package commands in the gitlab pipelines.
 
+### Debug the build infrastructure
+
+From time to time, someone will have to look into `lib/taks/packaging.rake` to understand why `rake package:build:debian` does not do what is expected or to update the way the package is built. This is what happens when you call the rake task
+
+1.  The rake task creates a temporary directory named /tmp/g5k-api_version, and extracts the lasted commited version of your code using `git archive HEAD` to it.
+2.  The rake task makes sure the build dependencies are installed using `mk-build-deps`, which in turn uses info in the `debian/control` file.
+3.  The changelog in the extracted version of the sources is updated with information from the latest commits.
+4.  The rake task finally calls `dpkg-buildpackage -us -uc -d` to generate the package. dpkg-buildpackage then uses the `debian/rules` makefile to go through all the steps needed for packaging. This in turn falls back to `dh` for most steps, using datafiles in the `debian` directory.
+    * Most tasks use the default implementation relying on the datafile found in the `debian` directory. Of particular interest are `logrotate`, `g5k-api.service`, `dirs`, `g5k-api.install` and `g5k-api.links`.
+	* The magic happens in the `debian/setup_bundle` script. That script handles all the instructions required so that the gems needed by the application are installed and usable on the target system.
+	    * It will prime the temporary directory from wich the application is packaged with the developper's bundle
+		* It will run bundle install to setup the gems to package
+		* It will generate a g5k-api binary so that the application is started in the context of the installed bundle without the user noticing bundler usage. This happens by generating a script to be installed in `/usr/bin` for all ruby executable found in `bin/`
+    * `debian/setup_bundle`'s work is completed by lines in `debian/dirs` and `debian/g5k-api.install` to setup the final execution context of the application
+
+## Releasing and Installing and new version
+
+* Once you've packaged the new version, you must release it to the APT
+  repository hosted on packages.grid5000.fr. Their is a manual step in gitlab's CI for this.
+
+* If you released on packages.grid5000.fr, then api-server-devel servers will pick it up, but you'll need to change the hiera data for the production versions.
 
 ## Statistics
 
@@ -295,16 +296,5 @@ Updating the OAR2 test db therefore requires either
 
 * <https://www.grid5000.fr/mediawiki/index.php/API_Maintenance>;
 
-* There exist monit recipes that send emails when an alert is raised. At the
-  time of writing (2011-10-18), these alerts are sent to
-  <cyril.rohr@irisa.fr>. You might want to change that.
-
-## Kadeploy update process
-
-Since Kadeploy3 uses DRb to communicate with clients, and that a lot of code is
-shared between the client and server, clients must have the whole kadeploy3 code
-accessible. So, each time a new version of Kadeploy is released and installed on
-the Grid5000 sites, you MUST remember to update the kadeploy-common package.
-
 ## Authors
 * Cyril Rohr <cyril.rohr@inria.fr>, David Margery <david.margery@inria.fr> and others

debian/dirs

 var/log/g5k-api
-var/db/g5k-api
\ No newline at end of file
+var/db/g5k-api
+etc/g5k-api/conf.d
\ No newline at end of file

debian/g5k-api.install

+config/*.yml etc/g5k-api/
+config opt/g5k-api
+Gemfile opt/g5k-api
+Gemfile.lock opt/g5k-api
+config.ru opt/g5k-api
+Rakefile opt/g5k-api
+app opt/g5k-api
+bin opt/g5k-api
+db opt/g5k-api
+lib opt/g5k-api
+public opt/g5k-api
+script opt/g5k-api
+spec opt/g5k-api
+bundle/bundler usr/lib/ruby/bundles/g5k-api
+bundle/ruby usr/lib/ruby/bundles/g5k-api
+bundle/bin usr
+bundle/.bundle opt/g5k-api
\ No newline at end of file

debian/g5k-api.links

+opt/g5k-api/config/database.yml etc/g5k-api/database.yml
+opt/g5k-api/config/config.yml etc/g5k-api/config.yml
+opt/g5k-api/config/secrets.yml etc/g5k-api/secrets.yml
+var/log/g5k-api opt/g5k-api/log

debian/g5k-api.preinst

+#!/bin/bash
+
+set -e
+
+export APP_USER="g5k-api"
+export APP_GROUP="g5k-api"
+
+if ! getent passwd "${APP_USER}" > /dev/null; then
+  if ! getent group "${APP_GROUP}" > /dev/null; then
+    addgroup "${APP_GROUP}" --system --quiet
+  fi
+  adduser "${APP_USER}" --disabled-login --ingroup "${APP_GROUP}" --system --quiet --shell /bin/bash
+fi

debian/install

-config/*.yml etc/g5k-api/
-config opt/local/g5k-api
-.bundle opt/local/g5k-api
-Gemfile opt/local/g5k-api
-Gemfile.lock opt/local/g5k-api
-config.ru opt/local/g5k-api
-Rakefile opt/local/g5k-api
-app opt/local/g5k-api
-bin opt/local/g5k-api
-db opt/local/g5k-api
-lib opt/local/g5k-api
-public opt/local/g5k-api
-script opt/local/g5k-api
-spec opt/local/g5k-api
-vendor opt/local/g5k-api
\ No newline at end of file

debian/links

-etc/g5k-api/database.yml opt/local/g5k-api/config/database.yml
-etc/g5k-api/defaults.yml opt/local/g5k-api/config/defaults.yml
-var/log/g5k-api opt/local/g5k-api/log
\ No newline at end of file

debian/postinst

@@ -17,24 +17,24 @@ set -e
 # for details, see http://www.debian.org/doc/debian-policy/ or
 # the debian-policy package
 
-NAME="g5k-api"
+NAME=g5k-api
 
 case "$1" in
     configure)
-
-        if [ ! -f /opt/local/$NAME/bin/$NAME.rb ]; then 
-          mv /opt/local/$NAME/bin/$NAME /opt/local/$NAME/bin/$NAME.rb
-        fi
-        echo "#!/usr/bin/env ruby
-require '/opt/local/$NAME/bin/$NAME'
-        " > /usr/bin/$NAME && chmod a+x /usr/bin/$NAME
-        # Creating the user if it does not exist
-        if ! getent passwd $NAME > /dev/null; then
-          adduser --no-create-home --system --group $NAME
-        fi
+        chmod a+x /usr/bin/$NAME
+        /usr/bin/$NAME rake assets:precompile
         chown -R $NAME.$NAME /etc/$NAME/
         chown -R $NAME.$NAME /var/db/$NAME/
         chown -R $NAME.$NAME /var/log/$NAME/
+        chown -R $NAME.$NAME /opt/$NAME/
+        chmod 0600 /opt/$NAME/config/*.yml
+        if ! [ -e /etc/${NAME}/conf.d/secret.sh ]; then
+          SECRET=`${NAME} rake secret`
+          echo "export SECRET_KEY_BASE=\${SECRET_KEY_BASE:-${SECRET}}" > /etc/${NAME}/conf.d/secret.sh
+        fi
+        `${NAME} rake db:migrate` || true #attempt to migrate db is there is one
+        chown $NAME.$NAME /etc/${NAME}/conf.d/secret.sh
+        chmod 0600 /etc/${NAME}/conf.d/secret.sh
     ;;
 
     abort-upgrade|abort-remove|abort-deconfigure)

debian/prerm

@@ -20,9 +20,8 @@ NAME="g5k-api"
 
 cleanup()
 {
-  /etc/init.d/$NAME stop || true
+  systemctl stop $NAME || true
   rm /usr/bin/$NAME
-  mv /opt/local/$NAME/bin/$NAME.rb /opt/local/$NAME/bin/$NAME
   return 0;
 }
 

debian/rules

@@ -2,6 +2,19 @@
 
 export DH_VERBOSE=1
 
-
 %:
 	dh $@
+
+# Run 'bundle install --standalone' at the beginning of the Debian build process
+override_dh_auto_build:
+	debian/setup_bundle --bundle-bundler --generate-binstubs
+	dh_auto_build
+
+# When started from bundler context, prime the build dir with a copy of the current bundle
+override_dh_auto_configure:
+	debian/setup_bundle --auto-configure
+	dh_auto_configure
+
+override_dh_auto_clean:
+	debian/setup_bundle --clean
+	dh_auto_clean

debian/setup_bundle

+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'ostruct'
+
+NAME = "g5k-api"
+
+options = OpenStruct.new
+options.verbose=ENV['DH_VERBOSE'].to_i==1
+options.bundle_bundler=false
+options.generate_binstubs=false
+
+def import_bundle_files(options)
+  if defined?(Bundler)
+    puts '  Running from bundler context: install local bundle files into source directory' if options.verbose
+    # Do not copy .bundle so as to bundle install with no persistant options
+    # system "cp -r #{Bundler.app_config_path} ."
+    if Bundler.bundle_path.to_s =~ /#{Bundler.ruby_scope}/
+      system "cp -r #{Bundler.bundle_path.to_s.gsub(Bundler.ruby_scope,'')} bundle"
+    else
+      system "mkdir -p bundle/#{Bundler.ruby_scope}"
+      system "cp -r #{Bundler.bundle_path.to_s}/* bundle/#{Bundler.ruby_scope}"
+    end
+  else
+    puts '  No bundler context found. Do not prime build directory with bundler files' if options.verbose
+  end
+  true
+end
+
+optparse = OptionParser.new do |opts|
+  opts.on('-h', '--help', 'show help') do
+    puts opts
+    exit
+  end
+
+  opts.on('--clean', 'dh_auto_clean helper') do
+    puts '   Clean source directory from bundler files' if options.verbose
+    system "rm -rf bundle .bundle"
+    exit(0)
+  end
+
+  opts.on('--auto-configure', 'dh_auto_configure helper') do
+    exit import_bundle_files(options)
+  end
+
+  opts.on('--[no-]bundle-bundler', 'build bundler as part of the bundle') do |opt|
+    options.bundle_bundler=opt
+  end
+
+  opts.on('--[no-]generate-binstubs', 'generate executables running in bundler context in bundle/bin for all ruby files in bin/') do |opt|
+    options.generate_binstubs=opt
+  end
+
+end
+optparse.parse!
+
+def without_bundle_env(options, &block)
+  to_clean=["BUNDLE_GEMFILE","RUBYOPT"]
+  saved_values={}
+  to_clean.each do |env_value|
+    saved_values[env_value]=ENV[env_value]
+    ENV.delete(env_value)
+  end
+  if File.directory?('.bundle')
+    puts '   Unexpected .bundle directory in sources. Removing it' if options.verbose 
+    #Remove saved bundler options
+    File.unlink(".bundle/config")
+    Dir.unlink(".bundle")
+  end
+
+  yield
+
+  to_clean.each do |env_value|
+    ENV[env_value]=saved_values[env_value]
+  end
+end
+
+without_bundle_env(options) do
+  puts "   Installing vendored #{NAME} dependencies..." if options.verbose
+  system 'bundle install --standalone --no-binstubs --without=development --path bundle '
+  system 'bundle clean'
+  abort "Error: bundle not installed" unless Dir.exist? 'bundle'
+  if options.bundle_bundler
+    require 'bundler' unless defined? Bundler
+    system "gem install bundler --no-ri --no-rdoc --version #{Bundler::VERSION} -i bundle/#{Bundler.ruby_scope}"
+  end
+end
+
+RUBY_INSTALL_ROOT = '/usr/lib/ruby'
+
+#Override bundle-generated setup.rb
+#Gem native extensions load paths are generated relative to current directory, which is incorrect
+File.open "bundle/bundler/setup.rb", "w" do |f|
+  f.puts "deps_root = '#{RUBY_INSTALL_ROOT}/bundles/#{NAME}'"
+  
+  Dir["bundle/ruby/**/lib"].each do |dir|
+    dir = dir.gsub('bundle/', '')
+    f.puts %Q[$:.unshift "\#{deps_root}/#{dir}"]
+  end
+end
+
+if options.generate_binstubs
+  # create binaries that will use the bundle
+  PACKAGED_DIR="bundle/bin"
+  system "mkdir -p #{PACKAGED_DIR}"
+  # use gem2deb JUNK_PATTERNS
+  JUNK_PATTERNS = [ /^#/, /^\.#/, /^cvslog/, /^,/, /^\.del-*/, /\.olb$/,
+                    /~$/, /\.(old|bak|BAK|orig|rej)$/, /^_\$/, /\$$/, /\.org$/, /\.in$/, /^\./ ]
+  
+  puts "  Looking for executable ruby scripts in bin" if options.verbose
+  
+  Dir.foreach ("bin") do |file|
+    if JUNK_PATTERNS.all? {|p| file !~ p}
+      next if ['rails','bundle','setup','rake'].include?(file)
+      filename=File.join('bin',file)
+      if File.file?(filename) && File.executable?(filename)
+        shebang=nil
+        File.open(filename,'r') {|f| shebang=f.gets}
+        if shebang =~ /ruby/ && shebang =~ /#!/
+          puts "    Creating wrapper for #{filename} in usr/bin/" if options.verbose
+          File.open "#{PACKAGED_DIR}/#{file}", "w" do |f|
+            f.puts "#!/bin/sh"
+            f.puts "# Generated during debian build process by #{$0} #{$@}"
+            f.puts "\n"
+            f.puts "export GEM_HOME=#{RUBY_INSTALL_ROOT}/bundles/#{NAME}/#{Bundler.ruby_scope}"
+            f.puts "export BUNDLE_GEMFILE=/opt/#{NAME}/Gemfile"
+            f.puts "export RUBYOPT=-rbundler/setup"
+            f.puts "export RUBYLIB=#{RUBY_INSTALL_ROOT}/bundles/#{NAME}/#{Bundler.ruby_scope}/gems/bundler-#{Bundler::VERSION}/lib"
+            f.puts "for config_file in /etc/#{NAME}/conf.d/*.sh ; do"
+            f.puts "  if [ -f $config_file ]; then" 
+            f.puts "    . $config_file"
+            f.puts "  fi"
+            f.puts "done"
+            f.puts "exec /opt/#{NAME}/bin/#{file} \"\$@\""
+          end
+        end
+      end
+    end
+  end
+end
+
+puts "  Generating fake bundle config file" if options.verbose
+# create fake bundle setup in application dir
+DOT_BUNDLE_DIR="bundle/.bundle"
+BUNDLE_CONFIG_FILE="#{DOT_BUNDLE_DIR}/config"
+system "mkdir -p #{DOT_BUNDLE_DIR}"
+File.open(BUNDLE_CONFIG_FILE, "w") do |f|
+  f.puts '----'
+  f.puts "BUNDLE_PATH: #{RUBY_INSTALL_ROOT}/bundles/#{NAME}"
+  f.puts "BUNDLE_DISABLE_SHARED_GEMS: '1'"
+  f.puts "BUNDLE_WITHOUT: development"
+end

lib/tasks/packaging.rake

+# Copyright (c) 2010-2012 INRIA Rennes Bretagne Atlantique by Cyril Rohr (Grid'5000 and BonFIRE projects)
+# Copyright (c) 2015-2018 INRIA Rennes Bretagne Atlantique by David Margery (Grid'5000)
+
 ROOT_DIR = File.expand_path("../../..", __FILE__)
 CHANGELOG_FILE = File.join(ROOT_DIR, "debian", "changelog")
-VERSION_FILE = File.join(ROOT_DIR, "lib", "grid5000", "version.rb")
+VERSION_FILE = File.join(ROOT_DIR, 'lib', 'grid5000', 'version.rb')
 
 NAME = ENV['PKG_NAME'] || "g5k-api"
-USER_NAME = `git config --get user.name`.chomp
-USER_EMAIL = `git config --get user.email`.chomp
-
-LSBDISTCODENAME= `lsb_release -s -c`.chomp
 
 require VERSION_FILE
 
-PACKAGING_DIR = '/tmp/'+NAME+'-'+Grid5000::VERSION
+PACKAGING_DIR = '/tmp/'+NAME+'_'+Grid5000::VERSION
+PACKAGES_DIR = File.join(ROOT_DIR, 'pkg')
+
+def lsb_dist_codename
+   return `lsb_release -s -c`.chomp
+end
+
+def date_of_commit(tag_or_commit)
+  date=`git show --pretty=tformat:"MyDate: %aD" #{tag_or_commit}`.chomp
+  if date =~ /MyDate\: (.*)$/
+    date=$1
+  end
+  date
+end
+
+def deb_version_from_date(date)
+  Time.parse(date).strftime("%Y%m%d%H%M%S")
+end
+
+def deb_version_of_commit(tag_or_commit)
+  deb_version_from_date(date_of_commit(tag_or_commit))
+end
+
+def purged_commits_between(version1,version2)
+  cmd = "git log --oneline"
+  cmd << " #{version1}..#{version2}" unless version1.nil?
+  commit_logs=`#{cmd}`.split("\n")
+  purged_logs=commit_logs.reject{|l| l =~ / v#{Grid5000::VERSION}/}.
+                reject{|l| l =~ / v#{version2}/}.
+                reject{|l| l =~ /Commit version #{Grid5000::VERSION}/}
+  purged_logs
+end
+
+def generate_changelog_entry(version, deb_version, logs, author, email, date)
+  return [
+    "#{NAME} (#{version.gsub('_','~')}-#{deb_version}) #{lsb_dist_codename}; urgency=low",
+    "",
+    logs.map{|l| "  * #{l}"}.join("\n"),
+    "",
+    " -- #{author} <#{email}>  #{date}",
+    ""
+  ].join("\n")
+end
 
-def in_working_dir(dir,&block)
-  #Dir.choidr does not change ENV['PWD']
-  old_wd=ENV['PWD']
-  Dir.chdir(dir) do
-    ENV['PWD']=dir
-    yield
+def changelog_for_version(version, deb_version, change_logs)
+  cmd="git show #{version}"
+  tagger=`#{cmd}`
+  if tagger =~ /Tagger\: ([^<]*)<([^>]*)>/
+    author=$1
+    email=$2
+  elsif tagger =~ /Author\: ([^<]*)<([^>]*)>/
+    author=$1
+    email=$2
+  else
+    puts "#{cmd} has #{tagger} as output: could not find Tagger or Author"
   end
-  ENV['PWD']=old_wd
+  date=date_of_commit(version)
+  if deb_version.nil?
+    deb_version=deb_version_from_date(date)
+  end
+  return generate_changelog_entry(version, deb_version, change_logs, author, email, date)
 end
 
-def without_bundle_env(forced_values={}, &block)
-  to_clean=["BUNDLE_GEMFILE","RUBYOPT"]
-  saved_values={}
-  to_clean.each do |env_value|
-    saved_values[env_value]=ENV[env_value]
-    if forced_values.has_key?(env_value)
-      ENV[env_value]=forced_values[env_value]
+def generate_changelog
+  versions = `git tag`.split("\n")
+  versions.sort! do |v1,v2|
+    major1,minor1,rest1=v1.split('.')
+    major2,minor2,rest2=v2.split('.')
+    unless major1==major2
+      major1 <=> major2
     else
-      ENV.delete(env_value)
+      unless minor1==minor2
+        minor1 <=> monor2
+      else
+        patch1,rc1=rest1.split('_rc')
+        patch2,rc2=rest2.split('_rc')
+        unless patch1==patch2
+          patch1.to_i <=> patch2.to_i
+        else
+          rc1.to_i <=> rc2.to_i
+        end
+      end
     end
   end
-  
-  yield
-  
-  to_clean.each do |env_value|
-    ENV[env_value]=saved_values[env_value]
+  versions.reject! {|v| v !~ /[0-9]+\.[0-9]+\..*/}
+  change_logs=[]
+  previous_version=versions.shift
+  change_logs << changelog_for_version(previous_version, nil, ["First version tagged for packaging"])
+  versions.each do |version|
+    purged_logs=purged_commits_between(previous_version, version)
+    if purged_logs.empty?
+      purged_logs=["Retagged #{previous_version}. No other changes"]
+    end
+    change_logs << changelog_for_version(version, nil, purged_logs)
+    previous_version=version
+  end
+  change_logs.reverse.join("\n")
+end
+
+def update_changelog(changelog_file,new_version)
+  content_changelog=''
+  if File.exists?(changelog_file)
+    changelog=File.read(changelog_file)
+    last_commit = changelog.scan(/\s+\* ([a-f0-9]{7}) /).flatten[0]
+    deb_version=deb_version_of_commit('HEAD')