Commit cb4d0a05 authored by Guillaume Melquiond's avatar Guillaume Melquiond

Update remake.

parent ba1befa5
......@@ -90,14 +90,16 @@ scripts. They are only allowed after a rule header.
Lines starting with <tt>#</tt> are considered to be comments and are ignored.
They do interrupt rule scripts though.
Any other line is either a rule header or a variable definition. If such a
Any other line is either a variable definition or a rule header. If such a
line ends with a backslash, the following line break is ignored and the line
extends to the next one.
Variable definitions are a single name followed by equal followed by a list
of names, possibly empty.
Rule headers are a nonempty list of names, followed by a colon, followed by
another list of names, possibly empty. Variable definitions are a single
name followed by equal followed by a list of names, possibly empty. Basically,
the syntax of a rule is as follows:
another list of names, possibly empty. Basically, the syntax of a rule is as
follows:
@verbatim
targets : prerequisites
......@@ -164,6 +166,39 @@ triggered the rule.
- <tt>$(addsuffix <i>suffix</i>, <i>list</i>)</tt> returns the list obtained
by appending its first argument to each element of its second argument.
\subsection sec-order Order-only prerequisites
If the static prerequisites of a rule contain a pipe symbol, prerequisites
on its right do not cause the targets to become obsolete if they are newer
(unless they are also dynamically registered as dependencies). They are
meant to be used when the targets do not directly depend on them, but the
computation of their dynamic dependencies does.
@verbatim
%.o : %.c | parser.h
gcc -MMD -MF $@.d -o $@ -c $<
remake -r < $@.d
rm $@.d
parser.c parser.h: parser.y
yacc -d -o parser.c parser.y
@endverbatim
\subsection sec-special-tgt Special targets
Target <tt>.PHONY</tt> marks its prerequisites as being always obsolete.
\subsection sec-special-var Special variables
Variable <tt>.OPTIONS</tt> is handled specially. Its content enables some
features of <b>remake</b> that are not enabled by default.
- <tt>variable-propagation</tt>: When a variable is set in the prerequisite
part of a rule, it is propagated to the rules of all the targets this rule
depends on. This option also enables variables to be set on the command
line. Note that, as in <b>make</b>, this features introduces non-determinism:
the content of some variables will depend on the build order.
\section sec-semantics Semantics
\subsection src-obsolete When are targets obsolete?
......@@ -217,7 +252,7 @@ When <b>remake</b> tries to build a given target, it looks for a specific rule
that matches it. If there is one and its script is nonempty, it uses it to
rebuild the target.
Otherwise, it looks for a generic rule that match the target. If there are
Otherwise, it looks for a generic rule that matches the target. If there are
several matching rules, it chooses the one with the shortest pattern (and if
there are several ones, the earliest one). <b>remake</b> then looks for
specific rules that match each target of the generic rule. All the
......@@ -273,6 +308,9 @@ Differences with <b>make</b>:
select one for which it could satisfy the dependencies.
- Variables and built-in functions are expanded as they are encountered
during <b>Remakefile</b> parsing.
- Target-specific variables are not propagated, unless specifically enabled,
since this causes non-deterministic builds. This is the same for variables
set on the command line.
Differences with <b>redo</b>:
......@@ -293,10 +331,11 @@ Remakefile: Remakefile.in ./config.status
\section sec-limitations Limitations
- When the user calls <b>remake</b>, the current working directory should be
the one containing <b>.remake</b>. Rules are understood relatively to this
directory. If a rule script calls <b>remake</b>, the current working
directory should be the same as the one from the original <b>remake</b>.
- If a rule script calls <b>remake</b>, the current working directory should
be the directory containing <b>Remakefile</b> (or the working directory
from the original <b>remake</b> if it was called with option <b>-f</b>).
- As with <b>make</b>, variables passed on the command line should keep
the same values, to ensure deterministic builds.
- Some cases of ill-formed rules are not caught by <b>remake</b> and can
thus lead to unpredictable behaviors.
......@@ -308,7 +347,7 @@ https://github.com/apenwarr/redo for an implementation and some comprehensive do
\section sec-licensing Licensing
@author Guillaume Melquiond
@version 0.9
@version 0.11
@date 2012-2013
@copyright
This program is free software: you can redistribute it and/or modify
......@@ -338,8 +377,8 @@ When building a target, the following sequence of events happens:
- #start calls #find_rule (and #find_generic_rule) to get the rule.
- It then creates a pseudo-client if the rule has static dependencies, or
calls #run_script otherwise. In both cases, a new job is created and its
targets are put into #job_targets.
calls #run_script otherwise. In both cases, a new job is created; the
rule and the variables are stored into #jobs.
- #run_script creates a shell process and stores it in #job_pids. It
increases #running_jobs.
- The child process possibly calls <b>remake</b> with a list of targets.
......@@ -357,7 +396,7 @@ When building a target, the following sequence of events happens:
- When a child process ends, #server_loop calls #finalize_job, which
removes the process from #job_pids, decreases #running_jobs, and calls
#complete_job.
- #complete_job removes the job from #job_targets and calls #update_status
- #complete_job removes the job from #jobs and calls #update_status
to change the status of the targets. It also removes the target files in
case of failure.
*/
......@@ -377,6 +416,7 @@ When building a target, the following sequence of events happens:
#include <vector>
#include <cassert>
#include <cstdlib>
#include <cstring>
#include <ctime>
#include <errno.h>
#include <fcntl.h>
......@@ -491,12 +531,11 @@ typedef std::map<std::string, status_t> status_map;
*/
struct assign_t
{
std::string name;
bool append;
string_list value;
};
typedef std::list<assign_t> assign_list;
typedef std::map<std::string, assign_t> assign_map;
/**
* A rule loaded from Remakefile.
......@@ -504,17 +543,28 @@ typedef std::list<assign_t> assign_list;
struct rule_t
{
string_list targets; ///< Files produced by this rule.
string_list deps; ///< Files used for an implicit call to remake at the start of the script.
assign_list vars; ///< Values of variables.
string_list deps; ///< Dependencies used for an implicit call to remake at the start of the script.
string_list wdeps; ///< Like #deps, except that they are not registered as dependencies.
assign_map assigns; ///< Assignment of variables.
std::string script; ///< Shell script for building the targets.
std::string stem; ///< String used to instantiate a generic rule.
};
typedef std::list<rule_t> rule_list;
typedef std::map<std::string, ref_ptr<rule_t> > rule_map;
typedef std::map<int, string_list> job_targets_map;
/**
* A job created from a set of rules.
*/
struct job_t
{
rule_t rule; ///< Original rule.
std::string stem; ///< Pattern used to instantiate the generic rule, if any.
variable_map vars; ///< Values of local variables.
};
typedef std::map<int, job_t> job_map;
typedef std::map<pid_t, int> pid_job_map;
......@@ -538,14 +588,16 @@ struct client_t
bool failed; ///< Whether some targets failed in mode -k.
string_list pending; ///< Targets not yet started.
string_set running; ///< Targets being built.
rule_t *delayed; ///< Rule that implicitly created a dependency client, and which script has to be started on request completion.
client_t(): socket(INVALID_SOCKET), job_id(-1), failed(false), delayed(NULL) {}
variable_map vars; ///< Variables set on request.
bool delayed; ///< Whether it is a dependency client and a script has to be started on request completion.
client_t(): socket(INVALID_SOCKET), job_id(-1), failed(false), delayed(false) {}
};
typedef std::list<client_t> client_list;
/**
* Map from variable names to their content.
* Initialized with the values passed on the command line.
*/
static variable_map variables;
......@@ -570,9 +622,9 @@ static rule_list generic_rules;
static rule_map specific_rules;
/**
* Map from jobs to targets being built.
* Map of jobs being built.
*/
static job_targets_map job_targets;
static job_map jobs;
/**
* Map from jobs to shell pids.
......@@ -620,7 +672,7 @@ static int waiting_jobs = 0;
/**
* Global counter used to produce increasing job numbers.
* @see job_targets
* @see jobs
*/
static int job_counter = 0;
......@@ -656,10 +708,31 @@ static bool show_targets = true;
*/
static bool echo_scripts = false;
/**
* Time at the start of the program.
*/
static time_t now = time(NULL);
/**
* Directory with respect to which command-line names are relative.
*/
static std::string working_dir;
/**
* Directory with respect to which targets are relative.
*/
static std::string prefix_dir;
/**
* Whether the prefix directory is different from #working_dir.
*/
static bool changed_prefix_dir;
/**
* Whether target-specific variables are propagated to prerequisites.
*/
static bool propagate_vars = false;
#ifndef WINDOWS
static volatile sig_atomic_t got_SIGCHLD = 0;
......@@ -777,7 +850,7 @@ static std::ostream &operator<<(std::ostream &out, escape_string const &se)
/**
* Initialize #working_dir.
*/
void init_working_dir()
static void init_working_dir()
{
char buf[1024];
char *res = getcwd(buf, sizeof(buf));
......@@ -787,16 +860,56 @@ void init_working_dir()
exit(EXIT_FAILURE);
}
working_dir = buf;
#ifdef WINDOWS
for (size_t i = 0, l = working_dir.size(); i != l; ++i)
{
if (working_dir[i] == '\\') working_dir[i] = '/';
}
#endif
prefix_dir = working_dir;
}
/**
* Normalize an absolute path with respect to the working directory.
* Paths outside the working subtree are left unchanged.
* Initialize #prefix_dir and switch to it.
*/
static std::string normalize_abs(std::string const &s)
static void init_prefix_dir()
{
size_t l = working_dir.length();
if (s.compare(0, l, working_dir)) return s;
for (;;)
{
struct stat s;
if (stat((prefix_dir + "/Remakefile").c_str(), &s) == 0)
{
if (!changed_prefix_dir) return;
if (chdir(prefix_dir.c_str()))
{
perror("Failed to change working directory");
exit(EXIT_FAILURE);
}
if (show_targets)
{
std::cout << "remake: Entering directory `" << prefix_dir << '\'' << std::endl;
}
return;
}
size_t pos = prefix_dir.find_last_of('/');
if (pos == std::string::npos)
{
std::cerr << "Failed to locate Remakefile in the current directory or one of its parents" << std::endl;
exit(EXIT_FAILURE);
}
prefix_dir.erase(pos);
changed_prefix_dir = true;
}
}
/**
* Normalize an absolute path with respect to @a p.
* Paths outside the subtree are left unchanged.
*/
static std::string normalize_abs(std::string const &s, std::string const &p)
{
size_t l = p.length();
if (s.compare(0, l, p)) return s;
size_t ll = s.length();
if (ll == l) return ".";
if (s[l] != '/')
......@@ -810,19 +923,24 @@ static std::string normalize_abs(std::string const &s)
}
/**
* Normalize a target name.
* Normalize path @a s (possibly relative to @a w) with respect to @a p.
*
* - If both @a p and @a w are empty, the function just removes ".", "..", "//".
* - If only @a p is empty, the function returns an absolute path.
*/
static std::string normalize(std::string const &s)
static std::string normalize(std::string const &s, std::string const &w, std::string const &p)
{
#ifdef WINDOWS
char const *delim = "/\\";
#else
char delim = '/';
#endif
size_t prev = 0, len = s.length();
size_t pos = s.find_first_of(delim);
if (pos == std::string::npos) return s;
if (pos == std::string::npos && w == p) return s;
bool absolute = pos == 0;
if (!absolute && w != p && !w.empty())
return normalize(w + '/' + s, w, p);
size_t prev = 0, len = s.length();
string_list l;
for (;;)
{
......@@ -832,8 +950,8 @@ static std::string normalize(std::string const &s)
if (n == "..")
{
if (!l.empty()) l.pop_back();
else if (!absolute)
return normalize(working_dir + '/' + s);
else if (!absolute && !w.empty())
return normalize(w + '/' + s, w, p);
}
else if (n != ".")
l.push_back(n);
......@@ -854,19 +972,19 @@ static std::string normalize(std::string const &s)
n.push_back('/');
n.append(*i);
}
if (absolute) return normalize_abs(n);
if (absolute && !p.empty()) return normalize_abs(n, p);
return n;
}
/**
* Normalize the content of a list of targets.
*/
static void normalize_list(string_list &l)
static void normalize_list(string_list &l, std::string const &w, std::string const &p)
{
for (string_list::iterator i = l.begin(),
i_end = l.end(); i != i_end; ++i)
{
*i = normalize(*i);
*i = normalize(*i, w, p);
}
}
......@@ -922,6 +1040,7 @@ enum
Rightpar = 1 << 5,
Comma = 1 << 6,
Plusequal = 1 << 7,
Pipe = 1 << 8,
};
/**
......@@ -946,6 +1065,7 @@ static int expect_token(std::istream &in, int mask)
case ',': tok = Comma; break;
case '=': tok = Equal; break;
case ')': tok = Rightpar; break;
case '|': tok = Pipe; break;
case '$':
if (!(mask & Dollarpar)) return Unexpected;
in.ignore(1);
......@@ -975,44 +1095,44 @@ static int expect_token(std::istream &in, int mask)
/**
* Read a (possibly quoted) word.
*/
static std::string read_word(std::istream &in)
static std::string read_word(std::istream &in, bool detect_equal = true)
{
int c = in.get();
int c = in.peek();
std::string res;
if (!in.good()) return res;
char const *separators = " \t\r\n:$(),=+\"";
char const *separators = " \t\r\n$(),:";
bool quoted = c == '"';
if (!quoted)
{
if (strchr(separators, c))
{
in.putback(c);
return res;
}
res += c;
}
if (quoted) in.ignore(1);
bool plus = false;
while (true)
{
c = in.get();
c = in.peek();
if (!in.good()) return res;
if (quoted)
{
in.ignore(1);
if (c == '\\')
res += in.get();
else if (c == '"')
return res;
quoted = false;
else
res += c;
continue;
}
else
if (detect_equal && c == '=')
{
if (strchr(separators, c))
{
in.putback(c);
return res;
}
res += c;
if (plus) in.putback('+');
return res;
}
if (plus)
{
res += '+';
plus = false;
}
if (strchr(separators, c)) return res;
in.ignore(1);
if (detect_equal && c == '+') plus = true;
else res += c;
}
}
......@@ -1049,68 +1169,38 @@ struct generator
struct variable_generator: generator
{
std::string name;
string_list::const_iterator cur1, end1;
assign_list::const_iterator cur2, end2;
variable_generator(std::string const &, assign_list const *);
string_list::const_iterator vcur, vend;
variable_generator(std::string const &, variable_map const *);
input_status next(std::string &);
};
variable_generator::variable_generator(std::string const &n,
assign_list const *local_variables): name(n)
variable_map const *local_variables): name(n)
{
bool append = true;
if (local_variables)
{
// Set cur2 to the last variable overwriter, if any.
cur2 = local_variables->begin();
end2 = local_variables->end();
for (assign_list::const_iterator i = cur2; i != end2; ++i)
variable_map::const_iterator i = local_variables->find(name);
if (i != local_variables->end())
{
if (i->name == name && !i->append)
{
append = false;
cur2 = i;
}
vcur = i->second.begin();
vend = i->second.end();
return;
}
}
else
{
static assign_list dummy;
cur2 = dummy.begin();
end2 = dummy.end();
}
static string_list dummy;
cur1 = dummy.begin();
end1 = dummy.end();
if (append)
{
variable_map::const_iterator i = variables.find(name);
if (i == variables.end()) return;
cur1 = i->second.begin();
end1 = i->second.end();
}
variable_map::const_iterator i = variables.find(name);
if (i == variables.end()) return;
vcur = i->second.begin();
vend = i->second.end();
}
input_status variable_generator::next(std::string &res)
{
restart:
if (cur1 != end1)
if (vcur != vend)
{
res = *cur1;
++cur1;
res = *vcur;
++vcur;
return Success;
}
while (cur2 != end2)
{
if (cur2->name == name)
{
cur1 = cur2->value.begin();
end1 = cur2->value.end();
++cur2;
goto restart;
}
++cur2;
}
return Eof;
}
......@@ -1121,9 +1211,9 @@ struct input_generator
{
std::istream &in;
generator *nested;
assign_list const *local_variables;
variable_map const *local_variables;
bool earliest_exit, done;
input_generator(std::istream &i, assign_list const *lv, bool e = false)
input_generator(std::istream &i, variable_map const *lv, bool e = false)
: in(i), nested(NULL), local_variables(lv), earliest_exit(e), done(false) {}
input_status next(std::string &);
~input_generator() { assert(!nested); }
......@@ -1147,11 +1237,11 @@ input_status input_generator::next(std::string &res)
switch (expect_token(in, Word | Dollarpar))
{
case Word:
res = read_word(in);
res = read_word(in, false);
return Success;
case Dollarpar:
{
std::string name = read_word(in);
std::string name = read_word(in, false);
if (name.empty()) return SyntaxError;
if (expect_token(in, Rightpar))
nested = new variable_generator(name, local_variables);
......@@ -1299,7 +1389,7 @@ input_status addsuffix_generator::next(std::string &res)
/**
* Return a generator for function @a name.
*/
generator *get_function(input_generator const &in, std::string const &name)
static generator *get_function(input_generator const &in, std::string const &name)
{
skip_spaces(in.in);
generator *g = NULL;
......@@ -1397,6 +1487,8 @@ static void save_dependencies()
/** @} */
static void merge_rule(rule_t &dest, rule_t const &src);
/**
* @defgroup parser Rule parser
*
......@@ -1433,8 +1525,7 @@ static void register_transparent_rule(rule_t const &rule, string_list const &tar
exit(EXIT_FAILURE);
}
assert(r->targets.size() == 1 && r->targets.front() == *i);
r->deps.insert(r->deps.end(), rule.deps.begin(), rule.deps.end());
r->vars.insert(r->vars.end(), rule.vars.begin(), rule.vars.end());
merge_rule(*r, rule);
}
for (string_list::const_iterator i = targets.begin(),
......@@ -1504,7 +1595,7 @@ static void load_rule(std::istream &in, std::string const &first)
else if (targets.empty()) goto error;
else DEBUG << "actual target: " << targets.front() << std::endl;
bool generic = false;
normalize_list(targets);
normalize_list(targets, "", "");
for (string_list::const_iterator i = targets.begin(),
i_end = targets.end(); i != i_end; ++i)
{
......@@ -1522,36 +1613,36 @@ static void load_rule(std::istream &in, std::string const &first)
bool assignment = false;
// Read dependencies.
if (expect_token(in, Word))
{
std::string d = read_word(in);
if (int tok = expect_token(in, Equal | Plusequal))
string_list v;
if (expect_token(in, Word))
{
rule.vars.push_back(assign_t());
string_list v;
if (!read_words(in, v)) goto error;
assign_t &a = rule.vars.back();
a.name = d;
a.append = tok == Plusequal;
a.value.swap(v);
assignment = true;
std::string d = read_word(in);
if (int tok = expect_token(in, Equal | Plusequal))
{
if (!read_words(in, v)) goto error;
assign_t &a = rule.assigns[d];
a.append = tok == Plusequal;
a.value.swap(v);
assignment = true;
goto end_line;
}
v.push_back(d);
}
else
if (!read_words(in, v)) goto error;
normalize_list(v, "", "");
rule.deps.swap(v);
if (expect_token(in, Pipe))
{
string_list v;
if (!read_words(in, v)) goto error;
v.push_front(d);
normalize_list(v);
rule.deps.swap(v);
normalize_list(v, "", "");
rule.wdeps.swap(v);
}
}
else
{
string_list v;
if (!read_words(in, v)) goto error;
normalize_list(v);
rule.deps.swap(v);
}
end_line:
skip_spaces(in);
if (!skip_eol(in, true)) goto error;
......@@ -1576,6 +1667,17 @@ static void load_rule(std::istream &in, std::string const &first)
}
rule.script = buf.str();
// Register phony targets.
if (rule.targets.front() == ".PHONY")
{
for (string_list::const_iterator i = rule.deps.begin(),
i_end = rule.deps.end(); i != i_end; ++i)
{
status[*i].status = Todo;
}
return;
}
// Add generic rules to the correct set.
if (generic)
{
......@@ -1625,6 +1727,8 @@ static void load_rules(std::string const &remakefile)
}
skip_empty(in);
string_list options;
// Read rules
while (in.good())
{
......@@ -1645,7 +1749,8 @@ static void load_rules(std::string const &remakefile)
DEBUG << "Assignment to variable " << name << std::endl;
string_list value;
if (!read_words(in, value)) goto error;
string_list &dest = variables[name];
string_list &dest =
*(name == ".OPTIONS" ? &options : &variables[name]);
if (tok == Equal) dest.swap(value);
else dest.splice(dest.end(), value);
if (!skip_eol(in, true)) goto error;
......@@ -1654,6 +1759,18 @@ static void load_rules(std::string const &remakefile)
}
else load_rule(in, std::string());
}
// Set actual options.
for (string_list::const_iterator i = options.begin(),
i_end = options.end(); i != i_end; ++i)
{
if (*i == "variable-propagation") propagate_vars = true;
else
{
std::cerr << "Failed to load rules: unrecognized option" << std::endl;
exit(EXIT_FAILURE);
}
}
}
/** @} */
......@@ -1664,6 +1781,26 @@ static void load_rules(std::string const &remakefile)
* @{
*/
static void merge_rule(rule_t &dest, rule_t const &src)
{
dest.deps.insert(dest.deps.end(), src.deps.begin(), src.deps.end());
dest.wdeps.insert(dest.wdeps.end(), src.wdeps.begin(), src.wdeps.end());
for (assign_map::const_iterator i = src.assigns.begin(),
i_end = src.assigns.end(); i != i_end; ++i)
{
if (!i->second.append)
{
new_assign:
dest.assigns[i->first] = i->second;
continue;
}
assign_map::iterator j = dest.assigns.find(i->first);
if (j == dest.assigns.end()) goto new_assign;
j->second.value.insert(j->second.value.end(),
i->second.value.begin(), i->second.value.end());
}
}
/**
* Substitute a pattern into a list of strings.
*/
......@@ -1673,7 +1810,7 @@ static void substitute_pattern(std::string const &pat, string_list const &src, s
i_end = src.end(); i != i_end; ++i)
{
size_t pos = i->find('%');
if (pos == std::string::npos)dst.push_back(*i);
if (pos == std::string::npos) dst.push_back(*i);
else dst.push_back(i->substr(0, pos) + pat + i->substr(pos + 1));
}
}
......@@ -1683,10 +1820,9 @@ static void substitute_pattern(std::string const &pat, string_list const &src, s
* - the one leading to shorter matches has priority,
* - among equivalent rules, the earliest one has priority.
*/
static rule_t find_generic_rule(std::string const &target)
static void find_generic_rule(job_t &job, std::string const &target)
{