Term types in Integraal are inconsistent

There are several problems when loading terms with Integraal parsers. There are already two other issues in this subject:

• Behavior of Literal's Number in Integraal
• RDFParser always parse subjects and objects as strings

There is another issue not listed here: there are cases where an IRI term is parsed as a functional term:

@prefix ex:<http://example.com>
p(X) :- r(X, ex:test)

In this example, ex:test is parsed as the functional term http://example.com/test().

There is an issue when loading a timestamp in RDF: currently, the timestamp is converted to a string representing a date. We obtain something like 1970-01-01 00:00:40.066669.

Finally, there is the issue with how the prefixes are managed for the terms (Currently, a prefix declared with @prefixed is just replaced by its value and a prefix declared with @computed just disappears).

These issues makes Integraal very difficult to use in practice as it can be very difficult to find the errors they cause. For example, it is hard to see that a rule is not applied because it tries to make a join between a Float and a Double. Or join between a number that was parsed as a Float (as in Dlgp) and another one parsed as a String (as in RDF). When we print the atoms, in appearance, they all appear to be the same value as printing an atom doesn't show the type of the terms.

I propose several default behaviours:

• By default, a literal should be parsed as Literal<?>, whatever the type of the file we parse;
• We should always select a specific type when possible, parsing everything as Literal<String> like in the RDF parsers is not a good behaviour;
• The parsing should be consistent between parsers: notably, an integer should have a unique type, the same for floating/decimal numbers;
• An IRI term should never be parsed as a functional term and should parsed the same way both in Dlgp and RDF ;
• When parsing types in RDF, we should register this type for a predicate position so that we build the good type when doing the parsing.

I have several ideas:

• Integers should be parsed as a unique type. For example, it could be a Python-like Integer, that is optimized when the integer is less than 64-bits in size but can grow bigger than that - it would give a great flexibility for the users. I put an implementation at the end of this issue.
• Decimal/Floating numbers should always be parsed as Double - for greater flexibility, we could implement a class IntegraalFloat that extends Double, so it will be easier to implement other functionalities later.
• Booleans should be, obviously, always be parsed as Boolean.

Implementation of IntegraalInteger:

import java.math.BigInteger;

public class IntegraalInteger extends Number implements Comparable<IntegraalInteger> {
    private boolean isBig;
    private long smallValue;
    private BigInteger bigValue;

    // Constructor from long
    public IntegraalInteger(long value) {
        this.smallValue = value;
        this.isBig = false;
    }

    // Constructor from BigInteger
    public IntegraalInteger(BigInteger value) {
        if (value.bitLength() <= 63) { // Fits in long
            this.smallValue = value.longValue();
            this.isBig = false;
        } else {
            this.bigValue = value;
            this.isBig = true;
        }
    }

    // Private helper to get BigInteger representation
    private BigInteger toBigInteger() {
        return isBig ? bigValue : BigInteger.valueOf(smallValue);
    }

    // Addition
    public IntegraalInteger add(IntegraalInteger other) {
        if (this.isBig || other.isBig) {
            return new IntegraalInteger(this.toBigInteger().add(other.toBigInteger()));
        }
        long result = this.smallValue + other.smallValue;
        if (((this.smallValue ^ result) & (other.smallValue ^ result)) < 0) { // Overflow detection
            return new IntegraalInteger(BigInteger.valueOf(this.smallValue).add(BigInteger.valueOf(other.smallValue)));
        }
        return new IntegraalInteger(result);
    }

    // Subtraction
    public IntegraalInteger subtract(IntegraalInteger other) {
        if (this.isBig || other.isBig) {
            return new IntegraalInteger(this.toBigInteger().subtract(other.toBigInteger()));
        }
        long result = this.smallValue - other.smallValue;
        if (((this.smallValue ^ other.smallValue) & (this.smallValue ^ result)) < 0) { // Overflow detection
            return new IntegraalInteger(BigInteger.valueOf(this.smallValue).subtract(BigInteger.valueOf(other.smallValue)));
        }
        return new IntegraalInteger(result);
    }

    // Multiplication
    public IntegraalInteger multiply(IntegraalInteger other) {
        if (this.isBig || other.isBig) {
            return new IntegraalInteger(this.toBigInteger().multiply(other.toBigInteger()));
        }
        long result = this.smallValue * other.smallValue;
        if (other.smallValue != 0 && result / other.smallValue != this.smallValue) { // Overflow detection
            return new IntegraalInteger(BigInteger.valueOf(this.smallValue).multiply(BigInteger.valueOf(other.smallValue)));
        }
        return new IntegraalInteger(result);
    }

    // Division
    public IntegraalInteger divide(IntegraalInteger other) {
        if (other.equals(new IntegraalInteger(0))) {
            throw new ArithmeticException("Division by zero");
        }
        return new IntegraalInteger(this.toBigInteger().divide(other.toBigInteger()));
    }

    // Modulo
    public IntegraalInteger mod(IntegraalInteger other) {
        if (other.equals(new IntegraalInteger(0))) {
            throw new ArithmeticException("Modulo by zero");
        }
        return new IntegraalInteger(this.toBigInteger().mod(other.toBigInteger()));
    }

    // Comparisons
    @Override
    public int compareTo(IntegraalInteger other) {
        return this.toBigInteger().compareTo(other.toBigInteger());
    }

    @Override
    public boolean equals(Object obj) {
        if (this == obj) return true;
        if (!(obj instanceof IntegraalInteger)) return false;
        IntegraalInteger other = (IntegraalInteger) obj;
        return this.toBigInteger().equals(other.toBigInteger());
    }

    @Override
    public int hashCode() {
        return this.toBigInteger().hashCode();
    }

    // Implementing Number abstract methods
    @Override
    public int intValue() {
        return isBig ? bigValue.intValue() : (int) smallValue;
    }

    @Override
    public long longValue() {
        return isBig ? bigValue.longValue() : smallValue;
    }

    @Override
    public float floatValue() {
        return isBig ? bigValue.floatValue() : smallValue;
    }

    @Override
    public double doubleValue() {
        return isBig ? bigValue.doubleValue() : smallValue;
    }

    @Override
    public String toString() {
        return isBig ? bigValue.toString() : Long.toString(smallValue);
    }
}

assigned to @gperutio

changed the description

changed the description

changed the description

changed the description

@prefix ex:<http://example.com>
p(X) :- r(X, ex:test)

While parsing this example, our parser tries to recognize a Term from ex:test. Among the choices in the ordered list of choices it has, there is Variable, Constant, Literal and Functional Term (not exactly what the integraal parser is soing, but I'm describing what it should do). Here it is not a Variable (does not begin with an uppercase), not a Literal (that is either a number or begin with a "). Both Constant and Functional Term begin with an identifier:

Term : ... | Functional_Term | ... | Constant ;
Constant : Identifier;
Functional_Term : Identifier "(" TermsList  ")";

It is important to test Functional_Term before Constant, because the parser is a LRR (otherwise, should we test Constant before, the parser would never be able to recognize a functional term, but this is not what is happening here).

So here the parser should recognize a Constant that is an Identifier. and that Identifier should even be a Prefixed_Identifier with Prefix ex (a Turtle PN_PREFIX) and Local_Name test (a Turtle PN_LOCAL). These parts are well recognized.

That May be the problem The parser builds an AST. Written as I did above, it would build something like:

["Atom",  
    ["Simple Identifier", "p"], 
    ["TermsList", 
        [
            ["Variable", "X"],
            ["Prefixed_Identifier", "ex", "test"]
        ]
    ]
]

Beacause the AST usually keeps only the most specific choice. And we see here that it has lost the information that the Prefixed_Identifier stood for a Constant. This is something I like in TextX grammars, because it allows me to write:

Constant : identifier = Identifier; 

And then, Constant is no longer an "Abstract" rule. The AST will contain something like:

["Atom",  
    ["Simple Identifier", "p"], 
    ["TermsList", 
        [
            ["Variable", "X"],
            ["Constant", {"name": ["Prefixed_Identifier", "ex", "test"]}]
        ]
    ]
]

I really think the problem is there: the parser loses the information that we are in a Constant, and the code attached to an Identifier does not handle it well.

Another thing: what is the full identifier built from this example ? Though I advocate using resolution of a relative IRI vs a Base IRI (and in that case it should generate http://www.example.com/test), I believe Graal and Integraal generate the concatenation of the base and the relative, so it should give the (bad) http://www.example.comtest.

If a / was indeed added, I don't know why because it has never been documented anywhere, and if it was just added to handle the worst cases of concatenation, wer should as well implement the correct resolution.

Finally, how to we interpret internally a Literal with type T and string representation S ? This is the general case, since 2 is only a DLGP shortcut for "2"^^xsd:integer and "foo" for "foo"^^xsd:string.

Now there is a java class, say MyInteger, associated with xsd:integer. This class has a constructor from String that returns the value of S^^T. This is the object we store internally (and may be we hash it first).

Now there is a problem when the same object can have different representations. For instance "2"^^xsd:float, "2.0"^^xsd:float, and "2.00"^^xsd:float. We must ensure that the instances of FLOAT built from those 3 strings are the same, or at least that they have the same hash, used in our HashTable of terms. Another solution is to correctly implement equals(), but it's less efficient since we lose direct access.

The problem is even more difficult (and I still struggle with it) when we want, for instance, join "2"^^xsd.integer and "2.0"^^xsd:float. We must now ensure that the hashes are "interclass compatible". That's already some work for integers and floats, and I have no idea how to specify what should do "user types" (for instance, should one want to implement ``VeryVeryLongIntegers`).

In my coding experiments, assume that 2 values can be equal only when they have a common subclass. So S1:T1 and S2:T2 are equal only when there exists a subclass T3 of T1 and T2 such that the values S1:T3 and S2:T3 exist and are equal. It allows to keep the equality test "inside" a class, by relying upon the normal form of T3.

Knowing thaty number > float > integer, when testing if "2"^^xsd:float == "2.0"^^xsd:integer , I observe that the common sublass is xsd:integer, and try to cast the float as an integer.

The problem is that we can only do it when we test equality, and not when we build the fact base, and we want to keep the joins working with direct access. A solution would be, when we read S:T, to check if there is a most specific subtype T' of T such that S:T'. In that case, we store S:T'. But we must ensure that our type hierarchy ensures that property to be verified.

To clarify the property to be held by a type hierarchy: let T be a type, and sT be the set of all subtypes of T. Then for any valid S:T, there is at most one T' in sT such that S:T' is valid.