ESTOCADA
Handy Links
- Download ESTOCADA Source code
- Code Requirements and Build
- System Prerequisites
- Description of Used Hardware
- Supported Native Stores Query Languages
- Adding New Model/Language Pair
- ESTOCADA MDD Tutorial
- Tutorial: ESTOCADA Quick Actions
- Testing
- Documentation
- Code Format
- Paper Reference
- Contact
Download ESTOCADA
$ git clone https://gitlab.inria.fr/cedar/estocada-public.git
Code Requirements and Build
Requirements
- A Unix-ish environment (Linux, Mac OS X). NOTE: the code build was tested on Mac OS X and CentOS7 Linux.
- Apache Maven 3.6.1 or newer (Download and Install)
- Java version 1.8
Used Libraries
- Antlr4.7
- Google Guice
- Google GSON
- Google Guava
- Google Findbugs
- JSON Jackson core
- Org MongoDB
- Org Spark Core
- Org SparkSQL
- Log4j - Version 2.17.1
Instructions for the code build
The code contains four main components:
- estocada-commons
- estocada-PACB
- estocada-main
- estoccada-engine-Tatooine
Code build (skipping tests). This will automatically build the source code, all related dependencies and generate APIs documentation. Note: documenting the code is work in progress.
$ chmod +x build-skip-tests.sh
$./build-skip-tests.sh
Code build (enabling tests). This will automatically build the source code, all related dependencies, run the tests and generate APIs documentation.
$ chmod +x build-tests.sh
$./build-tests.sh
System Prerequisites
AsterixDB v0.9.3
- Link to AsterixDB source code: https://github.com/apache/asterixdb/tree/apache-asterixdb-0.9.3
- Recommended installtion using Ansible: https://nightlies.apache.org/asterixdb/ansible.html
- Example of
cc.conf
configuration file used for MacOS with 16GB RAM and 8cores:
[common]
log.level=INFO
compiler.framesize=64KB
compiler.sortmemory=64MB
compiler.groupmemory=128MB
compiler.joinmemory=256MB
compiler.textsearchmemory=256MB
log.dir=logs/
storage.max.active.writable.datasets=2
[nc]
jvm.args=-Xmx12g
txn.log.dir=txnlog
iodevices=io0,io1,io2,io3
command=asterixnc
storage.buffercache.pagesize=128KB
storage.buffercache.size=8GB
storage.memorycomponent.globalbudget=1GB
storage.memorycomponent.pagesize=64KB
[nc/1]
address=localhost
[cc]
address=localhost
MongoDB v4.0.3
- MacOS: https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-os-x/
- CentOS: https://www.mongodb.com/docs/v5.0/tutorial/install-mongodb-on-red-hat/
PostgreSQL v9.6
- MacOS: brew install postgresql@9.6
- CentOS: https://wiki.postgresql.org/wiki/YUM_Installation
SparkSQL v2.4.6
-
SparkSQL 2.4.6
Download and Install
wget https://archive.apache.org/dist/spark/spark-2.4.6/spark-2.4.6.tgz
tar zxvf spark-2.4.6.tgz
cd spark-2.4.6/
./build/mvn -DskipTests clean package
Solr v6.0
wget https://archive.apache.org/dist/lucene/solr/6.0.0/solr-6.0.0.tgz
tar zxvf solr-6.0.0.tgz
cd solr-6.0.0/
# Running Solr Server
./solr start -e cloud
# Stop Running Solr Server
./solr stop -all
BigDAWG
-
BigDAWG
Download and Build
cd bigdawg/
mvn clean install -DskipTests
Description of Used Hardware
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 40
On-line CPU(s) list: 0-39
Thread(s) per core: 2
Core(s) per socket: 10
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
Stepping: 1
CPU MHz: 2893.212
CPU max MHz: 3400.0000
CPU min MHz: 1200.0000
BogoMIPS: 4799.55
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,3
Memory: 123G RAM
OS: CentOS Linux release 7.9.2009 (Core)
Supported Native Stores Query Languages
- For each supported native store, you can find the supported fragment of the language in
.g4
file underESTOCADA
compiler componentestocada.compiler.model.{X}
, whereX
is the model/language notation (e.g., aj,pj,sj,rk etc).
Adding New Model/Language Pair
- To add a new model/language pair, you need to do the following steps:
- Declare the relational schema and constraints used to capture the model relationally under
src/main/resources/{NameOfTheModel}
. You can checksrc/main/resources/json
directory for an example. - Provide
{NewModelAnnotation}.properties
file for each model configuration. For example, definingfresh_variable_name_prefix
for the new model. Check existing files for an example. - Create a new package
{NewModelAnnotation}
for the new model/language pair underestocada.compiler.model
. - Add the supported grammar in
.g4
file (We use the ANTLR tool for generating ASTs) underestocada.compiler.model.{NewModelAnnotation}
and implement the encoder of the language. Check existing encoders for an example. - Implement the decoder (translate a relatioanl query into a store's native language) under
estocada.rewriting.decoder.{NewModelAnnotation}
.
ESTOCADA MDD Tutorial
Check out the ESTOCADA MDD Tutorial.
ESTOCADA Quick Actions
Check out the tutorial for some quick actions using ESTOCADA
intermmediate abstraction!
Testing
- We added severl Unit Tests for each mode/language pair (encoder+decoder).
-
estocada.test.framework.EncoderTestSuite
undersrc/test/java/
runs all encoder Unit Tests -
estocada.test.framework.DecoderTestSuite
undersrc/test/java/
runs all decoder Unit Tests
Documentation
cd documentation/
-
ESTOCADA
Code documentation:sites/apidocs/help-doc.html
. ESTOCADA APIs documentation is automatically generated from the code usingmvn javadoc:javadoc
. To read the doc, openhelp-doc.html
using your default browser. ** Detailed Documentation is Work in Progress**
Code Format
# Eclipse IDE:
Step 1: Eclipse > Preferences > Java > Code Style > Formatter > import > ESTOCADACodeFormatProfile.XML
Step 2: Eclipse > Preferences > Java > Editor:
1- Select "Perform the selected actions on save"
2- Select "Format source code" and "Format all lines"
3- Select "Organize imports"
Step 3: Select "Apply and Close"
# IntelliJ IDE:
Step 1: IntelliJ > Preferences > Editor > Code Style > Java > Scheme > Import Scheme > Eclipse XML Profile > ESTOADACodeFormatProfile.XML
Step 2: Select "Apply", then "OK".
Paper Reference
If you use ESTOCADA
, please cite our SIGMOD'19 and PVLDB'20 papers
@inproceedings{DBLP:conf/sigmod/AlotaibiBDMZ19,
author = {Rana Alotaibi and
Damian Bursztyn and
Alin Deutsch and
Ioana Manolescu and
Stamatis Zampetakis},
title = {Towards Scalable Hybrid Stores: Constraint-Based Rewriting to the
Rescue},
booktitle = {Proceedings of the 2019 International Conference on Management of
Data, {SIGMOD} Conference 2019, Amsterdam, The Netherlands, June 30
- July 5, 2019},
pages = {1660--1677},
publisher = {{ACM}},
year = {2019}
}
@article{DBLP:journals/pvldb/AlotaibiCDLMY20,
author = {Rana Alotaibi and
Bogdan Cautis and
Alin Deutsch and
Moustafa Latrache and
Ioana Manolescu and
Yifei Yang},
title = {{ESTOCADA:} Towards Scalable Polystore Systems},
journal = {Proc. {VLDB} Endow.},
volume = {13},
number = {12},
pages = {2949--2952},
year = {2020}
}
Contact
Email: ralotaib@eng.ucsd.edu