Commit 7e69065f authored by calocedre TAC's avatar calocedre TAC
Browse files

merge develop into feature/clea/scoring-enh

parents d17e072a 939bc9a8
## CLEA coding guidelines
[[_TOC_]]
## What is Cléa ?
Cluster Exposure Verification (Cléa) is a protocol meant to warn the participants of a private event (e.g., wedding or private party) or the persons present in a commercial or public location (e.g., bar, restaurant, sport center, or train) that became a cluster because a certain number of people who were present at the same time have been tested COVID+.
It is based: (1) on a central server, under the responsibility of an authority (e.g., a health authority), that performs an automatic and centralized cluster detection; (2) on the display a QR code at the location, either in a dynamic manner (e.g., via a dedicated device, smartphone, or tablet) or static manner (e.g., printed); and (3) on a smartphone application used by people present at the location. This smartphone application enables to scan this QR code when entering the location, to store it locally (encrypted) for the next 14 days, and to perform periodic risk analyses, in a decentralized manner, after downloading information about the new clusters.
This protocol is also meant to be used by the location employees in order to warn them if their work place is qualified as cluster, or on the opposite to let them upload information to the server if they are tested COVID+.
This repository hosts a full implementation of the CLEA protocol.
For a full description of the CLEA protocol, see https://gitlab.inria.fr/stopcovid19/CLEA-exposure-verification
## Cléa Architecture
![Cléa Architecture diagram](clea-archi.png "Cléa Architecture")
### Cléa crypto library
This component holds all the logic needed to encode and decode Cléa QR codes as described in the Cléa specification. It is composed of:
- a C encoder that will be used in embedded devices to generate and display dynamic QR codes,
- a JavaScript encoder that will be used to generate QR codes from a web browser,
- a Java decoder that will be used server-side.
There are also a Java encoder used for interoperability testing and a JavaScript decoder for quick checks.
This library is hosted at https://gitlab.inria.fr/stopcovid19/CLEA-exposure-verification/-/tree/master/CLEA-lib
### QR code generation web site
A web site is available for location managers to generate QR codes for their location.
It is available at https://qrcode.tousanticovid.gouv.fr/.
The web site is hosted in a dedicated repository. It uses Cléa crypto library component.
### Report web service
When a TousAntiCovid user is diagnosed Covid+, he/she can upload his/her visited locations while being contagious to the Cléa server to allow cluster detection based on the anonymous data. This information is very important to allow warning of people that were present at the same location and at the same time that he/she was exposed to a risk.
The [clea-ws-rest](clea-ws-rest) is the component that will receive records of infected visits. It will first check if it is a valid report, remove duplicated scans, and then send one message per infected visit that will be consumed by the [clea-venue-consumer](clea-venue-consumer) component.
The Clea venue consumer component is in charge of decoding a QR-code, checking its validity and then to compute the exposure time of the visit to save this information in the database. It uses the Java version of Cléa crypto library.
### Cluster identification and publication batch
At regular intervals, a cluster identification job is triggered to detect clusters from exposed visits registered in the server database. When clusters, if any, are identified, a list of clusters is generated and published so that TousAntiCovid (mobile) apps can check if their user have been exposed to a risk.
This component is hosted in the [clea-batch](clea-batch) folder.
### Scenarios in natural language
This implementation of Cléa also comes with end-to-end scenarios written in natural language allowing to easily express the expected behavior of the Cléa application.
Scenarios are written and executed with the [Cucumber](cucumber.io/) tool. They are hosted in the [scenarios](scenarios) folder.
To enable easy and fast end-to-end testing, 2 components have been written to mimick the behavior of:
- a TousAntiCovid mobile app, hosted in the [clea-client](clea-client) folder
- and a QR code generator allowing to simulate QR code generation for a given location. It is hosted in the [clea-qr-simulator](clea-qr-simulator) folder.
## Cléa coding guidelines
* Contributions are done through Merge Requests on the develop branch (gitflow). MR must be reviewed by another dev than
the code author. a MR can be merged when approved by at least one reviewer.
......
......@@ -13,6 +13,7 @@
<properties>
<java.version>11</java.version>
<org.mapstruct.version>1.4.2.Final</org.mapstruct.version>
</properties>
<dependencies>
......@@ -79,6 +80,12 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.mapstruct</groupId>
<artifactId>mapstruct</artifactId>
<version>1.4.2.Final</version>
<scope>compile</scope>
</dependency>
</dependencies>
<build>
<plugins>
......@@ -88,6 +95,7 @@
<configuration>
<verbose>false</verbose>
<offline>true</offline>
<failOnNoGitDirectory>false</failOnNoGitDirectory>
</configuration>
</plugin>
<plugin>
......@@ -149,7 +157,154 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.7.0</version>
<configuration>
<source>11</source>
<target>11</target>
<annotationProcessorPaths>
<path>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
</path>
<path>
<groupId>org.mapstruct</groupId>
<artifactId>mapstruct-processor</artifactId>
<version>${org.mapstruct.version}</version>
</path>
</annotationProcessorPaths>
<compilerArgs>
<compilerArg>
-Amapstruct.defaultComponentModel=spring
</compilerArg>
</compilerArgs>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>test-bats</id>
<activation>
<property>
<name>test.bats</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-scm-plugin</artifactId>
<version>1.11.2</version>
<executions>
<execution>
<id>clone-bats-core</id>
<goals>
<goal>checkout</goal>
</goals>
<phase>test</phase>
<configuration>
<checkoutDirectory>${project.basedir}/target/bats/core</checkoutDirectory>
<connectionType>connection</connectionType>
<connectionUrl>scm:git:https://github.com/bats-core/bats-core.git</connectionUrl>
</configuration>
</execution>
<execution>
<id>clone-bats-support</id>
<goals>
<goal>checkout</goal>
</goals>
<phase>test</phase>
<configuration>
<checkoutDirectory>${project.basedir}/target/bats/support</checkoutDirectory>
<connectionType>connection</connectionType>
<connectionUrl>scm:git:https://github.com/ztombol/bats-support.git</connectionUrl>
</configuration>
</execution>
<execution>
<id>clone-bats-assert</id>
<goals>
<goal>checkout</goal>
</goals>
<phase>test</phase>
<configuration>
<checkoutDirectory>${project.basedir}/target/bats/assert</checkoutDirectory>
<connectionType>connection</connectionType>
<connectionUrl>scm:git:https://github.com/ztombol/bats-assert.git</connectionUrl>
</configuration>
</execution>
</executions>
<configuration>
<providerImplementations>
<git>jgit</git>
</providerImplementations>
</configuration>
<dependencies>
<dependency>
<groupId>org.apache.maven.scm</groupId>
<artifactId>maven-scm-provider-jgit</artifactId>
<version>1.11.2</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>chmod</id>
<goals>
<goal>exec</goal>
</goals>
<phase>test</phase>
<configuration>
<executable>chmod</executable>
<arguments>
<argument>+x</argument>
<argument>target/bats/core/bin/bats</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>chmod-libexec</id>
<goals>
<goal>exec</goal>
</goals>
<phase>test</phase>
<configuration>
<executable>chmod</executable>
<arguments>
<argument>+x</argument>
<argument>target/bats/core/libexec/bats-core/bats-exec-suite</argument>
<argument>target/bats/core/libexec/bats-core/bats-exec-file</argument>
<argument>target/bats/core/libexec/bats-core/bats-exec-test</argument>
<argument>target/bats/core/libexec/bats-core/bats-preprocess</argument>
<argument>target/bats/core/libexec/bats-core/bats-format-pretty</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>bats</id>
<goals>
<goal>exec</goal>
</goals>
<phase>test</phase>
<configuration>
<executable>target/bats/core/bin/bats</executable>
<arguments>
<argument>src/test/bats</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
\ No newline at end of file
......@@ -4,9 +4,11 @@ export LANG=fr_FR.utf8
export PATH=$PATH:~/.local/bin
SCRIPTPATH="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
cd $SCRIPTPATH
CLEA_BATCH_CLUSTER_OUTPUT_PATH=${CLEA_BATCH_CLUSTER_OUTPUT_PATH:-/tmp/v1}
mkdir -p ${CLEA_BATCH_CLUSTER_OUTPUT_PATH}
export FLASK_APP=web.py
export FLASK_ENV=development
......
package fr.gouv.clea;
import fr.gouv.clea.config.BatchProperties;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.domain.EntityScan;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.transaction.annotation.EnableTransactionManagement;
@SpringBootApplication
@EntityScan
@EnableBatchProcessing
@EnableBatchProcessing
@EnableTransactionManagement
@EnableConfigurationProperties(BatchProperties.class)
public class CleaBatchApplication {
public static void main(String[] args) {
SpringApplication.run(CleaBatchApplication.class, args);
}
public static void main(String[] args) {
SpringApplication.run(CleaBatchApplication.class, args);
}
}
package fr.gouv.clea.config;
import static fr.gouv.clea.config.BatchConstants.CLUSTERMAP_JOB_CONTEXT_KEY;
import static fr.gouv.clea.config.BatchConstants.EXPOSED_VISITS_TABLE;
import static fr.gouv.clea.config.BatchConstants.LTID_COLUMN;
import static fr.gouv.clea.config.BatchConstants.PERIOD_COLUMN;
import static fr.gouv.clea.config.BatchConstants.TIMESLOT_COLUMN;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import javax.sql.DataSource;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.core.listener.ExecutionContextPromotionListener;
import org.springframework.batch.core.partition.PartitionHandler;
import org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler;
import org.springframework.batch.item.database.Order;
import org.springframework.batch.item.database.PagingQueryProvider;
import org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.task.SimpleAsyncTaskExecutor;
import org.springframework.core.task.TaskExecutor;
import fr.gouv.clea.dto.SinglePlaceCluster;
import fr.gouv.clea.dto.SinglePlaceExposedVisits;
import fr.gouv.clea.identification.ExposedVisitPartitioner;
import fr.gouv.clea.identification.SinglePlaceExposedVisitsItemWriter;
import fr.gouv.clea.identification.SinglePlaceExposedVisitsProcessor;
import fr.gouv.clea.identification.reader.ExposedVisitItemReader;
import fr.gouv.clea.identification.reader.SinglePlaceExposedVisitItemReader;
import fr.gouv.clea.indexation.model.output.ClusterFile;
import fr.gouv.clea.indexation.processors.IndexationProcessor;
import fr.gouv.clea.indexation.readers.IndexationReader;
import fr.gouv.clea.indexation.writers.IndexationWriter;
import fr.gouv.clea.scoring.configuration.risk.RiskConfiguration;
@Configuration
public class BatchConfig {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Autowired
private RiskConfiguration riskConfiguration;
@Autowired
private BatchProperties properties;
@Autowired
private DataSource dataSource;
@Bean
public Job identificationPartitionedJob(Step identificationPartitionedMasterStep) {
return this.jobBuilderFactory.get("partitionedJob")
.incrementer(new RunIdIncrementer())
.start(identificationPartitionedMasterStep)
.next(clusterIndexation())
.build();
}
@Bean
public Step identificationPartitionedMasterStep(PartitionHandler partitionHandler) {
return this.stepBuilderFactory.get("identification-partitioned-step-master")
.partitioner("partitioner", partitioner())
.partitionHandler(partitionHandler)
.build();
}
@Bean
public Step clusterIndexation() {
return stepBuilderFactory.get("clusterIndexation")
.<List<SinglePlaceCluster>, HashMap<String, ClusterFile>> chunk(1)
.reader(new IndexationReader())
.processor(new IndexationProcessor(properties))
.writer(new IndexationWriter(properties))
.build();
}
@Bean
public Step identificationStepWorker(ExposedVisitItemReader exposedVisitItemReader) {
final SinglePlaceExposedVisitItemReader reader = new SinglePlaceExposedVisitItemReader();
reader.setDelegate(exposedVisitItemReader);
return stepBuilderFactory.get("identification-step-worker")
.listener(promotionListener())
.<SinglePlaceExposedVisits, SinglePlaceCluster>chunk(properties.getChunkSize())
.reader(reader)
.processor(new SinglePlaceExposedVisitsProcessor(properties, riskConfiguration))
.writer(new SinglePlaceExposedVisitsItemWriter())
.build();
}
@Bean
public TaskExecutorPartitionHandler partitionHandler(TaskExecutor taskExecutor, Step identificationStepWorker) {
TaskExecutorPartitionHandler partitionHandler = new TaskExecutorPartitionHandler();
partitionHandler.setGridSize(properties.getGridSize());
partitionHandler.setStep(identificationStepWorker);
partitionHandler.setTaskExecutor(taskExecutor());
return partitionHandler;
}
@Bean
public ExposedVisitPartitioner partitioner() {
return new ExposedVisitPartitioner(this.dataSource);
}
@Bean
@StepScope
public ExposedVisitItemReader exposedVisitItemReader(PagingQueryProvider pagingQueryProvider,
@Value("#{stepExecutionContext['ltid']}") UUID ltid) {
return new ExposedVisitItemReader(this.dataSource, pagingQueryProvider, Map.of("ltid", ltid));
}
@Bean
@StepScope
public SqlPagingQueryProviderFactoryBean pagingQueryProvider(@Value("#{stepExecutionContext['ltid']}") UUID ltid) {
SqlPagingQueryProviderFactoryBean factoryBean = new SqlPagingQueryProviderFactoryBean();
factoryBean.setDataSource(dataSource);
factoryBean.setSelectClause("*");
factoryBean.setFromClause(EXPOSED_VISITS_TABLE);
factoryBean.setWhereClause(LTID_COLUMN + " = :ltid");
factoryBean.setSortKeys(Map.of(PERIOD_COLUMN, Order.ASCENDING, TIMESLOT_COLUMN,Order.ASCENDING));
return factoryBean;
}
@Bean
public ExecutionContextPromotionListener promotionListener() {
ExecutionContextPromotionListener listener = new ExecutionContextPromotionListener();
listener.setKeys(new String[] { CLUSTERMAP_JOB_CONTEXT_KEY });
return listener;
}
@Bean
TaskExecutor taskExecutor() {
return new SimpleAsyncTaskExecutor("batch-ident");
}
}
......@@ -5,11 +5,32 @@ import lombok.experimental.UtilityClass;
@UtilityClass
public class BatchConstants {
public static final String LTID_COL = "ltid";
public static final String VENUE_TYPE_COL = "venue_type";
public static final String VENUE_CAT1_COL = "venue_category1";
public static final String VENUE_CAT2_COL = "venue_category2";
public static final String PERIOD_START_COL = "period_start";
public static final String FIRST_TIMESLOT_COL = "first_timeslot";
public static final String LAST_TIMESLOT_COL = "last_timeslot";
public static final String CLUSTER_START_COL = "cluster_start";
public static final String CLUSTER_DURATION_COL = "cluster_duration_in_seconds";
public static final String RISK_LEVEL_COL = "risk_level";
public static final String CLUSTER_INDEX_FILENAME = "clusterIndex.json";
public static final String JSON_FILE_EXTENSION = ".json";
public static final String PREFIXES_PARTITION_KEY = "prefixes";
public static final String LTIDS_LIST_PARTITION_KEY = "ltids";
// SQL properties
public static final String EXPOSED_VISITS_TABLE = "exposed_visits";
public static final String LTID_COLUMN = "ltid";
public static final String SINGLE_PLACE_CLUSTER_PERIOD_TABLE = "cluster_periods";
public static final String PERIOD_COLUMN = "period_start";
public static final String TIMESLOT_COLUMN = "timeslot";
public static final String LTID_PARAM = "ltid";
public static final String CLUSTERMAP_JOB_CONTEXT_KEY = "clusterMap";
// JDBC SQL Queries
public static final String SQL_SELECT_BY_LTID_IN_SINGLEPLACECLUSTERPERIOD = "select * from " + SINGLE_PLACE_CLUSTER_PERIOD_TABLE + " WHERE ltid= ?";
public static final String SQL_SELECT_DISTINCT_LTID_FROM_EXPOSEDVISITS = "select distinct " + LTID_COL + " from " + EXPOSED_VISITS_TABLE + " order by " + LTID_COL;
public static final String SQL_SELECT_DISTINCT_FROM_CLUSTERPERIODS_ORDERBY_LTID = "select distinct " + LTID_COL + " from " + SINGLE_PLACE_CLUSTER_PERIOD_TABLE + " ORDER BY " + LTID_COL;
public static final String SQL_SELECT_FROM_EXPOSEDVISITS_WHERE_LTID_ORDERBY_PERIOD_AND_TIMESLOT = "select * from " + EXPOSED_VISITS_TABLE + " WHERE ltid= ? ORDER BY " + PERIOD_COLUMN + ", " + TIMESLOT_COLUMN;
public static final String SQL_TRUNCATE_TABLE_CLUSTERPERIODS = "truncate table " + SINGLE_PLACE_CLUSTER_PERIOD_TABLE + ";";
}
package fr.gouv.clea.config;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.springframework.boot.context.properties.ConfigurationProperties;
@Getter
@Setter
@ToString
@Component
@NoArgsConstructor
@ConfigurationProperties(prefix = "clea.batch.cluster")
public class BatchProperties {
/**
* Duration unit of a timeSlot
*/
@Value("${clea.batch.duration-unit-in-seconds}")
public int durationUnitInSeconds;
private int durationUnitInSeconds;
@Value("${clea.batch.cluster.files-output-path}")
public String clusterFilesOutputPath;
private String filesOutputPath;
@Value("${clea.batch.cluster.static-prefix-length}")
public int prefixLength;
private int staticPrefixLength;
@Value("${clea.batch.cluster.grid-size}")
private int gridSize;
@Value("${clea.batch.cluster.chunk-size}")
private int chunkSize;
private int identificationStepChunkSize;
private int indexationStepChunkSize;
private int prefixesComputingStepChunkSize;
}
package fr.gouv.clea.config;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class CleaBatchJobConfig {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Bean
public Job cleaBatchJob(Step clusterIdentification, Step clustersIndexation, Step prefixesComputing, Step purgeIntermediateTable, Step clusterIndexGeneration) {
return this.jobBuilderFactory.get("clea-batch-job")
.incrementer(new RunIdIncrementer())
.start(purgeIntermediateTable)
.next(clusterIdentification)
.next(prefixesComputing)
.next(clustersIndexation)
.next(clusterIndexGeneration)
.build();
}
}
package fr.gouv.clea.config;
import com.fasterxml.jackson.databind.ObjectMapper;
import fr.gouv.clea.indexation.index.GenerateClusterIndexTasklet;
import fr.gouv.clea.service.PrefixesStorageService;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class ClusterIndexGenerationStepBatchConfig {
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Autowired
private PrefixesStorageService prefixesStorageService;
@Autowired
private BatchProperties batchProperties;
@Autowired
private ObjectMapper objectMapper;
@Bean
public Step clusterIndexGeneration() {
return stepBuilderFactory.get("clusterIndexGeneration")
.tasklet(generateClusterIndex())
.build();
}
@Bean
public Tasklet generateClusterIndex() {
return new GenerateClusterIndexTasklet(batchProperties, prefixesStorageService, objectMapper);
}
}
package fr.gouv.clea.config;
import static fr.gouv.clea.config.BatchConstants.SQL_SELECT_DISTINCT_LTID_FROM_EXPOSEDVISITS;
import java.util.List;
import javax.sql.DataSource;