Attention une mise à jour du service Gitlab va être effectuée le mardi 30 novembre entre 17h30 et 18h00. Cette mise à jour va générer une interruption du service dont nous ne maîtrisons pas complètement la durée mais qui ne devrait pas excéder quelques minutes. Cette mise à jour intermédiaire en version 14.0.12 nous permettra de rapidement pouvoir mettre à votre disposition une version plus récente.

Commit d878c89c authored by Jujube Orange's avatar Jujube Orange
Browse files

Merge branch 'feature/TAC-710' into 'develop'

Documentation clea-ws & clea-consumer (TAC-710, TAC-711)

See merge request !25
parents a70f2bfa 45c6d648
Pipeline #326943 canceled with stages
in 39 minutes and 6 seconds
## CLEA Venue Consumer
Listener meant to retrieve decoded visits from the kafka queue anonymously, and calculating expositions.
Listener intended to anonymously retrieve decoded visits from the kafka queue and calculate exposures.
In addition, this module processes statistics on reports issued by clea-ws and generates its own statistics by location type.
Statistics are stored on an ELK stack.
### Visit inputs
The structure of a visit received from Kafka is:
- qrCodeScanTime: scan date as NTP Timestamp
- isBackward: A visit can be backward (scanned before pivot/reference date) or forward (scanned after pivot date)
- version: (internally used by the decoder library)
- type: (internally used by the decoder library)
- locationTemporaryPublicId: UUID as string
- encryptedLocationMessage: an array of bytes to be decrypted
An EncryptedLocationMessage is decrypted as :
- staff: The qrcode was generate for a staff (with longueur exposure time)
- qrCodeRenewalIntervalExponentCompact: 2^n seconds of validity of this qrCode before the need to create a new one with same UUID, new qrCodeValidityStartTime
- compressedPeriodStartTime: stored validity date as Hour (periodStartTimeAsNtpTimestamp = compressedPeriodStartTime\*3600)
- locationTemporarySecretKey: used to check the validity of unencrypted locationTemporaryPublicId
- encryptedLocationContactMessage: currently ignored, can only be decrypted by MSS
- venueType: used to determine the number of exposure period (slots) to consider, used for statistics
- venueCategory1: used to determine the number of exposure period (slots) to consider, used for statistics
- venueCategory2: used to determine the number of exposure period (slots) to consider, used for statistics
The LocationSpecificPart class combines both structures
### Verifications
For each record read from the kafka queue, the visit will be decrypted using the configured secret key and the following
verifications will be applied:
- if there's an error while decrypting a specific visit, it will be rejected.
- if there an error happens while decrypting a specific visit, it will be rejected.
- if
the [drift check]("https://gitlab.inria.fr/stopcovid19/CLEA-exposure-verification/-/blob/master/documents/CLEA-specification-EN.md#processing-of-a-user-location-record-by-the-backend-server")
the [drift check]("https://hal.inria.fr/hal-03146022v3/document#processing-of-a-user-location-record-by-the-backend-server")
fails, it will be rejected.
- if the temporary public if of the location, is different from the one calculated from the location specific part, it
will be rejected.
- if the public location temporary id is different from the one calculated from the location specific part, it
will be rejected (see [hasValidTemporaryLocationPublicId]("src/main/java/fr/gouv/clea/consumer/service/impl/DecodedVisitService.java)).
### Exposition slots calculation
......@@ -23,6 +50,55 @@ specs https://gitlab.inria.fr/stopcovid19/backend-server/-/blob/clea-doc/documen
### Purge
Each day, at 01:00 PM UTC (configurable), a croned job will launch to purge all outdated entries from DB.
Each day, at 01:00 PM UTC (configurable), a scheduled job will launch to purge all outdated entries from DB.
Outdated entries are processed following the retention date (14 days).
### Statistics
#### By Reports
Stats pushed by clea-ws in a specific topic are moved without transformation to ElasticSearch.
If the transfert fails (ElasticSearch not available), clea-ws constantly tries to transfer it again.
#### By Location
Each valid decrypted visit is recorded in ElasticSearch (without locationTemporaryPublicId).
Document sent to ElasticSearch is:
- qrCodeScanTime : scan date truncated per statistic slot duration (property clea.conf.statSlotDurationInSeconds)
A statistic Slot can be different from a cluster slot.
- venueType (int): a location type (ex restaurant)
- venueCategory1: a caracteristic for the location (ex number of client)
- venueCategory2: a caracteristic for the location (ex surface)
- backward: Number of backward visits received for the same period and location type (type,categ1/categ2)
- forward: Number of forward visits received for the same period and location type
After 14 days (retention date), we save an aggregate sum of backward and forward visits per period (1/2 hours),
and the detailed documents are removed from Elasticsearch.
If a visit cannot be pushed to ElasticSearch (ElasticSearch is not available), a failback pushes the document to an error topic.
A scheduled job may try to process this topic periodically.
### Run
This module is an uber-jar. You can use this command to execute it:
```bash
java -jar target/clea-venue-consumer*-exec.jar
```
This module can be executed by maven:
```bash
mvn spring-boot:run
```
the docker-compose.yml at the root of clea-server execute this service
```bash
$ docker-compose up
#or
$ docker-compose up clea-venue-consumer
```
......@@ -13,26 +13,113 @@ When a report is sent, the following verifications are applied to the request:
### Reporting verifications:
Pivot Date must be contained between the retention date and today, if not it will be set to the retention Date.
Pivot Date must be between the retention date (today minus 14 days by default) and today, or else it will be set to the retention Date.
After the validity of the report is checked, all visits are decoded, and the following verifications are applied:
- if there's an error while decoding a specific visit, it will be individually purged.
- if there's an error while decoding a specific visit, it will be individually purged (decoding=extract the different fields without decryption).
- if a visit has its scan time before the retention date, it will be considered as outdated and rejected individually.
- if a visit has its scan time after today, it will be considered in future and rejected individually.
- Multiple visits for the same location(based on temporary id) will be considered as duplicates, if their qr scan time
is withing the configured duplicate scan threshold. Only 1 is kept.
- Multiple visits to the same location (based on temporary id) will be considered duplicates, if their qr scan time
is withing the configured duplicate scan threshold (half an hour by example). Only 1 is kept.
### Publishing reports:
After all verifications are applied, the decoded visits are sent to a kafka queue, to be processed by the next chain.
### Data Models:
When a user scan an image (qrcode), the mobile application receives an URL with this form:
[https://tac.gouv.fr?v=0#AK4e..u6k]("https://tac.gouv.fr?v=0#AK4ebl4J7VLSQh5hCBQaim7yghhwBGoW8bVgg3iNYKcPwDMFgnFo_DVChp5KZAqyoI85nwMtt5iSLhYp9EM1JZvBP1FmnoPbxSiKyCADPP63hs4_Qa1YbzvcVoe8fU8L3G8nFQvlfOx3XJ3fu6k")
The anchor part (AK...u6k) is stored on the phone with the scan date. This part is also called LocationSpecificPart.
When a user declares himself positive, he completes the date of the first symptom or the date of his positive test.
the mobile application determines a pivot date of contagiousness.
The structure received by this module is:
- pivotDateAsNtpTimestamp : pivotDate as NTP Timestamp. It pivot date is less than now - property clea.conf.retentionDurationInDays days, then
the pivot date used to detect backward or forward visits will be (now - property clea.conf.retentionDurationInDays days)
- An array of visits composed of:
. qrCodeScanTime : date of scan as NTP Timestamp
. qrCode: a LocationSpecificPart
When a LocationSpecificPart is decoded (not decrypted) the new structure is:
LocationSpecificPart:
- version: (=0) for future uses
- type: (=0/1) : synchronous/asynchronous scan. Controls on dates only manage synchronous (0) scans.
- locationTemporaryPublicId: a UUID representing a location
- encryptedLocationMessage: a binary structure that needs a private key to be decrypted.
The data model pushed to Kafka is:
- qrCodeScanTime: scan date as NTP Timestamp
- isBackward: boolean, true if qrCodeScanTime < pivotDate
- version: 0
- type: 0
- locationTemporaryPublicId: UUID as string
- encryptedLocationMessage: array of bytes
### Statistiques
This module a row of statistics for each report to a dedicated Kafka topic.
the data model used for each report is:
- timestamp (long): time the report was received as an NTP Timestamp.
- reported (int): number of visits reported in the report
- rejected (int): the causes of the rejections are:
- outdate : number of days between now and visit.qrCodeScanTime > property clea.conf.retentionDurationInDays (14 days by default)
- future: visit.qrCodeScanTime > now.
- duplicated scan: For the same LocationId, a second scan take place before property clea.conf.duplicateScanThresholdInSeconds (1800 sec/ 30 minutes)
- backwards (int): visit.qrCodeScanTime < pivotDate
- forwards (int): visit.qrCodeScanTime >= pivotDate
- close (int): count of non-rejected visits, those with a different locationTemporaryPublicId and a scan date interval less than the duration of a slot (property clea.conf.exposureTimeUnitInSeconds 1800sec/30 min by default)
### Security
In production, this module expects to receive a valid JWT token generated by the Robert-Server component.
The mobile application authenticates against Robert server by doing a "/status" call which returns a token used to report positivity to Robert server.
The same token can be used to report to Clea.
### Build
This module is part of Clea-server project.
Build clea-server project to build all the modules.
### Run
This module is an uber-jar. You can use this command to execute it:
```bash
java -jar target/clea-ws-rest*-exec.jar
```
This module can be executed by maven:
```bash
$ docker build -t tac/clea-ws-rest:latest .
$ docker run -it --rm --name clea-ws-rest -p 8080:8080 clea-ws-rest:latest --spring.profiles.active=dev,docker
```
the docker-compose.yml at the root of clea-server execute this service
```bash
mvn spring-boot:run
$ docker-compose up
#or
$ docker-compose up clea-ws-rest
```
Display the Clea API and use it: http://localhost:8088/swagger-ui/
Display the Clea API and use it: http://localhost:8080/swagger-ui/
Display the Clea OpenAPI generated from the code: http://localhost:8080/v3/api-docs?group=clea
### Dependencies
Display the Clea OpenAPI generated from the code: http://localhost:8088/v3/api-docs?group=clea
This module can start standalone but requires a running Kafka cluster to function.
The property "spring.kafka.bootstrap-servers" must point to a kafka server.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment