CI: test is failing with out-of-order database requests
Test "connect to c-service create, write twice, read and delete" (currently disabled)
in CServiceAdapterTest.kt
pass locally but fails in CI.
CouchDB logs show that requests reach the database out of order, and some may be lost.
In the log extract below, first database creation request is received after requests on objects
and one of the two DELETEs never reaches the DB.
[notice] 2021-03-12T12:30:43.853822Z nonode@nohost <0.4428.0> bc5a0cd878 couchdb:5984 172.17.0.3 admin GET /myapp 404 ok 1049
[notice] 2021-03-12T12:30:43.970418Z nonode@nohost <0.4427.0> e03e5716d9 couchdb:5984 172.17.0.3 admin GET /myapp 404 ok 1226
[notice] 2021-03-12T12:30:44.091078Z nonode@nohost <0.4426.0> 44f3ff30d1 couchdb:5984 172.17.0.3 admin GET /myapp 404 ok 1347
[notice] 2021-03-12T12:30:44.105345Z nonode@nohost <0.4425.0> 646f4cd9c6 couchdb:5984 172.17.0.3 admin GET /myapp 404 ok 1361
[notice] 2021-03-12T12:30:44.124073Z nonode@nohost <0.4424.0> 871e6698e7 couchdb:5984 172.17.0.3 admin GET /myapp 404 ok 1380
[notice] 2021-03-12T12:30:44.132861Z nonode@nohost <0.4423.0> ae188c6ac4 couchdb:5984 172.17.0.3 admin GET /myapp 404 ok 2088
[error] 2021-03-12T12:30:44.276978Z nonode@nohost <0.4439.0> 7e3c97c68d Request to create N=3 DB but only 1 node(s)
[notice] 2021-03-12T12:30:45.773455Z nonode@nohost <0.4439.0> 7e3c97c68d couchdb:5984 172.17.0.3 admin PUT /myapp 201 ok 1629
[notice] 2021-03-12T12:37:19.609300Z nonode@nohost <0.7688.0> ac385d750d couchdb:5984 172.17.0.3 admin GET /myapp 200 ok 2970
[notice] 2021-03-12T12:37:19.673151Z nonode@nohost <0.7687.0> c74d5225c9 couchdb:5984 172.17.0.3 admin GET /myapp 200 ok 3083
[notice] 2021-03-12T12:37:19.737741Z nonode@nohost <0.7686.0> d2d38edfa5 couchdb:5984 172.17.0.3 admin GET /myapp 200 ok 3209
[notice] 2021-03-12T12:37:19.749115Z nonode@nohost <0.7685.0> b1830f63bd couchdb:5984 172.17.0.3 admin GET /myapp 200 ok 3216
[notice] 2021-03-12T12:37:19.759181Z nonode@nohost <0.7684.0> afc77f504c couchdb:5984 172.17.0.3 admin GET /myapp 200 ok 3227
[notice] 2021-03-12T12:37:19.795337Z nonode@nohost <0.7683.0> 8fa3eaf872 couchdb:5984 172.17.0.3 admin GET /myapp 200 ok 3335
[notice] 2021-03-12T12:37:20.339186Z nonode@nohost <0.7757.0> 37a7b71d56 couchdb:5984 172.17.0.3 admin GET /myapp/%7B%22collectionUId%22%3A%22myCollection%22%2C%22type%22%3A%22PNCounter%22%2C%22name%22%3A%22myPNCounter%22%7D 404 ok 108
[notice] 2021-03-12T12:37:20.342950Z nonode@nohost <0.7758.0> 60cacc4b6d couchdb:5984 172.17.0.3 admin GET /myapp/%7B%22collectionUId%22%3A%22myCollection%22%2C%22type%22%3A%22PNCounter%22%2C%22name%22%3A%22myPNCounter%22%7D 404 ok 49
[notice] 2021-03-12T12:37:20.344732Z nonode@nohost <0.7759.0> 1cfe28a036 couchdb:5984 172.17.0.3 admin GET /myapp/%7B%22collectionUId%22%3A%22myCollection%22%2C%22type%22%3A%22PNCounter%22%2C%22name%22%3A%22myPNCounter%22%7D 404 ok 51
[notice] 2021-03-12T12:37:20.345251Z nonode@nohost <0.7761.0> 45a4555e8b couchdb:5984 172.17.0.3 admin GET /myapp/%7B%22collectionUId%22%3A%22myCollection%22%2C%22type%22%3A%22PNCounter%22%2C%22name%22%3A%22myPNCounter%22%7D 404 ok 51
[notice] 2021-03-12T12:37:20.662845Z nonode@nohost <0.7760.0> 2ba1d1e9c0 couchdb:5984 172.17.0.3 admin DELETE /myapp 200 ok 369
[notice] 2021-03-12T12:37:22.739707Z nonode@nohost <0.7838.0> ee957388f2 couchdb:5984 172.17.0.3 admin PUT /myapp/%7B%22collectionUId%22%3A%22myCollection%22%2C%22type%22%3A%22PNCounter%22%2C%22name%22%3A%22myPNCounter%22%7D 404 ok 14
[notice] 2021-03-12T12:37:22.741216Z nonode@nohost <0.7839.0> 668b4774cf couchdb:5984 172.17.0.3 admin PUT /myapp/%7B%22collectionUId%22%3A%22myCollection%22%2C%22type%22%3A%22PNCounter%22%2C%22name%22%3A%22myPNCounter%22%7D 404 ok 3
[notice] 2021-03-12T12:37:22.825167Z nonode@nohost <0.7840.0> 087298f65f couchdb:5984 172.17.0.3 admin GET /myapp 404 ok 2
[notice] 2021-03-12T12:37:22.832241Z nonode@nohost <0.7841.0> 3cc2349507 couchdb:5984 172.17.0.3 admin GET /myapp 404 ok 1
This leads to 404 errors (database does not exists) and some others:
[SERVER] Deleting database: 'myapp'
[SERVER] Creating database: 'myapp'
[SERVER] Getting document '{"collectionUId":"myCollection","type":"PNCounter","name":"myPNCounter"}' in database 'myapp'
[SERVER] Updating document '{"collectionUId":"myCollection","type":"PNCounter","name":"myPNCounter"}' in database 'myapp'
[SERVER] Updating document '{"collectionUId":"myCollection","type":"PNCounter","name":"myPNCounter"}' in database 'myapp'
[SERVER] Getting document '{"collectionUId":"myCollection","type":"PNCounter","name":"myPNCounter"}' in database 'myapp'
[SERVER][ERROR] Failed getting object '{"collectionUId":"myCollection","type":"PNCounter","name":"myPNCounter"}' in database 'myapp'
Error: Database does not exist.
at Request._callback (/builds/concordant/software/c-client/node_modules/nano/lib/nano.js:168:15)
at Request.self.callback (/builds/concordant/software/c-client/node_modules/request/request.js:185:22)
at Request.emit (events.js:315:20)
at Request.<anonymous> (/builds/concordant/software/c-client/node_modules/request/request.js:1154:10)
at Request.emit (events.js:315:20)
at IncomingMessage.<anonymous> (/builds/concordant/software/c-client/node_modules/request/request.js:1076:12)
at Object.onceWrapper (events.js:421:28)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12) {
error: 'not_found',
reason: 'Database does not exist.',
scope: 'couch',
statusCode: 404,
request: {
method: 'GET',
headers: { 'content-type': 'application/json', accept: 'application/json' },
uri: 'http://couchdb:5984/myapp',
qsStringifyOptions: { arrayFormat: 'repeat' }
},
headers: {
uri: 'http://couchdb:5984/myapp',
statusCode: 404,
'cache-control': 'must-revalidate',
connection: 'close',
'content-type': 'application/json',
date: 'Fri, 12 Mar 2021 11:51:23 GMT',
'set-cookie': [
'AuthSession=YWRtaW46NjA0QjU1QkI6s1O5G9j15Iugj1BlJh2OBpDLS8g; Version=1; Expires=Fri, 12-Mar-2021 12:24:43 GMT; Max-Age=2000; Path=/; HttpOnly'
],
'x-couch-request-id': 'a672e72c34',
'x-couchdb-body-time': '0'
},
errid: 'non_200',
description: 'couch returned 404'
}