swarm resources not freed
(to be investigated)
We observed old terminated jobs that remain in the swarm, even when the prod-controller is restarted. This means that the controller is unaware of these jobs and considers the resources available. The most likely explanation is that the job was destroyed in the db while being executed (maybe through the REST API, because the job destroy button is broken).
This is very likely a duplicate of #108 (closed).
Update: the jobs are still present in the db, it is definitely not related to #108 (closed)
root@worker0:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9320ae9f8919 cargo.irisa.fr:8000/allgo/prod/webapp/gatbcompiler:1.0 "/bin/sh -c '\n ..." 33 minutes ago Up 33 minutes 22/tcp prod-job-9188-gatbcompiler
8edbab79c05a swarm:1.2.6 "/swarm join --add..." 12 hours ago Up 12 hours 2375/tcp prod-swarm-node
274d849ad015 allgo/swarm-proxy "socat TCP-LISTEN:..." 12 hours ago Up 12 hours prod-swarm-proxy
17d89c96ec05 cargo.irisa.fr:8000/allgo/prod/webapp/massiccc:1.0 "/bin/sh -c '\n ..." 4 days ago Exited (255) 2 days ago 22/tcp prod-job-8725-massiccc
1d892b44bc32 6cf910dfd599 "/bin/sh -c '\n ..." 9 days ago Exited (255) 2 days ago 22/tcp prod-job-8619-gatbcompiler
root@worker1:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e7443b8c5867 cargo.irisa.fr:8000/allgo/prod/webapp/gatbcompiler:1.0 "/bin/sh -c '\n ..." 29 minutes ago Up 29 minutes 22/tcp prod-job-9185-gatbcompiler
d3469eddc570 cargo.irisa.fr:8000/allgo/prod/webapp/gatbcompiler:1.0 "/bin/sh -c '\n ..." 31 minutes ago Up 31 minutes 22/tcp prod-job-9179-gatbcompiler
b54ec624329c swarm:1.2.6 "/swarm join --add..." 12 hours ago Up 12 hours 2375/tcp prod-swarm-node
510beace80ff allgo/swarm-proxy "socat TCP-LISTEN:..." 12 hours ago Up 12 hours prod-swarm-proxy
d644bcf2a1b3 cargo.irisa.fr:8000/allgo/prod/webapp/massiccc:1.0 "/bin/sh -c '\n ..." 7 days ago Exited (137) 2 days ago prod-job-8691-massiccc
ce3d92c1d85e 6cf910dfd599 "/bin/sh -c '\n ..." 9 days ago Exited (137) 2 days ago prod-job-8617-gatbcompiler