Mentions légales du service

Skip to content

multiprocessing issue in preprocessing

Just launching :

dnadna preprocess BactDemo/BactDemo_training_config.yml With loader_num_workers set to 10, or even to 1, on titanic or on my laptop with the current master branch, it takes ages to preprocess the dataset.

The reason is (htop from titanic):

Capture_d_écran_de_2020-10-09_00-19-44

with a load of least 5 times (it keeps increasing) as big as its maximum (32 for 32 cpus), there is something clearly wrong there.

Unfortunately, nothing obvious could explain this after a quick look into the code.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information