Work with another dataset: Training: TypeError
I am going to spam issues to have ourselves a memory of common log errors and solutions for future documentation.
files:
cattle_000_training_config.yml
cattle_000_preprocessed_params.csv
Setup:
data_root: /home/pjobic/Work/DNADNA/master/cattle
Format: data_root/scenario_XXX/cattle_NNN_NN.npz
dataset_path_on_titanic: titanic:/home/tau/thsanche/data/cattle/scenario_{0_to_99}
working directory: /home/pjobic/Work/DNADNA/master
Issue
I am trying to work with dnadna
with another dataset:
Here are the steps I did:
Command: dnadna simulation init cattle
=> Then, modified cattle/cattle_simulation_config.yml (find in attached file)
Command: dnadna init --simulation-config=cattle/cattle_simulation_config.yml
=> Then, modified cattle/cattle_training_config.yml (find in attached file)
Command: dnadna --debug preprocess cattle/cattle_training_config.yml
Command: dnadna --debug train cattle/run_000/cattle_000_training_config.yml
TypeError: __init__() missing 1 required positional argument: 'msg'
Notice: I print a lot of stuff for debugging
dnadna --debug train cattle/run_000/cattle_000_training_config.yml
trainng_seires: scenario_idx
0 True
1 True
2 True
3 True
4 True
...
95 True
96 True
97 True
98 True
99 True
Name: training_set, Length: 100, dtype: bool
20/05/2020 15:24:27; INFO; 100 samples in the validation set and 9900 in the training set
training_loader: <torch.utils.data.dataloader.DataLoader object at 0x7f2f4e1ec790>
scenario_idx: 0
input_val: tensor([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 1, ..., 0, 0, 0],
...,
[1, 0, 1, ..., 0, 1, 0],
[0, 0, 1, ..., 1, 0, 0],
[0, 0, 0, ..., 0, 0, 1]])
target tensor([0.6654, 1.1194, 1.6442, 1.6025, 1.3646, 1.2572, 0.5795, 0.9207, 1.3488,
1.2579, 1.6739, 1.1383, 1.4908, 1.5662, 1.7496, 1.1881, 1.3855, 1.3249,
1.1209, 0.7589, 1.0839], dtype=torch.float64)
<torch.utils.data.dataloader.DataLoader object at 0x7f2f4e1ec790>
<torch.utils.data.dataloader.DataLoader object at 0x7f2f4e1ec790>
20/05/2020 15:24:27; INFO; Start training
20/05/2020 15:24:27; INFO; Starting Epoch #0
test 0
scenario_idx:
0
input_val:
tensor([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 1, ..., 0, 0, 0],
...,
[1, 0, 1, ..., 0, 1, 0],
[0, 0, 1, ..., 1, 0, 0],
[0, 0, 0, ..., 0, 0, 1]])
target
tensor([0.6654, 1.1194, 1.6442, 1.6025, 1.3646, 1.2572, 0.5795, 0.9207, 1.3488,
1.2579, 1.6739, 1.1383, 1.4908, 1.5662, 1.7496, 1.1881, 1.3855, 1.3249,
1.1209, 0.7589, 1.0839], dtype=torch.float64)
test enumerate:
(0, (0, tensor([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 1, ..., 0, 0, 0],
...,
[1, 0, 1, ..., 0, 1, 0],
[0, 0, 1, ..., 1, 0, 0],
[0, 0, 0, ..., 0, 0, 1]]), tensor([0.6654, 1.1194, 1.6442, 1.6025, 1.3646, 1.2572, 0.5795, 0.9207, 1.3488,
1.2579, 1.6739, 1.1383, 1.4908, 1.5662, 1.7496, 1.1881, 1.3855, 1.3249,
1.1209, 0.7589, 1.0839], dtype=torch.float64)))
epoch 0/1: 0%| | 0/9900 [00:00<?, ?batch/s]
an unexpected error occurred: __init__() missing 1 required positional argument: 'msg'; run again with --debug to view the full traceback | 0/9900 [00:00<?, ?batch/s]
an unexpected error occurred: __init__() missing 1 required positional argument: 'msg'; run again with --debug to view the full traceback
Traceback (most recent call last):
File "/home/pjobic/anaconda3/envs/dnadna/bin/dnadna", line 11, in <module>
load_entry_point('dnadna', 'console_scripts', 'dnadna')()
File "/home/pjobic/Work/DNADNA/master/dnadna/utils/__init__.py", line 820, in main
raise exc
File "/home/pjobic/Work/DNADNA/master/dnadna/utils/__init__.py", line 812, in main
ret2 = cls.run_subcommand(args)
File "/home/pjobic/Work/DNADNA/master/dnadna/utils/__init__.py", line 781, in run_subcommand
return command_cls.main(command[1:], namespace=args)
File "/home/pjobic/Work/DNADNA/master/dnadna/utils/__init__.py", line 820, in main
raise exc
File "/home/pjobic/Work/DNADNA/master/dnadna/utils/__init__.py", line 804, in main
ret = cls.run(args)
File "/home/pjobic/Work/DNADNA/master/dnadna/training.py", line 669, in run
model_trainer.run_training()
File "/home/pjobic/Work/DNADNA/master/dnadna/training.py", line 404, in run_training
best_loss = self.train()
File "/home/pjobic/Work/DNADNA/master/dnadna/training.py", line 386, in train
return best_loss
File "/home/pjobic/Work/DNADNA/master/dnadna/utils/__init__.py", line 403, in __exit__
raise exc_value
File "/home/pjobic/Work/DNADNA/master/dnadna/training.py", line 376, in train
best_loss = self._train_outer_loop(bar)
File "/home/pjobic/Work/DNADNA/master/dnadna/training.py", line 436, in _train_outer_loop
for batch, data in enumerate(self.training_loader):
File "/home/pjobic/anaconda3/envs/dnadna/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 819, in __next__
return self._process_data(data)
File "/home/pjobic/anaconda3/envs/dnadna/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/pjobic/anaconda3/envs/dnadna/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
raise self.exc_type(msg)
TypeError: __init__() missing 1 required positional argument: 'msg'
Problem Partially Solved:
line 436, in _train_outer_loop
for batch, data in enumerate(self.training_loader):
type(self.training_loader) = torch.utils.data.dataloader.DataLoader
Replaced by:
line 436, in _train_outer_loop
for batch, data in enumerate(self.training_loader.**dataset**):
Looks like it outputs what we want (i.e. batch, scenario_idx, inputs, targets).
However, I don't know why I couldn't enumerate
over self.training_loader
(Notice: it is slightly different from torch.utils.data.DataLoader
, maybe it has to do with it)