Bug with the transform ReformatPosition
If I have in my training config file the following :
dataset_transforms:
- reformat_position:
distance: true
- Crop:
max_snp: 400
max_indiv: 50
keep_polymorphic_only: true
- snp_format: concat
- validate_snp:
uniform_shape: false
and I run dnadna train config
I got:
dnadna train SPIBAC_models/SPIBAC_models_training_config.yml --overwrite
2021-10-29 18:51:12; ERROR; RefResolutionError: no known deserializer for the given filename: "////home/tau/jcury/DNADNA_project/dnadna/dnadna/schemas/"
2021-10-29 18:51:12; ERROR; run again with --debug to view the full traceback, or with --pdb to drop into a debugger
long version, with --debug
dnadna train SPIBAC_models/SPIBAC_models_training_config.yml --overwrite --debug
2021-10-29 18:52:06; DEBUG; loaded built-in plugin pluggable:network (<class 'dnadna.nets.Network'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin network:spidna (<class 'dnadna.nets.SPIDNA'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin network:custom_cnn (<class 'dnadna.nets.CustomCNN'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin network:mlp (<class 'dnadna.nets.MLP'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin network:cnn (<class 'dnadna.nets.CNN'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin pluggable:optimizer (<class 'dnadna.optim.Optimizer'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:adadelta (<class 'dnadna.optim.Adadelta'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:adagrad (<class 'dnadna.optim.Adagrad'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:adam (<class 'dnadna.optim.Adam'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:adam_w (<class 'dnadna.optim.AdamW'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:sparse_adam (<class 'dnadna.optim.SparseAdam'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:adamax (<class 'dnadna.optim.Adamax'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:asgd (<class 'dnadna.optim.ASGD'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:sgd (<class 'dnadna.optim.SGD'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:rprop (<class 'dnadna.optim.Rprop'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:rm_sprop (<class 'dnadna.optim.RMSprop'>)
2021-10-29 18:52:06; DEBUG; loaded built-in plugin optimizer:lbfgs (<class 'dnadna.optim.LBFGS'>)
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.nets (modified 2021-09-29 14:15:40.822448) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.optim (modified 2021-07-02 16:16:54.399747) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.simulator (modified 2021-09-28 16:43:36.574327) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.transforms (modified 2021-10-28 23:10:51.818095) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.nets (modified 2021-09-29 14:15:40.822448) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.optim (modified 2021-07-02 16:16:54.399747) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.simulator (modified 2021-09-28 16:43:36.574327) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.transforms (modified 2021-10-28 23:10:51.818095) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.nets (modified 2021-09-29 14:15:40.822448) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.optim (modified 2021-07-02 16:16:54.399747) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.simulator (modified 2021-09-28 16:43:36.574327) providing:
2021-10-29 18:52:06; DEBUG; loaded plugin dnadna.transforms (modified 2021-10-28 23:10:51.818095) providing:
2021-10-29 18:52:06; ERROR; RefResolutionError: no known deserializer for the given filename: "////home/tau/jcury/DNADNA_project/dnadna/dnadna/schemas/"
Traceback (most recent call last):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 774, in resolve_from_url
document = self.store[url]
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_utils.py", line 22, in __getitem__
return self.store[self.normalize(uri)]
KeyError: ''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/serializers.py", line 312, in load
serializer = cls.serializer_for(filename)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/serializers.py", line 245, in serializer_for
f'no known serializer for the given filename: "{filename}"')
NotImplementedError: no known serializer for the given filename: "////home/tau/jcury/DNADNA_project/dnadna/dnadna/schemas/"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 777, in resolve_from_url
document = self.resolve_remote(url)
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 856, in resolve_remote
result = self.handlers[scheme](uri)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 438, in _resolver_fallback_handler
return cls._resolver_file_handler(urljoin(base, uri))
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 454, in _resolver_file_handler
return DictSerializer.load(filename)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/serializers.py", line 315, in load
f'no known deserializer for the given filename: "{filename}"')
NotImplementedError: no known deserializer for the given filename: "////home/tau/jcury/DNADNA_project/dnadna/dnadna/schemas/"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/cli.py", line 254, in main
ret = cls.run(args)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/cli/train.py", line 42, in run
args.config, progress_bar=True)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/training.py", line 140, in from_config_file
return cls(config=config, validate=True, progress_bar=progress_bar)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/training.py", line 120, in __init__
super().__init__(config=config, validate=validate)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/config.py", line 1322, in __init__
self.validate_config(config)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/config.py", line 1377, in validate_config
config.validate(schema=self.config_schema)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/config.py", line 1088, in validate
validator.validate(self)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/config.py", line 1579, in validate
return super().validate(config, *args, **kwargs)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 221, in validate
key=self.relevance_with_const_select)
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/exceptions.py", line 367, in best_match
best = next(errors, None)
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 315, in allOf
for error in validator.descend(instance, subschema, schema_path=index):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 592, in validate_config_properties
schema))
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 286, in properties
schema_path=property,
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 263, in ref
for error in validator.descend(instance, resolved):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 337, in oneOf
errs = list(validator.descend(instance, subschema, schema_path=index))
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 263, in ref
for error in validator.descend(instance, resolved):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 81, in items
for error in validator.descend(item, items, path=index):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 263, in ref
for error in validator.descend(instance, resolved):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 337, in oneOf
errs = list(validator.descend(instance, subschema, schema_path=index))
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 592, in validate_config_properties
schema))
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 286, in properties
schema_path=property,
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 592, in validate_config_properties
schema))
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 286, in properties
schema_path=property,
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 205, in descend
schema_path=schema_path):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
for error in self.iter_errors(instance, schema):
File "/home/tau/jcury/DNADNA_project/dnadna/dnadna/utils/jsonschema.py", line 171, in iter_errors
for error in super().iter_errors(instance, _schema=_schema):
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
for error in errors:
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/_validators.py", line 259, in ref
scope, resolved = validator.resolver.resolve(ref)
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 766, in resolve
return url, self._remote_cache(url)
File "/home/tau/jcury/miniconda3/envs/dnadna/lib/python3.7/site-packages/jsonschema/validators.py", line 779, in resolve_from_url
raise exceptions.RefResolutionError(exc)
jsonschema.exceptions.RefResolutionError: no known deserializer for the given filename: "////home/tau/jcury/DNADNA_project/dnadna/dnadna/schemas/"
While trying to see whether I could fix this myself, I noticed that the doc on transformations was a bit light. I opened issue #125 for that.
Edited by Jean Cury