Mentions légales du service

Skip to content

Handling multiple dataset name format

JOBIC Pierre requested to merge pj/filename into master

Issue

the actual dataset format is too rigid because consistent means "always the same s_padding,r_padding scheme for scenario, replicates".
The actual code doesn't take into account not padded dataset format, which I add as a functionality (I needed it for cattle dataset)

Actual dataset format

_ simulations/  # root directory of all simulation datasets; arbitrary name
    \_ model-A/
        \_ model-A_params.csv  # the scenario parameters table
        |_ model-A_simulation_config.yml  # the simulation config file
        |_ scenario_000/
            \_ model-A_000_00.npz  # scenario 0 replicate 0
            ...
            |_ model-A_000_NN.npz
        |_ scenario_001/
            \_ model-A_001_00.npz  # scenario 1 replicate 0
            ...
            |_ model-A_001_NN.npz
        ...
        |_ scenario_NNN/
            \_ model-A_NNN_00.npz
            ...
            |_ model-A_NNN_NN.npz

which stands for:
scenario_{scenario:0{s_padding}d}/{dataset_name}_{scenario:0{s_padding}d}_{replicate:0{r_padding}d}.npz

Now it can handles:

_ simulations/  # root directory of all simulation datasets; arbitrary name
    \_ model-A/
        \_ model-A_params.csv  # the scenario parameters table
        |_ model-A_simulation_config.yml  # the simulation config file
        |_ scenario_000/
            \_ model-A_000_00.npz  # scenario 0 replicate 0
            ...
            |_ model-A_000_NN.npz
        |_ scenario_001/
            \_ model-A_001_00.npz  # scenario 1 replicate 0
            ...
            |_ model-A_001_NN.npz
        ...
        |_ scenario_N/
            \_ model-A_N_0.npz
            ...
            |_ model-A_N_N.npz
        |_ scenario_NNN/
            \_ model-A_NNN_00.npz
            ...
            |_ model-A_NNN_NN.npz

which stands for:
scenario_{scenario:0{s_padding}d}/{dataset_name}_{scenario:0{s_padding}d}_{replicate:0{r_padding}d}.npz
or, in case this file doesn't exist, look for:
scenario_{scenario}/{dataset_name}_{scenario}_{replicate}.npz

Merge request reports