Slurm restarts work, just take your old (unfinished) Slurm dir and set it with slurm_restart_dir=<whatever>

doesn't work for slurm yet, but small steps

also fixed a bug in keys_to_floats()

add CPU time to metadata output, and fix an issue with a tuple rather than a string from platform.something()

fixed issues with grid spacings

new gridtype: "discrete". This samples the grid points and does not set the probability.

I've added a new grid_option, num_processes, which is the number of processes launched by Python's multiprocessing. num_cores is used to set this:

if > 0 use the number specified (as previously, so backwards compatibility is fine)
if == 0 use the number of logical cores
if == -1 use the number of physical cores

Try running it with a command like:

---
rm -rf /tmp/slurm ; nice python3.9 ./src/python/ensemble.py dists=Moe binaries=False r=100 verbosity=1 max_evolution_time=10 slurm_dir=/tmp/slurm slurm_partition=debug slurm_memory=100MB monte_carlo_kicks=0 save_ensemble_chunks=False num_cores=-1 slurm=1 slurm_njobs=2 num_cores=2
---

You will want to change num_cores and slurm_njobs to suit. Each Slurm job gets num_processes cores allocated to it.

Note: you should set your slurm directory to be empty. This isn't really required, but makes debugging a lot easier.

You also have to set the slurm_partition by hand - this is something you need to find out based on your cluster. In the above example I use "debug" because this is the default.

There are quite a few changes internally, particularly new functions to load, save and merge Population objects and their data (mostly) correctly, and updates to the dict merging functions that this required.

please report bugs because there will be many!

beware though: there are many debugging options still built in, and the gridcode should (but doesn't) have a unique filename

add a fix to distinguish between centred and edge grids when computing number of points over which we should loop and phasevols

also added a warning in the const_dt spacing function to catch the edge case where tmax or max_evolution_time are shorter than the first timestep, which would lead to an empty interpolation table

Decided the chunks / ensemble results per thread will be written as jsons and not as raw strings. Changed tests accordingly

Fixing tests that failed. Added a function to filter dicts based on value instead of key. Added M&S data to grid options rather than custom options

remove deprecated resolution setting

own source files

add option "save_ensemble_chunks" to make sure the ensemble chunks are saved : I am chasing up a grid that completely froze (and gave segfaults) ...

add modulo cost in tpr calculation: if modulo is 10, we underestimate the cost by ~ a factor 10, so we should multiply by the modulo