I've added a new grid_option, num_processes, which is the number of processes launched by Python's multiprocessing. num_cores is used to set this:

if > 0 use the number specified (as previously, so backwards compatibility is fine)
if == 0 use the number of logical cores
if == -1 use the number of physical cores

Try running it with a command like:

---
rm -rf /tmp/slurm ; nice python3.9 ./src/python/ensemble.py dists=Moe binaries=False r=100 verbosity=1 max_evolution_time=10 slurm_dir=/tmp/slurm slurm_partition=debug slurm_memory=100MB monte_carlo_kicks=0 save_ensemble_chunks=False num_cores=-1 slurm=1 slurm_njobs=2 num_cores=2
---

You will want to change num_cores and slurm_njobs to suit. Each Slurm job gets num_processes cores allocated to it.

Note: you should set your slurm directory to be empty. This isn't really required, but makes debugging a lot easier.

You also have to set the slurm_partition by hand - this is something you need to find out based on your cluster. In the above example I use "debug" because this is the default.

There are quite a few changes internally, particularly new functions to load, save and merge Population objects and their data (mostly) correctly, and updates to the dict merging functions that this required.

please report bugs because there will be many!

Fixing tests that failed. Added a function to filter dicts based on value instead of key. Added M&S data to grid options rather than custom options

own source files

add option to convert dict keys to floats when loading ensemble data : this is required for matplotlib/seaborn otherwise the plots are completely messed up

added options to the Moe loader so we can change the JSON data and reload it after clearing the cache

the period distribution renormalization now gets the low and high limits from the loaded data

update_dicts() now copies the type of dict_1 for the new dict, rather than using OrderedDict (maybe merge_dicts should do this too?)

I've fixed the _boxed formatting to strip ANSI codes when calculating string lengths (so you can use ANSI colours properly)

changed the colours for the "failed systems" warning so it's absolutely obvious

wrapped the JSON ensemble output in a write_ensemble() function which automatically gzips or bzips depending on the filename extension (or this can be explictly set in grid_options['compress_ensemble'] ).

shows:
current grid status (#/# and modulo)
estimate of time remaining and time to finish
memory use (of all processes)
time per run

add methods to grid.py:

rename_grid_variable
update_grid_variable
clean

add functions to functions.py:

convfloat
datalinedict
pad_output_distribution