Slurm restarts work, just take your old (unfinished) Slurm dir and set it with slurm_restart_dir=<whatever>

doesn't work for slurm yet, but small steps

also fixed a bug in keys_to_floats()

add CPU time to metadata output, and fix an issue with a tuple rather than a string from platform.something()

I've added a new grid_option, num_processes, which is the number of processes launched by Python's multiprocessing. num_cores is used to set this:

if > 0 use the number specified (as previously, so backwards compatibility is fine)
if == 0 use the number of logical cores
if == -1 use the number of physical cores

Try running it with a command like:

---
rm -rf /tmp/slurm ; nice python3.9 ./src/python/ensemble.py dists=Moe binaries=False r=100 verbosity=1 max_evolution_time=10 slurm_dir=/tmp/slurm slurm_partition=debug slurm_memory=100MB monte_carlo_kicks=0 save_ensemble_chunks=False num_cores=-1 slurm=1 slurm_njobs=2 num_cores=2
---

You will want to change num_cores and slurm_njobs to suit. Each Slurm job gets num_processes cores allocated to it.

Note: you should set your slurm directory to be empty. This isn't really required, but makes debugging a lot easier.

You also have to set the slurm_partition by hand - this is something you need to find out based on your cluster. In the above example I use "debug" because this is the default.

There are quite a few changes internally, particularly new functions to load, save and merge Population objects and their data (mostly) correctly, and updates to the dict merging functions that this required.

please report bugs because there will be many!

beware though: there are many debugging options still built in, and the gridcode should (but doesn't) have a unique filename

Fixing tests that failed. Added a function to filter dicts based on value instead of key. Added M&S data to grid options rather than custom options

remove deprecated resolution setting

add option "save_ensemble_chunks" to make sure the ensemble chunks are saved : I am chasing up a grid that completely froze (and gave segfaults) ...

also prevented the Moecache from being freed on grid exit: we can have nested grid objects, and in that case it is required to remain

attempted to harmoize the Moe and di Stefano (2017) variable names so they all use Moe2017 (or MOE2017)

added options to the Moe loader so we can change the JSON data and reload it after clearing the cache

the period distribution renormalization now gets the low and high limits from the loaded data

update_dicts() now copies the type of dict_1 for the new dict, rather than using OrderedDict (maybe merge_dicts should do this too?)

I've fixed the _boxed formatting to strip ANSI codes when calculating string lengths (so you can use ANSI colours properly)

changed the colours for the "failed systems" warning so it's absolutely obvious

wrapped the JSON ensemble output in a write_ensemble() function which automatically gzips or bzips depending on the filename extension (or this can be explictly set in grid_options['compress_ensemble'] ).

implemented binary_grid-style statistics over last n_logging_stats verbose outputs (50 by default)

added ANSI colour to output if in a terminal (i.e. with a tty)

fixed the cmd-line parser to work in the form x=y as intended (downside: no help output, but that just said how to use the old parser which doesn't exist now)

add option such that when resolution==0, the resolution is calculated automatically from the sample list.

add delete_grid_variable function

In my CEMP simulations, I use the above with something like this to zoom in around 0.8Msun in M1:

emp_population.update_grid_variable(
    name="lnm1",
    spacingfunc="gaussian_zoom(np.log(0.4),np.log(6.0),np.log(0.8),0.05,0.99,100)",
    resolution=0
)

shows:
current grid status (#/# and modulo)
estimate of time remaining and time to finish
memory use (of all processes)
time per run

Integrated the new functions, population_id will now be renewed upon running a grid, prevented evolve_single from running if there are no arguments passed, added some check in the export_all_info function that raises an error if the user tries to export without outfile OR data_dir