Code owners
Assign users and groups as approvers for specific file changes. Learn more.
grid_options_descriptions.rst 21.45 KiB
Population grid code options
The following chapter contains all grid code options, along with their descriptions There are 28 options that are not described yet.
Public options
The following options are meant to be changed by the user.
C_auto_logging: Dictionary containing parameters to be logged by binary_c. The structure of this dictionary is as follows: the key is used as the headline which the user can then catch. The value at that key is a list of binary_c system parameters (like star[0].mass)
C_logging_code: Variable to store the exact code that is used for the custom_logging. In this way the user can do more complex logging, as well as putting these logging strings in files.
HPC_force_join: Integer, default 0. If 1, and the HPC variable ("slurm" or "condor") is 3, skip checking our own job and force the join.
HPC_rebuild_joinlist: Integer, default 0. If 1, ignore the joinlist we would usually use and rebuild it automatically
Moe2017_options: No description available yet
cache_dir: No description available yet
combine_ensemble_with_thread_joining: Boolean flag on whether to combine everything and return it to the user or if false: write it to data_dir/ensemble_output_{population_id}_{thread_id}.json
command_line: No description available yet
condor: Integer flag used to control HTCondor (referred to as Condor here) jobs. Default is 0 which means no Condor. 1 means launch Condor jobs. Do not manually set this to 2 (run Condor jobs) or 3 (join Condor job data) unless you know what you are doing, this is usually done for you.
condor_ClusterID: Integer. Condor ClusterID variable, equivalent to Slurm's jobid. Jobs are numbered <ClusterID>.<Process>
condor_Process: Integer. Condor Process variable, equivalent to Slurm's jobarrayindex. Jobs are numbered <ClusterID>.<Process>
condor_bash: String. Points the location of the "bash" command, e.g. /bin/bash, that is used in Condor launch scripts. This is set automatically on the submit machine, so if it is different on the nodes, you should set it manually.
condor_batchname: String. Condor batchname option: this is what appears in condor_q. Defaults to "binary_c-condor"
condor_date: String. Points the location of the "date" command, e.g. /usr/bin/date, that is used in Condor launch scripts. This is set automatically on the submit machine, so if it is different on the nodes, you should set it manually.
condor_dir: String. Working directory containing e.g. scripts, output, logs (e.g. should be NFS available to all jobs). This directory should not exist when you launch the Condor jobs.
condor_env: String. Points the location of the "env" command, e.g. /usr/bin/env or /bin/env, that is used in Condor launch scripts. This is set automatically on the submit machine, so if it is different on the nodes, you should set it manually.
condor_extra_settings: Dictionary. Place to put extra configuration for the CONDOR submit file. The key and value of the dict will become the key and value of the line in te slurm batch file. Will be put in after all the other settings (and before the command). Take care not to overwrite something without really meaning to do so.
condor_getenv: Boolean. If True, the default, condor takes the environment at submission and copies it to the jobs. You almost certainly want this to be True.
condor_initial_dir: String. Directory from which condor scripts are run. If set to the default, None, this is the directory from which your script is run.
condor_kill_sig: String. Signal Condor should use to stop a process. Note that grid.py expects this to be "SIGINT" which is the default.
condor_memory: Integer. In MB, the memory use (ImageSize) of the job.
condor_njobs: Integer. Number of jobs that Condor will run
condor_postpone_join: Integer. Use to delay the joining of Condor grid data. If 1, data is not joined, e.g. if you want to do it off the condor grid (e.g. with more RAM). Default 0.
condor_postpone_submit: Integer. Debugging tool. If 1, the condor script is not submitted (useful for debugging). Default 0.
condor_pwd: String. Points the location of the "pwd" command, e.g. /bin/pwd, that is used in Condor launch scripts. This is set automatically on the submit machine, so if it is different on the nodes, you should set it manually.
condor_q: String. The Condor_q command, usually "/usr/bin/condor_q" but will depend on your HTCondor installation.
condor_requirements: String. Condor job requirements. These are passed to Condor directly, you should read the HTCondor manual to learn about this. If no requirements exist, leave as an string.
condor_should_transfer_files: Integer. Condor's option to transfer files at the end of the job. You should set this to "YES"
condor_snapshot_on_kill: Integer. If 1 we save a snapshot on SIGKILL before exit.
condor_stream_error: Boolean. If True, we activate Condor's stderr stream. If False, this data is copied at the end of the job.
condor_stream_output: Boolean. If True, we activate Condor's stdout stream. If False, this data is copied at the end of the job.
condor_submit: String. The Condor_submit command, usually "/usr/bin/condor_submit" but will depend on your HTCondor installation.
condor_universe: String. The HTCondor "universe": this is "vanilla" by default.
condor_warn_max_memory: Integer. In MB, the memory use (ImageSize) of the job.
condor_when_to_transfer_output: Integer. Condor's option to decide when output files are transferred. You should usually set this to "ON_EXIT_OR_EVICT"
custom_generator: No description available yet
custom_logging_func_memaddr: Memory address where the custom_logging_function is stored. Input: int
do_analytics: No description available yet
do_dry_run: Whether to do a dry run to calculate the total probability for this run
dry_run_hook: Function hook to be called for every system in a dry run. The function is passed a dict of the system parameters. Does nothing if None (the default).
dry_run_num_cores: No description available yet
ensemble_factor_in_probability_weighted_mass: Flag to multiply all the ensemble results with 1/probability_weighted_mass
evolution_type: Variable containing the type of evolution used of the grid. Multiprocessing, linear processing or possibly something else (e.g. for Slurm or Condor).
exit_after_dry_run: If True, exits after a dry run. Default is False.
exit_code: No description available yet
failed_systems_threshold: Variable storing the maximum number of systems that are allowed to fail before logging their command line arguments to failed_systems log files
function_cache: Boolean, default True. If True, we use a cache for certain function calls.
function_cache_TTL: No description available yet
function_cache_default_maxsize: Integer, default 256. The default maxsize of the cache. Should be a power of 2.
function_cache_default_type: String. One of the following types: LRUCache, LFUCache, FIFOCache, MRUCache, RRCache, TTLCache, NullCache, NoCache. You can find details of what these mean in the Python cachetools manual, except fo NoCache which means no cache is used at all, and NullCache is a dummy cache that never matches, used for testing overheads.
function_cache_functions: No description available yet
gridcode_filename: Filename for the grid code. Set and used by the population object. TODO: allow the user to provide their own function, rather than only a generated function.
joinlist: No description available yet
log_args: Boolean to log the arguments.
log_args_dir: Directory to log the arguments to.
log_dt: Time between verbose logging output.
log_file: Log file for the population object. Unused
log_newline: Newline character used at the end of verbose logging statements. This is n (newline) by default, but x0d (carriage return) might also be what you want.
log_runtime_systems: Whether to log the runtime of the systems . Each systems run by the thread is logged to a file and is stored in the tmp_dir. (1 file per thread). Don't use this if you are planning to run a lot of systems. This is mostly for debugging and finding systems that take long to run. Integer, default = 0. if value is 1 then the systems are logged
max_queue_size: Maximum size of the queue that is used to feed the processes. Don't make this too big! Default: 1000. Input: int
modulo: No description available yet
multiplicity_fraction_function: Which multiplicity fraction function to use. 0: None, 1: Arenou 2010, 2: Rhagavan 2010, 3: Moe and di Stefano (2017) 2017
n_logging_stats: Number of logging statistics used to calculate time remaining (etc.). E.g., if you set this to 10 the previous 10 calls to the verbose log will be used to construct an estimate of the time remaining.
num_cores: The number of cores that the population grid will use. You can set this manually by entering an integer great than 0. When 0 uses all logical cores. When -1 uses all physical cores. Input: int
num_cores_available: No description available yet
original_command_line: No description available yet
original_submission_time: No description available yet
original_working_diretory: No description available yet
parse_function: Function that the user can provide to handle the output the binary_c. This function has to take the arguments (self, output). Its best not to return anything in this function, and just store stuff in the self.grid_results dictionary, or just output results to a file
print_stack_on_exit: If True, prints a stack trace when the population's exit method is called.
repeat: Factor of how many times a system should be repeated. Consider the evolution splitting binary_c argument for supernovae kick repeating.
restore_from_snapshot_dir: No description available yet
restore_from_snapshot_file: No description available yet
return_after_dry_run: If True, return immediately after a dry run (and don't run actual stars). Default is False.
run_zero_probability_system: Whether to run the zero probability systems. Default: True. Input: Boolean
rungrid: No description available yet
save_ensemble_chunks: No description available yet
save_population_object: No description available yet
save_snapshots: No description available yet
slurm: Integer flag used to control Slurm jobs. Default is 0 which means no Slurm. 1 means launch Slurm jobs. Do not manually set this to 2 (run Slurm jobs) or 3 (join Slurm job data) unless you know what you are doing, this is usually done for you.
slurm_array: String. Override for Slurm's --array option, useful for rerunning jobs manually. Default None.
slurm_array_max_jobs: Integer. Override for the max number of concurrent Slurm array jobs. Default None.
slurm_bash: String. Points the location of the "bash" command, e.g. /bin/bash, that is used in Slurm scripts. This is set automatically on the submit machine, so if it is different on the nodes, you should set it manually.
slurm_date: String. Points the location of the "date" command, e.g. /usr/bin/date, that is used in Slurm scripts. This is set automatically on the submit machine, so if it is different on the nodes, you should set it manually.
slurm_dir: String. Working directory containing e.g. scripts, output, logs (e.g. should be NFS available to all jobs). This directory should not exist when you launch the Slurm jobs.
slurm_env: String. Points the location of the "env" command, e.g. /usr/bin/env or /bin/env, that is used in Slurm scripts. This is set automatically on the submit machine, so if it is different on the nodes, you should set it manually.
slurm_extra_settings: Dictionary of extra settings for Slurm to put in its launch script. Please see the Slurm documentation for the many options that are available to you.
slurm_jobarrayindex: Integer. Slurm job array index. Each job is numbered <slurm_jobid>.<slurm_jobarrayindex>.
slurm_jobid: Integer. Slurm job id. Each job is numbered <slurm_jobid>.<slurm_jobarrayindex>.
slurm_jobname: String which names the Slurm jobs, default "binary_c-python".
slurm_memory: String. Memory required for the job. Should be in megabytes in a format that Slurm understands, e.g. "512MB" (the default).
slurm_njobs: Integer. Number of Slurm jobs to be launched.
slurm_ntasks: Integer. Number of CPUs required per array job: usually only need this to be 1 (the default).
slurm_partition: String containing the Slurm partition name. You should check your local Slurm installation to find out partition information, e.g. using the sview command.
slurm_postpone_join: Integer, default 0. If 1 do not join job results with Slurm, instead you have to do it later manually.
slurm_postpone_sbatch: Integer, default 0. If set to 1, do not launch Slurm jobs with sbatch, just make the scripts that would have.
slurm_pwd: String. Points the location of the "pwd" command, e.g. /bin/pwd, that is used in Slurm scripts. This is set automatically on the submit machine, so if it is different on the nodes, you should set it manually.
slurm_sbatch: String. The Slurm "sbatch" submission command, usually "/usr/bin/sbatch" but will depend on your Slurm installation. By default is set automatically.
slurm_time: String. The time a Slurm job is allowed to take. Default is 0 which means no limit. Please check the Slurm documentation for required format of this option.
slurm_warn_max_memory: String. If we set slurm_memory in excess of this, warn the user because this is usually a mistake. Default "1024MB".
source_file_filename: Variable containing the source file containing lines of binary_c command line calls. These all have to start with binary_c.
start_at: No description available yet
start_time: No description available yet
status_dir: Directory where grid status is stored
stop_queue: No description available yet
symlink_latest_gridcode: No description available yet
tmp_dir: Directory where certain types of output are stored. The grid code is stored in that directory, as well as the custom logging libraries. Log files and other diagnostics will usually be written to this location, unless specified otherwise
verbosity: Verbosity of the population code. Default is 0, by which only errors will be printed. Higher values will show more output, which is good for debugging.
weight: Weight factor for each system. The calculated probability is multiplied by this. If the user wants each system to be repeated several times, then this variable should not be changed, rather change the _repeat variable instead, as that handles the reduction in probability per system. This is useful for systems that have a process with some random element in it.
working_diretory: No description available yet
Moe & di Stefano sampler options
The following options are meant to be changed by the user.
JSON: No description available yet
Mmin: Minimum stellar mass
multiplicity_model:
multiplicity model (as a function of log10M1)
You can use 'Poisson' which uses the system multiplicity given by Moe and maps this to single/binary/triple/quad fractions.
Alternatively, 'data' takes the fractions directly from the data, but then triples and quadruples are combined (and there are NO quadruples).
multiplicity_modulator:
[single, binary, triple, quadruple]
- e.g. [1,0,0,0] for single stars only
- [0,1,0,0] for binary stars only
defaults to [1,1,0,0] i.e. singles and binaries
normalize_multiplicities:
'norm': normalise so the whole population is 1.0
after implementing the appropriate fractions
S/(S+B+T+Q), B/(S+B+T+Q), T/(S+B+T+Q), Q/(S+B+T+Q)
given a mix of multiplicities, you can either (noting that
here (S,B,T,Q) = appropriate modulator * model(S,B,T,Q) )
note: if you only set one multiplicity_modulator
to 1, and all the others to 0, then normalising
will mean that you effectively have the same number
of stars as single, binary, triple or quad (whichever
is non-zero) i.e. the multiplicity fraction is ignored.
This is probably not useful except for
testing purposes or comparing to old grids.
- 'raw' : stick to what is predicted, i.e.
- S/(S+B+T+Q), B/(S+B+T+Q), T/(S+B+T+Q), Q/(S+B+T+Q) without normalisation (in which case the total probability < 1.0 unless all you use single, binary, triple and quadruple)
- 'merge' : e.g. if you only have single and binary,
add the triples and quadruples to the binaries, so binaries represent all multiple systems ... * this is canonical binary population synthesis *
It only takes the maximum multiplicity into account, i.e. it doesn't multiply the resulting array by the multiplicity modulator again. This prevents the resulting array to always be 1 if only 1 multiplicity modulator element is nonzero
Note: if multiplicity_modulator == [1,1,1,1]. this option does nothing (equivalent to 'raw').
q_high_extrapolation_method: Same as q_low_extrapolation_method
q_low_extrapolation_method:
q extrapolation (below 0.15) method
none
flat
linear2
plaw2
nolowq
ranges:
resolutions:
samplerfuncs: No description available yet
Private options
The following options are not meant to be changed by the user, as these options are used and set internally by the object itself. The description still is provided, but just for documentation purposes.
_Moe2017_JSON_data: Location to store the loaded Moe&diStefano2017 dataset
_actually_evolve_system: Whether to actually evolve the systems of just act as if. for testing. used in _process_run_population_grid
_binary_c_config_executable: Full path of the binary_c-config executable. This options is not used in the population object.
_binary_c_dir: Director where binary_c is stored. This options are not really used
_binary_c_executable: Full path to the binary_c executable. This options is not used in the population object.
_binary_c_shared_library: Full path to the libbinary_c file. This options is not used in the population object
_commandline_input: String containing the arguments passed to the population object via the command line. Set and used by the population object.
_count: Counter tracking which system the generator is on.
_custom_logging_shared_library_file: filename for the custom_logging shared library. Used and set by the population object
_end_time_evolution: Variable storing the end timestamp of the population evolution. Set by the object itself
_errors_exceeded: Variable storing a Boolean flag whether the number of errors was higher than the set threshold (failed_systems_threshold). If True, then the command line arguments of the failing systems will not be stored in the failed_system_log files.
_errors_found: Variable storing a Boolean flag whether errors by binary_c are encountered.
_evolution_type_options: List containing the evolution type options.
_failed_count: Variable storing the number of failed systems.
_failed_prob: Variable storing the total probability of all the failed systems
_failed_systems_error_codes: List storing the unique error codes raised by binary_c of the failed systems
_grid_variables: Dictionary storing the grid_variables. These contain properties which are accessed by the _generate_grid_code function
_killed: No description available yet
_loaded_Moe2017_data: Internal variable storing whether the Moe and di Stefano (2017) data has been loaded into memory
_main_pid: Main process ID of the master process. Used and set by the population object.
_population_id: Variable storing a unique 32-char hex string.
_probtot: Total probability of the population.
_queue_done: No description available yet
_set_Moe2017_grid: Internal flag whether the Moe and di Stefano (2017) grid has been loaded
_start_time_evolution: Variable storing the start timestamp of the population evolution. Set by the object itself.
_store_memaddr: Memory address of the store object for binary_c.
_system_generator: Function object that contains the system generator function. This can be from a grid, or a source file, or a Monte Carlo grid.
_total_mass_run: To count the total mass that thread/process has ran
_total_probability_weighted_mass_run: To count the total mass * probability for each system that thread/process has ran
_total_starcount: Variable storing the total number of systems in the generator. Used and set by the population object.
_zero_prob_stars_skipped: Internal counter to track how many systems are skipped because they have 0 probability