Configuration

Configuration and parameter management is done via the Dynaconf Framework.

Warning

Because of bugs #926 and #896 in dynaconf (and because we apply merge_enabled=True) it is currently not straight forward to prevent the merging of lists and dicts in the configuration, i.e. items of a list or dict defined in settings_default.toml will be kept even if the setting is overridden in other places. Therefore, defaults of list and dict settings which may be overriden should be initialised empty. A workaround is using CLEAR to override list entries as the first entry of the new list. The according setting key needs to be added to config/dynaconf_hooks.py.

Order of configuration consideration

Default values for all settings are meant to be defined in settings_defaults.toml in the repos. Your personal settings that might override the default ones usually go into config/settings.toml.

1. Configuration in settings_defaults.toml

2. Environment variables

3. Configuration in settings.toml

   1. consider env var SN_SETTINGS_FILE_FOR_DYNACONF if set

   2. read ./input/settings.toml relative to current working directory

4. Configuration in Scenario XLSX file (if MAIN__EXCEL_SCENARIO_FILE is set)

5. Environment variables

6. Type casting in config_post.py

Background

Environment variables are considered twice to set e.g. SCENARIO_ID for ScenarioParameters.xlsx but also allow environment variables to override values e.g. in the Scenario XLSX file

Hint

You can apply settings for specific environments (context such as the cluster) by passing the according settings filename to the environment variable SN_SETTINGS_FILE_FOR_DYNACONF.

Basic configuration

You can obtain default settings file and template parameter files by executing:

import shobnetpy as sn
sn.prepare()

According files will be copied to the -/input folder of the working directory.

Defining Scenarios

Often, many runs need to be conducted with changes in only one or few parameters. In this case it is convenient to fill an excel table with lines for each particular scenario parameter set, indexed by a so-called scenario ID. The scenario ID (MAIN__SCENARIO_ID) can then be switched via an environment variable, e.g. in the SLURM configuration file when running on a cluster or in the personal settings.toml file.

The Scenario excel file looks like this (there is a template input_data/Scenario_Template.xlsx):

Scenario_Template.xlsx

	MAIN	MAIN	MAIN	RNG	RNG
ScenarioID	STARTYEAR	STEPSIZE	FINALYEAR	seed_buildinginit	seed_buildingnew
1	2020	5	2050	1	1
2	2020	1	2050	2	2
3	2022	5	2050	1	1
4	2022	1	2050	2	2

The part of the SLURM script may look like this:

Part of SLURM config file

 echo "started..."
 export SN_MAIN__SCENARIO_ID=1
 export SN_OUTPUT_OUTPUT_SETTINGS=TRUE
 srun python ./building_stock_model/run.py
 echo "finished"

The excel file is specified via MAIN__EXCEL_SCENARIO_FILE (relative to MAIN__INPUT_PATH or absolute).

Settings Documentation

main

Setting	Unit	Type	Default	Scope	Description	Annotations
output_dir			“./output”			Output ‘root’ for logs, settings history, graphml files etc.
project			“Example”			Appended to output directory to separate different projects
task			“TaskA”			Appended to output path (see below) to separate different tasks of a project
scenario_id			1			Scenario ID that is looked up in excel_scenario_file
runid			0
excel_scenario_file			“Scenario_Template.xlsx”			Excel file to look up parameters by ScenarioID
excel_scenario_sheetname			“Parameter”			Sheetname of the excel file to look up parameters in
input_path			“./input”			Used for parameter files. Relative to folder “shobnetpy” or absolute:
output_path			“@format {this.main.output_dir}/{this.main.project}”			Used as target for output files (specified excluding task but including project):

input

Setting	Unit	Type	Default	Scope	Description	Annotations
param_file_agent			“./param_agents.xlsx”
sheetname_degree			“degree”
sheetname_distance			“distance”
sheetname_milieucomposition			“milieucomposition”
sheetname_params			“params”

param

Setting	Unit	Type	Default	Scope	Description	Annotations
colname_probag_forw			“probag_forw”
colname_probag_back			“probag_back”
colname_dim_weight_milieu			“dim_weight_milieu”
colname_dim_weight_geo			“dim_weight_geo”
colname_distribution			“distribution”
colname_dist_xmin			“xmin”

preprocess

Setting	Unit	Type	Default	Scope	Description	Annotations
output_dir_hexagon			“@format {this.main.output_dir}/hexagons”
source_dir_boundaryShapes			“@format {this.main.output_dir}/boundaryShapes”

geo

Setting	Unit	Type	Default	Scope	Description	Annotations
hexagon_initialise			“onelayer”
hexagon1_shapefile			“shapes/hexagons1/hexagons1”
hexagon2_shapefile			“shapes/hexagons2/hexagons2”
hexagon1_shapefile_select			“shapes/hexagonsSelected1/selected1”
hexagon2_shapefile_select			“shapes/hexagonsSelected2/selected2”
agent_shapefile_logging			“shapes/agents/agents”
crs			“epsg:2163”
qgis_app			“/usr/bin/qgis”
hexagon1_width			150
hexagon2_width			50
removeshapefiles			false

Note

To apply changes in the properties of hexagons (e.g. hexagon1_width or hexagon1_width) existing shapefiles need to be removed or the path to selected shapes (hexagon1_shapefile_select and possibly hexagon2_shapefile_select) and all shapes (hexagon1_shapefile and hexagon1_shapefile) needs to be changed.

network

Setting	Unit	Type	Default	Scope	Description	Annotations
deg_dist			“nbinom”			distribution of scipy.stats
deg_dist_param1			3.11
deg_dist_param2			0.17
distance_dist			“weibull_min”			distribution of scipy.stats
distance_dist_param1			1.02
distance_dist_param2			174.24
distance_xmin			20.0
distance_plocal			0.7
distance_factor_for_distribution			1000			Usually, agent coordinates are in meters and the distance distribution assumes kilometres.
max_search_radius			inf
prob_burning_forward			0.0
prob_burning_backward			0.2
dim_weight_geo			0.5
dim_weight_milieu			0.5
num_recruiting_attempts			100			Number of attempts to find a ambassador, and to increase the degree.
agent_shuffle_interval			inf			The order of agent selection during forest fire linking is relevant since the later agents have more links to follow. In order to mitigate this effect the network service allows to shuffle the ordering after the number of turns specified here. Default: Integer.MAX_VALUE
attempts_ambassador			9999			absolute or relativ to input dir (main.input_path)

rng

Setting	Unit	Type	Default	Scope	Description	Annotations
seed_degree			1
seed_milieu			2
seed_dist			3
seed_shuffle			4
seed_explore			5
logger_degree			“test_degree_”
logger_milieu			“test_milieu_”
logger_distance			“test_distance_”
logger_shuffle			“test_shuffle_”
logger_explore			“test_explore_”
agent_shuffle_interva			100			The order of agent selection during forest fire linking is relevant since the later agents have more links to follow. In order to mitigate this effect the network service allows to shuffle the ordering after the number of turns specified here. Default: 100

output

Setting	Unit	Type	Default	Scope	Description	Annotations
subfolder			“output”
output_settings			true
output_settings_path			“conf”
output_settings_filename			“settings.toml”
output_settingshistory_filename			“settings_history.json”
output_settingshistory_key			“@none”

debug

Setting	Unit	Type	Default	Scope	Description	Annotations
output_randomstreaminvocations			false

logging

Setting	Unit	Type	Default	Scope	Description	Annotations
log_path			“log”
configfile			“./shobnetpy/config/config_logging.conf”			absolute or relativ to project dir (shobnetpy)
write_agentshapes			true

testing

Setting	Unit	Type	Default	Scope	Description	Annotations
agentshapefile			“testdata/agents.geojson”

Hint

CSV files can be created automatically for new settings (sections) via the script documentation/sphinx/source/application/concepts/settings/parseSet2doc.py To add unit, type, scope, description and annotation values, use LibreOffice and store with “_edit” added. Use then the above mentioned script to convert line endings in multiline cells. Adapt the filename of edited CSV files in settings_doc.rst.

Agent-specific parameters

Agent-specific parameters are loaded from shobnetpy/input/param_agents.xlsx per default. Agent types are indexed by agent’s milieu attribute. Parameters for degree and distance distributions are obtained from [1].

degree

type	distribution	n	p	loc
1	nbinom	3.6	0.2	0
2	nbinom	2.57	0.14	0
3	nbinom	3.35	0.21	0
4	nbinom	2.63	0.14	0
5	nbinom	3.18	0.17	0

distance

Note

Appropriate distance parameters very much depend on the spatial extent of the simulation region. If this is too small (say below 100 km) the default parameters may no lead to suitable results, at least network generation make take longer to finish. Therefore, consider different values for the scale parameter or completely different distributions.

type	distribution	c	loc	scale	plocal	xmin
1	weibull_min	1.087	0	184.4	0.479	6.619
2	weibull_min	0.533	0	60.1	0.65	9.173
3	weibull_min	0.533	0	60.1	0.65	9.173
4	weibull_min	1.429	0	213.6	0.626	402.4
5	weibull_min	3.6	0	174.2	0.6	200

milieu composition

type	1	2	3	4	5
1	0.2	0.2	0.2	0.2	0.2
2	0.2	0.2	0.2	0.2	0.2
3	0.2	0.2	0.2	0.2	0.2
4	0.2	0.2	0.2	0.2	0.2
5	0.2	0.2	0.2	0.2	0.2

params

type	probag_forw	probag_back	dim_weight_milieu	dim_weight_geo
1	0.4	0.4	0.5	0.5
2	0.4	0.4	0.5	0.5
3	0.4	0.4	0.5	0.5
4	0.4	0.4	0.5	0.5
5	0.4	0.4	0.5	0.5

[1] Holzhauer, S. Dynamic Social Networks in Agent-based Modelling kassel university press, 2017, doi: 10.19211/KUP9783737602631

Inspecting Settings History

In case of application of unexpected settings it can be helpful to inspect the history of settings. Simply make sure the switch output_settings is True (default) and optionally filter by a settings key (part):

Config for output of settings history

 [output]
 output_settings = true
 output_settingshistory_filename = "settings_history.json"
 output_settingshistory_key = "main"

The according dynconf feature is documented as Inspecting History. It shows first the finally applied settings and afterwards exports the history with the according loader and its parameters from last to first.