.. _concepts_config:

*************
Configuration
*************

Configuration and parameter management is done via the `Dynaconf Framework <https://www.dynaconf.com>`_.

.. warning::

	Because of bugs #926 and #896 in dynaconf (and because we apply `merge_enabled=True`) it is currently
	not straight forward to prevent the merging of lists and dicts in the configuration, i.e. items of a list or dict
	defined in `settings_default.toml` will be kept even if the setting is overridden in other places.
	Therefore, defaults of list and dict settings which may be overriden should be initialised empty. A workaround is 
	using `CLEAR` to override list entries as the first entry of the new list. The according setting key needs to be added
	to `config/dynaconf_hooks.py`.
	 
	
Order of configuration consideration
====================================

Default values for all settings are meant to be defined in `settings_defaults.toml` in the repos. Your personal settings that
might override the default ones usually go into `config/settings.toml`.

1. Configuration in `settings_defaults.toml`
2. Environment variables
3. Configuration in `settings.toml`
	a. consider env var SN_SETTINGS_FILE_FOR_DYNACONF if set
	b. read `./input/settings.toml` relative to current working directory
	
4. Configuration in Scenario XLSX file (if MAIN__EXCEL_SCENARIO_FILE is set)
5. Environment variables
6. Type casting in `config_post.py`

.. admonition:: Background

   Environment variables are considered twice to set e.g. SCENARIO_ID for ScenarioParameters.xlsx but also allow environment variables
   to override values e.g. in the Scenario XLSX file
   

.. hint::
	You can apply settings for specific environments (context such as the cluster) by passing the according settings filename to 
	the environment variable `SN_SETTINGS_FILE_FOR_DYNACONF`.
	
Basic configuration
===================

You can obtain default settings file and template parameter files by executing:

.. code-block:: python

	import shobnetpy as sn
	sn.prepare()
	
According files will be copied to the `-/input` folder of the working directory.
	

Defining Scenarios
==================

Often, many runs need to be conducted with changes in only one or few parameters. In this case it is convenient to fill an
excel table with lines for each particular scenario parameter set, indexed by a so-called scenario ID. The scenario ID
(MAIN__SCENARIO_ID) can then be switched via an environment variable, e.g. in the SLURM configuration file when running
on a cluster or in the personal settings.toml file.

The Scenario excel file looks like this (there is a template `input_data/Scenario_Template.xlsx`):

.. csv-table:: Scenario_Template.xlsx
    :file: scenario_excel.csv
    :header-rows: 2
    :stub-columns: 1
    :align: left
    :widths: 15, 15, 15, 15, 20, 20
    :width: 100%
    
The part of the SLURM script may look like this:

.. code-block:: sh
   :caption: Part of SLURM config file

    echo "started..."
    export SN_MAIN__SCENARIO_ID=1
    export SN_OUTPUT_OUTPUT_SETTINGS=TRUE
    srun python ./building_stock_model/run.py 
    echo "finished"
    
The excel file is specified via `MAIN__EXCEL_SCENARIO_FILE` (relative to `MAIN__INPUT_PATH` or absolute).


.. _app_concepts_configuration_settings:

Settings Documentation
======================
   
.. include::
	../settings/settings_doc.rst
	

.. hint::
	CSV files can be created automatically for new settings (sections) via the script
	``documentation/sphinx/source/application/concepts/settings/parseSet2doc.py``
	To add unit, type, scope, description and annotation values, use LibreOffice and store with "_edit" added.
	Use then the above mentioned script to convert line endings in multiline cells. Adapt the filename of edited CSV files
	in ``settings_doc.rst``.
	
Agent-specific parameters
=========================

Agent-specific parameters are loaded from `shobnetpy/input/param_agents.xlsx` per default.
Agent types are indexed by agent's milieu attribute. Parameters for degree and distance
distributions are obtained from [1].

.. include::
	../settings/settings_agents.rst

[1] Holzhauer, S. Dynamic Social Networks in Agent-based Modelling
kassel university press, 2017, doi: 10.19211/KUP9783737602631 

Inspecting Settings History
===========================

In case of application of unexpected settings it can be helpful to inspect the history of settings.
Simply make sure the switch `output_settings` is `True` (default) and optionally filter by a settings key (part):

.. code-block:: toml
   :caption: Config for output of settings history
   
    [output]
    output_settings = true
    output_settingshistory_filename = "settings_history.json"
    output_settingshistory_key = "main"
    
The according dynconf feature is documented as `Inspecting History <https://www.dynaconf.com/advanced/#inspecting-history>`_.
It shows first the finally applied settings and afterwards exports the history with the according loader and its parameters from last to first.