eGSIM

A Python library and web application to select and test Ground Motion models. Online at: https://egsim.gfz-potsdam.de

2
contributors

What eGSIM can do for you

eGSIM is a web service for selecting and testing ground shaking models (GSIM) in Europe, developed a the GFZ in the framework of the Thematic Core Services for Seismology of EPOS under the umbrella of EFEHR

The web portal (and API documentation) is available at:

https://egsim.gfz-potsdam.de

Citation

Zaccarelli, Riccardo; Weatherill, Graeme (2020): eGSIM - a Python library and web application to select and test Ground Motion models. GFZ Data Services. https://doi.org/10.5880/GFZ.2.6.2023.007

Table of contents

Installation

DISCLAIMER: This document covers installation in development (or debug) mode, i.e. when the program is deployed locally, usually for testing, fixing bug or adding features.

For installation in production mode, see deploy.html

Requirements

sudo apt-get update # pre-requisite
sudo apt-get install gcc  # optional
sudo apt-get install git python3-venv python3-pip python3-dev

(The command above are Ubuntu specific, in macOS install brew and type brew install instead of apt-get install. Remove python3-dev as it does not exist on macOS).

This web service uses a specific version of Python (Open setup.py and check python_requires=. As of January 2022, it's 3.9.7) which you must install in addition to the Python version required by your system, and use it. Any command python3 hereafter will refer to the required Python version.

Clone repository

Select a root directory (e.g. /root/path/to/egsim), and clone egsim into the so-called egsim directory:

git clone https://github.com/rizac/eGSIM.git egsim

Create and activate Python virtual env

Move to whatever directory you want (usually the egsim directory above) and then:

python3 -m venv .env/<ENVNAME>  # create python virtual environment (venv)
source .env/<ENVNAME>/bin/activate  # activate venv

NOTE: From now on, all following operations must have the virtualenv activated FIRST

Install

Note: if the installation is done for upgrading all dependencies and pip freeze into new requirements files, please go to dependency upgrade

On the terminal, execute:

pip install --upgrade pip setuptools && pip install -r requirements.dev.txt

(use requirements.txt if you don't need to run tests, e.g. you are not installing as developer)

Run Test

Note: the value of DJANGO_SETTINGS_MODULE in the examples below must be changed in production

Move in the egsim directory and type:

export DJANGO_SETTINGS_MODULE=egsim.settings_debug; pytest -xvvv --ds=egsim.settings_debug ./tests/

(x=stop at first error, v*=increase verbosity).

For PyCharm users, you need to configure the environment variable for all tests. Go to: Run -> Edit Configurations -> Edit Configurations templates -> Python tests -> pytests -> ENVIRONMENT VARIABLES

and add there the environment variable DJANGO_SETTINGS_MODULE

Other test options from the command line: with coverage ( showing the amount of code hit by tests):

export DJANGO_SETTINGS_MODULE=egsim.settings_debug; pytest --ignore=./tests/tmp/ -xvvv --cov=./egsim/ --cov-report=html ./tests/

(you can also invoke the commands without export ... but using the --ds option: pytest -xvvv --ds=egsim.settings_debug ./tests/)

Usage

Note: the value of DJANGO_SETTINGS_MODULE in the examples below must be changed in production

If you didn't do already, perform a Complete DB reset (one-time only operation)

If you want to access the admin panel, see the admin panel.

To run the program in your local browser, type:

export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py runserver 

(you can also invoke the commands without export ... but using the --settings option: python manage.py --settings=egsim.settings_debug [command])

Maintenance

  • Settings file: A Django settings file contains all the configuration of your Django installation. The settings file referred in this document, included in this git repo, is for debug and local deployment only. On production, a separate settings file is used, located on the server outside the git repo and not shared for security reasons.

  • manage.py or django-admin is Django’s command-line utility for administrative tasks. It is invoked from the terminal within your Python virtualenv (see examples in this document) by providing the settings file via:

    export DJANGO_SETTINGS_MODULE=<settings_file_path> python manage.py <command>
    

    Django allows also the implementation of custom management commands. eGSIM implements egsim-init in order to populate the db (more details below)

  • app a Django app is a Python package that is specifically intended for use in a Django project. An application may use common Django conventions, such as having models, tests, urls, and views submodules. In our case, the Django project is the egsim root directory (created with the command django-admin startproject egsim), and the Django apps inside it are "api" (the core web API) and "app" (the web app, i.e. the part of eGSIM delivered over the Internet through a browser interface), that relies on the "api" code. Inside the settings file (variable INSTALLED_APPS) is configured the list of all applications that are enabled in the eGSIM project. This includes not only our "api" app, that tells Django to create the eGISM tables when initializing the database, but also several builtin Django apps, e.g. the Django admin app, visible through the Admin panel).

Starting a Python terminal shell

Note: the value of DJANGO_SETTINGS_MODULE in the examples below must be changed in production

Typing python on the terminal does not work if you need to import django stuff, as there are things to be initialized beforehand. The Django shell command does this:

export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py shell 

Complete DB reset

Note: the value of DJANGO_SETTINGS_MODULE in the examples below must be changed in production

We perform a complete DB reset every time we change something in the Database schema (see egsim.api.models.py), e.g. a table, a column, a constraint.

The usual way to change a DB in a web app is to create and run migrations (Django full details here), which allow to keep track of all changes (moving back and forth if necessary) whilst preserving the data stored in the DB. However, none of those features is required in eGSIM: DB data is predefined and would be regenerated from scratch in any case after any new migration. Consequently, upon changes in the DB, a complete DB reset is an easier procedure.

In any case (just for reference), the steps to create and run migrations in eGSIM are the following:

export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py makemigrations egsim --name <migration_name>
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py migrate egsim

And then remember to Repopulate the db

Notes:

  • The make_migration command just generates a migration file, it doesn't change the db. The migrate command does that, by means of the migration files generated. For details on Django migrations, see:
  • <migration_name> will be a suffix appended to the migration file, use it like you would use a commit message in git).
  • When running migrate, if the migration will introduce new non-nullable fields, maybe better to run manage.py flush first to empty all tables, to avoid conflicts "egsim" above is the app name. If you omit the app, all apps will be migrated. The command migrate does nothing if it detects that there is nothing to migrate

To perform a complete db reset:

  • delete or rename db.sqlite3 (or wherever the database is. See settings file in case)

  • [Optional: skip if the db schema hasn't changed] delete all migrations, i.e. all *.py files except __init__.py currently under egsim/api/migrations, and re-generate the initial (and only) migration file:

    export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py makemigrations
    
  • Run migrations (re-create the DB via the generated migration file):

    export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py migrate
    
  • Repopulate the db

  • [Optional] re-add the Django admin superuser(s) as explained in the admin panel above

Repopulate the DB

NOTE: If you don't have created and initialized the db yet, go to Complete DB reset

When the db is created for the first time or reset, OpenQuake is upgraded, or new data is implemented (new regionalization or flatfile), you need to repopulate the database and the filesystem, to make changes available to eGSIM users. Execute the custom management command egsim_init:

   export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py egsim_init

Admin panel

Note: the value of DJANGO_SETTINGS_MODULE in the examples below must be changed in production

This command allows the user to check database data from the web browser. For further details, check the Django doc

The database must have been created and populated (see Usage).

Create a super user (to be done once only ):

export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py createsuperuser

and follow the instructions.

Start the program (see Usage) and then navigate in the browser to [SITE_URL]/admin (in development mode, http://127.0.0.1:8000/admin/)

Note: Theoretically, you can modify db data from the browser, but to make changes persistent after a database reset you should implement them in the management command egsim_init (see Repopulate the DB)

Create a custom management command

See egsim/api/management/commands/README.md.

The next two sections will describe how to store new data (regionalizations and flatfiles) that will be made available in eGSIM when repopulating the db

Add new predefined flatfiles

  • Add the file (CSV or zipped CSV) in managements/commands/data/predefined_flatfiles. Do not commit files too big (max few tens of Mb). When zipping in macOS you will probably need to exclude or remove (after zipping) the ___MACOSX folder

  • Implement a new FlatfileParser class in management/commands/flatfile_parsers.py The parser goal is to read the file and convert it into a harmonized HDF table (see ESM parser in the Python file)

  • Add binding file -> parser in the Python dict: management.commands._egsim_flatfiles.Command.PARSER

  • (Optional) Add the file data source in management/commands/data/data_sources.yaml, e.g. data reference, url, (see examples in the YAML file).

    NOTE: In the data source you can also set the data name, i.e. a unique, usually short alphanumeric string that will identify the flatfile in user requests. If no data source or name is provided, the public name will be the file name before the first ".".

  • Repopulate all eGSIM tables (command egsim_init)

Implemented flatfiles sources (click on the items below to expand)

  • Go to https://esm.mi.ingv.it//flatfile-2018/flatfile.php (with username and password, you must be registered beforehand it's relatively fast and simple)

  • Download ESM_flatfile_2018.zip, uncompress and extract ESM_flatfile_SA.csv from there

  • ESM_flatfile_SA.csv is our raw flatfile, compress it again (it's big) into this directory as ESM_flatfile_2018_SA.zip

  • If on macOS, type the command above to remove the macOS folder from the zip

Add new regionalization

  • Add two files with the same basename and extensions

    • .geojson (regionalization, aka regions collection) and
    • .json (region -> gsim mapping)

    in managements/commands/data/regionalization_files. Usually, these files are copied and pasted from the shakyground2 project (see GFZ gitlab), but if you neeed to implement your own see examples in the given directory or ask the developers

  • (Optional) Add the file data source in management/commands/data/data_sources.yaml, e.g. data reference, url, (see examples in the YAML file).

    NOTE: In the data source you can also set the data name, i.e. a unique, usually short alphanumeric string that will identify the regionalization in user requests. If no data source or name is provided, the public name will be the file name before the first ".".

  • Repopulate all eGSIM tables (command egsim_init)

Dependencies upgrade

Please note that it is safer (even mandatory with pip) to upgrade all dependencies instead of single packages in order to avoid conflicts.

Consequently, we recommend to follow this procedure also in case of a GitHub security issue (or dependency alert) on a single package.

First create a new virtual env (in the example below, in the git-ignored .env inside the eGSIM root directory, but you can also create it wherever you want)

python3 -m venv .env/<ENVNAME>  # create python virtual environment (venv)
source .env/<ENVNAME>/bin/activate  # activate venv
pip install --upgrade pip setuptools

Then cd into eGSIM if you are not there already (basically, the directory where setup.py is located), install and freeze the installed packages into a requirements.txt file:

pip install -e .
pip freeze > requirements.txt

Do the same for testing:

pip install -e ".[test]"
pip freeze > requirements.dev.txt

Run tests (see above) to check that everything wortks as expected, and then commit and push.

Keywords
Programming languages
  • Python 67%
  • JavaScript 20%
  • HTML 13%
  • CSS 1%
License
  • AGPL-3.0-only
</>Source code

Contributors

RZ
Riccardo Zaccarelli
GFZ German Research Centre for Geosciences, Potsdam, Germany
GW
Graeme Weatherill
GFZ German Research Centre for Geosciences, Potsdam, Germany