Get started
1377 commits | Last commit ≈ 4 days ago
A Python library and web application to select and test Ground Motion models. Online at: https://egsim.gfz-potsdam.de
eGSIM is a web service for selecting and testing ground shaking models (GSIM) in Europe, developed by the GFZ in the framework of the Thematic Core Services for Seismology of EPOS under the umbrella of EFEHR
The web portal (and API documentation) is available at:
Zaccarelli, Riccardo; Weatherill, Graeme (2020): eGSIM - a Python library and web application to select and test Ground Motion models. GFZ Data Services. https://doi.org/10.5880/GFZ.2.6.2023.007
DISCLAIMER: This document does not cover the server installation of the web app, which is publicly available at the URL above. Here you can find instructions on:
import egsim.smtk in your code)sudo apt-get update # pre-requisite
sudo apt-get install gcc # optional
sudo apt-get install git python3-venv python3-pip python3-dev
(The command above are Ubuntu specific, in macOS install brew and type
brew install instead of apt-get install. Remove python3-dev as it does not
exist on macOS).
This web service uses a specific version of Python (Open setup.py and
check python_requires=. As of January 2022, it's >=3.11) which you must
install in addition to the Python version required by your system, and use
it. Any command python3 hereafter will refer to the required Python version.
Select a root directory (e.g. /root/path/to/egsim), and clone egsim into the
so-called egsim directory:
git clone https://github.com/rizac/eGSIM.git egsim
Move to whatever directory you want (usually the egsim directory above) and then:
python3 -m venv .env/<ENVNAME> # create python virtual environment (venv)
source .env/<ENVNAME>/bin/activate # activate venv
NOTE: From now on, all following operations must have the virtualenv activated FIRST
Assuming you are in the egsim directory with a virtualenv :
source .env/<ENVNAME>/bin/activate
pip install -r ./requirements.txt
If you want to use eGSIM locally using the
strong motion toolkit package only (from egsim.smtk import ...
in your code):
source .env/<ENVNAME>/bin/activate
pip install -r ./requirements.lib.txt
(remember to pip install pytest first)
pytest -vvv ./tests/smtk
(web app tests. For testing the library only, see above)
Note: the value of
DJANGO_SETTINGS_MODULEin the examples below must be changed in production
Move in the egsim directory and type:
export DJANGO_SETTINGS_MODULE=egsim.settings_debug; pytest -xvvv ./tests/
(x=stop at first error, v*=increase verbosity).
with coverage report:
export DJANGO_SETTINGS_MODULE=egsim.settings_debug; pytest --cov=egsim --cov-report=html -xvvv ./tests/
And then under Environment variables: add:
DJANGO_SETTINGS_MODULE=egsim.settings_debug
(type several env vars separated by ;)
Note: the value of
DJANGO_SETTINGS_MODULEin the examples below must be changed in production
If you didn't do already, perform a Complete DB reset (one-time only operation)
If you want to access the admin panel, see the admin panel.
To run the program in your local browser, type:
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py runserver
then under Run:
script and module (should be a combo box) choose script,
and in the next text field put manage.pyrunserverDJANGO_SETTINGS_MODULE=egsim.settings_debug
(type several env vars separated by ;)You should see in the Services tab appearing the script name, so you can
run / debug it normally
source .env/<ENVNAME>/bin/activate
pip install --upgrade pip setuptools
Upgrade OpenQuake (optional). The operation below should be performed in
very specific cases only (important bugfixes or features) because
being OpenQuake often backward incompatible it might require additional
code fixes and feedbacks from scientific experts or OpenQuake developers.
First, open setup.py and comment the line of install_requires where OpenQuake
is installed (should be starting with openquake.engine). Then
(note that pip install openquake works but is not the recommended way):
pip install -r "https://raw.githubusercontent.com/gem/oq-engine/master/requirements-py311-macos_x86_64.txt"
# pip install -r "https://raw.githubusercontent.com/gem/oq-engine/master/requirements-py311-linux64.txt"
Install eGSIM Python library, upgrading its dependencies:
pip install -U . && pip freeze >./requirements.lib.txt && pip install pytest
Run tests:
pytest -vvv ./tests/smtk
Install eGSIM web app, upgrading its dependencies:
pip install -U --upgrade-strategy eager ".[web]"
pip freeze > ./requirements.txt
Run tests:
export DJANGO_SETTINGS_MODULE=egsim.settings_debug; pytest -xvvv ./tests/
Change setup.py and set the current OpenQuake version in
install_requires (uncomment it if commented). Optionally,
remove egsim from requirements.txt (it might interfere with Django web?*).
Eventually, commit and push
Remember to activate the Python virtualenv in all examples below
Settings file: A Django settings file contains all the configuration of your Django installation. The settings file referred in this document, included in this git repo, is for debug and local deployment only. On production, a separate settings file is used, located on the server outside the git repo and not shared for security reasons.
manage.py or
django-admin is Django’s command-line utility for administrative tasks.
It is invoked from the terminal within your Python virtualenv (see examples
in this document) by providing the settings file via:
export DJANGO_SETTINGS_MODULE=<settings_file_path> python manage.py <command>
Django allows also the implementation of custom management commands.
eGSIM implements egsim-init in order to populate the db (more details
below)
app a
Django app is a Python package that is specifically intended for use in
a Django project. An application may use common Django conventions, such as
having models, tests, urls, and views submodules. In our case, the Django
project is the egsim root directory (created with the command
django-admin startproject egsim), and the Django apps inside it are
"api" (the core web API) and "app" (the web app, i.e. the part of eGSIM
delivered over the Internet through a browser interface), that relies on
the "api" code.
Inside the settings file (variable INSTALLED_APPS) is configured the list
of all applications that are enabled in the eGSIM project. This includes not
only our "api" app, that tells Django to create the eGISM tables when
initializing the database, but also several builtin Django apps, e.g. the
Django admin app, visible through the Admin panel.
Note: the value of
DJANGO_SETTINGS_MODULEin the examples below must be changed in production
Typing python on the terminal does not work as one needs to
initialize Django settings. The Django shell command does this:
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py shell
If you did not set it explicitly in settings.FILE_UPLOAD_TEMP_DIR
(by default is missing), then Django will put uploaded files
in the standard temporary directory which you can get easily by
typing:
python -c "import tempfile;print(tempfile.gettempdir())"
Note: the value of
DJANGO_SETTINGS_MODULEin the examples below must be changed in production
We perform a complete DB reset every time we change something
in the Database schema (see egsim.api.models.py), e.g. a table,
a column, a constraint.
The usual way to change a DB in a web app is to create and run migrations (full details here), which allow to keep track of all changes (moving back and forth if necessary) whilst preserving the data stored in the DB. However, none of those features is required in eGSIM: DB data is predefined and would be regenerated from scratch in any case after any new migration. Consequently, upon changes in the DB, a complete DB reset is an easier procedure.
In any case (just for reference), the steps to create and run migrations in eGSIM are the following:
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py makemigrations egsim --name <migration_name>
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py migrate egsim
And then repopulate the db:
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py egsim_init
Notes:
make_migration command just generates a migration file, it doesn't
change the db. The migrate command does that, by means of the migration
files generated. For details on Django migrations, see:
git).migrate, if the migration
will introduce new non-nullable fields, maybe better to run
manage.py flush first to empty all tables, to avoid conflicts
"egsim" above is the app name. If you omit the app, all apps will be
migrated. The command migrate does nothing if it detects that there is
nothing to migrateTo perform a complete db reset:
egsim/db.sqlite3egsim/api/migrations/0001_initial.py (there should be only one. If there
are others, delete all of them)export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py makemigrations && python manage.py migrate && python manage.py egsim_init
git add the newly created migration file (in dev mode it's
egsim/api/migrations/0001_initial.py)Notes:
makemigrations creates the necessary migration file(s) from Python
code and existing migration file(s)migrate re-create the DB via the generated migration file(s)egsim_init repopulates the db with eGSIM dataWe repopulate the DB when its schema has not changed but its data needs to, e.g., OpenQuake is upgraded, or new data is implemented (new regionalization or flatfile), or a bug in the code has been fixed. The operations are similar but simpler than a complete Db Rest:
egsim/db.sqlite3export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py migrate && python manage.py egsim_init
Note: the value of
DJANGO_SETTINGS_MODULEin the examples below must be changed in production
This command allows the user to check database data from the web browser. For further details, check the Django doc
The database must have been created and populated (see Usage).
Create a superuser (to be done once only ):
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py createsuperuser
and follow the instructions.
Start the program (see Usage) and then navigate in the browser to
[SITE_URL]/admin (in development mode, http://127.0.0.1:8000/admin/)
Note: Theoretically, you can modify db data from the browser, e.g., hide some model, regionalization or predefined flatfile. Persistent changes should be implemented in Python code and then run a Complete DB reset
See egsim/api/management/commands/README.md.
The next two sections will describe how to store
new data (regionalizations and flatfiles) that will be
made available in eGSIM with the egsim_init command
(see Complete DB reset for details)
Add the file (CSV or zipped CSV) in
managements/commands/data/flatfiles.
If the file is too big try to zip it.
If it is more than few tens of Mb, then do not commit it (explain in
the section details - see below - how to get the source file).
When zipping in macOS you will probably need to
exclude or remove (after zipping) the MACOSX folder~~
Implement a new FlatfileParser class in
management/commands/flatfile_parsers. Take another parser, copy it
and follow instructions.
The parser goal is to read the file and convert it into a harmonized HDF
table
Add binding file -> parser in the Python dict:
management.commands._egsim_flatfiles.Command.PARSER
(Optional) Add the file refs
in management/commands/data/references.yaml, e.g. reference, url, the
file name that will be used in the API (if missing, defaults to the file
name without extension)
Repopulate all eGSIM tables (command egsim_init)
Implemented flatfiles sources (click on the items below to expand)
Go to https://esm.mi.ingv.it//flatfile-2018/flatfile.php (with username and password, you must be registered beforehand it's relatively fast and simple)
Download ESM_flatfile_2018.zip, uncompress and extract
ESM_flatfile_SA.csv from there
ESM_flatfile_SA.csv is our raw flatfile, compress it
again (it's big) into this directory as
ESM_flatfile_2018_SA.zip
If on macOS, type the command above to remove the macOS folder from the zip
Add two files with the same basename and extensions in
managements/commands/data/regionalization_files:
See already implemented files for an example
(Optional) Add the file refs
in management/commands/data/references.yaml, e.g. reference, url, the
file name that will be used in the API (if missing, defaults to the file
name without extension)
Repopulate all eGSIM tables (command egsim_init)