Get started
912 commits | Last commit ≈ 1 month ago
A Python library and web application to select and test Ground Motion models. Online at: https://egsim.gfz-potsdam.de
eGSIM is a web service for selecting and testing ground shaking models (GSIM) in Europe, developed a the GFZ in the framework of the Thematic Core Services for Seismology of EPOS under the umbrella of EFEHR
The web portal (and API documentation) is available at:
Zaccarelli, Riccardo; Weatherill, Graeme (2020): eGSIM - a Python library and web application to select and test Ground Motion models. GFZ Data Services. https://doi.org/10.5880/GFZ.2.6.2023.007
DISCLAIMER: This document covers installation in development (or debug) mode, i.e. when the program is deployed locally, usually for testing, fixing bug or adding features.
For installation in production mode, see deploy.html
sudo apt-get update # pre-requisite
sudo apt-get install gcc # optional
sudo apt-get install git python3-venv python3-pip python3-dev
(The command above are Ubuntu specific, in macOS install brew and type
brew install instead of apt-get install. Remove python3-dev as it does not
exist on macOS).
This web service uses a specific version of Python (Open setup.py and
check python_requires=. As of January 2022, it's 3.9.7) which you must
install in addition to the Python version required by your system, and use
it. Any command python3 hereafter will refer to the required Python version.
Select a root directory (e.g. /root/path/to/egsim), and clone egsim into the
so-called egsim directory:
git clone https://github.com/rizac/eGSIM.git egsim
Move to whatever directory you want (usually the egsim directory above) and then:
python3 -m venv .env/<ENVNAME> # create python virtual environment (venv)
source .env/<ENVNAME>/bin/activate # activate venv
NOTE: From now on, all following operations must have the virtualenv activated FIRST
Note: if the installation is done for upgrading all dependencies and
pip freeze into new requirements files,
please go to dependency upgrade
On the terminal, execute:
pip install --upgrade pip setuptools && pip install -r requirements.dev.txt
(use requirements.txt if you don't need to run tests, e.g. you are not
installing as developer)
Note: the value of DJANGO_SETTINGS_MODULE in the examples below
must be changed in production
Move in the egsim directory and type:
export DJANGO_SETTINGS_MODULE=egsim.settings_debug; pytest -xvvv --ds=egsim.settings_debug ./tests/
(x=stop at first error, v*=increase verbosity).
For PyCharm users, you need to configure the environment variable for all tests. Go to: Run -> Edit Configurations -> Edit Configurations templates -> Python tests -> pytests -> ENVIRONMENT VARIABLES
and add there the environment variable DJANGO_SETTINGS_MODULE
Other test options from the command line: with coverage ( showing the amount of code hit by tests):
export DJANGO_SETTINGS_MODULE=egsim.settings_debug; pytest --ignore=./tests/tmp/ -xvvv --cov=./egsim/ --cov-report=html ./tests/
(you can also invoke the commands without export ... but
using the --ds option: pytest -xvvv --ds=egsim.settings_debug ./tests/)
Note: the value of DJANGO_SETTINGS_MODULE in the examples below
must be changed in production
If you didn't do already, perform a Complete DB reset (one-time only operation)
If you want to access the admin panel, see the admin panel.
To run the program in your local browser, type:
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py runserver
(you can also invoke the commands without export ... but using the
--settings option:
python manage.py --settings=egsim.settings_debug [command])
Settings file: A Django settings file contains all the configuration of your Django installation. The settings file referred in this document, included in this git repo, is for debug and local deployment only. On production, a separate settings file is used, located on the server outside the git repo and not shared for security reasons.
manage.py or
django-admin is Django’s command-line utility for administrative tasks.
It is invoked from the terminal within your Python virtualenv (see examples
in this document) by providing the settings file via:
export DJANGO_SETTINGS_MODULE=<settings_file_path> python manage.py <command>
Django allows also the implementation of custom management commands.
eGSIM implements egsim-init in order to populate the db (more details
below)
app a
Django app is a Python package that is specifically intended for use in
a Django project. An application may use common Django conventions, such as
having models, tests, urls, and views submodules. In our case, the Django
project is the egsim root directory (created with the command
django-admin startproject egsim), and the Django apps inside it are
"api" (the core web API) and "app" (the web app, i.e. the part of eGSIM
delivered over the Internet through a browser interface), that relies on
the "api" code.
Inside the settings file (variable INSTALLED_APPS) is configured the list
of all applications that are enabled in the eGSIM project. This includes not
only our "api" app, that tells Django to create the eGISM tables when
initializing the database, but also several builtin Django apps, e.g. the
Django admin app, visible through the Admin panel).
Note: the value of DJANGO_SETTINGS_MODULE in the examples below
must be changed in production
Typing python on the terminal does not work if you need to import django
stuff, as there are things to be initialized beforehand. The Django shell
command does this:
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py shell
Note: the value of DJANGO_SETTINGS_MODULE in the examples below
must be changed in production
We perform a complete DB reset every time we change something in the Database
schema (see egsim.api.models.py), e.g. a table, a column, a constraint.
The usual way to change a DB in a web app is to create and run migrations (Django full details here), which allow to keep track of all changes (moving back and forth if necessary) whilst preserving the data stored in the DB. However, none of those features is required in eGSIM: DB data is predefined and would be regenerated from scratch in any case after any new migration. Consequently, upon changes in the DB, a complete DB reset is an easier procedure.
In any case (just for reference), the steps to create and run migrations in eGSIM are the following:
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py makemigrations egsim --name <migration_name>
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py migrate egsim
And then remember to Repopulate the db
Notes:
make_migration command just generates a migration file, it doesn't
change the db. The migrate command does that, by means of the migration
files generated. For details on Django migrations, see:
git).migrate, if the migration
will introduce new non-nullable fields, maybe better to run
manage.py flush first to empty all tables, to avoid conflicts
"egsim" above is the app name. If you omit the app, all apps will be
migrated. The command migrate does nothing if it detects that there is
nothing to migrateTo perform a complete db reset:
delete or rename db.sqlite3 (or wherever the database is. See settings file in case)
[Optional: skip if the db schema hasn't changed] delete all migrations,
i.e. all *.py files except __init__.py currently under egsim/api/migrations,
and re-generate the initial (and only) migration file:
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py makemigrations
Run migrations (re-create the DB via the generated migration file):
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py migrate
[Optional] re-add the Django admin superuser(s) as explained in the admin panel above
NOTE: If you don't have created and initialized the db yet, go to Complete DB reset
When the db is created for the first time or reset,
OpenQuake is upgraded, or new data is implemented (new regionalization
or flatfile), you need to repopulate the database and the filesystem,
to make changes available to eGSIM users.
Execute the custom management command egsim_init:
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py egsim_init
Note: the value of DJANGO_SETTINGS_MODULE in the examples below
must be changed in production
This command allows the user to check database data from the web browser. For further details, check the Django doc
The database must have been created and populated (see Usage).
Create a super user (to be done once only ):
export DJANGO_SETTINGS_MODULE="egsim.settings_debug";python manage.py createsuperuser
and follow the instructions.
Start the program (see Usage) and then navigate in the browser to
[SITE_URL]/admin (in development mode, http://127.0.0.1:8000/admin/)
Note: Theoretically, you can modify db data from the browser, but to make
changes persistent after a database reset you should implement them in
the management command egsim_init (see Repopulate the DB)
See egsim/api/management/commands/README.md.
The next two sections will describe how to store new data (regionalizations and flatfiles) that will be made available in eGSIM when repopulating the db
Add the file (CSV or zipped CSV) in
managements/commands/data/predefined_flatfiles.
Do not commit files too big (max few tens of Mb). When zipping in
macOS you will probably need to
exclude or remove (after zipping) the ___MACOSX folder
Implement a new FlatfileParser class in
management/commands/flatfile_parsers.py
The parser goal is to read the file and convert it into a harmonized HDF
table (see ESM parser in the Python file)
Add binding file -> parser in the Python dict:
management.commands._egsim_flatfiles.Command.PARSER
(Optional) Add the file data source
in management/commands/data/data_sources.yaml, e.g. data reference, url,
(see examples in the YAML file).
NOTE: In the data source you can also set the data name, i.e. a unique, usually short alphanumeric string that will identify the flatfile in user requests. If no data source or name is provided, the public name will be the file name before the first ".".
Repopulate all eGSIM tables (command egsim_init)
Implemented flatfiles sources (click on the items below to expand)
Go to https://esm.mi.ingv.it//flatfile-2018/flatfile.php (with username and password, you must be registered beforehand it's relatively fast and simple)
Download ESM_flatfile_2018.zip, uncompress and extract
ESM_flatfile_SA.csv from there
ESM_flatfile_SA.csv is our raw flatfile, compress it
again (it's big) into this directory as
ESM_flatfile_2018_SA.zip
If on macOS, type the command above to remove the macOS folder from the zip
Add two files with the same basename and extensions
in managements/commands/data/regionalization_files. Usually, these files
are copied and pasted from the shakyground2 project (see GFZ gitlab), but
if you neeed to implement your own see examples in the given directory
or ask the developers
(Optional) Add the file data source
in management/commands/data/data_sources.yaml, e.g. data reference, url,
(see examples in the YAML file).
NOTE: In the data source you can also set the data name, i.e. a unique, usually short alphanumeric string that will identify the regionalization in user requests. If no data source or name is provided, the public name will be the file name before the first ".".
Repopulate all eGSIM tables (command egsim_init)
Please note that it is safer (even
mandatory
with pip) to upgrade all dependencies
instead of single packages in order to avoid conflicts.
Consequently, we recommend to follow this procedure also in case of a GitHub security issue (or dependency alert) on a single package.
First create a new virtual env (in the example below,
in the git-ignored .env inside
the eGSIM root directory,
but you can also create it wherever you want)
python3 -m venv .env/<ENVNAME> # create python virtual environment (venv)
source .env/<ENVNAME>/bin/activate # activate venv
pip install --upgrade pip setuptools
Then cd into eGSIM if you are not there already
(basically, the directory where setup.py is located),
install and freeze
the installed packages into a requirements.txt file:
pip install -e .
pip freeze > requirements.txt
Do the same for testing:
pip install -e ".[test]"
pip freeze > requirements.dev.txt
Run tests (see above) to check that everything wortks as expected, and then commit and push.