LiberTEM is an open source platform for high-throughput distributed processing of large-scale binary data sets and live data streams using a stream-based programming model. The current focus is pixelated scanning transmission electron microscopy (4DSTEM) and scanning electron beam diffraction data.
LiberTEM is an open source platform for high-throughput distributed processing of large-scale binary data sets and live data streams using a modified MapReduce programming
model. The current focus is pixelated scanning transmission electron microscopy
(STEM) and scanning electron beam diffraction data.
MapReduce-like processing allows to specify an algorithm through two functions: One function that is mapped on portions of the input data, and another function that merges (reduces) a partial result from this mapping step into the complete result. A wide range of TEM and 4D STEM
processing tasks can be expressed in this fashion, see Applications.
The UDF interface of LiberTEM offers a standardized, versatile API to decouple the mathematical core of an algorithm from details of data source, parallelism, and use of results. Mapping and merging can be performed in any order and with different subdivisions of the input
data, including running parts of the calculation concurrently. That means the same implementation can be used in a wide range of modalities, including massive scaling on clusters. Since each merge step produces an intermediate result, this style of processing is suitable for displaying
live results from a running calculation in a GUI application and for processing live data
streams. A closed-loop feedback between processing and instrument control can be realized as well. See User-defined functions for more details on the LiberTEM UDF interface.
The LiberTEM back-end offers high throughput and scalability on PCs, single server nodes, clusters and cloud services. On clusters it can use fast distributed local storage on high-performance SSDs. That way it achieves very high aggregate IO performance on a compact and cost-efficient system built from stock components. All CPU cores and CUDA devices in a system can be used in parallel.
LiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow installation of Python 3.7+ and the required packages will likely work as well. The GUI is running in a web browser.
The short version:
$ virtualenv -p python3 ~/libertem-venv/ $ source ~/libertem-venv/bin/activate (libertem-venv) $ python -m pip install "libertem[torch]" # optional for GPU support # See also https://docs.cupy.dev/en/stable/install.html (libertem-venv) $ python -m pip install cupy
Please see our documentation for details!
Alternatively, to run the LiberTEM Docker image:
$ docker run -p localhost:9000:9000 --mount type=bind,source=/path/to/your/data/,dst=/data/,ro libertem/libertem
$ singularity exec docker://libertem/libertem /venv/bin/libertem-server
Deployment for offline data processing on a single-node system for a local user is thoroughly tested and can be considered stable. Deployment on a cluster is experimental and still requires some additional work, see Issue #105. Back-end support for live data processing is still experimental as well, see https://github.com/LiberTEM/LiberTEM-live.
Since LiberTEM is programmable through user-defined functions (UDFs), it can be used for a wide range of processing tasks on array-like data and data streams. The following applications have been implemented already:
The Python API and user-defined functions (UDFs) can be used for complex operations such as arbitrary linear operations and other features like data export. Example Jupyter notebooks are available in the examples directory. If you are having trouble running the examples, please let us know by filing an issue or by joining our Gitter chat.
LiberTEM is suitable as a high-performance processing backend for other applications, including live data streams. Contact us if you are interested!
LiberTEM is evolving rapidly and prioritizes features following user demand and contributions. Currently we are working on live data processing, improving application support for sparse data and event-based detectors, performance improvements for GPU processing, and implementing analysis methods for various applications of pixelated STEM and other large-scale detector data. If you like to influence the direction this project is taking, or if you'd like to contribute, please join our gitter chat and our general mailing list.
LiberTEM is licensed under GPLv3. The I/O parts are also available under the MIT license, please see LICENSE files in the subdirectories for details.
We are very grateful for your continuing support for LiberTEM!