Performance

Python scripts

This chapter explains how to control OVITO’s use of processor cores and how to process trajectories in parallel using Python’s multiprocessing module. It closes with general tips for writing efficient scripts.

Controlling how many processor cores OVITO uses

Many of OVITO’s computations run in parallel on all available processor cores. This includes, for example, analysis modifiers such as the PolyhedralTemplateMatchingModifier, the VoronoiAnalysisModifier, and the ComputePropertyModifier, as well as the software rendering engines TachyonRenderer and OSPRayRenderer. The number of cores OVITO uses by default is given by Qt’s QThread.idealThreadCount() function:

>>> from ovito.qt_compat import QtCore
>>> print(QtCore.QThread.idealThreadCount())
8

Sometimes you want to restrict OVITO to fewer cores, for example, when running several scripts in parallel on the same machine. To do so, set the environment variable OVITO_THREAD_COUNT before starting the Python interpreter or importing the ovito module:

export OVITO_THREAD_COUNT=1
python3 yourscript.py

The ovitos interpreter additionally supports the command line option --nthreads (see Number of CPU cores):

ovitos --nthreads 1 yourscript.py

Using OVITO with Python’s `multiprocessing` module

Some of OVITO’s computations always run on a single core, because the algorithm is hard to parallelize or relies on a third-party library without multithreading support. The same is true for any Python code in your own modifier functions. If you need to process many trajectory frames or files, which are independent of each other, you can use Python’s multiprocessing module to process them in parallel in several worker processes.

In a nutshell:

Multiprocessing works only in Python scripts run by a Python interpreter, not in code running inside the OVITO Pro desktop application.
Objects from the ovito package cannot be pickled and therefore cannot be passed between processes. The worker function must accept and return only regular Python objects, e.g. numbers, strings, or NumPy arrays.
Because OVITO starts background threads, it is not compatible with the fork start method of the multiprocessing module. You must use the spawn start method on all platforms.

Consider the following script, which computes a result for each frame of a trajectory in a serial loop:

pipeline = import_file('input/simulation.*.dump')
pipeline.modifiers.append(ClusterAnalysisModifier(cutoff = 3.0))

results = []
for data in pipeline.frames:
    num_clusters = data.attributes['ClusterAnalysis.cluster_count']
    results.append(num_clusters)

The order in which the frames are processed doesn’t matter here, which is the prerequisite for parallelization. To parallelize the computation, move the per-frame work into a function:

def process_frame(frame):
    data = pipeline.compute(frame)
    return data.attributes['ClusterAnalysis.cluster_count']

A multiprocessing.Pool then calls the function for all frames, distributing the work across several worker processes:

with multiprocessing.Pool(None) as pool:
    results = list(pool.imap(process_frame, range(pipeline.num_frames)))

Since the multiprocessing module now distributes the work across the processor cores, you should restrict each OVITO instance to a single thread by setting OVITO_THREAD_COUNT=1 to avoid an oversubscription of the cores. Whether this improves the overall performance depends on the problem; measure the execution times to find out.

The following complete program demonstrates the approach. It includes a flag to switch between parallel and serial processing, which lets you compare the execution times:

# This configuration flag enables the use of the Python multiprocessing module.
# Set it to false to switch back to conventional for-loop processing (included for comparison).
use_multiprocessing = True

if use_multiprocessing:
    # Disable internal parallelization of OVITO's pipeline system, which otherwise would perform certain
    # computations using all available processor cores. Setting the environment variable OVITO_THREAD_COUNT=1
    # must be done BEFORE importing the ovito module and will restrict OVITO to a single CPU core per process.
    import os
    os.environ["OVITO_THREAD_COUNT"] = "1"

from ovito.io import *
from ovito.modifiers import *

# Set up the OVITO data pipeline for performing some heavy computations for each trajectory frame.
# Keep in mind that this initialization code gets executed by EVERY worker process in the multiprocessing pool.
# So we should only prepare but not carry out any expensive operations here.
pipeline = import_file('input/simulation.*.dump')
pipeline.modifiers.append(ClusterAnalysisModifier(cutoff = 3.0))

# Define the worker function which gets called for every trajectory frame. It evaluates the pipeline
# at the requested simulation time and returns the computed results for that frame back to the caller.
#
# IMPORTANT: The function may only return regular Python objects but not OVITO objects
# such as DataCollection or Property. That's because OVITO objects do not currently support pickling,
# which means they cannot be sent back to the main program across process boundaries.
#
# In this simple example, the return value is a simple scalar (a global attribute) computed per frame.
# Note that it is possible to return NumPy arrays, which includes NumPy views of OVITO property arrays.
# Thus, to return the cluster assignment of each atom computed by the ClusterAnalysisModifier
# we could write:
#
#    return data.particles['Cluster'][...]
#
def process_frame(frame):
    data = pipeline.compute(frame)
    return data.attributes['ClusterAnalysis.cluster_count']

# Main program entry point:
if __name__ == '__main__':

    # Measure time for benchmarking purposes.
    from time import time
    t_start = time()

    if use_multiprocessing:

        # Force "spawn" start method on all platforms, because OVITO is not compatible with "fork" method.
        import multiprocessing as mp
        mp.set_start_method('spawn')

        # Create a pool of processes which will process the trajectory frames in parallel.
        with mp.Pool(None) as pool:
            results = list(pool.imap(process_frame, range(pipeline.num_frames)))

    else:

        # Conventional for-loop iterating over all frames of the trajectory and processing one by one.
        results = [process_frame(frame) for frame in range(pipeline.num_frames)]

    t_end = time()

    print(f"Computation took {t_end - t_start} seconds")

Note that the code at the top level of the script, outside the if __name__ == '__main__': block, gets executed by every worker process. Keep it cheap: set up the pipeline there, but don’t evaluate it.

Tips for efficient scripts

Set up the pipeline once. Build the pipeline before looping over frames or files, and only change its input or parameters inside the loop (see Processing simulation trajectories and Processing many files in a row). Creating a new pipeline for every file discards the caches OVITO maintains.
Use NumPy. Operate on entire property arrays with vectorized NumPy functions instead of looping over particles in Python.
Don’t recompute unnecessarily. Store the DataCollection returned by compute() in a variable if you need to access it several times.
Save memory for very large systems. Load particle data in single precision using the float_precision='float32' option of import_file() (see Floating-point precision of standard properties).
Find out where the time goes. Call ovito.enable_logging() to let OVITO print status messages about long-running operations.

Performance

Controlling how many processor cores OVITO uses

Using OVITO with Python’s multiprocessing module

Tips for efficient scripts

Using OVITO with Python’s `multiprocessing` module