Manage and Retrieve Results

Manage and Retrieve Results#

The output results of any simulation are the focal point of the entire simulation process. It is, therefore, important to adequately manage and retrieve these outputs in a way that is both efficient and organized. In this section, we will show how to configure where the simulation outputs are saved in the remote storage space and how to retrieve them to the user’s local machine. In this document, we will cover the following topics:

  • Saving simulation outputs: Learn how to configure where the outputs of a simulation are stored in the user’s remote storage;

  • Downloading outputs: Learn how to download the outputs of a task and control where the files are downloaded to.

Most of the topics that follow rely on the Task class. Please refer to the section about Tasks on our tutorials for more information about it.

Saving simulation outputs#

By default, the outputs generated by a simulation are saved to a directory named after the task id. In the following snippet, the outputs of task with id i4ir3kvv62odsfrhko4y8w2an are saved to a directory named i4ir3kvv62odsfrhko4y8w2an:

import inductiva

# Instantiate machine group
machine_group = inductiva.resources.MachineGroup("c2-standard-4")
machine_group.start()

simulator = inductiva.simulators.REEF3D()

task = simulator.run(input_dir="path-to-directory-with-input-files-for-reef3d",
                     on=machine_group)
print(task.id)  # will print i4ir3kvv62odsfrhko4y8w2an

# Terminate the machine group
machine_group.terminate()

Using the command line interface (CLI), the user can list the contents of his remote storage space to confirm the presence of the directory i4ir3kvv62odsfrhko4y8w2an:

$ inductiva storage list --max-results 2 --order-by creation_time --sort-order desc

       NAME                             SIZE           CREATION TIME
       i4ir3kvv62odsfrhko4y8w2an/       21.67 MB        07 Feb, 13:47:03

If the user wishes to save the outputs of the simulation to a different directory, he can do so by setting the storage_dir argument in the run method of the simulator:

task = simulator.run(input_dir="path-to-directory-with-input-files-for-reef3d"
                     storage_dir="my_reef3d_simulation")

Again, using the CLI, the user can now confirm that the directory my_reef3d_simulation was indeed created in his remote storage space:

$ inductiva storage list --max-results 2 --order-by creation_time --sort-order desc

       NAME                             SIZE           CREATION TIME
       my_reef3d_simulation/            21.67 MB       07 Feb, 14:57:11
       i4ir3kvv62odsfrhko4y8w2an/       21.67 MB       07 Feb, 13:47:03

Listing of the contents of the user’s remote storage space can also be accomplished programmatically using the inductiva.storage.listdir function:

>>> import inductiva
inductiva.storage.listdir(max_results=10, order_by="creation_time", sort_order="desc")

Refer to the Storage how-to documentation for more information on how to manage the user’s remote storage space.

Downloading outputs#

Downloading the outputs of a task is a common operation. In the following snippets, we show how to download the outputs of a finished task and how to control which files and where those files are downloaded to.

# Download all the files produced by the task
>>> output_dir = task.download_outputs()
Downloading simulation outputs to inductiva_output/i4ir3kvv62odsfrhko4y8w2an/output.zip.
100%|██████████████████████████████████████| 5.04M/5.04M [00:00<00:00, 13.3MB/s]
Uncompressing the outputs to inductiva_output/i4ir3kvv62odsfrhko4y8w2an/
>>> print(ouptut_dir)
PosixPath('inductiva_output/i4ir3kvv62odsfrhko4y8w2an')

The downloaded files can be inspected using the ls command:

$ ls -1 inductiva_output/i4ir3kvv62odsfrhko4y8w2an
log
stderr.txt
stdout.txt
vtk

By default, all output files are downloaded to a directory named after the task into a parent directory named inductiva_output. For example, the outputs of the task with id abc123 would be downloaded to inductiva_output/abc123/. The name of the parent directory is a package-level variable that applies to all downloads with a session. It prevents the user from cluttering the current working directory with a large number of downloaded files. However, the user can change this behavior by setting the parent directory name using the inductiva.set_output_dir function.

>>> import inductiva
# set the parent directory name
>>> inductiva.get_output_dir()
'inductiva_output'
>>> inductiva.set_output_dir("my_vault")
>>> inductiva.get_output_dir()
'my_vault'

# download the outputs again
>>> output_dir = task.download_outputs()
Downloading simulation outputs to my_vault/i4ir3kvv62odsfrhko4y8w2an/output.zip.
100%|██████████████████████████████████████| 5.04M/5.04M [00:00<00:00, 13.3MB/s]
Uncompressing the outputs to my_vault/i4ir3kvv62odsfrhko4y8w2an/
>>> print(ouptut_dir)
PosixPath('my_vault/i4ir3kvv62odsfrhko4y8w2an')

# Unset the parent directory name so that all downloads are done to the current
# working directory
>>> inductiva.set_output_dir(None)
>>> output_dir = task.download_outputs()
Downloading simulation outputs to i4ir3kvv62odsfrhko4y8w2an/output.zip.
100%|██████████████████████████████████████| 5.04M/5.04M [00:00<00:00, 13.3MB/s]
Uncompressing the outputs to i4ir3kvv62odsfrhko4y8w2an/
>>> print(ouptut_dir)
PosixPath('i4ir3kvv62odsfrhko4y8w2an')

The download_outputs method of the Task class has the option to specify the name of the folder where the outputs are downloaded inside the parent directory.

# reset the parent directory name to the default
>>> inductiva.set_output_dir("inductiva_outputs")
>>> output_dir = task.download_outputs(output_dir="my_outputs")
Downloading simulation outputs to inductiva_output/my_outputs/output.zip.
100%|██████████████████████████████████████| 5.04M/5.04M [00:00<00:00, 13.3MB/s]
Uncompressing the outputs to inductiva_output/my_outputs/
>>> print(ouptut_dir)
PosixPath('inductiva_output/my_outputs')

In addition to controlling how downloaded files are organized, the user can also configure which files to download. By default, all files produced by the task are downloaded. However, the user can select to retrieve only a subset of those files by specifying the filenames of interest:

# Download only the files of interest
>>> output_dir = task.download_outputs(filenames=["stdout.txt", "stderr.txt"]
                                       output_dir="my_outputs")
Downloading simulation outputs to inductiva_output/my_outputs/output.zip.
100%|██████████████████████████████████████| 1.04k/1.04k [00:00<00:00, 671kB/s]
Uncompressing the outputs to inductiva_output/my_outputs/

In this case, we can confirm that only the files stdout.txt and stderr.txt were indeed downloaded:

$ ls -1 inductiva_output/my_outputs
stderr.txt
stdout.txt