Reef3D

In this guide, we will walk you through setting up and running REEF3D simulations using the Inductiva API.

We will cover:

  • Configuring REEF3D simulations using the Inductiva API.

  • Allowed commands for running Reef3D

  • Example code to help you get started with simulations.

  • An advanced example that uses the API to run the “3D Dam Break Scenario with Obstacle” that can be found in Reef3D tutorials.

Reef3D#

REEF3D is an open-source computational fluid dynamics (CFD) solver designed for simulating free-surface flows and coastal engineering applications. It supports a wide range of functionalities, including wave modeling, sediment transport, and hydrodynamics.

REEF3D employs a fully coupled numerical approach to handle complex fluid-structure interactions, making it an essential tool for projects in coastal protection, offshore engineering, and environmental assessments. The solver’s ability to model free-surface flows, wave energy devices, and sediment transport allows engineers to simulate real-world conditions with high accuracy.

Reef3D Key Modules#

The modular programming approach allows the framework to incorporate a range of different flow solvers which together represent all relevant length scales. Depending on the wave or flow conditions, the following optimized hydrodynamic modules are available:

  • REEF3D::CFD solves the Navier-Stokes equations in three dimensions. For near-field simulations with a complex free surface pattern, it uses a two-phase flow approach with the level set method for interface capturing.

  • REEF3D::FNPF is a three-dimensional fully nonlinear potential flow solver. It is massively parallelized and can be used to create large-scale phase-resolved sea states at all water depths.

  • REEF3D::SFLOW is a depth-averaged model, solving the non-hydrostatic shallow water equations ideal for near-shore hydrodynamics and river flow.

Running a Reef3D Simulation#

Reef3D in Inductiva API executes two sequential steps:

  • the meshing with DiveMESH;

  • the simulation with Reef3D.

Each step is configured with input files, control.txt and ctrl.txt, respectively. Other files may be used to inform the simulator about the grid, geographical data or wave information. Reef3D has strict naming policies for each file and we recommend users to follow their guidelines.

To run the simulation users pass a folder with the above files and all others required to run the simulation. The folder is then uploaded to Inductiva API and the simulation is executed. The output files are then downloaded to the user’s machine.

The parallelization of the simulation is handled automatically by Inductiva API, based on the number of cores available in the machine.

General Arguments:

  • on: set the machines where the simulations will run. Check here for further detail.

  • storage_dir: set the directory where the output files will be stored in the cloud. If not selected the output files will be stored in a folder named with the task id of the simulation.

  • n_vcpus: number of virtual CPUs / threads that will be used to configure the MPI parallism. This number needs to be set consistently with parameter M 10 to be set in both control.txt and ctrl.txt configurations files.

For further information on handling the task of the simulation see here.

Example Code#

In this example, we follow the tutorial with regular wave propagation from Reef3D repository.

"""Reef3D example."""
import inductiva

# Instantiate machine group
machine_group = inductiva.resources.MachineGroup("c2-standard-4")
machine_group.start()

input_dir = inductiva.utils.download_from_url(
    "https://storage.googleapis.com/inductiva-api-demo-files/"
    "reef3d-input-example.zip",
    unzip=True)

reef3d = inductiva.simulators.REEF3D()

task = reef3d.run(input_dir=input_dir, on=machine_group)

task.wait()
task.download_outputs()

machine_group.terminate()

Advanced Tutorial: 3D Dam Break Scenario with Obstacle#

Let’s now run a more advanced example, one that will also require a lot more compute power, and will illustrate more advanced features of the API. More specifically, we will use the API to run the “3D Dam Break Scenario with Obstacle” that can be found in Reef3D tutorials.

Video with the magnitude of the velocity field produced with Paraview using VTK files generated by Reef3D

Aim of This Tutorial#

We’ll run this simulation on:

  1. Single 112 vCPU Machine.

  2. MPI Cluster using two 112 vCPU machines to improve performance.

Prerequisites#

  1. Download Input Files: Get the input files from the Reef3D tutorials.

    Directory Structure:

    ls -lasgo .
 total 16
 0 drwxrwxr-x@  4   128 Sep  4 08:46 .
 0 drwxrwxr-x@ 19   608 Nov  5 09:03 ..
 8 -rw-rw-r--@  1   142 Sep  4 08:46 control.txt
 8 -rw-rw-r--@  1   141 Sep  4 08:46 ctrl.txt

    control.txt (for DiveMESH):

    C 11 21
C 12 21
C 13 21
C 14 21
C 15 21
C 16 21

B 1 0.025
B 10 0.0 2.0 0.0 1.0 0.0 1.0
O 10 1.2 1.4 0.4 0.6 0.0 1.0

M 10 4    ---- defines the nr. of processors for parallel computations (4)

    ctrl.txt (for Reef3D):

    D 10 4
D 20 2
D 30 1
F 30 3
F 40 3
F 54 0.5
F 56 0.7
N 40 3
N 41 25.0    ---- set the maximum modeled time (25 seconds).
N 45 50000
N 47 0.2
M 10 4    ---- defines the nr. of processors for parallel computations (4)
P 10 1
P 30 0.01    ---- defines the rate of paraview results (1 frame per 0.01 s)
T 10 0
W 22 -9.81

    Before proceeding, let’s review three key parameters in the Reef3D configuration files:

    • N 41: Maximum simulation time

    • P 30: Result output frequency

    • M 10: Number of processors for parallel computation

Overview#

Here’s the code you’ll be working on as we progress through the tutorial. Don’t worry if it doesn’t all make sense right now; everything will become clearer in the upcoming steps.

import inductiva

machine_group = inductiva.resources.MachineGroup(
					machine_type="c2d-highcpu-112",
					spot=True,
					data_disk_gb=20,
					auto_resize_disk_max_gb=100)

machine_group.start()

input_dir = "path/to/10_2 3D Dam Break with Obstacle"

#Choose your simulator
reef3d = inductiva.simulators.REEF3D()

task = reef3d.run(
	input_dir=input_dir,
	on=machine_group,
	n_vcpus=56,
	use_hwthread=False,
	storage_dir="3D_dam_break_with_obstacle")

task.wait()
machine_group.terminate()

Step 1: Adjust Simulation Parameters#

For a faster simulation, modify the following parameters in both files:

  • Level of parallelism (M 10): 56

  • Simulation time (N 41): 25.0

  • Paraview results rate (P 30): 0.01

These parameters will enable us to run the simulation more quickly, limiting it to 25 seconds. Additionally, we’ll generate a Paraview output every 0.01 seconds.

Step 2: Running the Simulation#

a. Configure and Start Machine#

  1. Pick your machine:

    In this simulation, we’ll use a c2d-highcpu-112 machine with a 20 GB disk. Automatic disk resizing is enabled, so the disk size will increase as needed, up to the specified maximum of auto_resize_disk_max_gb.

    import inductiva
machine_group = inductiva.resources.MachineGroup(
					machine_type="c2d-highcpu-112",
					spot=True,
					data_disk_gb=20,
					auto_resize_disk_max_gb=100)

    Note: spot machines are a lot cheaper but can be terminated by the provider if needed.

  2. Start your machine

    machine_group.start()

b. Define Simulation Inputs#

  1. Specify Simulation Directory: Let’s start by defining a variable that points to the 10_2_3D_Dam_Break_with_Obstacle folder where all your simulation files are located.

    input_dir = "./10_2_3D_Dam_Break_with_Obstacle"

c. Run Your Simulation#

  1. Run the simulation: We now have everything we need to run our simulation.

    #Choose your simulator
reef3d = inductiva.simulators.REEF3D()

task = reef3d.run(
	input_dir=input_dir,
	on=machine_group,
	n_vcpus=56,
	use_hwthread=False,
	storage_dir="3D_dam_break_with_obstacle")

    In this snippet, two arguments might need clarification:

    • n_vcpus: This sets the number of virtual CPUs (vCPUs) for your simulation, essentially determining how many parts your simulation will be split into to run in parallel. Here, we’re dividing the simulation into 56 parts and running each part simultaneously.

    • use_hwthread: This argument is set to False to disable hyper-threading. Since we are only using 56 vCPUs, we don’t need hyper-threading to run the simulation.

    • storage_dir: This is the directory where the simulation outputs will be stored. You can access these outputs once the simulation is complete.

  2. Wait for the simulation to finish: That is it. Our simulation is now running on the cloud. We can wait for the simulation to finish while we grab a coffee (☕️). We will be skipping the download_outputs due to the fact that this simulation generates arround 12 GB of data. If you need to download your outputs just run task.download_outputs().

    task.wait()

    Note: run inductiva logs task_id to check the stdout of the simulation process in real time.

  3. Terminate Machine: Once our simulation completes, we can/should terminate our machine to save on costs. If you forget, don’t worry—we’ve got you covered. By default, the machine will automatically shut down if idle for 30 minutes with no simulation running.

    machine_group.terminate()

  4. Check your simulation summary: Now that our simulation is complete, we can print a summary with key details, including execution times, generated outputs, and more.

    task.print_summary()

Step 3: Enhancing Performance with MPI Cluster#

You might’ve noticd that simulations can take quite a while. To cut down on runtime, we can change our machine configuration to an MPI cluster with two machines:

mpi_cluster = inductiva.resources.MPICluster(
                  machine_type="c2d-highcpu-56",
                  data_disk_gb=100,
                  num_machines=2)
mpi_cluster.start()

# Re-run the simulation with adjusted `n_vcpus`
task = reef3d.run(
		input_dir=input_dir,
		on=mpi_cluster,
		n_vcpus=56,
		use_hwthread=True,
		storage_dir="3D_dam_break_with_obstacle_cluster")

As you can see, scaling up (or down) is easy—just select a new resource. We encourage you to experiment with different machines and configurations!

Conclusion#

Running the simulation on a high-performance machine or scaling it across an MPI cluster can significantly reduce computation time. However, in our case, this particular simulation does not benefit much from these optimizations due to its relatively low computational demands. The performance gain depends heavily on both the simulation’s complexity and the capabilities of the simulator being used.

Download and analyze your results locally once complete.

Happy simulations!