One of our core tenets at Convergent Science is that we regard our customer relationships as partnerships. We work closely with our clients to identify needs, track industry trends, and direct future development efforts.

Achieving a quick time-to-solution is always a high priority for our clients, and graphics processing units (GPUs) have garnered significant interest in recent years as a potential avenue to accelerate CFD simulations. Hoping to take advantage of their powerful parallel processing capabilities, our clients are eager to test their simulations on GPU architectures.

To support our clients in this endeavor, we began developing the CONVERGE GPU solver. The first version of this solver was released last year in CONVERGE 5. This year, we released an enhanced version of the GPU solver in CONVERGE 6 with significantly expanded capabilities. While development efforts for a fully featured GPU solver are ongoing, the enhanced GPU solver in version 6 allows for multi-GPU simulations and can handle cases with compressible and incompressible flows, RANS and LES turbulence modeling, and detailed chemistry for combustion modeling.

To test the capabilities of the new CONVERGE 6 GPU solver, we turned to our partners at CoreWeave, a cloud computing company that specializes in providing cloud-based GPU infrastructure for AI and high-performance workloads. The goals of this collaborative project were twofold: (1) validate the accuracy of the GPU solver with a well-known test case, and (2) assess the scalability of the GPU solver on two types of GPU instances provided by CoreWeave.

To verify the accuracy of the GPU solver, we selected the widely studied isotropic turbulence decay case, which ensured we would have access to literature data to validate our results. Turbulence decay occurs at the small scales of all turbulent systems—think of air flow over an airplane wing or gas in a centrifugal compressor—making this phenomenon relevant to a wide range of engineering problems.

We simulated the turbulent flow in a cubic domain with a turbulent Mach number of 0.2, and we used direct numerical simulation (DNS) to solve the flow field. Figure 1 shows a visualization of the case.

The cube was parameterized by N grid points on each side, resulting in a total grid size of N3. We ran the case using the CONVERGE 6 GPU solver with three different grid sizes: N = 128, 256, and 512. The cases were run on 1, 8, and 16 NVIDIA RTX PRO 6000 GPUs, respectively.

We compared the results for the time decay of turbulent kinetic energy (TKE) with filtered DNS results from literature.1 As we can see from Figure 2, the CONVERGE results for all three grid sizes match well with the literature data, demonstrating that the GPU solver is accurately predicting the turbulence decay.

Having validated the accuracy of the GPU solver for our test case, we moved on to the second aim of the study: assessing scalability. We conducted scaling tests on two types of GPUs provided by CoreWeave: workstation GPUs (NVIDIA RTX PRO 6000) and data center GPUs (NVIDIA HGX B200).

We assessed both weak and strong scaling on each GPU. Weak scaling involves increasing both the size of the case and the computing resources proportionally, ensuring that the computational load on each GPU remains the same. The goal of weak scaling is to run a larger or more detailed case without increasing the overall simulation runtime. Consequently, to achieve ideal weak scaling, the wall clock time would stay constant as both the case size and the number of GPUs are increased.

The goal of strong scaling, on the other hand, is to run the same case faster. The case is thus kept the same size while the computational resources on which you're running the simulation are increased. Ideal strong scaling efficiency would be obtaining a 2x speedup when the number of GPUs is doubled.

For our scaling studies, we used the N = 256 case setup described above as our base case and elongated the domain in a single direction to achieve the desired cell count for each simulation.

Digging into the results, we'll start with the weak scaling study on the RTX PRO 6000 GPUs. This study looked at intranode scaling, meaning all of the GPUs were on a single node. We ran the case on 2, 4, and 8 GPUs, keeping the cell count per GPU constant (~8.4 million cells). The results are shown in Figure 3(a); on 8 GPUs, the CONVERGE GPU solver achieved a scaling efficiency of 74%.

For the strong scaling study, we used an elongation factor of 4, resulting in approximately 67.1 million cells. We ran the case on 2, 4, and 8 GPUs across a single node. The results are shown in Figure 3(b). On 8 GPUs, we saw a 2.75x speedup compared to 2 GPUs.

Compared to the RTX PRO 6000 GPUs, the NVIDIA HGX B200 GPUs provide significantly more compute power and memory, enabling us to run cases with larger cell counts.

For the HGX B200 intranode weak scaling study, we ran the case on 1, 2, and 4 GPUs using a constant cell count per GPU of ~33.6 million. Because we were using a larger cell count, we were only able to scale up to 4 GPUs before the case size exceeded the memory constraints of the hardware. Figure 4(a) shows the results of the weak scaling study, demonstrating a 71.6% scaling efficiency on 4 GPUs.

For our strong scaling study on the B200 GPUs, we used the N = 512 case setup with an elongation factor of 2, for a total of ~268.4 million grid points. This study tested scaling across multiple nodes (i.e., internode scaling) to allow more memory per CPU for the larger case. We ran the fixed-size case on 8, 12, and 16 GPUs; the results are shown in Figure 4(b). On 16 GPUs, we saw a 1.36x speedup compared to 8 GPUs.

Generally, we expect a lower internode scaling efficiency compared to intranode scaling, since communication between nodes tends to be slower than within a single node. In this case, the B200 nodes employ NVLink, a high-speed direct GPU-to-GPU interconnect technology. This allows GPUs within a node to communicate with very high bandwidth and very low latency. InfiniBand is used for internode communication, and while InfiniBand is still fast, it has significantly lower bandwidth and higher latency, which directly leads to lower scaling efficiency.

Furthermore, achieving near-ideal scaling on GPUs is particularly tricky for a CFD code like CONVERGE. We employ high numerical precision (FP64) for our calculations, which is required for combustion applications—an area in which CONVERGE excels. The problem arises because the high-precision numerics create a high memory load. GPUs have a fixed amount of memory, so a CONVERGE simulation will typically exceed the available memory before the GPU is fully saturated. Especially for a high-performance node like a B200, this means the GPU won't be operating at maximum efficiency, which negatively impacts scaling.

"My team at Convergent Science is actively working on methods to improve scaling by reducing memory usage and exploring more scalable algorithms for GPUs. With these changes, users can expect to see a significant improvement in scaling efficiency in future versions of the GPU solver," says Cooper Burns, HPC Team Lead at Convergent Science. "Given where we are in the development cycle, we're pleased with the speedup we're achieving across multiple GPUs."

Outside of our scaling studies, we wanted to see how far we could push our current setup. We used the N = 512 case with an elongation factor of 4 for a total of 536 million cells. On 16 HGX B200 GPUs, we were able to simulate five eddy turnovers in under four hours.

"This is the largest case we have run so far with the CONVERGE GPU solver. These results demonstrate that the solver is capable of handling cases with very large cell counts when high mesh resolution is required for increased accuracy," says Cooper.

When running large jobs like this, it helps to know the platform you're running on is designed to handle demanding caseloads and comes equipped with safety measures to mitigate any potential problems that might interrupt your job.

"Scaling a solver from one GPU to 16 across nodes puts pressure on everything, not just the GPUs," says Justin Hodges, Distinguished Engineer for Physical AI at CoreWeave. "Convergent Science ran on bare metal, on a fabric built for the collective operations CFD depends on. Take those runs into production and CoreWeave Mission Control handles the next part: catching a degrading GPU or link before it takes down a multi-day job."

In this study, we set out to validate the accuracy and test the scalability of the CONVERGE GPU solver. We chose a canonical test case of decaying turbulence, which is commonly used to validate CFD codes, and teamed up with CoreWeave to run our simulations on their state-of-the-art GPU cloud offerings. We demonstrated that the CONVERGE GPU solver accurately predicted the relevant physics and scaled efficiently on two types of GPUs.

"Running the CONVERGE GPU solver on CoreWeave hardware was a seamless experience for our team," says Cooper. "We received excellent technical support from the CoreWeave team, and their easy-to-use platform, including comprehensive monitoring dashboards, enabled very short turnaround times for performing these simulations."

What do the results of this study mean for our clients? Our customers expect a high level of accuracy and fast time-to-solution when they run CONVERGE simulations, and we're working hard to ensure this experience is consistent across both CPUs and GPUs. Demonstrating accuracy on canonical turbulence benchmarks is a fundamental requirement to modeling real-world engineering systems, and the results we've shown here should give our clients confidence that they can trust our GPU solver to help them solve tough problems in their industry.

The CONVERGE GPU solver is under active development, and additional modeling capabilities will be released in upcoming versions. We're excited to continue to support our clients in an evolving computing landscape, ensuring that they can take advantage of the latest and greatest hardware options on the market to run ever larger, faster, and more accurate simulations.

Check out CoreWeave's blog post to learn more about the hardware used in this study, and contact us below for more information on the CONVERGE GPU solver!