Global Arrays underpins a number of computational chemistry codes, NWChem among them, which makes the correctness of its distributed operations a practical concern rather than an academic one. We recently built GA 5.9.2 on an AMD EPYC compute node with GCC, Open MPI 5.0.10 and OpenBLAS, added the test suite to our build recipe for the first time, and watched three of the seventy-nine tests fail. This post walks through how we established that the failures were upstream defects rather than anything wrong with our configuration, and what we decided to do about them.

The symptom

The build itself completed without a single error. The trouble only appeared once ctest ran:

96% tests passed, 3 tests failed out of 79

Total Test time (real) = 1174.68 sec

The following tests FAILED:
	 41 - bin (Timeout)
	 50 - mulmatpatch (Failed)
	 76 - nga-scatter (Failed)
Errors while running CTest

Each failing test aborted with the same line:

1:exiting ...:Received an Error in Communication
MPI_ABORT was invoked on rank 1 in communicator MPI COMM 3 DUP FROM 0

The error message is misleading

The MPI transport was the obvious first suspect, and that assumption proved incorrect. The wording of the message is the reason it misleads. Tracing the string back through the source leads to a single site in the two-sided COMEX backend:

/* comex/src-mpi/comex.c:1333 */
void comex_error(const char *msg, int code)
{
    fprintf(stderr, "%s", msg);
    fprintf(stderr,"Received an Error in Communication\n");
    MPI_Abort(l_state.world_comm, code);
}

That function prints the same sentence for every GA abort, whatever the cause. When a test detects a wrong numerical result and calls ga_error('exiting ...', 0) itself, the output still reads “Received an Error in Communication”. These were not transport failures at all. They were the tests correctly reporting that the data they got back was wrong.

Eliminating configuration as a cause

We rebuilt Global Arrays four times in separate scratch directories, varying the two settings most likely to be responsible, and reran the three failing tests against each build.

GA_RUNTIME	ENABLE_I8	Result
MPI_2SIDED	ON	All three fail
MPI_PROGRESS_RANK	ON	All three fail
MPI_RMA	ON	All three fail
MPI_2SIDED	OFF	All three fail

The same three tests fail identically across every combination tried.

The communication backend is therefore not responsible: progress ranks and MPI-3 RMA fail exactly as the two-sided path does. Nor are the eight-byte Fortran integers, which also clears the combination of SIZEOF_F77_INTEGER 8 against an LP64 OpenBLAS with BLAS_SIZE 4 — the mismatch we had suspected first, and which turns out to be handled correctly through the BlasInt conversion in GAI_DGEMM.

Dependence on process count

Running the test binaries by hand across different process counts sharpened the picture considerably:

mulmatpatch    n=1  PASS
mulmatpatch    n=2  FAIL
mulmatpatch    n=4  FAIL
nga-scatter    n=1  PASS
nga-scatter    n=2  FAIL
nga-scatter    n=4  FAIL
bin            n=1  PASS
bin            n=2  PASS
bin            n=4  PASS

Two observations follow. Firstly, the defects live in the distributed paths, since a single process never exercises them. Secondly, bin passed all three runs here despite having failed twice previously — once aborting in 0.2 seconds and once hanging past the 600 second timeout. It is not slow; it is subject to a race condition, and identical reruns of the identical binary produce different outcomes.

OpenBLAS is not involved

Because mulmatpatch exercises matrix multiplication, the BLAS pairing was an obvious suspect. It is exonerated by the tests that passed. Every test that drives BLAS directly came through cleanly: gemmtest, testmult, testmatmultc, testmultrect and mmatrix. We also confirmed that the Fortran hidden string-length convention was generated correctly as F2C_HIDDEN_STRING_LENGTH_AFTER_ARGS 1, which is right for gfortran.

A related observation: that macro is tested in galinalg.h with #if defined(...) whilst being emitted by #cmakedefine01. Anyone who switches the option off will get #define ... 0, which defined() still evaluates as true, and the wrong calling convention silently. It did not bite us, but it is a trap sitting there.

C passes where Fortran fails

The decisive observation was that test 10, mulmatpatchc, passed. It is the C version of the very test that fails in Fortran, and both call the same underlying routine — NGA_Matmul_patch from C, nga_matmul_patch from Fortran. The core algorithm is therefore sound, and the defect sits in the Fortran binding of that routine on the distributed path.

A bug in the test generator itself

The scatter tests are not written directly. They are generated from m4 templates, and reading the generated Fortran revealed an anomaly:

v(i) = int(drand(0) *  * 2)

Two asterisks separated by a space. The template at ngatest_src/ndim_NGA_SCATTER.src:61 calls v(i) = m4_rand with no argument, but ngatest.def:116 defines the macro as taking one:

define(m4_rand, `int(drand(0) * $1 * 2)')   # integer
define(m4_rand, `drand(0) * $1 * 2')        # double precision

With no argument supplied, $1 expands to nothing. Blanks are insignificant in fixed-form Fortran, so the surviving * * parses as exponentiation and the line becomes int(drand(0)**2). Since drand returns a value in the interval from zero to one, squaring it keeps it there, and the integer truncation makes it zero every single time.

The consequence is significant. The integer scatter test does not pass because scatter works. It passes because it compares zeros against zeros. The double precision variant has no int() wrapped around it to disguise the problem, which is precisely why that is the one reporting a mismatch. One of the tests that failed is the honest one.

Operational impact

Excluding a failing test does not repair the thing it was testing. On this build, nga_matmul_patch called from Fortran returns wrong results with more than one rank, and nga_scatter mismatches for double precision data. Any application sitting on top of Global Arrays that uses those routines from Fortran — and NWChem is the obvious candidate — can therefore receive silently incorrect numbers. That is a deployment decision to be taken deliberately, not a build problem to be dismissed.

Version 5.9.2 is the most recent release, so there is no upgrade available that resolves these.

Resolution

We kept the test suite as an installation gate and excluded the three known-broken tests, with the reasoning recorded in the recipe so that the next person does not repeat the investigation:

GA_EXCLUDED_TESTS="^(bin|mulmatpatch|nga-scatter)$"

ctest --test-dir build-${COMPILER}-${MPI} \
      --output-on-failure \
      --timeout 600 \
      -E "${GA_EXCLUDED_TESTS}" || exit

The remaining seventy-six tests still have to pass before anything is installed, and removing the -E argument restores the full suite once upstream fixes arrive.

Conclusions

• Read the error message back to its source before believing it. “Received an Error in Communication” is printed for every abort in Global Arrays, including ones a test raises itself.
• Vary one setting at a time and rebuild. Four builds took less time than the speculation they replaced, and each one eliminated a whole class of explanation.
• When a C test and its Fortran counterpart disagree, the binding is the suspect, not the algorithm.
• A passing test is not automatically evidence of anything. It is worth checking occasionally that a test can actually fail.
• Excluding a test is a way of recording a known defect, not of fixing it. Write down why, and say plainly what remains broken.