Towards full adoption of LLVM: building a productive SQLite3 library installation required by TensorFlow

Towards full adoption of LLVM: building a productive SQLite3 library installation required by TensorFlow

Nowadays, the management of TensorFlow models using the SQLite3 library is becoming more and more demanded. It has become a standard practice for applicants for HPC resources to insist on accessing an optimized installation of the SQLite3 library, and avoid the one provided by the Linux distribution or delivered by Conda. Until one becomes involved in dealing with the constant changing of large sets of models, that might seem like an exaggeration or overstatement. It should be noted that the adoption of an optimized Sqlite3 library installation can reduce the overall time for processing by 3 to 5%.

Discoverer already has its LLVM 19 compiler infrastructure installation. That installation permits compiling the SQLite3 library based on aggressive optimization (-O3) and comparing the achieved productivity with the one derived by utilizing the Intel oneAPI compilers. The results are based on the execution of a comprehensive floating-point selection. It searches those of the 1000000 3D vectors stored in an in-memory database that are located within a sphere with a specified radius. Here are the numbers:

Compiler collectionResult, seconds
Intel oneAPI (latest)0.72 ± 0.12
LLVM 19.1.20.71 ± 0.14
Comparing the time for performing the SQLite3 selection on 1000000 floating point vectors, on AMD Zen2 processors. The standard deviation is computed based on 1000 consecutive tests.

It is evident that within the confidence interval shaped by the error estimate, the results obtained for both builds are statistically indistinguishable. Consequently, the LLVM 19 build of SQLite3 can be used as a reliable driver for fast model editing in TensorFlow. It is known to create many more troubles with satisfying dependencies, which makes if preferable to the one compiled by Intel oneAPI compilers.

Leave a Reply

Your email address will not be published. Required fields are marked *

WordPress Appliance - Powered by TurnKey Linux