Python 3.9.7 ============ Instructions to compile Python on both clusters are very similar. We will follow generic instructions for any Python 3.9.x version. The latest version at the time of writing this document (August 30, 2021) is 3.9.7. A few particularities related to architecture will be mentioned in due place. Modules ------- We will compile Python 3.9 using GCC 9.3 which is a relatively recent version compared to those provided by RedHat 6.x and 7.x (4.4 and 4.7 respectively). Download -------- Download the sources on the canonical location for sources ``/shared/src``. Uncompress and move into the folder:: $> wget https://www.python.org/ftp/python/3.9.7/Python-3.9.7.tgz $> tar -zxvf Python-3.9.7.tgz $> cd Python-3.9.7 Configure --------- Edit the file ``setup.py`` from the base source dir. Add a line pointing to the include folder of the corresponding GCC version:: def detect_sqlite(self): # The sqlite interface sqlite_setup_debug = False # verbose debug prints from this script? # We hunt for #define SQLITE_VERSION "n.n.n" # We need to find >= sqlite version 3.3.9, for sqlite3_prepare_v2 sqlite_incdir = sqlite_libdir = None sqlite_inc_paths = [ '/shared/software/lang/gcc/9.3.0/include', '/usr/include', '/usr/include/sqlite', '/usr/include/sqlite3', '/usr/local/include', '/usr/local/include/sqlite', '/usr/local/include/sqlite3', ] Configure Python 3.9.x enabling optimizations, shared libraries, and setting the prefix for installation:: $> ../configure --enable-shared --enable-optimizations \ --prefix=/shared/software/lang/python/3.9.7_gcc93 \ --with-tcltk-libs="-L/shared/software/lang/gcc/9.3.0/lib -ltcl8.6 -ltk8.6" \ --with-tcltk-includes="-L/shared/software/lang/gcc/9.3.0/include" --with-lto LDFLAGS="-L${MD_GCC}/lib" Compile and install:: $> make $> make install The test suite:: $> make test The results:: Ran 3 tests in 0.010s FAILED (failures=2, skipped=1) test test_pathlib failed 0:02:28 load avg: 1.97 Re-running test_zipfile in verbose mode (matching: test_add_file_after_2107) test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... ERROR ====================================================================== ERROR: test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ---------------------------------------------------------------------- Traceback (most recent call last): File "/gpfs20/shared/src/Python-3.9.7/Lib/test/test_zipfile.py", line 618, in test_add_file_after_2107 os.utime(TESTFN, (ts, ts)) OSError: [Errno 75] Value too large for defined data type ---------------------------------------------------------------------- Ran 1 test in 0.002s FAILED (errors=1) test test_zipfile failed 2 tests failed again: test_pathlib test_zipfile == Tests result: FAILURE then FAILURE == 409 tests OK. 2 tests failed: test_pathlib test_zipfile 14 tests skipped: test_devpoll test_gdb test_ioctl test_kqueue test_msilib test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio test_winreg test_winsound test_zipfile64 Total duration: 2 min 28 sec Tests result: FAILURE then FAILURE make: *** [Makefile:1209: test] Error 2 Numpy and Scipy ---------------------------- Download the latest versions of Numpy and Scipy:: wget https://github.com/numpy/numpy/releases/download/v1.21.2/numpy-1.21.2.tar.gz wget https://github.com/scipy/scipy/releases/download/v1.7.1/scipy-1.7.1.tar.gz On each folder, create a file ``site.cfg`` with the following lines:: [openblas] libraries = openblas library_dirs = /shared/software/libs/openblas/0.3.13_gcc93/lib include_dirs = /shared/software/libs/openblas/0.3.13_gcc93/include runtime_library_dirs = /shared/software/libs/openblas/0.3.13_gcc93/lib Load the module:: module load libs/openblas/0.3.13_gcc93 Install dependencies:: python3 -m pip install pybind11 Cython Install numpy with:: python3 -m pip install -v --compile -v . Similarly inside Scipy folder:: python3 -m pip install -v --compile -v . Install a set:: python3 -m pip install pandas pyyaml networkx seaborn matplotlib ipython jupyter jupyterlab scikit-learn