System Application , Freeware¶
Info
The command line examples on this page use the following notation
[login]$ : login node
[rNnN]$ : compute node
[login/rNnN]$ : login node or compute node
[yourPC]$ : environment from which the login node is connected
How to use Python packages on PyPI (e.g. Theano)¶
You can install modules into your home directory (Example: Theano case)
[login]$ pip install --user theano
How to install numpy, mpi4py, chainer, tensorflow, cupy etc. using python/3.9.18¶
If you want to install numpy, mpi4py, chainer etc. using python/3.9.18, do as follows.
[rNnN]$ module purge
[rNnN]$ module load intel cuda openmpi
[rNnN]$ python3 -m pip install --user python_modules
[login/rNnN]$ python3 -m pip install --user python_modules==version
I want to install and use my library in R¶
In TSUBAME4.0, R-4.3.2 is available.
In addition to the basic package, the libraries available as default are as follows.
Rmpi, rpud
If you wish to use a library other than the above, you will need your own installation operation.
Since the installation directory of R is impossible due to the permission relationship, you can install / manage your own library after specifying the library path. The procedure is as follows.
Assuming that the library path is $HOME/Rlib, the library name is testlib, and the testlib.tar.gz is the source package, and operate as follows.
[rNnN]$ module purge # purge module setting
[rNnN]$ module load cuda openmpi R # Load modules
[rNnN]$ mkdir ~/Rlib # Create library installation directory (if nothing)
[rNnN]$ wget https://cran.r-project.org/src/contrib/testlib.tar.gz # Downlod package
[rNnN]$ R CMD INSTALL -l $HOME/Rlib testlib.tar.gz # Install library
[rNnN]$ export R_LIBS_USER=$HOME/Rlib # Your own installation library settings
[rNnN]$ R # Use your library
> library(testlib)
An error occurs when mpi4py.futures.MPIPoolExecutor with openmpi is called¶
Sometimes like the following error occurs when mpi4py.futures.MPIPoolExecutor with openmpi is called.
[rNnN:xxxxx] [[60041,0],0] ORTE_ERROR_LOG: Not found in file orted/pmix/pmix_server_dyn.c at line 87
1. mpirun -np
- use mpi4py with intel MPI
file output stops by doing mpirun ... >& log.txt & with intel MPI¶
With intel MPI, output might stop by doing background execution like the following.
[rNnN]$ mpirun ... ./a.out >& log.txt &
[rNnN]$ mpirun ... ./a.out **< /dev/null >**& log.txt &
I want to link Intel oneAPI Math Kernel Library, what are the link options?¶
If you want to link Intel oneAPI Math Kernel Library、please fill the appropriate contents into Intel oneAPI Math Kernel Library Link Line Advisor, and get the link opthion from "Use this link line”.
ifort, icc, icpc commands cannot be used with Intel OneAPI¶
Starting with Intel oneAPI 2024, the icc and icpc commands are no longer available.
Please use the icx and icpx commands.
And, starting with Intel oneAPI 2025, the ifort command is no longer available.
Please use the ifx command.
C++17 related errors occur when using Intel oneAPI.¶
Since Intel oneAPI 2024, the default standard for C++ has changed from C++14 to C++17.
As a result, the following error may occur
- error: ISO C++17 does not allow 'register' storage class specifier [-Wregister]
- error: ISO C++17 does not allow dynamic exception specifications [-Wdynamic-exception-spec]
For more information, see here.
If this error occurs, do one of the following
- Modify the source code to conform to the C++17 standard.
- Specify the option -std=c++14 during translation to compile with the C++14 standard.
Unable to module load python¶
Python 3 is available by default in TSUBAME4.0.
No module load required.
When running OpenMPI/Intel MPI, an hcoll-related error or segmentation fault occurs.¶
This may be improved by specifying the following environment variables.
OpenMPI：export OMPI_MCA_coll=^hcoll
Intel MPI：export I_MPI_COLL_EXTERNAL=0
UCX_ROOT=/apps/t4/rhel9/free/ucx/1.16.0-rc4/gcc11.4.1/lib
export LD_PRELOAD=$UCX_ROOT/libucm.so:$UCX_ROOT/libucp.so:$UCX_ROOT/libucs.so:$UCX_ROOT/libucs_signal.so:$UCX_ROOT/libuct.so
Info
When implementing the LD_PRELOAD export described in this topic, ensure it is executed on a single node. It has been confirmed that running QUANTUM ESPRESSO with this configuration on a multi-node system causes a kernel panic, and similar issues may occur with other applications.
Binaries compiled with the -lblas option in Intel Compiler terminate abnormally at runtime¶
Intel Compiler provides a numerical calculation library called Intel MKL.
When using BLAS with Intel Compiler, please change the option to link the Intel MKL library instead of -lblas. For details, please refer to How to link Intel MKL.
Note that a wide variety of numerical calculation libraries exist, including dedicated libraries such as Intel MKL.Some of them have dependencies or conflicts among libraries.
If an error occurs in a numerical calculation library, please consider checking for conflicts or using another library.
GPU not used when running TensorFlow¶
When running TensorFlow on a GPU, there are requirements for the combination of TensorFlow, python, cudnn, and cuda.
Please refer to Tensorflow for details.
I want to use Alphafold related databases¶
We have prepared databases on TSUBAME for use with the following software. These database files are large in size, so please avoid downloading them individually if at all possible.
Please refer to the respective links for details on how to use them.
- Alphafold2 database
- Alphafold3 database
- LocalColabfold database
If you want to use a version of python that is not installed in TSUBAME4.0¶
If you want to use a version of python that is not installed in TSUBAME4.0, you need to install it by yourself.
We will show you three different ways to install python.
Info
- The procedures described here and your own installation of python are not supported.
- Here is an example of installing version 3.13.4. If you want to install a different version, replace "3.13.4".
- This work must be done on compute node, assuming Interactive job queue(iqrsh) or Interactive job(qrsh) is used.
Install in conda environment¶
The following is an example of using miniconda installed in TSUBAME4.0.
[rNnN]$ module load miniconda
# When you module load miniconda , you will get a "please do: eval~~~" message, so refer to the miniconda page to configure the settings.
[rNnN]$ eval "$(/apps/t4/rhel9/free/miniconda/24.1.2/bin/conda shell.bash hook)"
# Check the version of the python you want to install.
[rNnN]$ conda search python | tail -n1 # This is a sample of displaying only the latest version. If you want to display all versions, do not specify " | tail -n1 ".
# Create a virtual environment containing python.
[rNnN]$ conda create -n [virtual_environment_name] python==[version_of_python ] -y
[login/rNnN]$ module load miniconda
[login/rNnN]$ conda activate [virtual_environment_name] # Activate the created virtual environment
(virtual_environment_name)[login/rNnN]$ python -V # Check the version of python (optional)
Install from source code¶
Info
Check python.org for available versions.
[rNnN]$ export pybase=[install_path]
[rNnN]$ export pyver=3.13.4 # Set the version to be installed
[rNnN]$ export pyinst=${pybase}/python${pyver} # Set the installation path
# Download and extract the source code.
[rNnN]$ mkdir ${pyinst} && cd ${pyinst}
[rNnN]$ wget https://www.python.org/ftp/python/${pyver}/Python-${pyver}.tgz
[rNnN]$ tar xf Python-${pyver}.tgz && cd Python-${pyver}
# configure, build, and install. ( change log_ filename and options if necessary)
[rNnN]$ ./configure --prefix=${pyinst} 2>&1 | tee -a log_config
[rNnN]$ make -j 2>&1 | tee -a log_make
[rNnN]$ make install 2>&1 | tee -a log_inst
Info
Replace ${pyinst} with the installation path you specified yourself.
[login/rNnN]$ export PATH=${pyinst}/bin:${PATH} # Specify the path to the installed python in the PATH environment variable
[login/rNnN]$ python -V # Check the version of python (optional)
Install in pyenv environment¶
Info
The configuration for using pyenv depends on your environment.
Please refer to official website. TSUBAME4.0 does not support pyenv.
# Install pyenv
[rNnN]$ export pybase=[Installation_path_of_pyenv] # Setting up the installation path of pyenv
[rNnN]$ export pyver=3.13.4 # Setting of the version to install
[rNnN]$ cd ${pybase}
[rNnN]$ git clone https://github.com/pyenv/pyenv
[rNnN]$ export PATH=${pybase}/pyenv/bin:${PATH}
[rNnN]$ pyenv --version
# Check the version of python provided by pyenv.
[rNnN]$ pyenv install --list | grep ${pyver}
# Install the specified version and change the version to the default for pyenv.
[rNnN]$ pyenv install ${pyver}
[rNnN]$ pyenv versions
[rNnN]$ pyenv global ${pyver}
To use the installed python, perform the following steps.
Info
- Replace ${pybase} and ${pyver} with the values you specified yourself during installation.
- "export PATH=${pybase}/pyenv/bin:${PATH}" can be omitted by defining it in .bashrc.
[login/rNnN]$ export PATH=${pybase}/pyenv/bin:${PATH} # Specify the path to the installed pyenv in the environment variable PATH
[login/rNnN]$ pyenv global ${pyver} # Switch python version to the version specified in ${pyver}
[login/rNnN]$ python -V # Check the version of python (optional)
When using OpenMPI, I want to configure the CPU core binding method¶
When configuring CPU core binding methods for OpenMPI, the following are typical approaches:
- Use the "numactl" command
- Use options such as --bind-to, --map-by, or --rand-by with the mpirun command
For details on using the numactl command, refer to "man numactl". For details on the various mpirun command options, refer to "man mpirun".
[Example Usage]
When running 1 process with 192 threads, threads were concentrated on a single core(CPU) by default.
$ export OMP_NUM_THREADS=192
$ mpirun -npernode 1 -n 1 -x LD_LIBRARY_PATH -x OMP_NUM_THREADS ./a.out
(Add P (Last Used CPU (SMP)) to the "top" command)
PID USER S %CPU TIME+ COMMAND P
3112554 ux00000 S 0.0 0:00.00 a.out 192
3112555 ux00000 S 0.0 0:00.00 a.out 192
3112556 ux00000 S 0.0 0:00.01 a.out 192
3112557 ux00000 S 0.0 0:00.00 a.out 192
3112558 ux00000 S 0.0 0:00.00 a.out 192
（Omitted）
By specifying the --bind-to numa option, it was distributed and assigned per NUMA node.
$ export OMP_NUM_THREADS=192
[ux00000@r5n10 ~]$ mpirun --bind-to numa -npernode 1 -n 2 -x LD_LIBRARY_PATH -x OMP_NUM_THREADS ./a.out
(Add P (Last Used CPU (SMP)) to the "top" command)
PID USER S %CPU TIME+ COMMAND P
3116060 ux00000 S 0.0 0:00.00 a.out 57
3116061 ux00000 S 0.0 0:00.00 a.out 193
3116062 ux00000 S 0.0 0:00.00 a.out 194
3116063 ux00000 S 0.0 0:00.00 a.out 196
3116064 ux00000 S 0.0 0:00.00 a.out 199
（Omitted）
Info
- The optimal CPU core allocation method varies depending on the task being executed. Please note that improper allocation may lead to performance degradation. (CPU core allocation methods are not covered by support.)
- On TSUBAME4.0, since multiple users share a single node, CPU numbers may start from values other than 0. For details, please refer to How to specify CPU number (%cpu) when using Gaussian?.
- For examples of using numactl, see run some programs on different CPUs/GPUs in a job script.
The combination of compiler and mpi module¶
GNU, Intel oneAPI, NVDIA HPC SDK and OpenMPI can be used in combination.
gcc is provided by the OS. Please check the available version with the following command.
[login/rNnN]$ gcc --version
[login/rNnN]$ module avail
Info
Please note that if you use any other OpenMPI than the following OpenMPI provided with TSUBAME4.0, the operation is not guaranteed and not supported.
The OpenMPI versions listed in this document may not be the latest. Please check the available version with module avail openmpi command. (Unless there is a specific reason not to, we recommend using the latest version.)
1. Intel OpenMPI
[rNnN]$ module purge
[rNnN]$ module load openmpi/5.0.2-intel
Loading openmpi/5.0.2-intel
Loading requirement: intel/2024.0.2 cuda/12.3.2
[rNnN]$ mpicc -v
Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2024.0.2.20231213)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /apps/t4/rhel9/isv/intel/compiler/2024.0/bin/compiler
Configuration file: /apps/t4/rhel9/isv/intel/compiler/2024.0/bin/compiler/../icx.cfg
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/11
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/11
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /apps/t4/rhel9/cuda/12.3.2, version
2. GNU OpenMPI
[rNnN]$ module purge
[rNnN]$ module load openmpi/5.0.2-gcc
Loading openmpi/5.0.2-gcc
Loading requirement: cuda/12.3.2
[rNnN]$ mpicc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-host-pie --enable-host-bind-now --enable-languages=c,c++,f
ortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/
bugzilla --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit -
-disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --en
able-plugin --enable-initfini-array --without-isl --enable-multilib --with-linker-hash-style=gnu --enable-offload-ta
rgets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_6
4=x86-64-v2 --with-arch_32=x86-64 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serial
ization=1
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.4.1 20230605 (Red Hat 11.4.1-2) (GCC)
3. NVIDIA HPC SDK OpenMPI
[rNnN]$ module purge
[rNnN]$ module load openmpi/5.0.2-nvhpc
Loading openmpi/5.0.2-nvhpc
Loading requirement: nvhpc/24.1
[rNnN]$ mpicc -v
Export NVCOMPILER=/apps/t4/rhel9/isv/nvidia/hpc_sdk/Linux_x86_64/24.1
Export PGI=/apps/t4/rhel9/isv/nvidia/hpc_sdk
nvc-Warning-No files to process
Can Docker be used with TSUBAME4.0?¶
Can Singularity be used with TSUBAME4.0?¶
Can container be used with TSUBAME4.0?¶
Docker cannot be used with TSUBAME4.0. Apptainer(Singularity) is available.
For more information, see Use containers.
Can I use Jupyter Lab with Tsubame 4.0?¶
Jupyter Lab is available in Tsubame4.0. See Open OnDemand User's Guide.
I want to use a debugger/profiler that supports multi-threading/multi-processing¶
Linaro Forge(ex:Arm forge) is available.
Please refer to "Parallel Programming" on the Lectures page for information on how to use it.
GLIBC not found error when using Apptainer¶
When using the fake root function (--fakeroot) in Apptainer, the libc version must be matched between the host and the container.
If they do not match, you may get the following error.
/.singularity.d/libs/faked: /lib/x86_64-linux-gnu/libc.so.6: version`GLIBC_2.33' not found (required by /.singularity.d/libs/faked)
/.singularity.d/libs/faked: /lib/x86_64-linux-gnu/libc.so.6: version`GLIBC_2.34' not found (required by /.singularity.d/libs/faked)
fakeroot: error while starting the `faked' daemon.
/.singularity.d/libs/fakeroot: 1: kill: Usage: kill [-s sigspec | -signum |-sigspec] [pid | job]... or
kill -l [exitstatus]
I want to use VS Code.¶
Access to login nodes using VS Code is prohibited because it places a heavy load on the login nodes.
Please refrain from occupying the CPU in the login nodes.
In TSUBAME4.0, it is possible to connect to compute nodes via Open OnDemand using code server.
Please consider using code server. Please refer to Open OnDemand User's Guide for details on how to use it.