Intel® Xeon Phi™ coprocessors are designed to extend the reach of applications that have demonstrated the ability to fully utilize the scaling capabilities of Intel Xeon processor-based systems and fully exploit available processor vector capabilities or memory bandwidth. For such applications, the Intel Xeon Phi coprocessors offer additional power-efficient scaling, vector support, and local memory bandwidth, while maintaining the programmability and support associated with Intel Xeon processors.

MesocentreFC provide one host with 4 XeonPhi devices:

1 Device (mic1)

3 Devices (mic0,mic2,mic3)

Host

• Card usage (ganglia)
• Node usage (ganglia)

The Xeon PHI hosting node is configured in new SGE queue called xphi.q on lumiere cluster.

First we access to the queue xphi.q (since the access is exclusive we can ask for full memory i.e 128 G)

From lumiere cluster (mesologin1.univ-fcomte.fr) :

[user@mesologin1 ~]$ qlogin -q xphi.q -l h_vmem=128G

A remote shell is opened to the node1-51

Your job 79206 ("QLOGIN") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 79206 has been successfully scheduled.
Establishing builtin session to host node1-51.cluster ...
 
[user@node1-51 ~]$

You can check the state of the MIC card by:

[user@node1-51 ~]$ micinfo

 
[user@node1-51 ~]$ module load intel/mic

Check the next section to learn how to compile and execute parallel programs on Xeon PHI.

In this section we will describe briefly how to effectively use Xeon PHI.

To benefit from MIC card performance you application MUST scale over hundreds of threads and heavily uses vectorization.

Applications exploiting the parallelism are often based on: OpenMP, MKL, MPI

We describe the two ways to execute OpenMP on MIC card :

• Native mode: compile on host run on MIC
• Offload mode: compile on host run on host/MIC

Let's use a classical hello world openMP example.

hello.c

 
#include <omp.h>
 
main ()  {
 
int nthreads, tid;
 
    /* Fork a team of threads with each thread having a private tid variable */
#pragma omp parallel private(tid)
 {
 
  /* Obtain and print thread id */
   tid = omp_get_thread_num();
   printf("Hello World from thread = %d\n", tid);
  /* Only master thread does this */
   if (tid == 0) 
    {
     nthreads = omp_get_num_threads();
      printf("Number of threads = %d\n", nthreads);
    }
 
   }  /* All threads join master thread and terminate */
 
}

The simplest model of running applications on the Intel Xeon Phi coprocessor is native mode. In native mode an application is compiled on the host using the compiler switch -mmic to generate code for the MIC architecture. The binary can then be copied to the coprocessor and has to be started there or by using micnativeloadex to automatically upload and execute code in co-processor.

To compile for mic:

$ icc -openmp -mmic hello.c -o hello.mic

To execute in mic0 (dynamic load):

$ micnativeloadex ./hello.mic

...
Hello World from thread = 207
Hello World from thread = 109
Hello World from thread = 219
Hello World from thread = 107
Hello World from thread = 95
Hello World from thread = 124
Hello World from thread = 110
Hello World from thread = 189
..
..

This execution use the all threads available on card. We can specify the number of threads by :

$ micnativeloadex ./hello.mic -e "OMP_NUM_THREADS=40"

$ micnativeloadex ./application -a "arg1 arg2 ..." -e "env1 env2 ..." -d mic3

Explicit controls of data transfer and remote execution using compiler offload progmas/directives.

Lets modify our example :

hello_offload.c

#include <omp.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define NAMELENGTH 256
 
main ()  {
 
char hostname[NAMELENGTH];
int nthreads, tid;
 
/* Fork a team of threads with each thread having a private tid variable */
#pragma offload target(mic)
gethostname(hostname,NAMELENGTH);
 
#pragma offload target (mic)
#pragma omp parallel private(tid)
 {
   /* Obtain and print thread id */
     tid = omp_get_thread_num();
     printf("Hello World from thread = %d\n", tid);
    /* Only master thread does this */
    if (tid == 0)
     {
        nthreads = omp_get_num_threads();
        printf("Number of threads = %d, running on %s\n", nthreads,hostname);
     }
  }  /* All threads join master thread and terminate */
 
}

To compile for offload mode:

$ icc -openmp hello_off.c -o hello_off

To execute on mic using 4 threads (on MIC) for example, just run :

$export MIC_ENV_PREFIX=MIC
$ MIC_OMP_NUM_THREADS=4 ./hello_off

Hello World from thread = 0
Number of threads = 4, running on node1-51-mic0
Hello World from thread = 2
Hello World from thread = 3
Hello World from thread = 1

1. Choose explicitly the device in the code, for example offload to mic1 ex :

    #pragma offload target (mic:1)

2. Use environment variable

   export OFFLOAD_DEVICES=0,1

   to offload on mic0 and mic1 for example

For hybrid OpenMP we can execute some program sections on MIC and others on HOST.

hybridOMP.c

#include <omp.h>
#include "stdio.h"
 
int main()
{
 
 
// run on host
#pragma omp parallel for
    for(int i=0; i<5; i++) {
        printf("x %d\n", i);
    }
 
 
// run on MIC
#pragma offload target (mic)
 
#pragma omp parallel for
    for(int i=0; i<5; i++) {
        printf(". %d\n", i);
    }
}

Q: How to control the number of threads on MIC and on HSOT?

A: We use environment variable, For example 240 threads on mic, 12 on host:

export MIC_ENV_PREFIX=MIC
export MIC_OMP_NUM8THREADS=240
export OMP_NUM_THREADS=12

and than run the application on host:

./hybridOMP

• Support for the Intel® Xeon Phi™ coprocessors is introduced starting Intel® MKL 11.0

• Heterogeneous computing: Takes advantage of both multicore host and many-core coprocessors

• All Intel MKL functions are supported: But optimized at different levels.

Highly Optimized Functions:

• BLAS Level 3, and much of Level 1 & 2
• Sparse BLAS: ?CSRMV, ?CSRMM
• Some important LAPACK routines (LU, QR, Cholesky Fast Fourier transforms)
• Vector Math Library
• Random number generators in the Vector Statistical Library

The following 3 usage models of MKL are available for the Xeon Phi:

1. Automatic Offload
2. Compiler Assisted Offload
3. Native Execution

In the case of automatic offload the user does not have to change the code at all. For automatic offload enabled functions the runtime may automatically download data to the Xeon Phi coprocessor and execute (all or part of) the computations there. The data transfer and the execution management is completely automatic and transparent for the user.

As of Intel MKL 11.0.2 only the following functions are enabled for automatic offload:

• Level-3 BLAS functions
  • GEMM (for m,n > 2048, k > 256)
  • ?TRSM (for M,N > 3072)
  • ?TRMM (for M,N > 3072)
  • ?SYMM (for M,N > 2048)

• LAPACK functions
  • LU (M,N > 8192)
  • QR
  • Cholesky

The AO is enabled either by using the function mkl_mic_enable() or by setting the following environment variable:

$ export MKL_MIC_ENABLE=1

To see more information about the Automatic Offload and debugging please enable the environment variable:

$ export OFFLOAD_REPORT=2

To build a program for automatic offload, the same way of building code as on the Xeon host is used:

$ icc -O3 -mkl file.c -o file

By default, the MKL library decides when to offload and also tries to determine the optimal work division between the host and the targets (MKL can take advantage of multiple coprocessors). In case of the BLAS routines the user can specify the work division between the host and the coprocessor by calling the routine:

mkl_mic_set_Workdivision(MKL_TARGET_MIC,0,0.5) 

or by setting the environment variable

$ export MKL_MIC_0_WORKDIVISION=0.5

Both examples specify to offload 50% of computation only to the 1st card (card #0).

Threading control in Automatic Offload

For example 2 threads on host and 200 on card :

$ export OMP_NUM_THREADS=2
 
$ export MIC_OMP_NUM_THREADS=200

In this mode of MKL the offloading is explicitly controlled by compiler pragmas or directives. In contrast to the automatic offload mode, all MKL function can be offloaded in CAO-mode.

A big advantage of this mode is that it allows for data persistence on the device.

For Intel compilers it is possible to use AO and CAO in the same program, however the work division must be explicitly set for AO in this case. Otherwise, all MKL AO calls are executed on the host.

MKL functions are offloaded in the same way as any other offloaded function. An example for offloading MKL’s sgemm routine looks as follows:

#pragma offload target(mic) \ 
  in(transa, transb, N, alpha, beta) \
  in(A:length(N*N)) \ 
  in(B:length(N*N)) \ 
  in(C:length(N*N)) \ 
  out(C:length(N*N) alloc_if(0)) { 
 
  sgemm(&transa, &transb, &N, &N, &N, &alpha, A, &N, B, &N, &beta, C, &N); 
 
  }

To build a program for compiler assisted offload, the following command is recommended by Intel:

$ icc –O3 -openmp -mkl \
–offload-option,mic,ld, “-L$MKLROOT/lib/mic -Wl,\
--start-group -lmkl_intel_lp64 -lmkl_intel_thread \
-lmkl_core -Wl,--end-group” file.c –o file

MIC_KMP_AFFINITY=explicit,granularity=fine,proclist=[1-240:1]

MIC_USE_2MB_BUFFERS=16K 

In this mode of MKL the Intel Xeon Phi coprocessor is used as an independent compute node.

To build a program for native mode, the following compiler settings should be used:

$ icc -O3 -mkl -mmic file.c -o file.mic

To execute program in Xeon PHI from the Host :

 
$ micnativeloadex ./file.mic -e "environment variables" -a "args"

The program will automatically loaded and executed in MIC.

provide environment variable

Before starting using Intel MPI in the mic you need to setup the environment (ssh password less to mic0):

• From mesologin1 type:

$ ssh-keygen -f $HOME/.ssh/id_rsa

• Don't enter passphrase if you wan't to connect without password
• Once key was generated:

$ mkdir $WORK/.ssh
$ cp $HOME/.ssh/id_rsa.pub $WORK/.ssh/authorized_keys

• Logout from mesologin1 and reconnect to mesologin1
• To test if it works:

$ qlogin -q xphi.q
$ ssh mic0

We use this example for testing

hello_mpi.c

#include <mpi.h>
 
int main(int argc, char** argv) {
 
MPI_Init(&argc, &argv);
 
 // Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
 
// Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
 
// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
 
// Print off a hello world message
printf("Hello world from processor %s, rank %d out of %d processors\n",
             processor_name, world_rank, world_size);
 
 // Finalize the MPI environment.
 MPI_Finalize();
}

To build and run an application in coprocessor-only mode, the following commands have to be executed:

1. Compile the program for the coprocessor (-mmic)

   $ mpiicc -mmic hello_mpi.c -o hello_mpi.mic

2. Set the I_MPI_MIC variable

   export I_MPI_MIC=1

3. Launch MPI jobs on the coprocessor mic0 from the host you MUST give the full path to program (because WORK is shared with MIC)

   mpirun -host mic0  -n 244 /Work/Users/kmazouzi/hello_mpi.mic

1. Compile the program for the coprocessor (-mmic)

   $ mpiicc -mmic hello_mpi.c -o hello_mpi.mic 

2. Compile the program for the host

   $ mpiicc -mmic hello_mpi.c -o hello_mpi

3. Set the I_MPI_MIC variable

   export I_MPI_MIC=1

4. Launch MPI jobs on the host node1-51 and on the coprocessor mic0

   $ mpirun -host node1-51 -n 1 ./hello_mpi : -n 244  -host mic0 /Work/Users/kmazouzi/hello_mpi.mic

   this will run 1 process on host (node1-51) and 244 on mic0.

This mode is also known as Hybride MPI/OpenMP.

Lets modify our example

hello_mpi_omp.c

include <mpi.h>;
#include <omp.h> 
#include <unistd.h>
int main(int argc, char** argv) {
 
MPI_Init(&argc, &argv);
 
  // Get the number of processes
  int world_size;
  MPI_Comm_size(MPI_COMM_WORLD, &world_size);
 
  // Get the rank of the process
  int world_rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
 
  // Get the name of the processor
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name, &name_len);
 
 char processor_name_target[MPI_MAX_PROCESSOR_NAME];
 int th_id;
// Only process 0 can offload to mic
if(world_rank==0)
{
#pragma offload target (mic)
#pragma omp parallel private(th_id) 
 {
    gethostname(processor_name_target,MPI_MAX_PROCESSOR_NAME) ;
    th_id=omp_get_thread_num();
    printf("Hello world from thread %d on processor %s, from mpi process %d\n",th_id,processor_name_target,world_rank);
 }
}
  // Print off a hello world message
  printf("Hello world from processor %s, rank %d"
         " out of %d processors\n",
         processor_name, world_rank, world_size);
 
  // Finalize the MPI environment.
  MPI_Finalize();
}

• Compile

$ mpiicc -openmp hello_mpi_omp.c -o  hello_mpi_omp

• Execute

$ MIC_NUM_THREADS=10 mpirun -np 3 ./hello_mpi_omp

Run 3 MPI process on host, process 0 will spawn 10 threads on mic0

OMP_NUM_THREADS=10 mpirun -np 3 ./a.out 
Hello world from processor node1-51, rank 1 out of 3 processors
Hello world from processor node1-51, rank 2 out of 3 processors
Hello world from processor node1-51, rank 0 out of 3 processors
Hello world from thread 0 on processor node1-51-mic0, from mpi process 0
Hello world from thread 2 on processor node1-51-mic0, from mpi process 0
Hello world from thread 4 on processor node1-51-mic0, from mpi process 0
Hello world from thread 6 on processor node1-51-mic0, from mpi process 0
Hello world from thread 3 on processor node1-51-mic0, from mpi process 0
Hello world from thread 7 on processor node1-51-mic0, from mpi process 0
Hello world from thread 8 on processor node1-51-mic0, from mpi process 0
Hello world from thread 9 on processor node1-51-mic0, from mpi process 0
Hello world from thread 5 on processor node1-51-mic0, from mpi process 0
Hello world from thread 1 on processor node1-51-mic0, from mpi process 0

Once you compiled your code for Xeon phi (see section above) you can submit your job using SGE.

Here is an example of an SGE script running OpenMP program on Mic using native mode:

xphi.q

#!/bin/bash 
 
#$ -q xphi.q
 
#$ -N myPhi_program
 
module load intel/mic
 
### You only to modify
 
micnativeloadex ./myprogram.mic a "my program argument" -e "MIC_KMP_AFFINITY=explicit,granularity=fine,proclist=[1-240:1]"

• Better use one core for the OS running on MIC so the maximum of threads to use us use 61*4 -1 = 243
• Try different threads affinity using

   $ export MIC_ENV_PREFIX=MIC
   $ export MIC_KMP_AFFINITY='...your affinity settings of choice...'
 

Where MIC_KMP_AFFINITY can takes several values, those often used:

1. granularity=fine,balanced
2. granularity=fine,compact

export MIC_KMP_AFFINITY="verbose,granularity=fine,balanced"

We used a naive matmul.c for benchmark, Matrix size: '10000x10000'.

We used BlAS 3 DGEMM subroutine, Matrix size: '10000x10000'

• Matlab with XeonPhi
• Using XeonPhi with Python

• Short presentation on Xeon Phi
• http://www.prace-ri.eu/Best-Practice-Guide-Intel-Xeon-Phi-HTML
• https://software.intel.com/sites/default/files/11MIC42_How_to_Use_MKL_Automatic_Offload_0.pdf
• https://software.intel.com/en-us/articles/intel-xeon-phi-coprocessor-developers-quick-start-guide
• http://www.drdobbs.com/parallel/programming-intels-xeon-phi-a-jumpstart/240144160