Qlogic OFED+ Host Specifications

IB6054601-00 HQLogic Host Channel Adapter and QLogicOFED Software Users GuideQLogic OFED Version 1.4

Page x IB6054601-00 HSQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4In the first and last paragraphs, added

6–Using Other MPIsManaging Open MPI, MVAPICH, and QLogic MPI with the mpi-selector Utility6-6 IB6054601-00 HSTo compile your program in C, type:$ mpi

6–Using Other MPIsHP-MPIIB6054601-00 H 6-7AFollowing is an example for listing and selecting an MPI:$ mpi-selector --list mpi-1.2.3mpi-3.4.5$ mpi-sel

6–Using Other MPIsHP-MPI6-8 IB6054601-00 HSInstallationFollow the instructions for downloading and installing HP-MPI from the HP web site. SetupNo sp

6–Using Other MPIsPlatform (Scali) MPIIB6054601-00 H 6-9APlatform (Scali) MPIPlatform MPI was formerly known as Scali MPI Connect. The version tested

6–Using Other MPIsIntel MPI6-10 IB6054601-00 HSRunning Platform MPI ApplicationsHere is an example of a simple mpirun command running with four proce

6–Using Other MPIsIntel MPIIB6054601-00 H 6-11Acompat-dapl-devel-static-1.2.12-1.x86_64.rpmcompat-dapl-utils-1.2.12-1.x86_64.rpm2. Verify that there

6–Using Other MPIsImproving Performance of Other MPIs Over IB Verbs6-12 IB6054601-00 HSTo compile your program in C using the default compiler, type:

6–Using Other MPIsImproving Performance of Other MPIs Over IB VerbsIB6054601-00 H 6-13AThe maximum supported MTU size of InfiniPath adapter cards are

6–Using Other MPIsImproving Performance of Other MPIs Over IB Verbs6-14 IB6054601-00 HSNotes

IB6054601-00 H A-1A mpirun Options SummaryThis section summarizes the most commonly used options to mpirun. See the mpirun (1) man page for a complet

QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4IB6054601-00 H Page xiSAdded program example. “ibv_devinfo” on

A–mpirun Options SummarySpawn OptionsA-2 IB6054601-00 HS-np np This option specifies the number of processes to spawn. If this option is not set, the

A–mpirun Options SummaryQuiescence OptionsIB6054601-00 H A-3AQuiescence Options-disable-mpi-progress-checkThis option disables the MPI communication

A–mpirun Options SummaryStats OptionsA-4 IB6054601-00 HS-k, -kill-timeout secondsThis option indicates the time to wait for other ranks after the fir

A–mpirun Options SummaryTuning OptionsIB6054601-00 H A-5A-statsfile file-prefixThis option specifies an alternate file to receive the output from the

A–mpirun Options SummaryShell OptionsA-6 IB6054601-00 HSShell Options-shell shell-nameThis option specifies the name of the program to use to log int

A–mpirun Options SummaryFormat OptionsIB6054601-00 H A-7AFormat Options-l, -label-outputThis option labels each line of output on stdout and stderr w

A–mpirun Options SummaryOther OptionsA-8 IB6054601-00 HSNotes

IB6054601-00 H B-1B Benchmark ProgramsSeveral MPI performance measurement programs are installed from the mpi-benchmark RPM. This appendix describes

B–Benchmark ProgramsBenchmark 1: Measuring MPI Latency Between Two NodesB-2 IB6054601-00 HSdefined. The program uses a loop, executing many such exch

B–Benchmark ProgramsBenchmark 2: Measuring MPI Bandwidth Between Two NodesIB6054601-00 H B-3ABenchmark 2: Measuring MPI Bandwidth Between Two NodesTh

Page xii IB6054601-00 HSQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Updated output file name and example

B–Benchmark ProgramsBenchmark 3: Messaging Rate MicrobenchmarksB-4 IB6054601-00 HSBenchmark 3: Messaging Rate Microbenchmarksmpi_multibw is the micro

B–Benchmark ProgramsBenchmark 4: Measuring MPI Latency in Host RingsIB6054601-00 H B-5ASearching for N/2 bandwidth. Maximum Bandwidth of 1958.400649

B–Benchmark ProgramsBenchmark 4: Measuring MPI Latency in Host RingsB-6 IB6054601-00 HSNotes

IB6054601-00 H C-1C Integration with a Batch Queuing SystemMost cluster systems use some kind of batch queuing system as an orderly way to provide us

C–Integration with a Batch Queuing SystemUsing SLURM for Batch QueuingC-2 IB6054601-00 HSFor more usage information, see the OSC mpiexec documentatio

C–Integration with a Batch Queuing SystemUsing SLURM for Batch QueuingIB6054601-00 H C-3AAllocating ResourcesWhen the mpirun command starts, it requi

C–Integration with a Batch Queuing SystemUsing SLURM for Batch QueuingC-4 IB6054601-00 HSThe sort | uniq -c component determines the number of times

C–Integration with a Batch Queuing SystemLock Enough Memory on Nodes when Using SLURMIB6054601-00 H C-5AThe following command terminates all processe

C–Integration with a Batch Queuing SystemLock Enough Memory on Nodes when Using SLURMC-6 IB6054601-00 HSThe following message displays after installa

IB6054601-00 H Preliminary D-1D TroubleshootingThis appendix describes some of the tools you can use to diagnose and fix problems. The following topi

QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4IB6054601-00 H Page xiiiSAdded Glossary entry for Verbs. “Verb

D–TroubleshootingBIOS SettingsD-2 IB6054601-00 HSBIOS SettingsThis section covers issues related to BIOS settings.The most important setting is Advan

D–TroubleshootingKernel and Initialization IssuesIB6054601-00 H D-3AKernel and Initialization IssuesIssues that may prevent the system from coming up

D–TroubleshootingKernel and Initialization IssuesD-4 IB6054601-00 HSIf there is no output at all, the driver initialization failed. For more informat

D–TroubleshootingKernel and Initialization IssuesIB6054601-00 H D-5AOpenFabrics Load Errors if ib_ipath Driver Load FailsWhen the ib_ipath driver fai

D–TroubleshootingOpenFabrics and InfiniPath IssuesD-6 IB6054601-00 HSIf the driver loaded, but MPI or other programs are not working, check to see if

D–TroubleshootingOpenFabrics and InfiniPath IssuesIB6054601-00 H D-7AManual Shutdown or Restart May Hang if NFS in UseIf you are using NFS over IPoIB

D–TroubleshootingOpenFabrics and InfiniPath IssuesD-8 IB6054601-00 HSSDP Module Not LoadingIf the settings for debug level and the zero copy threshol

D–TroubleshootingSystem Administration TroubleshootingIB6054601-00 H D-9ASystem Administration TroubleshootingThe following sections provide details

D–TroubleshootingPerformance IssuesD-10 IB6054601-00 HSThe exact symptoms can vary with BIOS, amount of memory, etc. When the driver starts, you may

D–TroubleshootingPerformance IssuesIB6054601-00 H D-11AErratic PerformanceSometimes erratic performance is seen on applications that use interrupts.

Page xiv IB6054601-00 HSQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Notes

D–TroubleshootingQLogic MPI TroubleshootingD-12 IB6054601-00 HSHere is an example:# cat /sys/class/infiniband/ipath*/device/irq98# ls /proc/irq0 10

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-13AMixed Releases of MPI RPMsMake sure that all of the MPI RPMs are from the same release

D–TroubleshootingQLogic MPI TroubleshootingD-14 IB6054601-00 HSResolving Hostname with Multi-Homed Head NodeBy default, mpirun assumes that ranks can

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-15ACompiler/Linker MismatchIf the compiler and linker do not match in C and C++ programs,

D–TroubleshootingQLogic MPI TroubleshootingD-16 IB6054601-00 HSCompiling on Development NodesIf the mpi-devel-* RPM is installed with the --prefix /p

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-17AFor the second option, change the file /etc/ld.so.conf on the compute nodes rather tha

D–TroubleshootingQLogic MPI TroubleshootingD-18 IB6054601-00 HSThe shell strips off the double quotes before handing the arguments to the mpicc scrip

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-19AIn the next case, mpi_nxnlatbw.c is compiled with the HP-MPI version of mpicc, and giv

D–TroubleshootingQLogic MPI TroubleshootingD-20 IB6054601-00 HSSee “Compiler Cannot Find Include, Module, or Library Files” on page D-15, “Compiling

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-21AUsing MPI.mod FilesMPI.mod (or mpi.mod) are the Fortran 90/Fortran 95 mpi modules file

IB6054601-00 H xvTable of Contents1 IntroductionWho Should Read this Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

D–TroubleshootingQLogic MPI TroubleshootingD-22 IB6054601-00 HSIf you know that an argument can accept a data type that the MPI module does not expli

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-23AThis is equally applicable if the module mpi provides only a lower-rank interface and

D–TroubleshootingQLogic MPI TroubleshootingD-24 IB6054601-00 HSThe following message displays after installation:$ mpirun -m ~/tmp/sm -np 2 -mpi_late

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-25Agdb Gets SIG32 Signal Under mpirun -debug with the PSM Receive Progress Thread Enabled

D–TroubleshootingQLogic MPI TroubleshootingD-26 IB6054601-00 HSGeneral Error MessagesThe following message may be generated by ipath_checkout or mpir

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-27AThe following message indicates a mismatch between the QLogic interconnect hardware in

D–TroubleshootingQLogic MPI TroubleshootingD-28 IB6054601-00 HSThe following message usually indicates a node failure or malfunctioning link in the f

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-29AThe following indicates an unknown host:$ mpirun -np 2 -m ~/tmp/q mpi_latency 100 100

D–TroubleshootingQLogic MPI TroubleshootingD-30 IB6054601-00 HSThe following indicates that one program on one node died:$ mpirun -np 2 -m ~/tmp/q mp

D–TroubleshootingQLogic MPI TroubleshootingIB6054601-00 H D-31AThis message occurs when a cable is disconnected, a switch is rebooted, or when there

xvi IB6054601-00 HQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4SOther Configuration: Changing the MTU Size

D–TroubleshootingQLogic MPI TroubleshootingD-32 IB6054601-00 HSStatistics other than MPI-level statistics are fairly low level; most users will not u

IB6054601-00 H E-1E Write CombiningIntroductionWrite combining improves write bandwidth to the QLogic chip by writing multiple words in a single bus

E–Write CombiningPAT and Write CombiningE-2 IB6054601-00 HS..If you do not see any of these messages on your console, but suspect this problem, check

E–Write CombiningMTRR Mapping and Write CombiningIB6054601-00 H E-3AFor systems with very large amounts of memory (32GB or more), it may also be nece

E–Write CombiningMTRR Mapping and Write CombiningE-4 IB6054601-00 HSNotes

IB6054601-00 H F-1F Useful Programs and FilesThe most useful programs and files for debugging, and commands for common tasks, are presented in the fo

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsF-2 IB6054601-00 HSSummary and Descriptions of Useful ProgramsUseful programs a

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsIB6054601-00 H F-3Admesgdmesg prints out bootup messages. It is useful for chec

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsF-4 IB6054601-00 HSibhostsThis tool determines if all the hosts in your InfiniB

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsIB6054601-00 H F-5AibtracertThe tool ibtracert determines the path that InfiniB

IB6054601-00 H xviiQLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4ATo Use Another Compiler . . . . . . . .

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsF-6 IB6054601-00 HSidentThe ident strings are available in ib_ipath.ko. Running

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsIB6054601-00 H F-7AIf you cannot see the problem, send the stdout output to you

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsF-8 IB6054601-00 HSThe options available with ipath_checkout are shown in Table

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsIB6054601-00 H F-9AMost of the functionality is accessed via the /sys filesyste

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsF-10 IB6054601-00 HSipath_mtrrMTRR is used by the InfiniPath driver to enable w

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsIB6054601-00 H F-11Aipath_pkt_testThis program is installed from the infinipath

F–Useful Programs and FilesSummary and Descriptions of Useful ProgramsF-12 IB6054601-00 HSmpirunmpirun determines whether the program is being run ag

F–Useful Programs and FilesCommon Tasks and CommandsIB6054601-00 H F-13ArpmTo check the contents of an installed RPM, use these commands:$ rpm -qa in

F–Useful Programs and FilesCommon Tasks and CommandsF-14 IB6054601-00 HSVerify hosts via an Ethernet pingipath_checkout --run=1 hostsfileVerify ssh

F–Useful Programs and FilesSummary and Descriptions of Useful FilesIB6054601-00 H F-15ASummary and Descriptions of Useful FilesUseful files are summa

xviii IB6054601-00 HQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4SCompiling MVAPICH Applications . . . . .

F–Useful Programs and FilesSummary and Descriptions of Useful FilesF-16 IB6054601-00 HSTable F-5 shows the possible contents of the file, with brief

F–Useful Programs and FilesSummary of Configuration FilesIB6054601-00 H F-17AversionYou can check the version of the installed InfiniPath software by

F–Useful Programs and FilesSummary of Configuration FilesF-18 IB6054601-00 HSConfiguration File Name Description/etc/modprobe.conf Specifies options

F–Useful Programs and FilesSummary of Configuration FilesIB6054601-00 H F-19ASample and Template Files Descriptionqlgc_vnic.cfg.sample Sample VNIC co

F–Useful Programs and FilesSummary of Configuration FilesF-20 IB6054601-00 HSNotes

IB6054601-00 H G-1G Recommended ReadingReference material for further reading is provided in this appendix.References for MPIThe MPI Standard specifi

G–Recommended ReadingClustersG-2 IB6054601-00 HSClustersGropp, William, Ewing Lusk, and Thomas Sterling, Beowulf Cluster Computing with Linux, Second

IB6054601-00 H Glossary-1GlossaryA glossary is provided for technical terms used in the documentation. Italicized terms in the definitions are define

GlossaryEE – kDAPLGlossary-2 IB6054601-00 HSEEStands for End to EndEECStands for End to End ContextfabricThe InfiniBand interconnect infrastructure,

IB6054601-00 H Glossary-3Glossarylatency — mpihosts fileAlatencyThe delay inherent in processing network data. In terms of MPI, it is the time requi

IB6054601-00 H xixQLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4AB Benchmark ProgramsBenchmark 1: Measuring

GlossaryMR – RCGlossary-4 IB6054601-00 HSMRStands for Memory RegionMTRR Stands for Memory Type Range Registers. Used by the InfiniPath driver to ena

IB6054601-00 H Glossary-5GlossaryRDMA — uDAPLARDMAStands for Remote Direct Memory Access. A communications protocol that enables data transmission f

Glossaryunmanaged switch – VNICGlossary-6 IB6054601-00 HSunmanaged switchA switch that does not have an active Subnet Manager (SM)VerbsIn the Infini

IB6054601-00 H Index-1IndexSymbols!!!ERROR!!! Lockable memory less than 4096KB on x nodes error message C-6, D-24./hpmpi-mpi_nxnlatbw: error while lo

QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Index-2 IB6054601-00 HSchkconfig F-2Cluster 1-2checker, Inte

IB6054601-00 H Index-3QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4Aopening shared memory object error mes

QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Index-4 IB6054601-00 HSibtracert F-2, F-5ibv_devinfo 4-30, F

IB6054601-00 H Index-5QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4AK-k A-4, F-8kDAPL Glossary-2--keep F-8

QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Index-6 IB6054601-00 HSperformance improvement over IB Verbs

IB6054601-00 H Index-7QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4AMulticast group Glossary-4Multihomed h

Page ii IB6054601-00 HSQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Information furnished in this manual i

xx IB6054601-00 HQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4SPerformance Issues . . . . . . . . . . . .

QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Index-8 IB6054601-00 HSPCIe Glossary-4bus width 4-23bus widt

IB6054601-00 H Index-9QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4A--run=LIST F-8-runscript A-4Run-time l

QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Index-10 IB6054601-00 HSTotalView debugger 5-26Transport ser

© 2005-2009 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic, the QLogic logo, and Infin

IB6054601-00 H xxiQLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4AMTRR Mapping and Write Combining . . . . .

xxii IB6054601-00 HQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4SNetworking . . . . . . . . . . . . . . .

IB6054601-00 H 1-11 IntroductionThis section describes the objectives, intended audience, and organization of the QLogic Host Channel Adapter and QLo

1–IntroductionOverview1-2 IB6054601-00 HS Section 4, InfiniPath Cluster Setup and Administration, describes the lower levels of the supplied InfiniP

1–IntroductionInteroperabilityIB6054601-00 H 1-3AThe QLogic host channel adapters are InfiniBand 4X. The Double Data Rate (DDR) QLE7240 and QLE7280 a

1–IntroductionConventions Used in this Guide1-4 IB6054601-00 HSConventions Used in this GuideThis guide uses the typographical conventions listed in

1–IntroductionDocumentationIB6054601-00 H 1-5ADocumentationThe product documentation includes: The QLogic HCA and QLogic OFED Software Install Guide

1–IntroductionContact Information1-6 IB6054601-00 HS

IB6054601-00 H 2-12 Feature OverviewThis section contains the features for this release, the supported QLogic adapter models, supported distributions

QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4IB6054601-00 H Page iiiSIn the second bullet, removed referenc

2–Feature OverviewFeatures2-2 IB6054601-00 HS The QLogic InfiniBand Fabric Suite (IFS) is available separately for purchase. It includes FastFabric,

2–Feature OverviewFeaturesIB6054601-00 H 2-3AThis version of the QLogic OFED software provides support for all of the QLogic adapters in Table 2-1. A

2–Feature OverviewSupported Distributions and Kernels2-4 IB6054601-00 HSSupported Distributions and KernelsThe QLogic interconnect runs on AMD™ Opter

2–Feature OverviewCompiler SupportIB6054601-00 H 2-5ACompiler SupportQLogic MPI supports the following compilers: GNU gcc 3.3.x, 3.4.x, 4.0, 4.1, 4.

2–Feature OverviewSoftware Components2-6 IB6054601-00 HS Reliable Datagram Sockets (RDS) iSCSI Extensions for RDMA (iSER) This release supports two

IB6054601-00 H 3-13 Step-by-Step Cluster Setup and MPI Usage ChecklistsThis section describes how to set up your cluster to run high-performance MPI

3–Step-by-Step Cluster Setup and MPI Usage ChecklistsUsing MPI3-2 IB6054601-00 HSUsing MPI1. Verify that the QLogic hardware and software has been in

IB6054601-00 H 4-14 InfiniPath Cluster Setup and AdministrationThis section describes what the cluster administrator needs to know about the InfiniPa

4–InfiniPath Cluster Setup and AdministrationInstalled Layout4-2 IB6054601-00 HSInstalled LayoutThis section describes the default installed layout f

4–InfiniPath Cluster Setup and AdministrationMemory FootprintIB6054601-00 H 4-3AThe InfiniPath driver modules in this release are installed in: /lib/

Page iv IB6054601-00 HSQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Deleted reference to MTRR BIOS setting

4–InfiniPath Cluster Setup and AdministrationMemory Footprint4-4 IB6054601-00 HSThe following paragraphs provide an example for a 1024 processor syst

4–InfiniPath Cluster Setup and AdministrationBIOS SettingsIB6054601-00 H 4-5ABIOS SettingsTo achieve the best performance with QLogic adapters, you n

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and Startup4-6 IB6054601-00 HS SRP (OFED and QLogic modu

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and StartupIB6054601-00 H 4-7AThe output from this comman

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and Startup4-8 IB6054601-00 HSIf you are using the rpm in

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and StartupIB6054601-00 H 4-9AUsing QLogic SRPIf you inst

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and Startup4-10 IB6054601-00 HS3. Choose the device you w

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and StartupIB6054601-00 H 4-11AConfiguring and Administer

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and Startup4-12 IB6054601-00 HScontroller[ 1]GUID:

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and StartupIB6054601-00 H 4-13Acontroller[ 1]GUID:

QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4IB6054601-00 H Page vSAdded ibv_devinfo command to see the cur

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and Startup4-14 IB6054601-00 HSLook at the qlgc_vnic.cfg.

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and StartupIB6054601-00 H 4-15Ab. Format 2: Define an IOC

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and Startup4-16 IB6054601-00 HS4. Create VirtualNIC inter

4–InfiniPath Cluster Setup and AdministrationOpenFabrics Drivers and Services Configuration and StartupIB6054601-00 H 4-17AIf you start the InfiniBan

4–InfiniPath Cluster Setup and AdministrationOther Configuration: Changing the MTU Size4-18 IB6054601-00 HSMore information for configuration, starti

4–InfiniPath Cluster Setup and AdministrationManaging the InfiniPath DriverIB6054601-00 H 4-19AManaging the InfiniPath DriverThe startup script for i

4–InfiniPath Cluster Setup and AdministrationManaging the InfiniPath Driver4-20 IB6054601-00 HSStart, Stop, or Restart InfiniPathRestart the software

4–InfiniPath Cluster Setup and AdministrationMore Information on Configuring and Loading DriversIB6054601-00 H 4-21AUnload the Driver/Modules Manuall

4–InfiniPath Cluster Setup and AdministrationPerformance Settings and Management Tips4-22 IB6054601-00 HSPerformance Settings and Management TipsThe

4–InfiniPath Cluster Setup and AdministrationPerformance Settings and Management TipsIB6054601-00 H 4-23A Use a PCIe Max Read Request size of at lea

Page vi IB6054601-00 HSQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4Changed Intel® processor name.“Hyper-T

4–InfiniPath Cluster Setup and AdministrationPerformance Settings and Management Tips4-24 IB6054601-00 HSIf you are using Network File System (NFS) o

4–InfiniPath Cluster Setup and AdministrationHost Environment Setup for MPIIB6054601-00 H 4-25AHost Environment Setup for MPIAfter the InfiniPath sof

4–InfiniPath Cluster Setup and AdministrationHost Environment Setup for MPI4-26 IB6054601-00 HSThe example in this section assumes the following: Bo

4–InfiniPath Cluster Setup and AdministrationHost Environment Setup for MPIIB6054601-00 H 4-27A812E5iw1GCZfNefBmWbegWVKFwGlNbqBnZK7kDRLSOKQtuhYbGPcrV

4–InfiniPath Cluster Setup and AdministrationHost Environment Setup for MPI4-28 IB6054601-00 HS2. Enter a passphrase for your key pair when prompted.

4–InfiniPath Cluster Setup and AdministrationChecking Cluster and Software StatusIB6054601-00 H 4-29AChecking Cluster and Software Statusipath_contro

4–InfiniPath Cluster Setup and AdministrationChecking Cluster and Software Status4-30 IB6054601-00 HSibv_devinfoibv_devinfo queries RDMA devices. Use

4–InfiniPath Cluster Setup and AdministrationChecking Cluster and Software StatusIB6054601-00 H 4-31AThe Intel Cluster CheckerThe Intel Cluster Check

4–InfiniPath Cluster Setup and AdministrationChecking Cluster and Software Status4-32 IB6054601-00 HSNotes

IB6054601-00 H 5-15 Using QLogic MPIThis section provides information on using the QLogic Message-Passing Interface (MPI). Examples are provided for

QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4IB6054601-00 H Page viiSRenamed Compiling and Linking section.

5–Using QLogic MPIIntroduction5-2 IB6054601-00 HSOther MPIsOther high-performance MPIs, such as HP-MPI version 2.3, Open MPI (release 1.2.8), Ohio St

5–Using QLogic MPIGetting Started with MPIIB6054601-00 H 5-3AGetting Started with MPIThis section shows how to compile and run some simple example pr

5–Using QLogic MPIGetting Started with MPI5-4 IB6054601-00 HSThen, run the program with several different specifications for the number of processes:

5–Using QLogic MPIQLogic MPI DetailsIB6054601-00 H 5-5Afpi.f is a Fortran 77 program that computes pi in a way similar to cpi.c. Compile and link, an

5–Using QLogic MPIQLogic MPI Details5-6 IB6054601-00 HSUse Wrapper Scripts for Compiling and LinkingThe scripts in Table 5-1 invoke the compiler and

5–Using QLogic MPIQLogic MPI DetailsIB6054601-00 H 5-7AConfiguring MPI Programs for QLogic MPIWhen configuring an MPI program (generating header file

5–Using QLogic MPIQLogic MPI Details5-8 IB6054601-00 HSTo Use Another CompilerQLogic MPI supports a number of compilers, in addition to the default G

5–Using QLogic MPIQLogic MPI DetailsIB6054601-00 H 5-9AAlso, use mpif77, mpif90, or mpif95 for linking; otherwise, .true. may have the wrong value. I

5–Using QLogic MPIQLogic MPI Details5-10 IB6054601-00 HSCompiler and Linker VariablesWhen you use environment variables (e.g., $MPICH_CC) to select w

5–Using QLogic MPIQLogic MPI DetailsIB6054601-00 H 5-11AWhen running a job in a batch system environment where multiple jobs may be running simultane

Page viii IB6054601-00 HSQLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4sysctl parameters do not help TCP/SD

5–Using QLogic MPIQLogic MPI Details5-12 IB6054601-00 HSEnabling and Disabling Software Context SharingBy default, context sharing is enabled; it can

5–Using QLogic MPIQLogic MPI DetailsIB6054601-00 H 5-13AAdditionally, you can explicitly configure for the number of contexts you want to use with th

5–Using QLogic MPIQLogic MPI Details5-14 IB6054601-00 HSmpihosts File DetailsAs noted in “Create the mpihosts File” on page 5-3, an mpihosts file (al

5–Using QLogic MPIQLogic MPI DetailsIB6054601-00 H 5-15A In the absence of both the -m option and the MPIHOSTS environment variable, mpirun uses the

5–Using QLogic MPIQLogic MPI Details5-16 IB6054601-00 HSGenerally, mpirun tries to distribute the specified number of processes evenly among the node

5–Using QLogic MPIQLogic MPI DetailsIB6054601-00 H 5-17Ampirun monitors the parallel MPI job, terminating when all the node programs in that job exit

5–Using QLogic MPIQLogic MPI Details5-18 IB6054601-00 HSEnvironment variables are specified in descending order, as follows:1. Set in the default she

5–Using QLogic MPIQLogic MPI DetailsIB6054601-00 H 5-19AIPATH_UNIT This variable is for context sharing. When multiple InfiniPath devices are present

5–Using QLogic MPIQLogic MPI Details5-20 IB6054601-00 HSRunning Multiple Versions of InfiniPath or MPIThe variable MPICH_ROOT sets a root prefix for

5–Using QLogic MPIPerformance TuningIB6054601-00 H 5-21ATo keep remote process checks but disable triggering quiescence for temporary IB link failure

QLogic Host Channel Adapter and QLogic OFED Software Users GuideQLogic OFED Version 1.4IB6054601-00 H Page ixSMoved mpirun Options Summary to new App

5–Using QLogic MPIPerformance Tuning5-22 IB6054601-00 HSUse the taskset utility with mpirun to specify the mapping of MPI processes to logical proces

5–Using QLogic MPIMPDIB6054601-00 H 5-23AThe complete list of options is contained in Appendix A.MPDThe Multi-Purpose Daemon (MPD) is an alternative

5–Using QLogic MPIQLogic MPI and Hybrid MPI/OpenMP Applications5-24 IB6054601-00 HSQLogic MPI and Hybrid MPI/OpenMP ApplicationsQLogic MPI supports h

5–Using QLogic MPIDebugging MPI ProgramsIB6054601-00 H 5-25ADebugging MPI ProgramsDebugging parallel programs is substantially more difficult than de

5–Using QLogic MPIQLogic MPI Limitations5-26 IB6054601-00 HSThe following features of QLogic MPI facilitate debugging: Stack backtraces are provided

IB6054601-00 H 6-16 Using Other MPIsThis section provides information on using other MPI implementations. IntroductionSupport for multiple high-perfo

6–Using Other MPIsInstalled Layout6-2 IB6054601-00 HSThese MPI implementations run on multiple interconnects, and have their own mechanisms for selec

6–Using Other MPIsOpen MPIIB6054601-00 H 6-3AOpen MPI Open MPI is an open source MPI-2 implementation from the Open MPI Project. Pre-compiled version

6–Using Other MPIsOpen MPI6-4 IB6054601-00 HSTo compile your program in C, type:$ mpicc mpi_app_name.c -o mpi_app_nameRunning Open MPI ApplicationsBy

6–Using Other MPIsMVAPICHIB6054601-00 H 6-5AMVAPICHPre-compiled versions of MVAPICH 1.1 built with the GNU, PGI, PathScale, and Intel compilers, and