Sizable code packages are typically compiled by Makefiles. These have the advantage of only recompiling files that have been changed (converting human readable .c, .cxx, .f, or .f90 C or Fortran files to .o) files and then relinking the .o files to produce an executable. Commonly "configure" or "cmake" is run to produce Makefiles. Alternatively, Makefiles may be customized by setting machine specific arch or Make.inc files in which compilers, flags, and paths to needed libraries can be set.
For your own code which you repeatedly recompile, you will likely want to create your own make file. See Make tutorial for a short introduction to the use of Makefiles. Makefile syntax is a bit tricky. For examples, blanks at the ends of lines may cause errors. Indented lines must sometimes be tabs as opposed to spaces.
If a make file is named Makefile or makefile, then you can execute it by
Else if the makefile has some other name, e.g. makefoo, you can execute it by
make -f makefoo
Lines in a makefile ending with :, e.g.
are parts of a makefile that can be individually executed,
make fooConstructing Makefiles tuned to an individual machine and set of compilers is often accomplished by "configure" and "cmake"
Configure files are produced by running Autoconf. The developer of code wants users to be able compile the code so they can run it on their own platform. Running autoconf, he can specify what libraries and other options are needed, producing a "configure" executable. The code user runs "configure" to learn what environmental variables to specify paths and compilers to use. -- flags to specify what code options to compile. Running configure creates makefiles. The hope is that after configure is run, the makefiles will work, i.e, the user can just type "make", "make test", and "make install". The following discussion goes through what is supposed to happen and how to fix the most common glitches. If you want more information about how to produce your own configure file, see the above web site or try "info autoconf" at the command line. Some packages are migrating their build system from autoconf to cmake and below Compiling with Cmake . On the NCSU hpc, you can get a fairly recent version of cmake by using the command line
(See for example running cmake). Hopefully, we'll also soon have a web page for compiling packages using cmake, the rest of this page addresses how to use "configure".
First you need to get the source code to compile. Likely this is from a web site. If you downloaded to your own computer, then in linux you could transfer it to the hpc by ftp,
where "cd" changes directories on the target machine you've "ftp"ed into, "lcd" changes directories on your local machine, "put foo" copies file foo onto the rtarget machine, "get foo" copies foo from the target machine to the local. For windows, WinSCP can transfer files, looks like there is a putty file transfer program?
If you're in luck the web site you found the code on may give a "git" command you can use directly from the HPC login blades, thereby avoiding having to transfer files to your local machine and then to the hpc. Another single transfer option is to use the HPC VCL image. There you can use the browser firefox and download files directly the HPC file system.
Suppose you've succesffuly transferred a foo.tar.gz file to a directory where you can unpack it.
produces a file foo.tar
tar tvf foo.tar
shows the contents. If the contents are not all prefixed with a directory name, you'll likely want to create a subdirectory and move the tar file to that directory, e.g.
mkdir foo mv foo.tar foo cd foo
tar xvf foo.tar
will extract the contents of the tar ball. If instead the downloaded file is *.tar.bz, you can use
tar xjvf foo.tar.bz
If the extraction is into directory foo-1.1
ls -l configure
shows there is an executable file called configure, then the configure executable is meant to produce make files that will compile the code package. As the INSTALL or README file in the same directory is likely to tell you, the action may be as easy as
configure make make install
But anyway look at the INSTALL file and see if there are helpful hints. If "configure" and "make" don't work, hopefully some of the rest of this page will help. "make install" typically will only work if you are "root" or if you have used the --prefix flag with configure (to specify a directory to install into, where yoU have write privileges).
If configure does exist,
./configure --help |& tee confhelp
will show you the flags and environmental variables you can set, conveniently piped to the file confhelp, where you can examine them at leisure.
lets you view confhelp, allowing you to scroll up and down through the confhelp file.
gets you out of confhelp (where ESC indicates the ESC key). Typically I would at this point open several windows, e.g. ssh or putty into the login nodes once per desired window. In one window view confhelp, in another one, you'll execute configure, set environmental variables, and then compile via "make". Typically you'll want a third window for hunting for needed libraries, so that the library paths can be set in the "execute configure" window.
At the bottom of the confhelp file (when using view, scroll down with the down arrow, or type G) , you'll find a list of environment variables you can set. Commonly, you'd set the C and/or Fortran compilers, paths to find include directories (containing .h and .mod files) and paths to find libraries. You'll set all of these in the window in which configure and later make is to be run. For example, to use intel parallel compilers, I might first use
Then in the default tcsh shell, set the parallel fortran and C compilers by setting environment variables by
setenv CC mpicc setenv FC mpif90 setenv CPPFLAGS "-I/usr/local/apps/intel/compilers_and_libraries_2016.3.210/linux/mkl/include -I/usr/local/apps/hdf-centos7/intel2016-1.8.15p1/include" setenv LDFLAGS "-L/usr/local/apps/intel/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64_lin -L/usr/local/apps/hdf-centos7/intel2016-1.8.15p1/lib"
for example. The above commands also set environment varaibles for include and library paths. In this case we suppose for example that mkl (math libraries) and hdf (a file format used for large data sets) libraries are needed. For the hdf libraries, notice that the directory tells the compiler and hdf version number. Libraries are typically compiler dependent, so use a library compiled with the compiler you want to use. Since most system libraries are from gnu compilers, compilers will often acept gnu compiled libraries. It's worth looking through subdirectories of /usr/local/apps, also of /usr/lib and /usr/lib64 for libraries you need. A useful command for your search is "find". For example,
cd /usr/local/apps find . -name "libhdf*" -print |& tee findings
will find all file names starting with libhdf in all subdirectories of /usr/local/apps. The find command takes a long time from /usr/local/apps, so is more practical from a subdirectory. So instead you might try
cd /usr/local/apps ls | grep hdf
to see if any of the files (including directory names) in /usr/local/apps happen to contain the string "hdf". To see what the "| grep hdf" does, you might just try the command "ls".
The mkl libraries include tuned blas and lapack (matrix) and fft (fast fourier transforms) libraries, which are among the few operations that can run at a reasonable fraction of the advertised processor speed, so using these tuned libraries can give orders of magnitude speedup over reference algorithms. mkl libraries are a frequent source of trial and error, much of which can be avoided by looking at the The Intel Math Kernel Library Link Line Advisor. mkl is also available for gnu compilers, but likely your code will execute faster if you compile with the intel compilers.
The above setenv commands set environment variables in the tcsh or csh shell. For bash shell use for example
For some supported compilers, see NCSU HPC Compilers. If you're likely to have to update the code package with a newer version, it's a good idea to save all environment setting commands in a file, so that you can duplicate them the next time you compile the code.
Searching through the confhelp file (0 or :1 will take you to the top in view), you'll likely find quite a few more flags you need to set if you want the package to perform as you would like. I would typically set both "--enable-shared" and "--enable-static" to get both static (.a) libraries (linked in at compile time) and and shared (.so) libraries, which are linked at run time, allowing a smaller executable file, but also requiring the executable to be able to find them at run time, a possible source of error. You can use the third window to search for required or otherwise useful packages. You can add library directories to the search path by
setenv LD_LIBRARY_PATH /usr/local/apps/foo/lib:$LD_LIBRARY_PATH
or by adding them to the list of directories in the LDFLAGS environment variable set above. Most of the confhelp file gives optional flags to be placed on the ./configure command line. Typically, you won't be logged in as root, so the default installation directories won't be available to install code. To install the code in directoriy /home/foouser/foo, you would use
where typically I would specify the compiler and version by
--prefix = /home/foouser/foo/1.1-intel2016
You might keep this convention in mind as you look through /usr/local/apps and /usr/local/usrapps to see if someone else has already compiled the pakcage you need. Since space in /home/foouser is quite limited (though backed up daily), you might instead have asked for space in /usr/local/usrapps.
Having specified environment variables and the flags you want, then type
./configure --prefix=/whatever/foo/1.1-intel2016 |& tee conf1
Ideally, and in fact often for small packages, configure is successful and then you just type
make |& tee log1
Somewhere, possibly in an INSTALL or README file, you'll find whether there is a test package. In which case,
make test |& tee test1
or plausibly "make check" will run. And after it succeeds, you type
make install |& tee install1
where "install" typically works, so we may choose not to tee to an install log file, but you should look in the prefix directory to see if there are the usual lib, bin, include subdirectories.
Some final steps. If you have created an executable package, create a .csh file which has the paths needed to run the code. You'll source this in the bsub file you use to run the job in batch mode. If the code is in "path/foo/1.1-intel2016" you'll typically put the .csh foo.csh in "path/foo/foo.csh". For example .csh files look in many subdirectories of /usr/local/apps. In particular the directory /usr/local/apps/env has many examples. If there was anything that took a while to figure out, be sure you put it in a compileNOTES file in "/path/foo/compileNOTES". The idea is to save you time when the next version needs compiling. Environmental variables you used and the configure command and flags are good things to put in compileNOTES. Also if you found some that only some subsets of flags worked.
You've got two log files to look at. Configure produces conf.log and your tee produced conf1. view will let you look at these files and scroll up and down. Inside view you may find
Most typically, configure can't find a package you need. Perhaps you copied the path wrong. Paths entered with -- flags may for example assume you are giving them the path that has both the lib and include subdirectories.
The log files will typically give you a line number xxxx in the configure file.
view configure :xxxx
will take you to like xxxx and you can scroll up and down to see what went wrong. "Compiler will not produce executables" is likely to mean something is wrong with a path ? or needs a -L or -I before a path ?
Googling an error message may help.
Something to (hopefully) avoid doing is going to the line in configure where there was a failure and commenting it. To be avoided as likely to cause problems later, in the "make" phase. But perhaps you'll find no other way. If you do edit configure make sure to make a backup so you can replace your hacking attempt with the original.
You may find after a successful configure command that there is some package you really would like to include, so you want to run configure again. Some packages are kind enough to display the options your successful configure has enabled, most are not. Or perhaps you've discovered you're using the wrong compiler, or that using both static and shared libraries gives problems. In these cases, you need to get rid of the make files that configure created on the last successful run. Typically
gets rid of the out of date makefiles. Then you can try configure again with revised environmental variables and/or -- flags.
An INSTALL or README file will typically give some instructions on how to compiler the code once configure has been successfully run. Most typically, from the same window in which you just ran configure, type one of the following
make |& tee log1 make all |& tee log1
Ideally the code compiles without incident. For the INSTALL or README file you know whether it helps to type one of
make check |& tee test1 make test |& tee test1
and when the check or test is successful, type
and find that the libraries, executables, etc. are in subdirectories of the directory you specified with the --prefix command.
For small packages, make is often in fact uneventful and successful.
Alas, life is not always so easy.
When the "make" or "make all" fails, the log1 (I keep numbering them till I get a success, under 10 such files is pretty good) is useful. In the window in which you had previously looked at confhelp
view log1 /error :1 /Error :1 /undefined
let us look at the file log1 and search for the first instances of error, Error and undefined. Useful in case the "make" did not cease on the first fatal error. Usually, I fix the first error, then try "make" or "make clean" again. "make clean" gets rid of the .o files already produced. so is a bit safer, but causes more compilation.
"undefined reference" is a common error. The undefined reference "foo4_all" is named. Typically "foo4_all" is to be found in some library. libxxxx.a or libxxxx.so. It's worthwhile to search some common directories, /usr/lib, /usr/lib64, less commonly /usr/local/apps/lib64. Depending on the compiler you are using, there is an associated lib directory. If you specified intel comilers by "source /usr/local/apps/mpich3/centos7/intelmpi2016.csh" directories you might look in are /usr/local/apps/intel/compilers_and_libraries_2016.0.109/linux/compiler/lib/intel64_lin,/usr/local/apps/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin, and /usr/local/apps/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64_lin. Having cd'ed to a directory you can search it for foo4_all in all of the static libraries lib*.a by
nm -A *.a | grep foo4_all | grep T
and in all the shared libraries lib*.so by
nm -AD *.so | grep foo4_all | grep T
A bit of explanation "nm -A *.a" would give a list of all the references in all the .a files, quite long, grepping for foo4_all gives only the lines with foo4_all, grepping for T gets rid of lines with "U" or "B". We're looking for the lines with the "T", which means the reference is actually defined in this library, not merely referenced.
Of course I don't actually think you can find foo4_all in any of these directories, so next you would google for it and find what package it is part of. You might then search /usr/local/apps to see if any of its subdirectories look like they might have been created to provide foo4_all. Then you can cd to that directory and see what you can see. Ideally you need to find a library compiled with the same compiler, though gnu (gcc, gfortran, g++) compiled packages will often work, e.g. all the libraries in /usr/lib and /usr/lib64 are gnu compiled.
If you can't find the reference, you may find you need to download and compile another library (perhaps one you found as a requirment you'd overlooked in your first read of INSTALL, or perhaps you found the requirement by looking in a user forum for compiling the package)
Once you have found the reference, how to fix it. A primitive method that works when this undefined reference occurs at the final "ar" step of producing a library or -o line to produced an executable is to find the command that failed, move the cursor to it in your view session of test1, find its line number and write that line to a file
:.= :1234w comp1 :q chmod +x comp1 vi comp1 i
comp1 is now an executable file, you can insert the needed file and path into the path, e.g. you'd put in "-L/pathtofoolib -lfoo" in the -o command. If
works you've made some progress or at least made it to the next undefined reference. Most usually when you again type
make |& tee log2
you find the same undefined reference occurs later. Some solutions is to put the library path in LD_LIBRARY_PATH by
setenv LD_LIBRARY_PATH /pathtofoolib:$LD_LIBRARY_PATH
which is in the default tcshrc shell. If you have set your shell to bash
Another possibililty, if you have a Makefile.include, you may be able to edit that file so that the pathtofoolib is in the link path, e.g. add -L/pathtofoolib to the LDFLAGS line. After any of these "fixes" you would again do
make |& tee log3
and pray. Yet another possibility is to look again at the confhelp file, see what envionment variables it suggests for library paths. One was is to scroll up the previous setting of the same environment variables. If before you typed
setenv LDFLAGS -L/usr/local/fo/lib
edit that line to be
setenv LDFLAGS "-L/usr/local/fo/lib -L/pathtofoolib"
If the problem is the code cannot find a .h file, then having found it in /usr/local/apps/fooful/include, you would similarly add -I/usr/local/apps/fooful/include to the environment variable CPPFLAGS. Then use
make distclean ./configure --sameflagsasthefirst time make |& tee log4
Many packages in the last decade have migrated to using cmake instead of the autoconf configure. There are quite a few optional ways to compile with cmake. Here we describe an interactive way using ccmake (/usr/local/apps/bin/ccmake is a current path on the NCSU HPC. )
Having unpacked the source code into a directory, path /pathto/dir/foo, do
cd /pathto/dir/foo mkdir build cd build /usr/local/apps/bin/ccmake /pathto/dir/foo
You get a window reminiscent of Midnight Commander, in which you can type c. Then you get some sort of error message about can't find a library or an include file. Getting out of ccmake by "CTRL C", you can give a library path /foopath/foodir/lib, adding it to the LD_LIBRARY_PATH by
setenv LD_LIBRARY_PATH /foopath/foodir/lib:$LD_LIBRARY_PATH
or similarly an include path /foopath/foodir/include by
setenv C_INCLUDE_PATH /foopath/foodir/include:$C_INCLUDE_PATH
Or possibly you'll need the path to an executable
setenv PATH /foopath/foodir/bin:$PATH
While still in the terminal in which you are using ccmake, you may also find it useful to do some "source" commands, e.g. for a newer flavor of python than the system version, use
(source commands set environmental variables). Or possibly ccmake will have told you an environmental variable to set. Having set the hopefully useful environmental variables, then repeat
and again try "c". Eventually, after several tries, ccmake will have found the needed environment variables (and you'll have selected the ON, OFF options you want from inside ccmake) and you'll get a "g" option which will generate the make files and exit. You can now type
make VERBOSE=1 |& tee log1
and see what happens.
Many packages which use cmake for a build packages are in active development, so may need up-to-date dependencies. It's worthwhile to check versions before starting. If you missed one, you may have to recompile that package and restart ccmake with a new set of environmental variables. Before restarting make, it's a good idea to do
After an apparently successful compilation, you'll likely want to do "make check" or "make test" to see if the build executes as expected, and "make install" to get the build into a simple directory structure.