Installation - CCTools Documentation (2024)

The CCTools software runs on Linux and Mac computers,whether laptops, desktops, or large high performance clusters.There are several ways to install:

• Install From Conda is best for most users on laptops or clusters.
• Install From Github is recommended for developers or those trying out the latest features.
• Install From Binary Tarballis recommended for use on specific supported platforms.

Install From Conda

Installing via the Conda package manager will give you a personalinstallation of the software in your home directory.You may already have Conda installed. To check:

$ conda -V

This displays the version of conda currently installed. If it fails, then you should install Miniconda.Miniconda is a light version of Anaconda, and we recommend it as it is much faster to install.We also recommend installing the version for Python 3.9

Once Conda is installed, then install CCTools with:

# run this command once$ conda create -n cctools-env -y -c conda-forge --strict-channel-priority python ndcctools# run this command every time you want to use cctools$ conda activate cctools-env# run this command every time you want to update your version of cctools (after `activate` as above).$ conda update -y -c conda-forge ndcctools

And that's it! You can test your setup following the instructions here.

Install From Github

If you wish to install the latest version of CCTools totry a new feature or develop new code then you can install from github.This requires that you first install the necessarysoftware dependencies, and the easiest way to do this is to createa new cctools-dev environment via Conda:

git clone git://github.com/cooperative-computing-lab/cctools.git cctools-srccd cctools-srcunset PYTHONPATHconda env create -y -f environment.ymlconda activate cctools-dev

This can run slowly, so, for a potential speedup, try running:

unset PYTHONPATHconda env create -y -f environment.yml --experimental-solver=libmamba

Now that you are inside the cctools-dev environment, you can check outthe software repository and build it:

./configure --with-base-dir $CONDA_PREFIX --prefix $CONDA_PREFIXmakemake install

The next time you log in, be sure to activate the environment again:

unset PYTHONPATHconda activate cctools-dev

Install From Source Tarball

In a similar way, you can download a source package from the download page.Then unpack the tarball and compile:

$ tar zxf cctools-*-source.tar.gz$ cd cctools-src$ ./configure --prefix $HOME/cctools$ make$ make install

Install From Binary Tarball

The CCTools software is pre-built for a small number of platforms.Download a source package from the downloadpage. And follow this recipe whilelogged in as any ordinary user:

$ tar zxf cctools-*-source.tar.gz$ cd cctools-*-source$ ./configure --prefix $HOME/cctools$ make$ make install

Setting Your Environment

If you installed CCTools from github, source tarball, or from a binary tarball, you will need to set some environment variables.

First determine the python and perl versions you are using:

# Note: your output may vary according to your python version.$ python -c 'from sys import version_info; print("{}.{}".format(version_info.major, version_info.minor))'3.7# Note: your output may vary according to your perl version.$ perl -e 'print("$^V\n")'5.16.3

Now update your environment variables with those versions:

$ export PATH=$HOME/cctools/bin:$PATH# Change 3.7 to the python version you found above.$ export PYTHONPATH=$HOME/cctools/lib/python3.7/site-packages:${PYTHONPATH}# Change 5.16.3 to the perl version you found above.$ export PERL5LIB=$HOME/cctools/lib/perl5/site_perl/5.16.3:${PERL5LIB}

Testing Your Installation

You can test that the key programs are available with:

$ work_queue_worker --versionwork_queue_worker version 7.4.3 FINAL (released 2022-02-03 16:26:52 +0000)$ makeflow --versionmakeflow version 7.4.3 FINAL (released 2022-02-03 16:26:52 +0000)...

You can test that the python and perl modules are available with:

$ python -c 'import work_queue; print(work_queue.WORK_QUEUE_DEFAULT_PORT)'9123$ perl -MWork_Queue -e 'print("$Work_Queue::WORK_QUEUE_DEFAULT_PORT\n")'9123

If the above commands fail, please make sure that you follow one (and onlyone!) of the methods above. For example, if you are using a conda installation,make sure that your PYTHONPATH is unset.

Special Cases

The software will happily build and run without installing any externalpackages. Optionally, the CCTools will interoperate with a variety of externalpackages for security, data access, and novel research methods.To use these, you must download andinstall them separately:

• Python
• Perl
• CVMFS
• FUSE
• iRODS (version 4.0.3)
• Globus (version 5.0.3)
• Hadoop
• xRootD

Once the desired packages are correctly installed, unpack the CCTools and thenissue a configure command that points to all of the other installations. Then,make and install as before. For example:

$ ./configure --with-globus-path /usr/local/globus$ make && make install

SWIG

CCTools needs SWIG during compilation to provide pythonand perl support. SWIG is available through conda, or as a package of manylinux distributions. Once SWIG is installed, the configure script shouldautomatically find it if the executable swig is somewhere in your PATH.

iRODS

Building Parrot with support for the iRODS service requires some custom buildinstructions, since Parrot requires access to some internals of iRODS that arenot usually available. To do this, first make a source build of iRODS in yourhome directory:

cd $HOMEgit clone https://github.com/irods/irods-sourcecd irods-sourcegit checkout 4.0.3$ packaging/build.sh --run-in-place icommandscd ..

Then, configure and build CCTools relative to that installation:

$ git clone https://github.com/cooperative-computing-lab/cctools cctools-source$ cd cctools-source$ ./configure --with-irods-path ../irods-src$ make && make install

MPI

Building with MPI requires a valid MPI installation to be in your path.Generally CCTools compiles with both intel-ompi and MPICH. If you do not havempi installed already, we suggest downloading the latest MPICH from the MPICHwebsite. The latest known supporting version of MPICHwith CCTools is MPICH-3.2.1. Simply build MPICH as is best for yoursite/system, and then place the binaries in your path. We also suggestconfiguring MPICH to use gcc as the compiling software. For example:

$ tar -xvf MPICH-3.2.1.tar.gz$ cd MPICH-3.2.1 CC=gcc CXX=g++$ ./configure$ make && make install

Once MPI is in your path, configure CCTools to use MPI and then install. For example:

$ cd ~/cctools$ ./configure --with-mpi-path=`which mpicc`$ make && install `

Now, our tools will be MPI enabled, allowing you to run Makeflow as an MPI job,as well as using both WorkQueue as MPI jobs, and submitting Makeflow andWorkQueue together as a single MPI job.

Build on Mac OSX

In order to build CCTools on Mac OS X you must first have the Xcode CommandLine Tools installed. For OS X 10.9 and later this can be done using thefollowing command:

xcode-select --install

Then, click "Install" in the window that appears on the screen. If the commandline tools are already installed, you will get an error and can proceed withthe instructions in the "Installing From Source" section above. For OS Xversions before 10.9, you will need to first install Xcode. Xcode can be foundin the App Store or on the installation disk.

License

The Cooperative Computing Tools are Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2022 The University of Notre Dame. All rights reserved. This software is distributed under the GNU General Public License.