Skip to end of metadata
Go to start of metadata

While we have installed a small number of highly-used packages to each version of Python, we encourage users to maintain their own Python packages. This ensures that the user has no discontinuity in their workflow waiting for packages to be installed and that packages work with the user's chosen, loaded Python version module. Python packages are built specifically to a version of Python and may not work properly if a different version of Python is loaded. For example, there is still a great disparity between Python 2 and Python 3. O2's "module" feature allows users to quickly and easily switch between Python versions.

Useful commands:

CommandMeaning
module avail python
shows the versions of Python installed on O2
module load python/version
loads an individual module (substitute version for an actual version)
module unload python/version
unloads an individual module
which python
shows the current version of Python loaded
pip freeze
shows what packages are installed for currently loaded Python module

Note that in order for module avail python to give any output, you must have gcc/6.2.0 loaded into your environment. Alternately, you can use module spider python to circumvent this, but you will still need gcc/6.2.0 loaded in order to load any python modules that are installed.

Setting Up a Virtual Environment

Setup

mfk8@login01:~$ module load gcc/6.2.0
mfk8@login01:~$ module avail python
mfk8@login01:~$ module load python/2.7.12       # or whichever version you'd like here
mfk8@login01:~$ which virtualenv
mfk8@login01:~$ virtualenv nameyourenvhere      # Please read on before executing this command as-is

These commands create a copy of our Python distribution in whichever directory you ran the command, placing it in a folder named after whatever you specified. You can name your environment however you'd like. The above examples generate a directories located at ~/nameyourenvhere or ~/foobar, respectively.

If you'd like to use the packages that are pre-installed in the Python module, include the --system-site-packages flag when creating the virtual environment:

mfk8@login01:~$ virtualenv nameyourenvhere --system-site-packages

The --system-site-packages flag will allow your virtual environment to inherit all existing installed packages in that Python distribution (which means you don't need to install your own numpy or scipy). On O2, we have purposely kept the list of installed python modules as light as possible so that the user is afforded the maximum amount of flexibility to install the versions they need.

You can create as many virtual environments as you need.

To uninstall a virtual environment, simply rm -rf the folder containing the directory:

mfk8@login01:~$ rm -rf nameyourenvhere

Basic Usage

To begin using the virtual environment, it needs to be activated. For a virtual environment located at ~/venv:

mfk8@login01:~$ source venv/bin/activate

Your prompt will now look (something like) this:

(venv)mfk8@login01:~$

From here you an confirm that you're using the python associated with your virtual environment:

(venv)mfk8@login01:~$ which python
~/venv/bin/python

If it points instead to something like /n/app/python/<version>/bin/python or /usr/bin/python, that means your environment is not active. Once you have confirmed that you are indeed inside your virtual environment, you may do whatever you'd like, such as testing code or installing packages.

To deactivate the environment, simply type deactivate, to see the corresponding change to your terminal prompt:

(venv)mfk8@login01:~$ deactivate
mfk8@login01:~$

Installing Packages (via pip)

PyPI (The Python Package Index) is a repository of software for the Python programming language, with packages available listed at https://pypi.python.org/pypi.

To install something, simply type:

(venv)mfk8@login01:~$ pip install nameofpackage

Notice that your virtual environment needs to be active. The above command will search PyPI and install the package to your virtual environment. If it is not active, you will get a permissions error because you will not have write access to the original installation.

You can also install packages manually; please refer to the instructions provided by the corresponding package if this is required (e.g. a README/INSTALL file or a documentation page).

Using Python Packages

In general, you will access packages by importing them at the top of your Python scripts:

#!/usr/bin/env python


import nameofpackage


# actual code here

This will allow you to access that package (called a "module")'s functions. Some modules will have extended functionality; please refer to the appropriate usage instructions provided by the package for more information on how to access them.

A Full Example

Here is an example of loading the desired version of Python on O2, setting up a virtual environment and activating it, installing a package to this personal environment, running a Python script, and deactivating the virtual environment (2.x chosen here arbitrarily):

mfk8@login01:~$ which python
/usr/bin/python
mfk8@login01:~$ module load gcc/6.2.0
mfk8@login01:~$ module avail python

--------------- /n/app/lmod/lmod/modulefiles/Compiler/gcc/6.2.0 ----------------

   python/2.7.12 (E)    python/3.6.0 (E,D)

  Where:

   E:  Experimental
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".

mfk8@login01:~$ module load python/2.7.12
mfk8@login01:~$ which python
/n/app/python/2.7.12/bin/python
mfk8@login01:~$ which virtualenv
/n/app/python/2.7.12/bin/virtualenv
mfk8@login01:~$ mkdir mypythonfolder && cd mypythonfolder
mfk8@login01:~/mypythonfolder$ virtualenv myvirtualenv
(truncated)
mfk8@login01:~/mypythonfolder$ source myvirtualenv/bin/activate
(myvirtualenv)mfk8@login01:~/mypythonfolder$ pip install numpy
(truncated)
(myvirtualenv)mfk8@login01:~/mypythonfolder$ echo "print 'hello world'" > myscript.py
(myvirtualenv)mfk8@login01:~/mypythonfolder$ python myscript.py
hello world
(myvirtualenv)mfk8@login01:~/mypythonfolder$ deactivate
mfk8@login01:~/mypythonfolder$



  • No labels