A beginners guide to scripting with UV-CDAT

The third step of my guide to getting started with Python mentions UV-CDAT as a useful tool for quickly viewing the contents of netCDF files (and the sub-component CDAT as a useful collection of Python libraries). While many people simply use UV-CDAT for this purpose, it actually has many more capabilities. In its entirety, UV-CDAT is an open-source application that links together numerous software packages to form an integrated environment for weather/climate data analysis and visualisation. It can be used as a complete data analysis environment like Matlab, IDL or Mathmatica, only it’s free, has broader capabilities, and is written explicitly with the weather/climate community in mind.

The American consortium in charge of developing UV-CDAT consists of four Department of Energy laboratories (Los Alamos, Lawrence Berkeley, Lawrence Livermore National Laboritory, and Oak Ridge), two universities (University of Utah and the Polytechnic Institute of New York University), NASA, and two private companies (Kitware and tech-X). The development team has done a wonderful job in a relatively short amount of time, bringing together a suite of climate data analysis tools (CDAT), a provenance tracking and workflow recording system (VisTrails), and a collection of some of the the most popular visualisation tools out there (ParaView, DV3D, VisIt, ViSUS and VTK) (see Williams et al 2013 for details of how this was achieved). The end product is:

1. An awesome graphical user interface where you can quickly and easily view your data from every angle and projection imaginable
2. A scripting environment that puts all the tools you’ll ever need right at your fingertips
3. A workflow builder to help manage the scheduling, tracking and recording of large data processing pipelines

The first of these is pretty self explanatory (e.g. check out these basic and more advanced video clip examples, in addition to the UV-CDAT YouTube channel itself), while the third is really only relevant if you’re in charge of coordinating very large data processing tasks (e.g. the post processing of CMIP5 climate model simulations). As such, my focus here is on the scripting environment. In particular, once you’ve installed UV-CDAT on your machine (see instructions at the end of the post), if you go ahead and fire up IPython (an excellent interactive environment from which to explore what’s going on) you’ll see something like the following:
 
$ /usr/local/uvcdat/1.4.0/bin/ipython
Python 2.7.4 (default, Oct 21 2013, 17:01:37)
Type "copyright", "credits" or "license" for more information.

IPython 0.13.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.

In [1]:


If you then type import at the prompt at hit the [tab] key, the interpreter will tell you that there’s a whopping 490 or so modules available to import into the environment!

In [1]: import
Display all 492 possibilities? (y or n)


The majority of these modules are only of interest to developers working on improving UV-CDAT, but among them is a collection of pretty much all the modules a typical weather/climate scientist could need in writing their own data analysis scripts. I’m not aware of any consolidated list of this collection of useful modules, let alone links to the relevant documentation (which can sometimes be difficult to track down), so I’ve tried to provide one below. My list isn’t exhaustive (i.e. it’s mainly the modules I find useful), so please let me know if there are other modules you think should be added!

 
Core CDAT modules

When scripting in the Python/UV-CDAT environment, you will use the following core CDAT modules most frequently:

cdms2	The Climate Data Management System (cdms) module is primarily used for netCDF input and output.
cdutil	The Climate Data Specific Utilities (cdutil) module contains a whole range of utilities for calculating things like climatologies and anomalies, for custom spatial regions and temporal seasons.
genutil	The General Utilities (genutil) module has utilities for all sorts of climate relevant statistics (e.g. correlation, covariance, auto-correlation, lagged correlation, root mean square, standard deviation, percentiles, linear regression, etc).
MV2	This module is basically a copy of the numpy.ma module used for dealing with masked data arrays, but it preserves the metadata contained in cdms2 variables.

On the UV-CDAT website and YouTube channel you’ll find links to papers and videos explaining the visualisation interface, but nothing on the actual data analysis/scripting capability. As mentioned previously, this is because they didn’t develop that capability themselves – they’ve simply linked together lots of useful packages that have their own documentation located elsewhere. The functionality of CDAT hasn’t changed substantially since the mid-2000s, so the original CDAT website has everything you need in terms of documentation for the core CDAT modules, including manuals, tutorials, tips and tricks, and the application programming interface (API). The teaching materials from the 2011 AMS Python Short Course are also pretty useful.

 
Regridding modules

Since regridding is performed extensively by the visualisation interface, there are a number of modules included:

regrid2	The basic CDAT regridding package.
css	A collection of modules from the ngmath library, which contains a range of interpolators and approximators for one-, two-, and three-dimensional data. Details can be found at the relevant NCAR page or CDAT contributed packages page.
ESMP	A Python interface to the Earth System Modeling Framework (ESMF) regridding utility. ESMF is software for building and coupling weather, climate, and related models.

 
General Python modules

Naturally, the UV-CDAT installation also comes with pretty much all the general Python modules that people in the weather/climate sciences find useful. These include:

matplotlib, basemap	The most widely used Python plotting packages. As mentioned in my getting started with Python post, I’d recommend using Iris (which is not included with UV-CDAT) if you’re going to be doing a lot of plotting with Python.
numpy, scipy	Numerical programming and scientific data analysis/statistics.
unittest, nose, doctest	For unit testing.
argparse	For handling command line arguments.

 
General climate packages

There are also numerous other user contributed packages included with the UV-CDAT install:

eofs	Empirical Orthogonal Function (EOF) analysis.
windspharm	Calculates wind related quantities such as the streamfunction, velocity potential and divergence/convergence.
lmoments	Routines for statistical analysis using L-moments (i.e. measures of the location, scale and shape of probability distributions).

 
Installation and help

Installing UV-CDAT can be a little tricky, as it requires that a number of common software development packages be installed on your machine first. The specific requirements differ depending on your operating system, however there are thorough instructions available. Failing that, the mailing list is well monitored and responses are usually prompt, and I’m hopeful that the recently launched support forum will take off in the months ahead.

If you only want the highly used CDAT libraries (like cdms2, MV2, etc) and have no interest in the graphical user interface or workflow builder, you might also consider installing cdat-lite instead. If you have Anaconda installed, then it can be installed with Conda via a single command line entry like:
conda install -c https://conda.binstar.org/scitools cdat-lite

See my post on software installation for details.