Tutorial on Scalable Programming at Notre Dame

Tutorial: Introduction to Scalable Programming with Makeflow and Work Queue

October 24th, 3-5PM, 303 Cushing Hall

Register here (no fee) to reserve your spot in the class:

http://www.nd.edu/~ccl/software/tutorials/ndtut12

Would you like to learn how to write programs that can scale up to
hundreds or thousands of machines?

This tutorial will provide an introduction to writing scalable
programs using Makeflow and Work Queue. These tools are used at Notre
Dame and around the world to attack large problems in fields such as
biology, chemistry, data mining, economics, physics, and more. Using
these tools, you will be able to write programs that can scale up to
hundreds or thousands of machines drawn from clusters, clouds, and
grids.

This tutorial is appropriate for new graduate students, undergraduate
researchers, and research staff involved in computing in any
department on campus. Some familiarity with Unix and the ability to
program in Python, Perl, or C is required.

The class will consist of half lecture and half hands-on instruction
in a computer equipped classroom. The instructors are Dinesh Rajan
and Michael Albrecht, developers of the software who are PhD students
in the CSE department.

For questions about the tutorial, contact Dinesh Rajan, dpandiar AT nd.edu.

Global Access to High Energy Physics Software with Parrot and CVMFS

Scientists searching for the Higgs boson have profited from Parrot's new support for the CernVM Filesystem (CVMFS), a network filesystem tailored to providing world-wide access to software installations. By using Parrot, CVMFS, and additional components integrated by the Any Data, Anytime, Anywhere project, physicists working in the Compact Muon Solenoid experiment have been able to create a uniform computing environment across the Open Science Grid. Instead of maintaining large software installations at each participating institution, Parrot is used to provide access to a single highly-available CVMFS installation of the software from which files are downloaded as needed and aggressively cached for efficiency. A pilot project at the University of Wisconsin has demonstrated the feasibility of this approach by exporting excess compute jobs to run in the Open Science Grid, opportunistically harnessing 370,000 CPU-hours across 15 sites with seamless access to 400 gigabytes of software in the Wisconsin CVMFS repository.
  
- Dan Bradley, University of Wisconsin and the Open Science Grid

CCTools 3.6.0 Released!

The Cooperative Computing Lab is pleased to announce the release ofversion 3.6.0 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.

The software may be downloaded here: http://www.cse.nd.edu/~ccl/software/download

This is a minor release which adds numerous features and fixes several bugs:

• [WQ] Added API for logging functionality. [Christopher Bauschka]


• [WQ] Python bindings have more complete access to the API available from C. Documentation has also been improved. [Dinesh Rajan]


• [WQ] No longer manually redirects stdin/stdout/stderr by editing the user provided shell string, it now sets file descriptors directly. User redirections are no longer overridden. [Patrick Donnelly]


• [WQ, Makeflow] The torque batch submission system is now supported. [Michael Albrecht, Douglas Thain]


• [Parrot] Now supports extended attributes. [Patrick Donnelly]


• [Makeflow] Now supports garbage collection of intermediate files. [Peter Bui]


• [Makeflow] Now supports lexical scoping of Makeflow variables. [Peter Bui]


• [Makeflow] New MAKEFLOW keyword for recursive Makeflows. [Peter Bui]


• [WQ] Bindings for WQ now support SWIG versions >= 1.3.29. [Peter Bui]


• [Parrot] iRODS now supports putfile/getfile operations for much faster file copies. [Douglas Thain]


• [Parrot] Now includes watchdog support that runs alongside Parrot. [Douglas Thain, Brian Bockelman]


• [*] CCTools now have been version information from the -v option. Version information is included in debug output with the `-d debug' flag. [Patrick Donnelly]


• [WQ] work_queue_status output has been cosmetically improved. [Douglas Thain]


• [WQ] New $WORK_QUEUE_SANDBOX environment variable. [Dinesh Rajan]

Thanks goes to the contributors for many minor features and bug fixes:

• Michael Albrecht


• Christopher Bauschka


• Brian Bockelman


• Dan Bradley


• Peter Bui


• Iheanyi Ekechukwu


• Patrick Donnelly


• Dinesh Rajan


• Douglas Thain

Please send any feedback to the CCTools discussion mailing list. Enjoy!

Papers at e-Science Conference

Members of the CCL will present two papers and two posters at the upcoming IEEE Conference on e-Science in Chicago:

• 
  Badi Abdul-Wahid, Li Yu, Dinesh Rajan, Haoyun Feng, Eric Darve, Douglas Thain, Jesus A. Izaguirre,
  Folding Proteins at 500 ns/hour with Work Queue
  
  
• Peter Sempolinski, Daniel Wei, Douglas Thain and Ahsan Kareem, A System for Management of Computational Fluid Dynamics Simulations for Civil Engineering
  
  
• Michael Albrecht, John Bent, and Douglas Thain, An Evaluation of Embedded Storage for Exascale and the Value of Asynchronous Data Transfer
  
  
• Dinesh Rajan, Badi Abdul-Wahid, Joe Fetsch, Li Yu, Jesus Izaguirre, Sekou Remy and Douglas Thain, A Case Study in Elastic Scientific Application Design using Work Queue
  

Lecture and Tutorial: Univ. of Arizona

We are doing a guest lecture and tutorial titled Building Scalable Data Intensive Applications with Makeflow and Work Queue at the University of Arizona as part of the Applied CI Concepts class on September 11 and 13, 2012.

Rapid Processing of LIDAR Data in the Field with Makeflow

Makeflow is used to manage the data processing workflow of the Airborne Lidar Processing System (ALPS) for the Experimental Advanced Airborne Research Lidar (EAARL) at the USGS. Over the course of several hours of flight time, about a gigabyte of raw LIDAR waveform data is collected. This data must be geo-referenced in order to convert it into point clouds broken into 2km tiles suitable for GIS use. When we collect this data in the field, it is critical that the field crew can process the data rapidly and take a look at it for obvious problems so they can be corrected before the next flight.

Using Makeflow, the data can be processed quickly on a portable 32-core cluster in the field in about 20 minutes. The data can be processed fast enough to do some cursory analysis and also re-process it a few times if needed to troubleshoot issues. Using Makeflow, it is easy to run the exactly same workflow in the field on the portable cluster or back in the office on a multi-core system.

- David Nagle, US Geological Survey

Tutorial at Cloud Summer School

We will be offering a tutorial titled Building Scalable Data Intensive Applications on the Cloud with Makeflow and Work Queue as part of the Science Cloud Summer School hosted by Indiana University and joined by universities around the country.

Talk at ICE Workshop

Prof. Thain gave a talk titled Computational Abstractions: Strategies for Scaling Up Applications at the Initiative for Computational Economics at the University of Chicago.

CCTools 3.5.2 Released

The Cooperative Computing Lab is pleased to announce the release of version 3.5.2 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.

This is a bug fix release of version 3.5.1. A shell script executable has been added for Torque worker compatibility.

The software may be downloaded here.

Changes:

• [WQ] Improved some debug messages. [Dinesh Rajan]


• [AP] Fixed minor bug for dealing with comparison commands that produce no output. [Douglas Thain]


• [WQ] Fixed a bug where the stdout buffer was not reset at the beginning of every task. [Douglas Thain]


• [WQ] Documented -C option for work_queue_status. [Patrick Donnelly]


• [WQ] Fixed a bug where pool configurations with an absolute path would result in a segfault. [Patrick Donnelly]


• [Parrot] Fixed a bug where Parrot mistakenly thought it correctly wrote to memory using /proc/pid/mem. [Patrick Donnelly]


• [*] Fixed a bug on OSX where non-blocking connects would result in an infinite loop. [Douglas Thain]


• [*] Support for SWIG 1.3.29 added. [Peter Bui]


• [*] Support has been added workers using Torque. [Michael Albrecht, Douglas Thain]


• [Makeflow] Fixed option parsing. [Patrick Donnelly]

CCTools 3.5.1 Released

The Cooperative Computing Lab is pleased to announce the release of version 3.5.1 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.

This is a bug fix release of version 3.5.0. No new features were added.

The software may be downloaded here: http://www.cse.nd.edu/~ccl/software/download

Changes:

• Fixed a file descriptor leak in WorkQueue. [Joe Fetsch, Dinesh Rajan, Patrick Donnelly]


• Better detection of fast memory access in Linux kernels >= 3.0. [Michael Hanke, Douglas Thain]