Teaching Distributed Computing with Work Queue

The undergraduate Programming Paradigms class at the University of Notre Dame introduces undergraduate students to a variety of parallel and distributed programming models. Work Queue is used as an example of large scale distributed computing. Using a solar system simulator developed in a previous assignment, students were tasked with splitting a trajectory of the planets' positions into individual frames, populating POVRay scene files, rendering the scenes in a distributed manner using Work Queue, and combining the frames into a movie using ImageMagick. Since the students had used Python extensively, they found it very easy to write a single Work Queue master using the Python bindings. Several of the students went above and beyond the requirements by adding textures to the planets and animating the movement of the camera. The students really enjoyed the assignment while learning about the advantages and pitfalls of distributed computing.

- Ronald J. Nowling and Jesus A. Izaguirre, University of Notre Dame

Scaling Up Comparative Genomics with Makeflow

The CoGe Comparative Genomics Portal provides on-the-fly genomic analysis and comparative tools for nearly 20,000 genomes from 15,000 organisms and has become more and more popular as genome sequence has become less expensive. The portal runs about 10,000 workflows a month and needed a robust solution for distributed computing of various workflows that range from simple to complex. Using Makeflow, the CoGe team is modularizing the workflows being run through CoGe, has early wins in delivering value to the system by easily monitoring/restarting workflows, and is now starting to work on distributing computation across multiple types of compute resources.

-  Eric Lyons, University of Arizona

CCTools 3.6.1 Released!

The Cooperative Computing Lab is pleased to announce the release of version 3.6.1 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.

This is a bug fix release of version 3.6.0. No new features were added.

The software may be downloaded here: http://www.cse.nd.edu/~ccl/software/download

Changes:

• [Work Queue] Fixes bugs that resulted in a cancelled task becoming a zombie. [Dinesh Rajan]


• [Makeflow] Various corrections to Makeflow manual and html documentation. [Li Yu]


• [Makeflow] -I and -O options now correctly output file list to stdout. [Li Yu]


• [*] Added missing debug flag for ldflags in configure. [Douglas Thain]


• [Work Queue] Now correctly removes directories during cleanup. [Dinesh Rajan]


• [Chirp] -b is now documented in the man/-h page. [Patrick Donnelly]


• [Sand] Fixed a wrong error message. [Peter Bui, Li Yu]


• [Catalog Server] -T option now properly accepts an argument. [Patrick Donnelly]


• [*] Fixed a bug where the wrong version of perl was used in configure. [Dinesh Rajan]

Thanks goes to the contributors for this release: Dinesh Rajan, Patrick Donnelly, Peter Bui, Li Yu, and Douglas Thain.

Enjoy!

Applied Cyber Infrastructure Class at U. Arizona

The Applied Cyber Infrastructure Concepts course at the University of Arizona makes use of the Cooperative Computing Tools to teach principles of large scale distributed computing. The project-based course introduced fundamental concepts and tools for analyzing large datasets on national computing resources and commercial cloud providers. For the final project, students developed an appliance for annotating multiple rice genomes using MAKER software, distributed across Amazon and FutureGrid resources via the Work Queue framework. They also developed a web application to visualize the performance and utilization of each cloud appliance in real-time, overall progress of the rice annotation process, and integrated the final results into a genome browser.

- Nirav Merchant and Eric Lyons, University of Arizona

NSF Grant: Data and Software Preservation for Open Science

Mike Hildreth, Professor of Physics, Jarek Nabrzyski, Director of the Center for Research Computing and Concurrent Associate Professor of Computer Science and Engineering, and Douglas Thain, Associate Professor of Computer Science and Engineering, are the lead investigators on a project that will explore solutions to the problems of preserving data, analysis software, and how these relate to results obtained from the analysis of large datasets.

Known as Data and Software Preservation for Open Science (DASPOS), it is focused on High Energy Physics data from the Large Hadron Collider (LHC) and the Fermilab Tevatron. The group will also survey and incorporate the preservation needs of other communities, such as Astrophysics and Bioinformatics, where large datasets and the derived results are becoming the core of emerging science in these disciplines

The three-year $1.8M program, funded by the National Science Foundation, will include several international workshops and the design of a prototype data and software-preservation architecture that meets the functionality needed by the scientific disciplines. What is learned from building this prototype will inform the design and construction of the global data and software-preservation infrastructure for the LHC, and potentially for other disciplines.

The multi-disciplinary DASPOS team includes particle physicists, computer scientists, and digital librarians from Notre Dame, the University of Chicago, the University of Illinois Urbana-Champaign, the University of Nebraska at Lincoln, New York University, and the University of Washington, Seattle.

Tutorial on Scalable Programming at Notre Dame

Tutorial: Introduction to Scalable Programming with Makeflow and Work Queue

October 24th, 3-5PM, 303 Cushing Hall

Register here (no fee) to reserve your spot in the class:

http://www.nd.edu/~ccl/software/tutorials/ndtut12

Would you like to learn how to write programs that can scale up to
hundreds or thousands of machines?

This tutorial will provide an introduction to writing scalable
programs using Makeflow and Work Queue. These tools are used at Notre
Dame and around the world to attack large problems in fields such as
biology, chemistry, data mining, economics, physics, and more. Using
these tools, you will be able to write programs that can scale up to
hundreds or thousands of machines drawn from clusters, clouds, and
grids.

This tutorial is appropriate for new graduate students, undergraduate
researchers, and research staff involved in computing in any
department on campus. Some familiarity with Unix and the ability to
program in Python, Perl, or C is required.

The class will consist of half lecture and half hands-on instruction
in a computer equipped classroom. The instructors are Dinesh Rajan
and Michael Albrecht, developers of the software who are PhD students
in the CSE department.

For questions about the tutorial, contact Dinesh Rajan, dpandiar AT nd.edu.

Global Access to High Energy Physics Software with Parrot and CVMFS

Scientists searching for the Higgs boson have profited from Parrot's new support for the CernVM Filesystem (CVMFS), a network filesystem tailored to providing world-wide access to software installations. By using Parrot, CVMFS, and additional components integrated by the Any Data, Anytime, Anywhere project, physicists working in the Compact Muon Solenoid experiment have been able to create a uniform computing environment across the Open Science Grid. Instead of maintaining large software installations at each participating institution, Parrot is used to provide access to a single highly-available CVMFS installation of the software from which files are downloaded as needed and aggressively cached for efficiency. A pilot project at the University of Wisconsin has demonstrated the feasibility of this approach by exporting excess compute jobs to run in the Open Science Grid, opportunistically harnessing 370,000 CPU-hours across 15 sites with seamless access to 400 gigabytes of software in the Wisconsin CVMFS repository.
  
- Dan Bradley, University of Wisconsin and the Open Science Grid

CCTools 3.6.0 Released!

The Cooperative Computing Lab is pleased to announce the release ofversion 3.6.0 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.

The software may be downloaded here: http://www.cse.nd.edu/~ccl/software/download

This is a minor release which adds numerous features and fixes several bugs:

• [WQ] Added API for logging functionality. [Christopher Bauschka]


• [WQ] Python bindings have more complete access to the API available from C. Documentation has also been improved. [Dinesh Rajan]


• [WQ] No longer manually redirects stdin/stdout/stderr by editing the user provided shell string, it now sets file descriptors directly. User redirections are no longer overridden. [Patrick Donnelly]


• [WQ, Makeflow] The torque batch submission system is now supported. [Michael Albrecht, Douglas Thain]


• [Parrot] Now supports extended attributes. [Patrick Donnelly]


• [Makeflow] Now supports garbage collection of intermediate files. [Peter Bui]


• [Makeflow] Now supports lexical scoping of Makeflow variables. [Peter Bui]


• [Makeflow] New MAKEFLOW keyword for recursive Makeflows. [Peter Bui]


• [WQ] Bindings for WQ now support SWIG versions >= 1.3.29. [Peter Bui]


• [Parrot] iRODS now supports putfile/getfile operations for much faster file copies. [Douglas Thain]


• [Parrot] Now includes watchdog support that runs alongside Parrot. [Douglas Thain, Brian Bockelman]


• [*] CCTools now have been version information from the -v option. Version information is included in debug output with the `-d debug' flag. [Patrick Donnelly]


• [WQ] work_queue_status output has been cosmetically improved. [Douglas Thain]


• [WQ] New $WORK_QUEUE_SANDBOX environment variable. [Dinesh Rajan]

Thanks goes to the contributors for many minor features and bug fixes:

• Michael Albrecht


• Christopher Bauschka


• Brian Bockelman


• Dan Bradley


• Peter Bui


• Iheanyi Ekechukwu


• Patrick Donnelly


• Dinesh Rajan


• Douglas Thain

Please send any feedback to the CCTools discussion mailing list. Enjoy!

Papers at e-Science Conference

Members of the CCL will present two papers and two posters at the upcoming IEEE Conference on e-Science in Chicago:

• 
  Badi Abdul-Wahid, Li Yu, Dinesh Rajan, Haoyun Feng, Eric Darve, Douglas Thain, Jesus A. Izaguirre,
  Folding Proteins at 500 ns/hour with Work Queue
  
  
• Peter Sempolinski, Daniel Wei, Douglas Thain and Ahsan Kareem, A System for Management of Computational Fluid Dynamics Simulations for Civil Engineering
  
  
• Michael Albrecht, John Bent, and Douglas Thain, An Evaluation of Embedded Storage for Exascale and the Value of Asynchronous Data Transfer
  
  
• Dinesh Rajan, Badi Abdul-Wahid, Joe Fetsch, Li Yu, Jesus Izaguirre, Sekou Remy and Douglas Thain, A Case Study in Elastic Scientific Application Design using Work Queue
  

Lecture and Tutorial: Univ. of Arizona

We are doing a guest lecture and tutorial titled Building Scalable Data Intensive Applications with Makeflow and Work Queue at the University of Arizona as part of the Applied CI Concepts class on September 11 and 13, 2012.