Tuesday, May 21, 2013
Dinesh Rajan Wins Best Talk at CCGrid 2013
Congratulations to CCL graduate student Dinesh Rajan, who won the Best Presentation Award at CCGrid 2013 for his presentation of
Case Studies in Designing Elastic Applications!
Friday, March 22, 2013
Tutorial on Makeflow and Work Queue at CCGrid 2013
Dinesh Rajan will present a tutorial on Building Elastic Applications with Makeflow and Work Queue as part of CCGrid 2013 in Delft, the Netherlands on May 13th. Come join us and learn how to write applications that scale up to hundreds or thousands of nodes running on clusters, clouds, and grids.
Elastic Apps Paper at CCGrid 2013
Dinesh Rajan will present his paper
Case Studies in Designing Elastic Applications at the IEEE International Conference on Clusters, Clouds, and Grids (CCGrid) in Delft, the Netherlands. This work was done in collaboration with Andrew Thrasher and Scott Emrich from the Notre Dame Bioinformatics Lab, and Badi Abdul-Wahid and Jesus Izaguirre from the Laboratory for Computational Life Sciences.
The paper describes our experience in designing three different elastic applications -- E-MAKER, Elastic Replica Exchange, and Folding at Work -- that run on hundreds to thousands of cores using the Work Queue framework. The paper offers six guidelines for designing similar applications:
- Abolish shared writes.
- Keep your software close and your dependencies closer.
- Synchronize two, you make company; synchronize
three, you make a crowd. - Make tasks of a feather flock together.
- Seek simplicity, and gain power.
- Build a model before scaling new heights.
Thursday, March 21, 2013
Genome Assembly Paper in IEEE TPDS
A recent article in IEEE Transactions on Parallel and Distributed Computing describes our work in collaboration with the Notre Dame Bioinformatics Laboratory on SAND - The Scalable Assembler at Notre Dame.
In this article, we describe how to refactor the standard Celera genome assembly pipeline into a scalable computation that runs on thousands of distributed cores using the Work Queue. By explicitly handling the data dependencies between tasks, we are able to significantly improve runtime over Celera on a standard cluster. In addition this technique allows the user to break free of the shared filesystem and run on hundreds thousands of nodes drawn from clusters, clouds, and grids.
Monday, February 18, 2013
CCTools 3.7.0 Released!
The Cooperative Computing Lab is pleased to announce the release of version 3.7.0 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.
The software may be downloaded here.
This is a minor release which adds numerous features and fixes several bugs:
- [WorkQueue] It is now possible to specify chunks (pieces) of an input file to be used as input for worker tasks. [Dinesh Rajan]
- [Chirp] File extended attributes are now supported. [Patrick Donnelly]
- [Makeflow] New -i switch now outputs pre-execution analysis of Makeflow DAG. [Li Yu]
- [WorkQueue/Makeflow] Support for submitting tasks to the PBS batch submission platform added. [Dinesh Rajan]
- [Makeflow] makeflow_log_parser now ignores comments in Makeflow logs. [Andrew Thrasher]
- [Catalog] New catalog_update which reports information to a catalog server. [Peter Bui, Dinesh Rajan]
- [WorkQueue] Various minor tweaks made to the API. [Li Yu, Dinesh Rajan]
- [Catalog/WorkQueue] Support added for querying workers and tasks at run-time. [Douglas Thain]
- [WorkQueue] Many environment variables removed in favor of option manipulation API. [Li Yu]
- [Makeflow] Deprecated -t option (capacity tolerance) removed.
- [WorkQueue] -W (worker status) now has working_dir and current_time fields.
- [WorkQueue] -T (task status) now reports working_dir, current_time, address_port, submit_to_queue_time, send_input_start_time, execute_cmd_start_time. [Li Yu]
- [WorkQueue] -Q (queue status) now reports working_dir.
- [Makeflow] Input file (dependency) renaming supported with new "->" operator. [Michael Albrecht, Ben Tovar]
- [WorkQueue] work_queue_pool now supports a new -L option to specify a log file. [Li Yu]
- [WorkQueue] Tasks are now killed using SIGKILL.
- [WorkQueue] Protocol based keep-alives added to workers. [Dinesh Rajan]
Thanks goes to the contributors for many minor features and bug fixes:
- Michael Albrecht
- Peter Bui
- Patrick Donnelly
- Brian Du Sell
- Kyle Mulholland
- Dinesh Rajan
- Douglas Thain
- Andrew Thrasher
- Ben Tovar
- Li Yu
Please send any feedback to the CCTools discussion mailing list.
Monday, February 11, 2013
CCTools 3.6.2 Released!
The Cooperative Computing Lab is pleased to announce the release of version 3.6.2 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.
This is a bug fix release of version 3.6.1. No new features were added.
The software may be downloaded here:
http://www.cse.nd.edu/~ccl/software/download
Changes:
- [WorkQueue] Corrected memory errors leading to a SEGFAULT. [Li Yu]
- [Makeflow] Properly interpret escape codes in Makeflow files: \n, \t, etc. [Brian Du Sell]
- [Parrot] Watchdog now properly honors minimum wait time. [Li Yu]
- [Parrot] Reports the logical executable name for /proc/self/exe instead of the physical name. [Douglas Thain]
- [WorkQueue] Race conditions in signal handling for workers were corrected. Tasks now have a unique process group to properly kill all task children on abort. [Dinesh Rajan, Li Yu]
- [WorkQueue] Corrected incorrect handling of -C option where worker would not use the same catalog server as work_queue_pool. [Li Yu]
Thanks goes to the contributors for this release: Patrick Donnelly, Brian Du Sell, Dinesh Rajan, Douglas Thain, and Li Yu.
Enjoy!
Tuesday, January 15, 2013
Teaching Distributed Computing with Work Queue
- Ronald J. Nowling and Jesus A. Izaguirre, University of Notre Dame
Tuesday, January 1, 2013
Scaling Up Comparative Genomics with Makeflow
- Eric Lyons, University of Arizona
Friday, November 2, 2012
CCTools 3.6.1 Released!
The Cooperative Computing Lab is pleased to announce the release of version 3.6.1 of the Cooperative Computing Tools, including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, and other software.
This is a bug fix release of version 3.6.0. No new features were added.
The software may be downloaded here: http://www.cse.nd.edu/~ccl/software/download
Changes:
- [Work Queue] Fixes bugs that resulted in a cancelled task becoming a zombie. [Dinesh Rajan]
- [Makeflow] Various corrections to Makeflow manual and html documentation. [Li Yu]
- [Makeflow] -I and -O options now correctly output file list to stdout. [Li Yu]
- [*] Added missing debug flag for ldflags in configure. [Douglas Thain]
- [Work Queue] Now correctly removes directories during cleanup. [Dinesh Rajan]
- [Chirp] -b is now documented in the man/-h page. [Patrick Donnelly]
- [Sand] Fixed a wrong error message. [Peter Bui, Li Yu]
- [Catalog Server] -T option now properly accepts an argument. [Patrick Donnelly]
- [*] Fixed a bug where the wrong version of perl was used in configure. [Dinesh Rajan]
Thanks goes to the contributors for this release: Dinesh Rajan, Patrick Donnelly, Peter Bui, Li Yu, and Douglas Thain.
Enjoy!
Thursday, November 1, 2012
Applied Cyber Infrastructure Class at U. Arizona
- Nirav Merchant and Eric Lyons, University of Arizona