CCTools 5.3.0 released

The Cooperative Computing Lab is pleased to announce the release of version 5.3.0 of the Cooperative Computing Tools including Parrot, Chirp, Makeflow, WorkQueue, Umbrella, SAND, All-Pairs, Weaver, and other software.

The software may be downloaded here:
http://www.cse.nd.edu/~ccl/software/download

This minor release adds several features and bug fixes. Among them:

• [Makeflow]  Several enhancements in garbage collection. (Nick Hazekamp)
• [Makeflow]  Better task state handling when recovering execution log. (Nick Hazekamp)
• [Parrot]    Correct handling of multi-threaded programs. (Patrick Donnelly)
• [Parrot]    Adds parrot_mount, to set arbitrary mount points while parrot is executing. (Douglas Thain)
• [Parrot]    Add --fake-setuid option, for executables that request setuid. (Tim Shaffer)
• [Parrot]    Update cvmfs uri to new convention. (Jakob Blomer)
• [Parrot]    Add --whitelist to restrict filesystem access. (Tim Shaffer)
• [Parrot]    Make special file descriptors invisible to tracee. (Patrick Donnelly)
• [Resource Monitor] Compute approximations of shared resident memory. (Ben Tovar)
• [Resource Monitor] Remove resource_monitorv for static binaries. resource_monitor now handles all cases.  (Ben Tovar)
• [Resource Monitor] Working directories are not tracked by default anymore. Use --follow-chdir instead.  (Ben Tovar)
• [Umbrella]  Support for curateND. (Haiyan Meng)
• [Umbrella]  Support for installing software from package managers. (Haiyan Meng)
• [Work Queue] Adds option -C to read a JSON configuration file. (Ben Tovar)
• [Work Queue] Several bug fixes regarding task/workflow statistics. (Ben Tovar)
• [Work Queue] Master's shutdown option now correctly terminates  workers. (Ben Tovar)
• [Work Queue] Adds --sge-paremeter to ./configure script to personalize the sge_submit_workers script. (Ben Tovar)
• [Work Queue] Adds the executable disk_allocator, to restrict disk usage at the workers. (Nate Kremer-Herman)

Incompatibility warnings:

• Workers from this release do not work correctly with masters from previous releases.

Thanks goes to the contributors for many features, bug fixes, and tests:

• Jakob Blomer
• Neil Butcher
• Patrick Donnelly
• Nathaniel Kremer-Herman
• Nicholas Hazekamp
• Peter Ivie
• Kevin Lannon
• Haiyan Meng
• Tim Shaffer
• Douglas Thain
• Ben Tovar
• Mathias Wolf
• Anna Woodard

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/software/help/

Enjoy!

Analyzing LHC Data on 10K Cores with Lobster

Prof. Thain gave a talk titled Analyzing LHC Data on 10K Cores with Lobster at the Workshop on Data Intensive Computing in the Clouds at Supercomputing 2015.  The talk gave an overview of our collaboration with members of the CMS experiment at Notre Dame.  Together, we have built a data analysis system which can deploy the complex CMS computing environment on large clusters of non-dedicated machines.

Global Filesystems Paper in IEEE CiSE

Our latest paper, in collaboration with Jakob Blomer and the CVMFS team at CERN, describes the evolution of global-scale filesystems to serve the needs of the world-wide LHC experiment collaborations:

• Jakob Blomer, Predrag Buncic, Rene Meusel, Gerardo Ganis, Igor Sfiligoi and Douglas Thain, The Evolution of Global Scale Filesystems for Scientific Software Distribution, IEEE/AIP Computing in Science and Engineering, 17(6), pages 61-71, December, 2015. DOI: 10.1109/MCSE.2015.111 

Delivering complex software across a worldwide distributed system is a major challenge in high-throughput scientific computing. The problem arises at different scales for many scientific communities that use grids, clouds, and distributed clusters to satisfy their computing needs. For high-energy physics (HEP) collaborations dealing with large amounts of data that rely on hundreds of thousands of cores spread around the world for data processing, the challenge is particularly acute. To serve the needs of the HEP community, several iterations were made to create a scalable, user-level filesystem that delivers software worldwide on a daily basis. The implementation was designed in 2006 to serve the needs of one experiment running on thousands of machines. Since that time, this idea evolved into a new production global-scale filesystem serving the needs of multiple science communities on hundreds of thousands of machines around the world. 

Preservation Talk at iPres 2015

Prof. Thain gave a talk titled "Preserving Scientific Software Executions: Preserve the Mess or Encourage Cleanliness" at the 2015 Conference on Digital Preservation.  This talk gives a high level overview of our work on preservation, encompassing packaging with Parrot, environment specification with Umbrella, and workflow preservation with Prune.