NSF Grant to Support CCTools Development

We are pleased to announce that our work will continue to be supported by the National Science Foundation through the division of Advanced Cyber Infrastructure.

The project is titled "SI2-SSE: Scaling up Science on Cyberinfrastructure with the Cooperative Computing Tools"  It will advance the development of the Cooperative Computing Tools to meet the changing technology landscape in three key respects: exploiting container technologies, making efficient use of local concurrency, and performing capacity management at the workflow scale.  We will continue to focus on active user communities in high energy physics, which rely on Parrot for global scale filesystem access in campus clusters and the Open Science Grid; bioinformatics users executing complex workflows via the VectorBase, LifeMapper, and CyVerse disciplinary portals, and ensemble molecular dynamics applications that harness GPUs from XSEDE and commercial clouds. 

Announcement: CCTools 6.0.0. released

The Cooperative Computing Lab is pleased to announce the release of version 6.0.0 of the Cooperative Computing Tools including Parrot, Chirp, Makeflow, WorkQueue, Umbrella, Prune, SAND, All-Pairs, Weaver, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a major which adds several features and bug fixes. Among them:

• [Catalog]   Automatic fallback to a backup catalog server. (Tim Shaffer)
• [Makeflow]  Accept DAGs in JSON format. (Tim Shaffer)
• [Makeflow]  Multiple documentation omission bugs. (Nick Hazekamp and Haiyan Meng)
• [Makeflow]  Send information to catalog server. (Kyle Sweeney)
• [Makeflow]  Syntax directives (e.g. .SIZE for to indicate file size). (Nick Hazekamp)
• [Parrot] Fix cvmfs logging redirection. (Jakob Blomer)
• [Parrot] Multiple bug-fixes. (Tim Shaffer, Patrick Donnelly, Douglas Thain)
• [Parrot] Timewarp mode for reproducible runs. (Douglas Thain)
• [Parrot] Use new libcvmfs interfaces if available. (Jakob Blomer)
• [Prune]     Use SQLite as backend. (Peter Ivie)
• [Resource Monitor] Record the time where a resource peak occurs. (Ben Tovar)
• [Resource Monitor] Report the peak number of cores used. (Ben Tovar)
• [Work Queue] Add a transactions log. (Ben Tovar)
• [Work Queue] Automatic resource labeling and monitoring. (Ben Tovar)
• [Work Queue] Better capacity worker autoregulation. (Ben Tovar)
• [Work Queue] Creation of disk allocation per tasks. (Nate Herman-Kremer)
• [Work Queue] Extensive updates to wq_maker. (Nick Hazekamp)
• [Work Queue] Improvements in computing master's task capacity. (Nate Herman-Kremer).
• [Work Queue] Raspberry Pi compilation fixes. (Peter Bui)
• [Work Queue] Throttle work_queue_factory with --workers-per-cycle. (Ben Tovar)
• [Work Queue] Unlabeled tasks are assumed to consume 1 core, 512 MB RAM and 512 MB disk. (Ben Tovar)
• [Work Queue] Worker disconnects when node does not longer have the resources promised. (Ben Tovar)
• [Work Queue] work queue statistics clean up (see work_queue.h for deprecated names). (Ben Tovar)
• [Work Queue] work_queue_status respects terminal column settings. (Mathias Wolf)

We will have tutorials on the new features in our upcoming workshop, October 19 and 20. Refer to http://ccl.cse.nd.edu/workshop/2016 for more information. We hope you can join us!

Thanks goes to the contributors for many features, bug fixes, and tests:

• Jakob Blomer
• Peter Bui
• Patrick Donnelly
• Nathaniel Kremer-Herman
• Kenyi Hurtado-Anampa
• Peter Ivie
• Kevin Lannon
• Haiyan Meng
• Tim Shaffer
• Douglas Thain
• Ben Tovar
• Kyle Sweeney
• Mathias Wolf
• Anna Woodard
• Chao Zheng

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!