CMS Analysis on 10K Cores Using Lobster

We have been working closely with the CMS physics group at Notre Dame for the last year to build Lobster, a data analysis system that runs on O(10K) cores to process data produced by the CMS experiment at the LHC.  At peak, Lobster at ND delivers capacity equal to that of a dedicated CMS Tier-2 facility!

Existing data analysis systems for CMS generally require that the user be running in a cluster that has been set up just so for the purpose: exactly the right operating system, certain software installed, various user identities present, and so on. This is fine for the various clusters dedicated to the CMS experiment, but it leaves unused the enormous amount of computing power that can be found at university computing centers (like the ND CRC), national computing resources (like XSEDE or the Open Science Grid), and public cloud systems.

Lobster is designed to harness clusters that are not dedicated to CMS.  This requires solving two problems:

1. The required software and data are not available on every node.  Instead, Lobster must bring them in at runtime and create the necessary execution system on the fly.
2. A given machine may only be available for a short interval of time before it is taken away and assigned to another user, so Lobster must be efficient at getting things set up, and handy at dealing with disconnections and failures.

To do this, we build upon a variety of technologies for distributed computing.  Lobster uses Work Queue to dispatch tasks to thousands of machines, Parrot with CVMFS to deliver the complex software stack from CERN, XRootD to deliver the LHC data, and Chirp and Hadoop to manage the output data.

Lobster runs effectively on O(10K) cores so far, depending on the CPU/IO ratio of the jobs.  These two graphs show the behavior of a production run on top of HTCondor at Notre Dame hitting up to 10K cores over the course of a 48-hour run.  The top graph shows the number of tasks running simultaneously, while the bottom shows the number of tasks completed or failed in each 10-minute interval.  Note that about two thirds of the way through, there is a big hiccup, due to an external network outages.  Lobster accepts the failures and keeps on going.

Lobster has been a team effort between Physics, Computer Science, and the Center for Research Computing: Anna Woodard and Matthias Wolf have taken the lead in developing the core software; Ben Tovar, Patrick Donnelly, and Peter Ivie have improved and debugged Work Queue, Parrot, and Chirp along the way; Charles Mueller, Nil Valls, Kenyi Anampa, and Paul Brenner have all worked to deploy the system at scale in production; Kevin Lannon, Michael Hildreth, and Douglas Thain provide the project leadership.

Anna Woodard, Matthias Wolf, Charles Nicholas Mueller, Ben Tovar, Patrick Donnelly, Kenyi Hurtado Anampa, Paul Brenner, Kevin Lannon, and Michael Hildreth, Exploiting Volatile Opportunistic Computing Resources with Lobster, Computing in High Energy Physics, January, 2015.

Anna Woodard, Matthias Wolf, Charles Mueller, Nil Valls, Ben Tovar, Patrick Donnelly, Peter Ivie, Kenyi Hurtado Anampa, Paul Brenner, Douglas Thain, Kevin Lannon and Michael Hildreth, Scaling Data Intensive Physics Applications to 10k Cores on Non-Dedicated Clusters with Lobster, IEEE Conference on Cluster Computing, September, 2015.

Haipeng Cai Defends Ph.D.

Haipeng Cai successfully defended his dissertation, "Cost-effective Dependence Analyses for Reliable Software Evolution", which studied methods for efficiently determining the scope of complex software system that is affected by a given change.

Haipeng will be taking a postdoctoral research position at Virginia Tech under the supervision of Prof. Barbara Ryder.

Congratulations to Dr. Haipeng Cai!

CCTools 5.1.0 released

The Cooperative Computing Lab is pleased to announce the release of version 5.1.0 of the Cooperative Computing Tools including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, Weaver, and other software.

The software may be downloaded here:
download

This minor release adds a couple of small features, and fixes the following
issues of version 5.0.0:

• [Prune]     Fix installation issue. (Haiyan Meng)
• [Umbrella]  Fix installation issue. (Haiyan Meng)
• [WorkQueue] Worker's --wall-time to specify maximum period of time a worker may be active. (Andrey Tovchigrechko, Ben Tovar)
• [WorkQueue] work_queue_status's --M to show the status of masters by name. (Names may be regular expressions). (Ben Tovar)
• [WorkQueue] Fix missing priority python binding.
• [WorkQueue] Fix incorrect reset of workers when connecting to different masters. (Ben Tovar)
• [WorkQueue] Fix segmentation fault when cloning tasks. (Ben Tovar)
• [WQ_Maker]  Cleanup, and small fixes. (Nick Hazekamp)

Thanks goes to our contributors:

Nicholas Hazekamp
Haiyan Meng
Ben Tovar
Andrey Tovchigrechko

Please send any feedback to the CCTools discussion mailing list:

mailing list

Enjoy!

~

CCTools 5.0.0 released

The Cooperative Computing Lab is pleased to announce the release of version 5.0.0 of the Cooperative Computing Tools including Parrot, Chirp, Makeflow, WorkQueue, SAND, All-Pairs, Weaver, and other software.
The software may be downloaded here: CCTools download
This is a major release that incorporates the preview of three new tools:

• [Confuga] An active storage cluster file system built on top of Chirp. It is used as a collaborative distributed file system and as a platform for execution of scientific workflows with full data locality for all job dependencies. (Patrick Donnelly)
• [Umbrella] A tool for specifying and materializing comprehensive execution environments. Once a task is specified, Umbrella determines the minimum mechanism necessary to run it such as, direct execution, a system container, a local virtual machine, or submission to a cloud or grid environment. (Haiyan Meng).
• [Prune] A system for executing and precisely preserving scientific workflows. Collaborators can verify research results and easily extend them at a granularity determined by the user. (Peter Ivie)

This release adds several features and several bug fixes. Among them:

• [AllPairs] Support for symmetric matrices. (Haiyan Meng)
• [Chirp] Perl and python bindings. (Ben Tovar)
• [Chirp] Improvements to the job interface. (Patrick Donnelly)
• [Makeflow] Improved Graphviz's dot output. (Nate Kremer-Herman)
• [Makeflow] Support for command wrappers. (Douglas Thain)
• [Parrot] Several bug fixes for CVMFS-based applications. (Jakob Blomer, Patrick Donnelly)
• [Parrot] Valgrind support. (Patrick Donnelly)
• [Resource Monitor] Library for polling resources. (Ben Tovar)
• [WorkQueue] Signal handling bug fixes. (Andrey Tovchigrechko)
• [WorkQueue] Log visualizer. (Ryan Boccabella)
• [WorkQueue] work_queue_worker support for Docker. (Charles Zheng)
• [WorkQueue] Improvements to perl bindings. (Ben Tovar)
• [WorkQueue] Support to blacklist workers. (Nick Hazekamp)

Incompatibility warnings: Workers from 5.0 do not work with masters pre 5.0.
Thanks goes to the contributors for many features and bug fixes: Matthew Astley, Jakob Blomer, Ryan Boccabella, Peter Bui, Patrick Donnelly, Nathaniel Kremer-Herman, Victor Hawley, Nicholas Hazekamp, Peter Ivie, Kangkang Li, Haiyan Meng, Douglas Thain, Ben Tovar, Andrey Tovchigrechko, and Charles Zheng.
Please send any feedback to the CCTools discussion mailing list: mailing list
Enjoy!

Preservation Framework for Computational Reproducibility at ICCS 2015

Haiyan Meng presented our work on Preservation Framework for Computational Reproducibility at the International Conference on Computational Science (ICCS) in Reykjavik, Iceland. This is a collaborative work between University of Notre Dame and University of Chicago for the DASPOS project both of these two universities are working on.

The preservation framework proposed in this paper includes three parts: 

• First, how to use light-weight application-level virtualization techniques to create a reduced package which only includes all the necessary dependencies; 
• Second, how to organize the data storage archive to preserve these packages; 
• Third, how to distribute applications through standard software delivery mechanisms like Docker and deploy applications through flexible deployment mechanisms such as Parrot, PTU, Docker, and chroot.

Haiyan Meng, Rupa Kommineni, Quan Pham, Robert Gardner, Tanu Malik and Douglas Thain, An Invariant Framework for Conducting Reproducible Computational Science, Journal of Computational Science 9 (2015): 137-142. DOI: 10.1016/j.jocs.2015.04.012

 

Umbrella and Containers at VTDC 2015

Two CCL students presented their latest work at the Virtualization Technologies in Distributed Computing (VTDC) at the Symposium on High Performance Distributed Computing (HPDC) in Portland, Oregon.

Haiyan Meng presented her work on Umbrella, a system for specifying and materializing execution environments in a portable and reproducible way.  Umbrella accepts a declarative specification for an application, and then determines the minimum technology needed to deploy it.   The application will be run natively if the local execution environment is compatible, but if not, Umbrella will deploy a container, a virtual machine, or make use of a public cloud if necessary.

Haiyan Meng and Douglas Thain, Umbrella: A Portable Environment Creator for Reproducible Computing on Clusters, Clouds, and Grids, Workshop on Virtualization Technologies in Distributed Computing (VTDC) at HPDC, June, 2015. DOI: 10.1145/2755979.2755982

Charles Zheng presented his work on integrating Docker containers into the Makeflow workflow engine and the Work Queue runtime system, each with different tradeoffs in performance and isolation.  These capabilities will be included in the upcoming 5.0 release of CCTools.

Charles Zheng and Douglas Thain, Integrating Containers into Workflows: A Case Study Using Makeflow, Work Queue, and Docker, Workshop on Virtualization Technologies in Distributed Computing (VTDC), June, 2015. DOI: 10.1145/2755979.2755984

Lobster Talk at Condor Week 2015

Ben Tovar gave an overview of Lobster in the talk High-Energy Physics workloads on 10k non-dedicated opportunistic cores with Lobster. The talk was part of Condor Week 2015, at the University of Wisconsin-Madison.

Lobster is a system for deploying data intensive high-throughput science applications on non-dedicated resources. It is build on top Work Queue, Parrot, and Chirp, which are part of CCTools.

Parrot and Lobster at CHEP 2015

CCL students gave two poster presentations at the annual Computing in High Energy Physics (CHEP) conference in Japan.  Both represent our close collaboration with the CMS HEP group at Notre Dame:


Haiyan Meng presented A Case Study in Preserving a High Energy Physics Application.  This poster describes the complexity of preserving a non-trivial application, the shows how Parrot packaging technology can be used to capture a program's
dependencies, and then re-execute it using a variety of technologies.





Anna Woodard and Matthias Wolf won the best poster presentation award for Exploiting Volatile Opportunistic Computing Resources with Lobster, which was rewarded with a lightning plenary talk.  Lobster is an analysis workload management system which has been able to harness 10-20K opportunistic cores at a time for large workloads at Notre Dame, making the facility comparable in size to the dedicated Tier-2 facilities of the WLCG!

Peter Sempolinski Defends Ph.D.

Dr. Peter Sempolinski successfully defended his PhD thesis titled "An Extensible System for Facilitating Collaboration for Structural Engineering Applications"

While at Notre Dame, Peter created a Virtual Wind Tunnel which enabled the crowdsourcing of structural design and evaluation by combining online building design with Google Sketchup and CFD simulation with OpenFoam.  The system was used in a variety of contexts, ranging from virtual engineering classes to managing work crowdsourced via Mechanical Turk. his work was recently accepted for publication in IEEE CiSE and PLOS1.

Congratulations to Dr. Sempolinski!

CMS Analysis on 10K Cores with Lobster

The CMS physics group at Notre Dame has created Lobster, a data analysis system that runs on O(10K) cores to process data produced by the CMS experiment at the LHC.  Lobster uses Work Queue to dispatch tasks to thousands of machines, Parrot with CVMFS to deliver the complex software stack from CERN, XRootD to deliver the LHC data, and Chirp and Hadoop to manage the output data. By using these technologies, Lobster is able to harness arbitrary machines and bring along the CMS computing environment wherever it goes.   At peak, Lobster at ND delivers capacity equal to that of a dedicated CMS Tier-2 facility!     (read more here)