Lobster Talk at Condor Week 2015

Ben Tovar gave an overview of Lobster in the talk High-Energy Physics workloads on 10k non-dedicated opportunistic cores with Lobster. The talk was part of Condor Week 2015, at the University of Wisconsin-Madison.

Lobster is a system for deploying data intensive high-throughput science applications on non-dedicated resources. It is build on top Work Queue, Parrot, and Chirp, which are part of CCTools.

Parrot and Lobster at CHEP 2015

CCL students gave two poster presentations at the annual Computing in High Energy Physics (CHEP) conference in Japan.  Both represent our close collaboration with the CMS HEP group at Notre Dame:


Haiyan Meng presented A Case Study in Preserving a High Energy Physics Application.  This poster describes the complexity of preserving a non-trivial application, the shows how Parrot packaging technology can be used to capture a program's
dependencies, and then re-execute it using a variety of technologies.





Anna Woodard and Matthias Wolf won the best poster presentation award for Exploiting Volatile Opportunistic Computing Resources with Lobster, which was rewarded with a lightning plenary talk.  Lobster is an analysis workload management system which has been able to harness 10-20K opportunistic cores at a time for large workloads at Notre Dame, making the facility comparable in size to the dedicated Tier-2 facilities of the WLCG!

Peter Sempolinski Defends Ph.D.

Dr. Peter Sempolinski successfully defended his PhD thesis titled "An Extensible System for Facilitating Collaboration for Structural Engineering Applications"

While at Notre Dame, Peter created a Virtual Wind Tunnel which enabled the crowdsourcing of structural design and evaluation by combining online building design with Google Sketchup and CFD simulation with OpenFoam.  The system was used in a variety of contexts, ranging from virtual engineering classes to managing work crowdsourced via Mechanical Turk. his work was recently accepted for publication in IEEE CiSE and PLOS1.

Congratulations to Dr. Sempolinski!

CMS Analysis on 10K Cores with Lobster

The CMS physics group at Notre Dame has created Lobster, a data analysis system that runs on O(10K) cores to process data produced by the CMS experiment at the LHC.  Lobster uses Work Queue to dispatch tasks to thousands of machines, Parrot with CVMFS to deliver the complex software stack from CERN, XRootD to deliver the LHC data, and Chirp and Hadoop to manage the output data. By using these technologies, Lobster is able to harness arbitrary machines and bring along the CMS computing environment wherever it goes.   At peak, Lobster at ND delivers capacity equal to that of a dedicated CMS Tier-2 facility!     (read more here)