Wednesday, April 25, 2018

VC3-Builder and WQ-MAKER at IC2E 2018

Ben Tovar presented the paper Automatic Dependency Management for Scientific Applications on Clusters and Nick Hazekamp presented the paper MAKER as a Service: Moving HPC applications to Jetstream Cloud at the IEEE International Conference on Cloud Engineering (IC2E 2018) on April 18, 2018 in Orlando, Florida.

In Automatic Dependency Management for Scientific workflows (paper slides) we introduce a tool for sofware environments deployments in clusters. This tool, called the vc3-builder, has minimal dependencies and a lightbootsrap, which allows it to be deployed along batch jobs. The vc3-builder then install any missing sofware using only user-priviliges (e.g., no sudo) so that the actual user payload can be executed. The vc3-builder is being developed as part of the DOE funded Virtual Clusters for Community Computation (VC3) project, in which users can construct custom short-lived virtual clusters across different computational sites.

In MAKER as a Service: Moving HPC applications to Jetstream Cloud (paper poster slides) we discussed the lessons learn in migrating MAKER, a traditional HPC application, to the cloud. This focused on issues like recreating the software stack using VC3-Bulder, addressing the lack of shared filesystems and inter-node communications with Work Queue, and building the application focused on user feedback allowing for informed decisions in the cloud. Using WQ-MAKER we were able to run MAKER not only on Jetstream, but also resources from Notre Dame's Condor cluster. Below you can see the systems architecture.

Monday, March 12, 2018

CCL at CyVerse Container Camp

Nick Hazekamp and Kyle Sweeney gave a talk, "Distributed Computing with Makeflow and Work Queue", at the CyVerse Container Camp workshop. This talk gives an overview of Makeflow and Work Queue, with an emphasis on how we approach using containers in a workflow. Highlights of the talk focused on different approaches to applying containers, either per-task or workflow-level, and what that looks like in practice on Jetstream, an Openstack based cloud platform.

You can find the slides and tutorial at: CCL at CyVerse Container Camp

Here you can see the active participants:

Monday, January 15, 2018

Submit Your CCL Highlight

We like to hear how you use our software!

If you have discovered a new result, published a paper, given a talk, or just done something cool, please take a few minutes to tell us about it on this simple form.

If we accept your submission, we will highlight your story on our website, include a mention in our annual report to the NSF, and send you some neat CCL stickers and swag as a little thank-you.

You can see what others have submitted on our community highlights page.

Monday, December 4, 2017

CCL on Chameleon Cloud with ACIC

As has been a tradition for several years, the CCL has had the opportunity to teach about the CCTools and distributed computing as part of Applied Cyberinfrastructure Concepts(ACIC) course at University of Arizona and taught by Dr. Nirav Merchant and Dr. Eric Lyons. Due to the number of features that have been added as of recently, this year primarily focused on Makeflow and how we use it on the Cloud and with containers. The topics we talk about were:
  • Thinking opportunistically
  • Overview of the Cooperative Computing Tools
  • Makeflow
  • Makeflow using Work Queue as a batch system
  • Makeflow using the Cloud as a batch system
  • Specifying and managing resources
  • Using containers in and on a Makeflow
  • Specifying Makeflow with JSON and JX
The major topic I wanted to focus on here is running Makeflow on the Cloud. For several months we have supported Makeflow submitting to Amazon EC2 directly, and there is an upcoming release that will incorporate support for the Amazon Batch system. For this class we also worked on deploying CCTools on Chameleon Cloud, which is a "configurable experimental environment for large-scale cloud research" as found Chameleon Cloud is a great test bed for researchers to deploy cloud instances and utilizes the OpenStack KVM interface.

TPDS Paper: Storage Management in Makeflow

As the scale of workflows and their data grow, it becomes increasingly difficult to execute within the provide storage. This issue is only exacerbated when attempting to run multiple workflows at the same time, often sharing the resources. Up until recently, the user would often make a guess at the total size of the workflow and execute until failure, having to remove older experiment and data to accommodate the needed room; creating a time consuming cycle of run->fail->clean until the required space is achieved.

The initial solution, which is effective in many cases, is to turn on garbage collection. Garbage collection in Makeflow track a created file from creation until it is no longer needed, at which point it is deleted. This initial solution works well to limit the active footprint of the workflow. However, the user is still left in a situation where they are not aware of the space needed to execute.

To resolve this added an algorithm that will estimate the size of the workflow, and what this minimum size needed to execute said workflow would be. This is done by determining the different paths of execution and finding the resulting minimum path(s) through the workflow. This is most accurately done by estimating and labeling the files in the Makeflow:
.SIZE test.dat 1024K

Using this information Makeflow can statically analyze the workflow and tell you the minimum and maximum storage needed to execute. This information can then be coupled with a run-time storage manager and garbage collection to stay within a user specified limit. Instead of actively trying to schedule in an order to prevent going over the limit, nodes are submitted when there is enough space to permit the it to run and have space for all of its children. This allows for the more concurrency if the space allows. Below is an image that shows how this limit can be used to at different levels.

This first image shows a bioinformatics workflow running using the minimum required space. We can see several(10) peaks in the workflow. Each of these correspond to a larger set of intermediate files that can be removed later. In the naive case where we don't track storage these can all occur at the same time using more storage than may be available.
In this second case, we set a limit on storage that is higher than the minimum. We can see similar spikes, run-time manager is only scheduling as many branches as can be run under the limit.
To use this feature, which is now released and available in the main branch of CCTools here are the steps:
  1. Label the files. A slight over-estimate will work as well as the exact number is not known ahead of time. The default size is 1G.
    .SIZE 5M
  2. Find the estimated size of the Makeflow.
    makeflow --storage-print storage-estimate.dat
  3. Run Makeflow setting a mode and limit. The limit can be anywhere between the min and the max. Type 1 indicates a min tracking which holds below the limit set.
    makeflow --storage-type 1 --storage-limit 10G

Monday, November 13, 2017

CCL at Supercomputing 2017

We are well represented at the annual Supercomputing conference this week:

Tim Shaffer is presenting "Taming Metadata Storms in Parallel Filesystems with MetaFS" at the Parallel Data Storage Workshop (PDSW).  This paper describes a technique for accelerating metadata-intensive program loading workloads that often cause trouble in parallel filesystems.

Kyle Sweeney is presenting "Lightweight Container Integration into Workflow Systems: A Case Study with Singularity and Makeflow" at Workflows in Support of Large Scale Science (WORKS).  This talk explains why integrating containers into workflows isn't as simple as it first appears, and describes a variety of approaches for assembling complete applications from containers and data.

Charles Zheng is presenting "Wharf: Sharing Docker Images across Hosts from a Distributed Filesystem" at the Monday poster session.  This work in progress aims to accelerate the use of containers on HPC clusters by sharing images and metadata within a parallel filesystem.

Thursday, October 26, 2017

TPDS Paper: Job Sizing

When submitting jobs for execution to a computing facility, a user must make a critical decision: how many resources (such as cores, memory and disk) should be requested for each job?
Broadly speaking, if the initial job size selected is too small, it is more likely that the job will fail and be returned, thus wasting resources on a failed run that must be retried. On the other hand, if the initial job size selected is too large, the job will succeed on the first try, but waste resources that go unused inside the job's allocation. If the waste is large enough, throughput will be reduced because those resources could have been used to run another job.
If the resources consumed by a collection of jobs were known and constant, then the solution would be easy: run one job at a large size, measure its consumption, and then use that smaller measured size for the remainder of the jobs. However, experience shows that real jobs have non-trivial distributions. For example, the figure shows the histogram of memory consumption for a set of jobs in a high energy physics workflow run on an HTCondor batch system at the University of Notre Dame.
Note that the histogram shows large peaks at approximately 900MB and 1300MB, but there are small number of outliers both above and below those values.
What memory size should we select for this workload? If we pick 3.8GB RAM for all jobs, then every job will succeed, but then most jobs would end up wasting several GB of memory that could be used to run other jobs. On the other hand, we could try a two-step approach, in which each job is run with a smaller value, wait to see which ones succeed or fail, and those that fail are run with the maximum 3.8GB memory allocation.
But precisely what smaller value should be used for the first attempt? The dotted line, at around 1.32GB, turns out to maximize the throughput when running the workflow under this two-step policy. Allowing for %8 of the tasks to be retried, throughput increases 2.54 times, and resources wasted decreased %44.
In our recent paper A Job Sizing Strategy for High-Throughput Scientific Workflows we fully describe the two-step strategy described above. These developments have also been integrated to makeflow and work queue in CCTools. For makeflow, the rules need to be labeled with the optimization mode:

.MAKEFLOW CATEGORY myfirstcategory

output_1: input_1
    cmdline input_1 -o output_1

output_2: input_2
    cmdline input_2 -o output_2

.MAKEFLOW CATEGORY myothercategory

output_3: input_3
    cmdline input_3 -o output_3

output_4: input_4
    cmdline input_4 -o output_4

Also, makeflow needs to run with the resource monitor enabled, as:
makeflow --monitor=my_resource_summaries_dir (... other options ...)
Rules in the same category will be optimized together.
Similarly, for work queue:

q = WorkQueue(...)

q.specify_category_mode('myfirstcategory', WORK_QUEUE_ALLOCATION_MODE_MAX_THROUGHPUT)

t = Task(...)

Additionally, we have made available a pure python implementation at:

Wednesday, October 18, 2017

Makeflow Feature: JX Representation

There are a number of neat new features in the latest versions of our software that I would like to highlight through some occasional blog posts.  If these sound interesting, please give them a try and send us your feedback.

First, I would like to highlight recent work by Tim Shaffer on JX, a new encoding for Makeflow that makes it easier to express complex workflows programmatically.  

For example, a traditional makeflow rule looks like this:

out.txt: in.txt calib.dat simulate.exe
    simulate.exe -i in.txt -p 10 > out.txt

In the latest version of Makeflow, you can write the same rule in JSON like this:

    "command" : "simulate.exe -i in.txt -p 10 > out.txt",
    "inputs" : [ "in.txt", "calib.dat", "simulate.exe" ],
    "outputs": [ "out.txt" ]

Now, just using JSON by itself doesn't give you a whole lot.  However, we extended JSON with a few new features like list comprehensions, variables substitutions, and operators.  This gives us a programmable way of generating a lot of rules easily.

For example, this represents 100 rules where the parameter varies from 0-99:

   "command" : format("simulate.exe -i in.txt -p %d > out.%d.txt",param,param),
   "inputs" : [ "in.txt", "calib.dat", "simulate.exe" ],
   "outputs": [ format("out.%d.txt",param) ]

} for param in range(100)

For a more detailed example, see these example BWA workflows expressed in three different ways:
Thanks to Andrew Litteken for converting and testing many of our example workflows into the new format.

Monday, October 9, 2017

Announcement: CCTools 6.2.0 released

The Cooperative Computing Lab is pleased to announce the release of version 6.2.0 of the Cooperative Computing Tools including Parrot, Chirp, Makeflow, WorkQueue, Umbrella, Prune, SAND, All-Pairs, Weaver, and other software.

The software may be downloaded here:

This is a major which adds several features and bug fixes. Among them:

  • [JX] A superset of JSON to dynamically describe workflows, see doc/jx.html. (Tim Shaffer)
  • [Makeflow] Support for Amazon EC2. (Kyle Sweeney, Douglas Thain)
  • [Makeflow]  Singularity support bug fixes. (Kyle Sweeney)
  • [Parrot] Fix CVMFS initialization. (Tim Shaffer)
  • [Prune] Several bug fixes. (Peter Ivie)
  • [ResourceMonitor] Measurement snapshots by observing log files, --snapshot-events. (Ben Tovar)
  • [WorkQueue] Compressed updates to the catalog server. (Nick Hazekamp, Douglas Thain)
  • [WorkQueue] work_queue_factory uses computed maximum worker capacity of the master. (Nate Kremer-Herman)
  • [WorkQueue] Several bug fixes. (Nick Hazekamp, Ben Tovar)
  • [WQ_Maker] Several bug fixes. (Nick Hazekamp)

Thanks goes to the contributors for many features, bug fixes, and tests:

  • Jakob Blomer
  • Nathaniel Kremer-Herman
  • Nicholas Hazekamp
  • Peter Ivie
  • Tim Shaffer
  • Douglas Thain
  • Ben Tovar
  • Kyle Sweeney
  • Chao Zheng

Please send any feedback to the CCTools discussion mailing list:


Wednesday, August 30, 2017

2017 DISC Summer REU Conclusion

This summer, we hosted 9 outstanding undergraduate students in our summer REU program in Data Intensive Scientific Computing (DISC).  Our guests spent the summer working with faculty in labs across campus in fields such as astronomy, high energy physics, bioinformatics, data visualization, and distributed systems.  And, they enjoyed some summer fun around South Bend.

Check out these short YouTube clips that explain each research project:

And here they are presenting at our summer research symposium: 

If you would like to participate, please apply for the 2018 edition of the DISC REU program at Notre Dame.