CCTools Version 7.3.0 released

The Cooperative Computing Lab is pleased to announce the release of version 7.3.0 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be installed from here: http://ccl.cse.nd.edu/software/download

This is a minor release with some new features and bug fixes. Among them:

- [WorkQueue] PythonTask to directly execute python functions as WorkQueue tasks. (Barry Sly-Delgado)
- [WorkQueue] Fix max mode allocation to work as a high-water mark when dispatching tasks. (Ben Tovar)
- [WorkQueue] Reworked documentation in https://cctools.readthedocs.io. (Douglas Thain)
- [WorkQueue] API to show summary of workers connected. (David Simonetti)
- [WorkQueue] Adds --wall-time limit to workers. (Thanh Son Phung)
- [Resource Monitor] Time is now reported in seconds, rather than microseconds. (Ben Tovar)
- [JX] jx_repl tool for JX language exploration. (Jack Rundle)


Thanks goes to the contributors for many features, bug fixes, and tests:

- David Rundle
- Barry Sly-Delgado
- Thanh Son Phung
- Tim Shaffer
- David Simonetti
- Douglas Thain
- Ben Tovar

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Lightweight Function Paper at IPDPS

Tim Shaffer, a Ph.D student in the CCL, will be presenting a paper "Lightweight Function Monitors for Fine-Grained Management in Large Scale Python Applications" at the International Parallel and Distributed Processing Symposium (IPDPS) in May 2021.  This work is the result of a collaboration between our group and the Parsl workflow team at the University of Chicago, led by Kyle Chard.

Emerging large scale science applications may consist of a large number of dynamic tasks to be run across a large number of workers.  When written in a Python-oriented framework like Parsl or FuncX those tasks are not heavyweight Unix processes, but rather lightweight invocations of individual functions.  Running these functions at large scale presents two distinct challenges:

1 - The precise software dependencies needed by the function must be made available at each worker node.  These dependencies must be chosen accurately: too few, and the function won't work; too many, and the cost of distribution is too high.  We show a method for determining, distributing, and caching the exact dependencies needed at runtime, without user intervention.

2 - The right number of functions must be "packed" into large worker nodes that may have hundreds of cores and many GB of memory.  Too few, and the system is badly underutilized; too many, and performance will suffer or invocations will crash.  We show an automatic method for monitoring and predicting the resources consumed by categories of functions.  This results in resource allocation that is much more efficient than an unmanaged approach, and is very close to an "oracle" that predicts perfectly.

The techniques shown in this paper are integrated into the Parsl workflow system from U-Chicago, and the Work Queue distributed execution framework from Notre Dame, both of which are open source software supported by the NSF CSSI program.

Citation:

• Tim Shaffer, Zhuozhao Li, Ben Tovar, Yadu Babuji, TJ Dasso, Zoe Surma, Kyle Chard, Ian Foster, and Douglas Thain, Lightweight Function Monitors for Fine-Grained Management in Large Scale Python Applications, IEEE International Parallel & Distributed Processing Symposium, May, 2021. 

Ph.D. Defense - Nathaniel Kremer-Herman

Congratulations to Dr. Kremer-Herman, who successfully defended his Ph.D. dissertation "Log Discovery, Log Custody, and the Web Inspired Approach for Open Distributed Systems Troubleshooting".  His work created a system TLQ (Troubleshooting via Log Queries) that enables the structured query of distributed data logged by independent components including workflows, batch systems, and application file access.  Prof. Kremer-Herman recently began a faculty position at Hanover College in Indiana.  Congrads!

CCTools 7.2.0 released

The Cooperative Computing Lab is pleased to announce the release of version 7.2.0 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a minor release with some new features and bug fixes. Among them:

• [Batch] Improved gpu handling with HTcondor. (Douglas Thain)
• [WorkQueue] Resource usage report with work_queue_status per task and worker. (Thanh Son Phung)
• [WorkQueue] Improved gpu handling. (Douglas Thain, Tim Shaffer)
• [WorkQueue] Assign tasks to specific gpus with CUDA_VISIBLE_DEVICES. (Douglas Thain)
• [Makeflow] Several fixes for sge_submit_makeflow. (Ben Tovar)

Thanks goes to the contributors for many features, bug fixes, and tests:

• Ben Tovar
• Douglas Thain
• Nathaniel Kremer-Herman
• Thanh Son Phung
• Tim Shaffer

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

CCTools version 7.1.12 released

 The Cooperative Computing Lab is pleased to announce the release of version 7.1.12 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a bug fix release:

• [Batch interface] Adds sge_submit_workers to installed scripts directory. (Ben Tovar)
• [Batch interface] Adds LSF as a batch type. (Douglas Thain)

Thanks goes to the contributors for many features, bug fixes, and tests:

• Ben Tovar
• Cami Carballo
• Douglas Thain
• Nathaniel Kremer-Herman
• Thanh Son Phung
• Tim Shaffer

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

OpenTopography + EEMT + Makeflow

The OpenTopography service provides online access to geospatial data and computational tools in support of earth sciences.  The Effective Energy and Mass Transfer (EEMT) tool allows for computations of energy transfer in the Earth's critical zone, taking into account topography, vegetation, weather, and so forth.  To scale these computations up to large clusters, the CCL's Makeflow and Work Queue frameworks are employed to construct large scale parallel workflows at the touch of a button from the OpenTopography website. 

Source: Tyson Swetnam, University of Arizona

Analyzing Agriculture with Work Queue

The Field Scanalyzer at the University of Arizona is a massive robot that uses sensors, cameras, and GPS devices to collect vast quantities of agricultural data from crop fields.  In the background, distributed computing and deep learning techniques are used to understand and improve agricultural efficiencies in hot, dry, climates.  Processing all this data requires reliable computation on large clusters: the PhytoOracle software from the Lyons Lab at UA makes this possible, building on the Work Queue software from the Cooperative Computing Lab at Notre Dame.

- Source: Eric Lyons University of Arizona

 

Now Recruiting Students

Research Opportunities in the Cooperative Computing Lab

Join the CCL team and work on challenging problems in the realm of parallel and distributed systems! We work closely with collaborators in physics, molecular dynamics, machine learning, and other fields to build systems that scale to tens of thousands of cores on national infrastructure such as clusters, clouds, and grids. We publish open source software that is used around the world.

We currently have positions for undergraduate, M.S. and Ph.D students.  For more information, see our lab web page.  To apply, send a resume and brief email cover letter to Prof. Douglas Thain (dthain@nd.edu).

 

CCTools version 7.1.9 released

The Cooperative Computing Lab is pleased to announce the release of version 7.1.9 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a bug release with some new features and bug fixes. Among them:

• [Batch] Improve missing jobs detection on slurm, torque, sge, pbs. (Ben Tovar)
• [Batch] WALL_TIME as a resource for slurm. (Ben Tovar)
• [Makeflow] Several fixes for nested workflows. (Ben Tovar)
• [Makeflow] Warn on redefinition of resources. (Ben Tovar)
• [Resource Monitor] --measure-only flag when limits are specified. (Ben Tovar)
• [Work Queue] API to define minimum resources for a category. (Ben Tovar)

Thanks goes to the contributors for many features, bug fixes, and tests:

• Ben Tovar
• Cami Carballo
• Douglas Thain
• HDsky
• Nathaniel Kremer-Herman
• Stefano Mangiola
• Tanner Judeman
• Tim Shaffer

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Autoscaling HTC at CLUSTER 2020

Recent CCL graduate Charles Zheng, Ph.D., presented his paper "Autoscaling High Throughput Workloads on Container Orchestrators" at the CLUSTER 2020 conference in September 2020.

In this paper, we explore the problem of how many machines to acquire for a high-throughput workload of known size when running on a container orchestrator like Kubernetes.

Most approaches to autoscaling are designed to scaling up web servers, or other services that respond to some unknown external request.  Generally, the autoscaler looks at some metric such as CPU utilization, and scales resources up or down in order to achieve some target like 90% CPU utilization.

However, when running a high throughput workload of, say, one thousand simulation runs, the situation is different.  First off, high CPU utilization is the norm: the simulator is likely to peg the CPU at 100% utilization, and adding or removing nodes isn't going to affect simulation.  And second, the offered load is not a mystery: we are in control of the workload, so we have some idea of the total size of the workload, or at least the number of jobs currently in the queue.

To address this, Charles built a High Throughput Autoscaler (HTA) that interfaces the Makeflow workflow system with the Kubernetes container orchestrator:

To learn more, check out the paper and accompanying video:

Chao Zheng, Nathaniel Kremer-Herman, Tim Shaffer, and Douglas Thain, Autoscaling High Throughput Workloads on Container Orchestrators, IEEE Conference on Cluster Computing, pages 1-10, September, 2020.