CCTools 7.1.0 released

The Cooperative Computing Lab is pleased to announce the release of version 7.1.0 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a minor release with some new features and bug fixes. Among them:

• [General]   Documentation available in https://cctools.readthedocs.io (Ben Tovar, Douglas Thain)
• [General]   Installation via conda: conda install -c conda-forge ndcctools (Ben Tovar)
• [General]   Installation via spack: spack install cctools (Tanner Juedeman)
• [General]   Several fixes for the batch job interface -T. (Ben Tovar)
• [JX]        New template("{VAR} ...") function to construct strings. (Tim Shaffer)
• [JX]        Improved parsing. (Tim Shaffer, Douglas Thain)
• [Makeflow]  Support for sub-makeflows. (Douglas Thain)
• [Makeflow]  Wrappers and hooks facility clean-up. (Nicholas Hazekamp, Tim Shaffer)
• [WorkQueue] Fix status connections being count as workers. (Ben Tovar)
• [Resource Monitor] Measurements of single python functions. (Ben Tovar)

Thanks goes to the contributors for many features, bug fixes, and tests:

Cami Carballo
T.J. Dasso
Nathaniel Kremer-Herman
Nicholas Hazekamp
Tanner Juedeman
Tim Shaffer
Emily Strout
Zoe Surma
Douglas Thain
Ben Tovar
Yifan Yu

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Announcement: CCTools version 7.0.22 released

The Cooperative Computing Lab is pleased to announce the release of version 7.0.22 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a minor release with some new features and bug fixes. Among them:

• [WorkQueue] Worker warning when files cannot be executed in the scratch directory. (Ben Tovar)
• [WorkQueue] Status connections are no longer counted as available workers. (Ben Tovar)
• [Makeflow] Fix issue with using quotes ("...",'...') when specifying files. (Ben Tovar)
• [Makeflow] Fix makeflow_monitor race condition on partially written log lines. (Ben Tovar)

Thanks goes to the contributors for many features, bug fixes, and tests:

Camila Carballo
T.J. Dasso
Nathaniel Kremer-Herman
Nicholas Hazekamp
Tanner Juedeman
Ryker McIntyre
Tim Shaffer
Francis Schickel
Zoe Surma
Douglas Thain
Ben Tovar
Yifan Yu

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Ph.D. Defense: Nick Hazekamp

Nick Hazekamp successfully defended his Ph.D. dissertation titled "Methods Enabling Portability of Scientific Workflows".  He has recently started a job at Atomic Object in Muskegon, Michigan.  Congratulations, Dr. Hazekamp!

Announcement: CCTools version 7.0.21 released

The Cooperative Computing Lab is pleased to announce the release of version 7.0.21 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a minor release which adds some bug fixes. Among them:

• [Makeflow] Fix segfault when using --submit-safe-mode. (Ben Tovar)
• [WorkQueue] Fix memory leak error when accumulating capacity statistics. (Ben Tovar)
• [WorkQueue] Do not count the factory as an available worker. (Ben Tovar)

Thanks goes to the contributors for many features, bug fixes, and tests:


T.J. Dasso
Nathaniel Kremer-Herman
Nicholas Hazekamp
Tanner Juedeman 
Ryker McIntyre
Tim Shaffer
Zoe Surma
Douglas Thain
Ben Tovar
Yifan Yu


Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

ACIC Tutorial on Makeflow and Work Queue

We enjoyed giving a tutorial on Building Scalable Applications with Makeflow and Work Queue for the Advanced Cyberinfrastructure taught by Nirav Merchant at the University of Arizona.  The midterm exam is to scale up a bioinformatics website using Makeflow and Work Queue! 

ReadTheDocs Preview

We are migration our documentation to the online ReadTheDocs service,
which offers an improved style and better navigation.  Check out the
preview here: cctools.readthedocs.io

Announcement: CCTools 7.0.17 released

The Cooperative Computing Lab is pleased to announce the release of version 7.0.17 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, SAND, All-Pairs, Weaver, and other software.

The software may be downloaded here.

This is a minor release which adds some bug fixes. Among them:

• [General] Fix for linking issue when using a conda environment. (Tim Shaffer)
• [General] Fix for compilation issue with python3 and old versions of swig. (Ben Tovar)
• [Chirp] Fix for the perl test. (Zoe Surma)
• [Makeflow] Fix typos in documentation. (Ryker Campbell)
• [Parrot] Fix a PATH issue with local execution. (Ben Tovar)
• [Parrot] LOCAL keyword in a mountlist acts now like --disable-service. (Tim Shaffer)

Thanks goes to the contributors for many features, bug fixes, and tests:

• Ryker Campbell
• Nathaniel Kremer-Herman
• Nicholas Hazekamp
• Tim Shaffer
• Zoe Surma
• Douglas Thain
• Ben Tovar
• Chao Zheng

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Ph.D. Defense: Chao "Charles" Zheng

Congratulations to Dr. Chao "Charles" Zheng, who defended his Ph.D. thesis on "The Challenges of Scaling Up High Throughput Workflows with Container Technology".  He will shortly be leaving for California to join Alibaba, Inc.  Here is Charles (second from right) after the successful defense:

Summer REU Projects

In summer 2019, REU students TJ Dasso and Eamon Marmion worked at the CCL on large scale distributed applications.  They completed the integration between the Parsl workflow language and the Work Queue execution framework, demonstrating Python applications running on thousands of cores, and also streamlined our software installation process via Pip and Conda.  They presented their work a the summer REU poster symposium in Jordan Hall:

PhD Proposal: Tim Shaffer

Congrads to Tim Shaffer on passing the PhD candidacy proposal stage:
"Proactive Storage Management for High Throughput Scientific Workloads"