Monday, June 15, 2020

CCTools version 7.1.6 released

The Cooperative Computing Lab is pleased to announce the release of version 7.1.6 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a bug release with some new features and bug fixes. Among them:

  • [Resource Monitor] Fix race condition that caused an overhead for very short running processes. (Ben Tovar)
  • [WorkQueue] Efficient recursive transfer of directories. (Douglas Thain)
  • [WorkQueue] Several work_queue_graph_log bug fixes. (Cami Carballo)

Thanks goes to the contributors for many features, bug fixes, and tests:

  • Cami Carballo
  • Nathaniel Kremer-Herman
  • Tim Shaffer
  • Douglas Thain
  • Ben Tovar

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Thursday, May 14, 2020

REU Project: Coffea + Work Queue

This spring, undergraduate researchers Zoe Surma and Emily Strout worked to integrate the Coffea data analysis framework for high energy physics with the Work Queue distributed execution framework from the CCL.  Zoe presented initial results from testing at the Coffea user's meeting last week.  We are seeing some good scaling behavior, but there is still some tuning needed to go to large scale.  Many thanks to Lindsay Grey and the Coffea team for help and support.  The next step is to get this up and running on the CMS HTCondor cluster at Notre Dame!


Monday, May 4, 2020

CCTools version 7.1.5 released

The Cooperative Computing Lab is pleased to announce the release of version 7.1.5 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a minor release with some new features and bug fixes. Among them:

  • [General]   Support scripts to execute tasks inside python environments. (Tim Shaffer, T.J. Dasso)
  • [General]   Fix problem with python 2.7 bindings and unicode support. (Ben Tovar)
  • [Parrot]    Kill all processes when initial ptrace attachment fails. (Ben Tovar)
  • [WorkQueue] Adds work_queue_factory python interface. (Tim Shaffer)

Thanks goes to the contributors for many features, bug fixes, and tests:

  • Cami Carballo
  • T.J. Dasso
  • Nathaniel Kremer-Herman
  • Tim Shaffer
  • Emily Strout
  • Zoe Surma
  • Douglas Thain
  • Ben Tovar
  • Yifan Yu

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Tuesday, April 7, 2020

CCTools version 7.1.2 released

The Cooperative Computing Lab is pleased to announce the release of version 7.1.2 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a bug fix release:

  • [Batch interface] Handle new date format in HTCondor for Makeflow and Work Queue factory. (Greg Thain)

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Tuesday, March 17, 2020

CCTools 7.1.0 released

The Cooperative Computing Lab is pleased to announce the release of version 7.1.0 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a minor release with some new features and bug fixes. Among them:

  • [General]   Documentation available in https://cctools.readthedocs.io (Ben Tovar, Douglas Thain)
  • [General]   Installation via conda: conda install -c conda-forge ndcctools (Ben Tovar)
  • [General]   Installation via spack: spack install cctools (Tanner Juedeman)
  • [General]   Several fixes for the batch job interface -T. (Ben Tovar)
  • [JX]        New template("{VAR} ...") function to construct strings. (Tim Shaffer)
  • [JX]        Improved parsing. (Tim Shaffer, Douglas Thain)
  • [Makeflow]  Support for sub-makeflows. (Douglas Thain)
  • [Makeflow]  Wrappers and hooks facility clean-up. (Nicholas Hazekamp, Tim Shaffer)
  • [WorkQueue] Fix status connections being count as workers. (Ben Tovar)
  • [Resource Monitor] Measurements of single python functions. (Ben Tovar)

Thanks goes to the contributors for many features, bug fixes, and tests:

Cami Carballo
T.J. Dasso
Nathaniel Kremer-Herman
Nicholas Hazekamp
Tanner Juedeman
Tim Shaffer
Emily Strout
Zoe Surma
Douglas Thain
Ben Tovar
Yifan Yu

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Wednesday, January 29, 2020

Announcement: CCTools version 7.0.22 released

The Cooperative Computing Lab is pleased to announce the release of version 7.0.22 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a minor release with some new features and bug fixes. Among them:

  • [WorkQueue] Worker warning when files cannot be executed in the scratch directory. (Ben Tovar)
  • [WorkQueue] Status connections are no longer counted as available workers. (Ben Tovar)
  • [Makeflow] Fix issue with using quotes ("...",'...') when specifying files. (Ben Tovar)
  • [Makeflow] Fix makeflow_monitor race condition on partially written log lines. (Ben Tovar)


Thanks goes to the contributors for many features, bug fixes, and tests:

Camila Carballo
T.J. Dasso
Nathaniel Kremer-Herman
Nicholas Hazekamp
Tanner Juedeman
Ryker McIntyre
Tim Shaffer
Francis Schickel
Zoe Surma
Douglas Thain
Ben Tovar
Yifan Yu

Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Thursday, November 7, 2019

Ph.D. Defense: Nick Hazekamp


Nick Hazekamp successfully defended his Ph.D. dissertation titled "Methods Enabling Portability of Scientific Workflows".  He has recently started a job at Atomic Object in Muskegon, Michigan.  Congratulations, Dr. Hazekamp!



Monday, November 4, 2019

Announcement: CCTools version 7.0.21 released

The Cooperative Computing Lab is pleased to announce the release of version 7.0.21 of the Cooperative Computing Tools including Parrot, Chirp, JX, Makeflow, WorkQueue, and other software.

The software may be downloaded here:
http://ccl.cse.nd.edu/software/download

This is a minor release which adds some bug fixes. Among them:

  • [Makeflow] Fix segfault when using --submit-safe-mode. (Ben Tovar)
  • [WorkQueue] Fix memory leak error when accumulating capacity statistics. (Ben Tovar)
  • [WorkQueue] Do not count the factory as an available worker. (Ben Tovar)

Thanks goes to the contributors for many features, bug fixes, and tests:


T.J. Dasso
Nathaniel Kremer-Herman
Nicholas Hazekamp
Tanner Juedeman 
Ryker McIntyre
Tim Shaffer
Zoe Surma
Douglas Thain
Ben Tovar
Yifan Yu


Please send any feedback to the CCTools discussion mailing list:

http://ccl.cse.nd.edu/community/forum

Enjoy!

Friday, October 25, 2019

ACIC Tutorial on Makeflow and Work Queue

We enjoyed giving a tutorial on Building Scalable Applications with Makeflow and Work Queue for the Advanced Cyberinfrastructure taught by Nirav Merchant at the University of Arizona.  The midterm exam is to scale up a bioinformatics website using Makeflow and Work Queue! 

ReadTheDocs Preview

We are migration our documentation to the online ReadTheDocs service,
which offers an improved style and better navigation.  Check out the
preview here: cctools.readthedocs.io