Prof. Thain gave the afternoon keynote (Data Intensive Computing with TaskVine) at the GCASR Workshop in Chicago on April 24th. TaskVine is our latest workflow execution system, just released in March 2023.
The key idea of TaskVine is to exploit the local storage already embedded into clusters for use by complex workflows:
To accomplish this, each node runs a general purpose worker process that manages cached data under unique names, and runs each task within a private namespace:
You can write workflows directly to the TaskVine API, declaring files and tasks and connecting them together. Or, TaskVine can be used as an executor under existing workflow systems:
No comments:
Post a Comment