-
Notifications
You must be signed in to change notification settings - Fork 247
Home
In a glow program, datasets are linked together by computation steps. Each dataset can be partitioned into dataset shards. Each computation step can also be partitioned into computation tasks.
Glow make full use of Golang's channel feature.
In local mode, data are fed into computation tasks by input channel(s), and output also via a channel to a new dataset.
In distributed mode, a group of tasks will run together in a server, pulling its own input dataset shards. Its output are streamed to local disk, which will be pulled by downstream tasks. But these plumbing work are hidden from the computation since the tasks only read from input and write to output via common go channels.
Here are the components in distributed mode. Master, Agents, Driver, TaskGroup,
Master collects data from agents about resources.
Resources are CPU, memroy, allocated CPUs, allocated memory, etc.
Currently there is only a single master. But since master only has soft states from agents, we can easily extend it to a master cluster to avoid SPOF.
Agent wear several hats:
- reports system resources and usage to master.
- accept tasks from driver program.
- fetch the binary executable from driver program.
- accept reads and writes for a dataset.
Driver program is actually just the code that a developer will write. If executed with "-glow" option, it will drive the distributed execution.
- create optimized execution plan, group tasks into task groups.
- request resources from master.
- allocate tasks to assigned servers.
- if a dataset has input or output channels, write or receive from those datasets.
- clean up intermediate datasets generated during run time.
Tasks usually can be grouped together. A taskgroup program actually also uses the same binary executable file as the driver program, but in task mode.
- setup inputs and outputs for the tasks.
- execute the tasks.
One of the Golang's missing feature is the capability to move execution closure across the network, and not likely available any time soon. Given current situation, we can just move the whole binary code, but run in different modes, i.g., task mode and driver mode.
However, the implication is that the computation flow will be static. The flow graph can not be changed. One future way to allow dynamic flow is to pre-register all the flows, and dynamically choose one or several flows to run.
Check this page first. https://github.com/chrislusf/glow_examples/tree/master/word_count
Fork it, code it, and send pull requests. Better first discuss about the feature you want on the mailing list. https://groups.google.com/forum/#!forum/glow-user-discussion