From a7df512f36a1414324d16649790ef35fc49edc1f Mon Sep 17 00:00:00 2001 From: arshiam2 <arshiam2@illinois.edu> Date: Sun, 2 Dec 2018 23:22:53 -0600 Subject: [PATCH] udpate --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index a229e94..b3d1899 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,13 @@ ## Prerequisites Have node and npm installed on each VM and clone the repo in each one. +## Streamming Architecture Explained +Through the cli tool, you can read the dataset as a stream, sending it row by row. This goes to the master, which is the spout that receives the data. It then sends it to multiple bolts, following the topology and deciding where to go based off the state machine. +Every machine is running an express server, and can serve as a bolt or a spout. It chooses what to do based off the response sent. To get the data, run get-stream, which will stream to you all the results from the last run and stream it to you through the cli tool. +To catch errors, it waits for a response, and after a given amount of time sends the request again. If the master dies, it goes to the backup master, which is the second lowest VM number, which already has all the state information saved. +To write your own applications, define a topology which defines the functions for the maps, filters, and reduces. Define a state machine or resolver at the bottom, which is exported. To run it, just run “put-stream <task_name>â€. If you want to do an aggregation, define a aggregation and it will be handled on the master. This is because the bolt’s don’t maintain state, but the master does and can handle the aggregations. + + ## Cli Tool The cli tool has a couple commands that allows it to interact with the introducers and machines to tell it to join, leave, or list that machine's membership lists. @@ -37,6 +44,9 @@ This links the ```mem``` command to your command line, so you can run all the co ```mem get-versions <sdfsfilename> <numversions> <localfilename>``` Writes the names of the latest numversions of sdfsfilename to a localfilename. +```mem put-stream <name>``` Streams data to system and executes the application +```mem get-stream ``` Streams data from sink + ## SDFS -- GitLab