Color-FaaS-2

Hanzhong

2023-10-27

Color FaaS Doc Final

All Videos: https://www.youtube.com/playlist?list=PLARUmtazqWjWrdYpJCU77CL7i0meZPqK9

Website & Bill Service: https://github.com/muchengl/tamu-sw-faas

Client: https://github.com/muchengl/Color-FaaS-Core

Function SDK: https://github.com/muchengl/Color-FaaS-SDK

Environment: https://drive.google.com/file/d/1UE5ZxRM66E4el5G-MV9gjSpM_WBBAFvZ/view?usp=drive_link

Catalog

[toc]

Introduction

In this project, we have undertaken a significant redesign of our existing system, transitioning to a microservices architecture and incorporating multi-modal client interfaces. This iteration represents an evolution of our system, focusing on enhanced scalability, modularity, and user experience.

Our submission for this iteration includes:

Requirements: Detailed descriptions and UI sketches of all use cases, incorporating standard format narratives and sequence diagrams. Video recordings demonstrating the system in action for each defined use case, highlighting functionality and user interaction.
Database Design: We redesigned our DB based on MongoDB. Comprehensive outlines of data entities, relationships, and an entity-relationship diagram, accompanied by sample data.
Architectural Design: In-depth descriptions of client and server components, communication protocols, data formats, and an overarching system architecture diagram.
Runnable System: A fully functional system with a personalized and polished UI.
Installation and Usage Manual: A detailed Handbook covering installation procedures, usage instructions, and information on necessary libraries and frameworks.

User cases and UI design

Items
Actor	User
Goal	Register a new account and login to the platform
Path	1.User visits the website landing page 2.System returns the html code of the page 3.User clicks “New User” 4.System display the form for registering a new user 5.User fill in the new user information and submit 6.System writes the new user information to the database. And jump back to the landing page 7.User enters the new user information and clicks Login 8.System checks whether the user exists. If yes, go to the home page

UI of this User case:

VIdeo: https://youtu.be/bLijflOhaDs?si=2M6rCZVapOltcmfz

User case : Development/Upload function

Items
Actor	Developer
Goal	A User want to deploy a application in cloud environment
Path	1. User develop a func in his/her computer by a SDK and upload this function from the webpage 2. System saves the function file (in HDFS), and the function file is successfully created 3. User views the corresponding information in the function list

UI of this User case

Video: https://youtu.be/zrBuOFd2jsk?si=BBQKRtWMwfU-F9Aj

User case : Test Function via web page

Items
Actor	User
Goal	Test the function from platform’s web page
Path	1. User enter the function test page 2. System return the web page to user 3. User test this function from web page. 4. System calls the function and returns the result and log

UI of this User case:

Video: https://youtu.be/nybjcanzMko?si=rx2toB0qJ9R3zZx8

User case : Test Functions via Public URL

Items
Actor	User
Goal	Test the function from platform’s web page
Path	1. User enter the function test page 2. System return the web page to user 3. User copy the “public url” and test it in Browser 4. System calls the function and returns the result and log as JSON. Show it in Browser.

Video: https://youtu.be/2_BJYx0JUBU?si=xXDeK_aF2hs8fj2u

User case : Test Functions in the Desktop Tool

There are two options to implement this Desktop tool, one is to write an interactive command line, and the other is to implement the program with UI. Since my project is a developer-oriented tool, I decided to implement an interactive command line program that more closely resembles a real-world product (like these tools in linux).

Items
Actor	Developer
Goal	Test functions in desktop tool
Path	1. User start the Desktop tool and login 2. System verifies that user information is correct 3. User enter list, history or test command to use different functions 4.System process the command and return the result 5.User see the output in screen.

UI of this User case:

Video: https://youtu.be/Uq23_F1yW-k?si=FnUfgjdBmk0kwb4i

User case : Manage tasks

Items
Actor	Developer
Goal	Trace the running status of a Func
Path	1. User enter Task manage page 2. System return the manage page 3. User look at the running status, output and time of the function

Video: https://youtu.be/dJ60XcTPhV0?si=MnkhJ3wmDE60XXmE

User case : Top up user’s account

Items
Actor	Developer
Goal	Top up
Path	1. User enter the account info page 2. System return the web page to user 3. User click “Recharge” button 4. System show the recharge form 5.User type info and click the button “Submit” 6.System add money to user’s account

UI of this User case:

Video: https://youtu.be/bL5l1LQHHrw?si=pWOBahXJLAHMEsHh

User case : Pay the bill

Items
Actor	Developer
Goal	Pay the bill
Path	1. User enter the account info page 2. System return the web page to user 3. User click “Update Bill” button 4. System queries all unpaid records and generates a bill. Platform will show the bill in web page 5. User click “Pay” button 6. System deducts the fee and completes the payment

UI of this User case:

Video: https://youtu.be/XO2TSqXJF0Y?si=Ozx1OBktopRhTn0i

Architectural design

High Level Design

This system comprises three microservices: Website Service, Billing Service, and Execution Client. The Website interact with the Execution Client and Billing Servers using the Http RestFul API protocol. The Website Service is responsible for user login, function upload, and function management. The Billing Server handles generating bills and processing user operations for recharge and payment.

The Execution Client is a multi-instance service, allowing for the deployment of numerous instances to achieve horizontal scaling and enhance system performance. The clients will register its information in ZooKeeper. The Website can then retrieve the list of clients from ZooKeeper and subsequently dispatch function calls to these clients.

Core cluster may contains many servers, so it can process many tasks in the same time. A Client program will run in In these servers. When a task is sent to a Client, it will stand up a Executor to process this task. The communitetion bewteen Client and Executor will been held by gPRC. Because, Function files are very big, so worker will use LRU to cache some function instances.

As for the architecture of this system：

Data Storage System

In Website service and Bill service, we use MongoDB to store user’s information. Website will provide user with functions include: add/update a Func, pay for the bill, trace Func status, debug function via webpage.

For function file storage, we use HDFS as the cloud storage system. User’s function files will be uploaded to HDFS and got by Executor.

For cluster management, we use ZooKeeper as the information register system. All clients’ info will be stored in ZooKeeper.

Data format between services

User ↔ Website

Request	Protocol	Data Type	Request Data	Response Data	URL
Login	POST	Form	Username Password	Redirect to the home page	/user
Register new user	POST	Form	Username Password	register success: “ok” fail to register: “fail”	/add/user
Upload a new function	POST	Form	FunctionName Description Function File	upload success: “ok” fail to upload: “fail”	/upload/func
Update function’s data	POST	Json	New FunctionName New Description	update success: “ok” fail to update: “fail”	/update/func
Get user’s function list	GET	url	User ID	A json list contains all user’s function	/funcs
Get user’s function access history	GET	url	User ID	A json list contains all user’s function access history	/user/tasks
Pay for the bill (Forward to the Bill service)	POST	Json	User ID Bill ID	pay success: “ok” fail to pay: “fail”	/pay/bill
Top up (Forward to the Bill service)	POST	Json	User ID Card Number Amount of money	success: “ok” fail: “fail”	/user/add/balance
Update Bill (Forward to the Bill service)	GET	Json	User ID	success: “ok” fail: “fail”	/update/bills
Get Bill List (Forward to the Bill service)	GET	Json	User ID	A json list contains all user’s bill history	/bills
Invoke a function	POST/GET	Json	Function ID Task Input	A json of output and logs	/func/invoke

Website ↔ Bill service

Request	Protocol	Data Type	Request Data	Response Data	URL
Pay for the bill	POST	Json	User ID Bill ID	pay success: “ok” fail to pay: “fail”	/pay/bill
Top up	POST	Json	User ID Card Number Amount of money	success: “ok” fail: “fail”	/user/add/balance
Update Bill	GET	Json	User ID	success: “ok” fail: “fail”	/update/bills
Get Bill List	GET	Json	User ID	A json list contains all user’s bill history	/bills

Website ↔ Client

Request	Protocol	Data Type	Request Data	Response Data	URL
Submit a task	POST	Json	TaskID FuncName FuncID FuncStorageType TaskFuncPath TaskRunningMode FuncType TaskCPUCore TaskMem TaskDiskSpace TaskMaxTime TaskInput	A JSON include: status, msg logs	/func/invoke

Database Design

This FaaS platform’s database(MongoDB) design encompasses multiple tables, intended to manage user information, bills, functions, function execution records, billing units, and account history details. Here’s a detailed explanation of these tables:

User Info Table:
- User_ID: A unique identifier for the user, of type Bigint.
- User_Email: The email address of the user.
- User_Name: The name of the user, which can be up to 20 characters.
- User_Password: The password of the user, capped at 20 characters.
- User_Balance: The balance in the user’s account, of type Bigint.
Bill Table:
- Bill_ID: A unique identifier for the bill.
- User_ID: A unique identifier for the user, serving as an index.
- Bill_Date: The date of the bill.
- Bill_Balance: The amount of the bill.
- Bill_tasks: A JSON document inclueds all unpaid function running records, supported by MongoDB
- Status: The status of the bill, where 0 stands for unpaid and 1 for paid.
Function Table:
- Func_ID: A unique identifier for the function.
- User_ID: A unique identifier for the user.
- Func_Path: The path to the function’s code in object storage services like S3.
- Func_Explanation: An explanation or description of the function.
- Func_Resource: Resources required by the function, in JSON format, e.g., {“CPU”:4,”MEM”:1024m,”DISK”:1024m}.
Func Running Record Table:
- ID: A unique identifier for the execution record.
- Func_ID: A unique identifier for the function.
- Running_Time: Execution time, e.g., 100ms.
- Status: Status of execution, where 0 stands for failure, 1 for success, and 2 for pending.
- Input: Input data in JSON format.
- Running_Date: The date of execution.
- Payment_Status: Payment status, where 0 stands for unpaid and 1 for paid.
Billing Unit Table:
- Price: The price, e.g., 0.01$/ms for functions.
Account Info History Table:
- ID: A unique identifier for the record.
- Date: The date.
- Type: The action type (add fund or payment).
- Account_num: Account number.
- Bill_ID: The ID of the bill.
- Money: The amount.
- User_ID: A unique identifier for the user.

- User Info Table:

Item	Type	Explanation/Sample Data
User_ID	String
User_Email	String
User_Name	String
User_Password	String
User_Balance	Bigint	1000.0$

- Bill Table

Item	Type	Explanation/Sample Data
Bill_ID	String
User_ID	String	Index
Bill_Date	String
Bill_Balance	Bigint
Bill_Tasks	Task	A JSON document inclueds all unpaid function running records, supported by MongoDB { {taskID:1,…}, {taskID:2,…} }
Status	int	0 unpay, 1 paid

- Function Table

Item	Type	Explanation/Sample Data
Func_ID	String
User_ID	String
Func_Args	String	{“num1”:int,”num2”,int}
Func_Path	String	path of func’s code in HDFS or S3
Func_Explanation	String

- Func Running Record

tem	Type	Explanation/Sample Data
Running_ID	String
Func_ID	String
Running_Time	String	Example: 100ms
Status	Int	0: fail, 1 success, 2 pending
Input	String	A input json
Running_Date	String
Payment_Status	Int	0 unpay, 1 paid

- Billing Unit

Item	Type	Explanation/Sample Data
ID	Bitint
Price	float	0.0001$ / ms

- Account Info History

Item	Type	Explanation/Sample Data
ID	String
Date	String
Type	Int	Payment or ADD
Account_num	String
Bill_ID	Int
Money	Bigint
User_ID	Bigint

- ER Graph

Handbook

Website & Bill Service Code: https://github.com/muchengl/tamu-sw-faas

Client Code: https://github.com/muchengl/Color-FaaS-Core

Docker Environment: https://drive.google.com/file/d/1UE5ZxRM66E4el5G-MV9gjSpM_WBBAFvZ/view?usp=drive_link

Environment

This service includes a Website, Bill service, and Client. Both the Website and Bill service rely on a Java environment (Java 1.8+), while the Client is developed using Golang and requires Golang 1.21. Website and Bill service use Maven to manager all dependent environment. Client can use Golang’s mod to get all dependents.

Start Website and Bill Service

Step 01: Download/Clone all code of project.

Step 02: Open the project with IDEA.

Step 03: Open pom.xml in project and init maven for both main-module and bill-module.

Step 04: Run project in IDEA. Website is in main module com.tamu.faas.FaasApplication. Bill is in bill module com.tamu.faas.bill.BillApplication

Step 05: com.tamu.faas.FaasApplication will use port 8080 and com.tamu.faas.bill.BillApplication will use 8081.

Start Client Cluster

Client uses makefile to manager the project, you can use the follows commands to init and build the project:

➜ cd [client project's dir in your computer]
➜ make mod
---> make mod <---
go mod tidy
go mod download

➜ make build
---> make build <---
./build.sh
  ---> make clean <---
  ---> init dir <---
  ---> init files <---
  ---> make server <---
  ---> make client <---
  ---> make executor <---

And than, use the follows commands to start the client:

1	➜ ./output/startup.sh --client

If you see the following output, the client has started successfully (only one client, if you want to start many clients to form a cluster, you need to run muit-clients in many Containers/VMs):

Start Zookeeper and HDFS

Start by docker-compose in project dir (May not work perfectly, may require manual installation) :

1 2	➜ cd [docker-compose's dir in your computer] ➜ docker-compose up -d

Test the project

Test the project by URL in your browser: http://localhost:8080/login

HDFS URL in browser: http://localhost:9870/

Test desktop tool:

1 2	➜ pip install prettytable requests ➜ python3 client.py

Color FaaS Doc Final

Introduction

User cases and UI design

**User case **: Register & Login

**User case **: Development/Upload function

**User case **: Test Function via web page

**User case **: Test Functions via Public URL

**User case **: Test Functions in the Desktop Tool

**User case **: Manage tasks

**User case **: Top up user’s account

**User case **: Pay the bill

Architectural design

High Level Design

Data Storage System

Data format between services

Database Design

- User Info Table:

- Bill Table

- Function Table

- Func Running Record

- Billing Unit

- Account Info History

- ER Graph

Handbook

Environment

Start Website and Bill Service

Start Client Cluster

Start Zookeeper and HDFS

Test the project

User case : Register & Login

User case : Development/Upload function

User case : Test Function via web page

User case : Test Functions via Public URL

User case : Test Functions in the Desktop Tool

User case : Manage tasks

User case : Top up user’s account

User case : Pay the bill