COSCO Implementation details
We present the implementation details of the simulator and framework, with the interface details. To implement the COSCO framework, we use an object-oriented programming model and realize it using the Python programming language. We now provide details of various Python classes which together consolidate to the complete framework.
The hosts are created/simulated using the Datacenter class which provides a list of Host objects to the simulator or framework. The workloads are generated by the Workload class. These are either time-series models of IPS, RAM, Disk, Bandwidth requirements in the case of simulation or are actual application programs in the case of physical experiments. Moreover, the Scheduler class allows taking scheduling decisions in the form of task selection getSelection() and placement getPlacement(). Task selection function selects the containers that need to be migrated to another host; task placement function returns the target host for each selected or new container.
| Folder/File | Utility |
|---|---|
| install.py | This is the automated installation script for the fog broker which installs the required python packages using pip. It sets up the required software packages including VirtualBox, Vagrant, Ansible, InfluxDB and WSL (in Windows systems). It also sets us secured SSH keys communication among fog agents. |
| main.py | This is the main entry point to the implementation which runs the simulation or execution of tasks in a physical environment. It uses pre-implemented schedulers, datacenter configurations, workload models and database modules to simulate/execute fog tasks. |
| grapher.py | This generates comparative graphs for the raw traces saved as Stats objects in the all_datasets directory and saves them in the results directory. |
| simulator/Simulator.py | This is the implementation of the Simulator class which allocates new workloads, determines if allocation/migration is possible, executes event driven simulation and destroys completed workload. |
| simulator/container/Container.py | This is the implementation of a simulated container object with IPS, RAM, Disk and Bandwidth time-series models. |
| simulator/container/*.py | Implementation of simulation environments with details of host characteristics. |
| simulator/host/Host.py | Implementation of simulated host objects with IPS, RAM, Disk, Bandwidth capacities and utilization values. |
| simulator/workload/*.py | This is the workload generation module which generated containers using traces of static/bitbrain applications. The number of workloads being generated at each interval can be set by the user. |
| framework/Framework.py | This is the implementation of the physical container management framework which monitors hosts and containers, determines if container placement is possible, checkpoints/migrates/restores containers and destroys completed containers. |
| framework/task/Task.py | This is the identifier object for a container executing in the framework with IPS, RAM, Disk and Bandwidth utilization values. |
| framework/node/Node.py | This is the identifier object for a host with IPS, RAM, Disk, Bandwidth capacities and utilization values. |
| framework/database/Database.py | This is the InfluxDB client which maintains time-stamped information of all hosts and containers. |
| framework/agent/* | These files enable interaction with the individual agents and contain communication and container management scripts based on Python DockerClient API. These files are transferred to the agents and agent.py runs as a flask service. |
| framework/server/* | These helper files enable interaction from the broker end. These include the REST Client which sends operation codes to each agent as decided by the Framework. |
| framework/config/* | These are the configuration files for the Vagrant/Ansible setups. |
| framework/datacenter/Datacenter.py | This is the implementation of Datacenter class which configures host machines in the physical testbed and obtains the IPS, RAM, Disk and Bandwidth capacities. |
| metrics/*.py | These store the IPS, RAM, Disk and Bandwidth consumption of tasks/containers at the end of each scheduling interval. |
| metrics/powermodels/*.py | Implementation of power models for different types of hosts (simulated or physical). |
| scheduler/*.py | Implementation of various schedulers compatible with both Simulator and Framework classes. |
| scheduler/BaGTI/*.py | Implementation of the training codes for the Back-propagation of gradients with respect to Input scheduler aka GOBI/GOBI*. |
| stats/Stats.py | Implementation of the stats object which monitors and logs information about workloads, hosts, containers and workloads for the complete execution (simulation or physical execution). It also runs the one-step simulation used by the GOBI* scheduler. |