From f0e652a621a7be1518d40272027547640b7bbb3b Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Thu, 10 May 2018 03:58:18 +0100 Subject: [PATCH 01/12] 03 and 04 --- docs/03_getting_started.rst | 2 + docs/04_installation.rst | 118 ++++++++++++++---------------------- 2 files changed, 46 insertions(+), 74 deletions(-) diff --git a/docs/03_getting_started.rst b/docs/03_getting_started.rst index 1b34d27bc7..b8f24e5e7e 100644 --- a/docs/03_getting_started.rst +++ b/docs/03_getting_started.rst @@ -475,6 +475,8 @@ Finally, you should now be able to see the list of the available databases from (3 rows) $ +|br| + Appendix: Building FogLAMP on CentOS ==================================== diff --git a/docs/04_installation.rst b/docs/04_installation.rst index 27ad7695df..d236af3053 100644 --- a/docs/04_installation.rst +++ b/docs/04_installation.rst @@ -268,18 +268,21 @@ You can install FogLAMP as a service following these simple steps: └─1817 python3 -m foglamp.services.south --port=46309 --address=127.0.0.1 --name=HTTP_SOUTH $ +|br| + Installing the Debian Package ============================= We have versions of FogLAMP available as Debian packages for you. Check the |Downloads page| to review which versions and platforms are available. + Obtaining and Installing the Debian Package ------------------------------------------- Check the |Downloads page| to find the package to install. -Once you have downloaded the package, install it using the ``dbpkg`` command. Note that you may need to install it as superuser (or by using the ``sudo`` command) and you need to install the pre-requisites first. +Once you have downloaded the package, install it using the ``apt-get`` command. You can use ``apt-get`` to install a local Debian package and automatically retrieve all the necessary packages that are defined as pre-requisites for FogLAMP. Note that you may need to install the package as superuser (or by using the ``sudo`` command) and move the package to the apt cache directory first (``/var/cache/apt/archives``). For example, if you are installing FogLAMP on an Intel x86/64 machine, you can type this command to download the package: @@ -297,61 +300,26 @@ For example, if you are installing FogLAMP on an Intel x86/64 machine, you can t 2018-04-24 18:22:10 (521 KB/s) - ‘foglamp-1.2-x86_64.deb’ saved [496094/496094] $ -Make sure the system has all the pre-requisites needed to run FogLAMP: - -- autoconf -- jq -- libboost-dev -- libboost-system-dev -- libboost-thread-dev -- libpq-dev -- libtool -- postgresql -- postgresql-server-dev-9.6 -- python-setuptools -- python3-pip -- sqlite3 +We recommend to execute an *update-upgrade-update* of the system first, then you may copy the FogLAMP package in the *apt cache* directory and install it. + .. code-block:: console $ sudo apt update Hit:1 http://gb.archive.ubuntu.com/ubuntu xenial InRelease ... - $ - $ sudo apt -y install postgresql postgresql-server-dev-9.5 - ... - $ sudo apt -y install libpq-dev autoconf - ... - $ sudo apt -y install libtool jq - ... - $ sudo apt -y install libboost-dev libboost-system-dev libboost-thread-dev - ... - $ sudo apt -y install sqlite3 + $ sudo apt upgrade ... - $ sudo apt -y install python-setuptools - ... - $ sudo apt install python3-pip + $ sudo apt update ... - $ - -If you are installing FogLAMP on an ARM platform, you may also need: - -.. code-block:: console - - $ sudo apt -y install postgresql postgresql-server-dev-9.6 + $ sudo cp foglamp-1.2-x86_64.deb /var/cache/apt/archives/. ... - $ sudo apt install python3.5-dev python3-dbus python3-setuptools - ... - $ - -Now you are ready to install the package: - -.. code-block:: console - - $ sudo dpkg -i foglamp-1.2-x86_64.deb - Selecting previously unselected package foglamp. - (Reading database ... 114963 files and directories currently installed.) - Preparing to unpack foglamp-1.2-x86_64.deb ... + $ sudo apt install /var/cache/apt/archives/foglamp-1.2-x86_64.deb + Reading package lists... Done + Building dependency tree + Reading state information... Done + Note, selecting 'foglamp' instead of '/var/cache/apt/archives/foglamp-1.2-x86_64.deb' + The following packages were automatically installed and are no longer required: ... Unpacking foglamp (1.2) ... Setting up foglamp (1.2) ... @@ -367,10 +335,9 @@ Now you are ready to install the package: foglamp.service is not a native service, redirecting to systemd-sysv-install Executing /lib/systemd/systemd-sysv-install enable foglamp Starting FogLAMP service - $ - + $ -The installation automatically registers FogLAMP as a service, so it will come up at startup and it is already up and running when you complete the command. +As you can see from the output, the installation automatically registers FogLAMP as a service, so it will come up at startup and it is already up and running when you complete the command. Check the newly installed package: @@ -387,20 +354,20 @@ You can also check the service currently running: $ sudo systemctl status foglamp.service ● foglamp.service - LSB: FogLAMP - Loaded: loaded (/etc/init.d/foglamp; bad; vendor preset: enabled) - Active: active (running) since Tue 2018-04-24 18:38:34 BST; 3min 30s ago - Docs: man:systemd-sysv-generator(8) - Process: 28451 ExecStart=/etc/init.d/foglamp start (code=exited, status=0/SUCCESS) - Tasks: 14 - Memory: 64.9M - CPU: 3.867s - CGroup: /system.slice/foglamp.service - ├─28516 python3 -m foglamp.services.core - ├─28521 /usr/local/foglamp/services/storage --address=0.0.0.0 --port=46141 - ├─28565 /bin/sh services/south --port=46141 --address=127.0.0.1 --name=COAP - ├─28566 python3 -m foglamp.services.south --port=46141 --address=127.0.0.1 --name=COAP - ├─28567 /bin/sh services/south --port=46141 --address=127.0.0.1 --name=HTTP_SOUTH - └─28568 python3 -m foglamp.services.south --port=46141 --address=127.0.0.1 --name=HTTP_SOUTH + Loaded: loaded (/etc/init.d/foglamp; bad; vendor preset: enabled) + Active: active (running) since Thu 2018-05-10 03:48:20 BST; 1min 31s ago + Docs: man:systemd-sysv-generator(8) + Process: 1088 ExecStart=/etc/init.d/foglamp start (code=exited, status=0/SUCCESS) + Tasks: 14 + Memory: 87.2M + CPU: 2.603s + CGroup: /system.slice/foglamp.service + ├─1218 python3 -m foglamp.services.core + ├─1226 /usr/local/foglamp/services/storage --address=0.0.0.0 --port=44530 + ├─1277 /bin/sh services/south --port=44530 --address=127.0.0.1 --name=COAP + ├─1278 /bin/sh services/south --port=44530 --address=127.0.0.1 --name=HTTP_SOUTH + ├─1279 python3 -m foglamp.services.south --port=44530 --address=127.0.0.1 --name=COAP + └─1280 python3 -m foglamp.services.south --port=44530 --address=127.0.0.1 --name=HTTP_SOUTH ... $ @@ -411,7 +378,7 @@ Check if FogLAMP is up and running with the ``foglamp`` command: $ /usr/local/foglamp/bin/foglamp status FogLAMP v1.2 running. - FogLAMP Uptime: 308 seconds. + FogLAMP Uptime: 162 seconds. FogLAMP records: 0 read, 0 sent, 0 purged. FogLAMP does not require authentication. === FogLAMP services: @@ -437,14 +404,19 @@ Upgrading or downgrading FogLAMP, starting from version 1.2, is as easy as insta Uninstalling the Debian Package ------------------------------- -Use the ``dpkg`` command to uninstall FogLAMP: +Use the ``apt`` or the ``apt-get`` command to uninstall FogLAMP: .. code-block:: console - $ sudo dpkg -l | grep foglamp - ii foglamp 1.2 amd64 FogLAMP, the open source platform for the Internet of Things - $ sudo dpkg --purge foglamp - (Reading database ... 115190 files and directories currently installed.) + $ sudo apt remove foglamp + Reading package lists... Done + ... + The following packages will be REMOVED + foglamp + 0 to upgrade, 0 to newly install, 1 to remove and 2 not to upgrade. + After this operation, 0 B of additional disk space will be used. + Do you want to continue? [Y/n] + (Reading database ... 211747 files and directories currently installed.) Removing foglamp (1.2) ... FogLAMP is currently running. Stop FogLAMP service. @@ -459,15 +431,13 @@ Use the ``dpkg`` command to uninstall FogLAMP: Remove FogLAMP service script Reset systemctl dpkg: warning: while removing foglamp, directory '/usr/local/foglamp' not empty so not removed - $ sudo dpkg --purge foglamp - dpkg: warning: ignoring request to remove foglamp which isn't installed $ -The command also removes the service installed, but it leaves the data directory, in case an administrator might want to analyze or reuse the data. - +The command also removes the service installed. |br| You may notice the warning in the last row of the command output: this is due to the fact that the data directory (``/usr/local/foglamp/data`` by default) has not been removed, in case an administrator might want to analyze or reuse the data. |br| + DEPRECATED: Installing the Snap Package --------------------------------------- From afcd32c12a55e77898ac8ea132a13f9528e022ea Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Thu, 10 May 2018 22:39:50 +0100 Subject: [PATCH 02/12] 06 plugins --- docs/06_plugins.rst | 775 ------------------------- docs/06_plugins/01_FogLAMP_plugins.rst | 94 +++ docs/06_plugins/02_writing_plugins.rst | 178 ++++++ docs/06_plugins/03_south_plugins.rst | 188 ++++++ docs/06_plugins/04_north_plugins.rst | 465 +++++++++++++++ docs/06_plugins/index.rst | 13 + docs/index.rst | 2 +- 7 files changed, 939 insertions(+), 776 deletions(-) delete mode 100644 docs/06_plugins.rst create mode 100644 docs/06_plugins/01_FogLAMP_plugins.rst create mode 100644 docs/06_plugins/02_writing_plugins.rst create mode 100644 docs/06_plugins/03_south_plugins.rst create mode 100644 docs/06_plugins/04_north_plugins.rst create mode 100644 docs/06_plugins/index.rst diff --git a/docs/06_plugins.rst b/docs/06_plugins.rst deleted file mode 100644 index 6aa67cf243..0000000000 --- a/docs/06_plugins.rst +++ /dev/null @@ -1,775 +0,0 @@ -.. Writing and Using Plugins describes how to implement a plugin for FogLAMP and how to use it -.. https://docs.google.com/document/d/1IKGXLWbyN6a7vx8UO3uDbq5Df0VvE4oCQIULgZVZbjM - -.. |br| raw:: html - -
- -.. Images - -.. |DHT11 in PI| image:: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg - :target: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg - -.. Links -.. _here: 05_testing.html#setting-the-omf-translator-plugin -.. _these steps: 04_installation.html - -.. |Getting Started| raw:: html - - here - -.. Links in new tabs - -.. |ADAFruit| raw:: html - - ADAFruit DHT Library - -.. |here BT| raw:: html - - here - -.. |DHT Description| raw:: html - - DHT11 Product Description - -.. |DHT Manual| raw:: html - - DHT11 Product Manual - -.. |DHT Resistor| raw:: html - - This picture - -.. |DHT Wired| raw:: html - - This picture - -.. |DHT Pins| raw:: html - - this - -.. |GPIO| raw:: html - - here - - -.. ============================================= - - -******* -Plugins -******* - -FogLAMP makes extensive use of plugin components to extend the base functionality of the platform. In particular, plugins are used to extend the set of sensors and actuators that FogLAMP supports, the set of services to which FogLAMP will push accumulated data gathered from those sensors and the mechanism by which FogLAMP buffers data internally. - -This chapter presents the plugins available in FogLAMP, how to write and use new plugins to support different sensors, protocols, historians and storage devices. It will guide you through the process and entry points that are required for the various different type of plugin. - - -FogLAMP Plugins -=============== - -In this version of FogLAMP you have three types of plugins: - -- **South Microservice Plugins** - They are responsible for communication between FogLAMP and the sensors and actuators they support. Each instance of a FogLAMP South microservice will use a plugin for the actual communication to the sensors or actuators that that instance of the South microservice supports. -- **North Plugins** - They are responsible for taking reading data passed to them from the South bound task and doing any necessary conversion to the data and providing the protocol to send that converted data to a north-side service. -- **Storage Plugins** - They sit between the Storage microservice and the physical data storage mechanism that stores the FogLAMP configuration and readings data. Storage plugins differ from other plugins in that they interface to a storage system which is written in C/C++ rather than Python, however they share the same common attributes and entry points that the Python based plugins must support. - - -Plugins in this version of FogLAMP ----------------------------------- - -This version of FogLAMP provides the following plugins: - -+---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| Type | Name | Initial | Description | Availability | Notes | -| | | |br| Status| | | | -+=========+============+============+=============================+============================+========================================+ -| Storage | Postgres | Enabled | PostgreSQL storage |br| | Ubuntu: x86 |br| | | -| | | | for data and metadata | Ubuntu Core: x86, ARM |br| | | -| | | | | Raspbian | | -+---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| South | COAP | Enabled | CoAP Listener | Ubuntu: x86 |br| | | -| | | | | Ubuntu Core: x86, ARM |br| | | -| | | | | Raspbian | | -+---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| South | CC2650POLL | Disabled | TI SensorTag CC2650 |br| | Ubuntu: x86 |br| | It requires BLE support. |br| | -| | | | in polling mode | Ubuntu Core: x86, ARM |br| | There are issues with Ubuntu Core |br| | -| | | | | Raspbian | on ARM, reported |here BT| | -+---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| South | CC2650ASYN | Disabled | TI SensorTag CC2650 |br| | Ubuntu: x86 |br| | It requires BLE support. |br| | -| | | | asynchronous |br| | Ubuntu Core: x86, ARM |br| | There are issues with Ubuntu Core |br| | -| | | | (listening) mode | Raspbian | on ARM, reported |here BT|. | -+---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| South | HTTP_SOUTH | Enabled | HTTP Listener | Ubuntu: x86 |br| | | -| | | | | Ubuntu Core: x86, ARM |br| | | -| | | | | Raspbian | | -+---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| South | dht11pi | Disabled | Wired DHT11 Sensor |br| | Ubuntu Core: ARM |br| | It requires the |ADAFruit|. |br| | -| | | | in polling mode | Raspbian | The plugin is still experimental. | -+---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| North | OMF | Disabled | OSIsoft Message Format |br| | Ubuntu: x86 |br| | It works with PI Connector |br| | -| | | | sender to PI Connector |br| | Ubuntu Core: x86, ARM |br| | Relay OMF 1.0 and 1.2. | -| | | | Relay OMF | Raspbian | | -+---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ - -|br| - - -Writing and Using Plugins -========================= - -Common FogLAMP Plugin API -------------------------- - -Every plugin provides at least one common API entry point, the *plugin_info* entry point. It is used to obtain information about a plugin before it is initialised and used. It allows FogLAMP to determine what type of plugin it is, e.g. a South bound plugin or a North bound plugin, obtain default configuration information for the plugin and determine version information. - - -Plugin Information -~~~~~~~~~~~~~~~~~~ - -The information entry point is implemented as a call, *plugin_info*, that takes no arguments. Data is returned from this API call as a JSON document with certain well known properties. - -A typical Python implementation of this would simply return a fixed dictionary object that encodes the required properties. - -.. code-block:: python - - def plugin_info(): - """ Returns information about the plugin. - - Args: - Returns: - dict: plugin information - Raises: - """ - - return { - 'name': 'DHT11 GPIO', - 'version': '1.0', - 'mode': 'poll', - 'type': 'south', - 'interface': '1.0', - 'config': _DEFAULT_CONFIG - } - -These are the properties returned by the JSON document: - -- **Name** - A textual name that will be used for reporting purposes for this plugin. -- **Version** - This property allows the version of the plugin to be communicated to the plugin loader. This is used for reporting purposes only and has no effect on the way FogLAMP interacts with the plugin. -- **Type** - The type of the plugin, used by the plugin loader to determine if the plugin is being used correctly. The type is a simple string and may be South, North or Storage. - -.. note:: If you browse the FogLAMP code you may find old plugins with type *device*: this was the type used to indicate a South plugin and it is now deprecated. - -- **Interface** - This property reports the version of the plugin API to which this plugin was written. It allows FogLAMP to support upgrades of the API whilst being able to recognise the version that a particular plugin is compliant with. Currently all interfaces are version 1.0. -- **Configuration** - This allows the plugin to return a JSON document which contains the default configuration of the plugin. This is in line with the extensible plugin mechanism of FogLAMP, each plugin will return a set of configuration items that it wishes to use, this will then be used to extend the set of FogLAMP configuration items. This structure, a JSON document, includes default values but no actual values for each configuration option. The first time FogLAMP’s configuration manager sees a category it will register the category and create values for each item using the default value in the configuration document. On subsequent calls the value already in the configuration manager will be used. |br| This mechanism allows the plugin to extend the set of configuration variables whilst giving the user the opportunity to modify the value of these configuration items. It also allow new versions of plugins to add new configuration items whilst retaining the values of previous items. And new items will automatically be assigned the default value for that item. |br| As an example, a plugin that wishes to maintain two configuration variables, say a GPIO pin to use and a polling interval, would return a configuration document that looks as follows: - -.. code-block:: console - - { - 'pollInterval': { - 'description': 'The interval between poll calls to the device poll routine expressed in milliseconds.', - 'type': 'integer', - 'default': '1000' - }, - 'gpiopin': { - 'description': 'The GPIO pin into which the DHT11 data pin is connected', - 'type': 'integer', - 'default': '4' - } - } - - -Plugin Initialization ---------------------- - -The plugin initialization is called after the service that has loaded the plugin has collected the plugin information and resolved the configuration of the plugin but before any other calls will be made to the plugin. The initialization routine is called with the resolved configuration of the plugin, this includes values as opposed to the defaults that were returned in the *plugin_info* call. - -This call is used by the plugin to do any initialization or state creation it needs to do. The call returns a handle which will be passed into each subsequent call of the plugin. The handle allows the plugin to have state information that is maintained and passed to it whilst allowing for multiple instances of the same plugin to be loaded by a service if desired. It is equivalent to a this or self pointer for the plugin, although the plugin is not defined as a class. - -In a simple example of a sensor that reads a GPIO pin for data, we might choose to use that configured GPIO pin as the handle we pass to other calls. - -.. code-block:: python - - def plugin_init(config): - """ Initialise the plugin. - - Args: - config: JSON configuration document for the device configuration category - Returns: - handle: JSON object to be used in future calls to the plugin - Raises: - """ - - handle = config['gpiopin']['value'] - return handle - - -Plugin Reconfigure ------------------- - -The plugin reconfigure method is called whenever the configuration of the plugin is changed. It allows for the dynamic reconfiguration of the plugin whilst it is running. The method is called with the handle of the plugin and the updated configuration document. The plugin should take whatever action it needs to and return a new or updated copy of the handle that will be passed to future calls. - -Using a simple example of our sensor reading a GPIO pin, we extract the new pin number from the new configuration data and return that as the new handle for the plugin instance. - -.. code-block:: python - - def plugin_reconfigure(handle, new_config): - """ Reconfigures the plugin, it should be called when the configuration of the plugin is changed during the - operation of the device service. - The new configuration category should be passed. - - Args: - handle: handle returned by the plugin initialisation call - new_config: JSON object representing the new configuration category for the category - Returns: - new_handle: new handle to be used in the future calls - Raises: - """ - - new_handle = new_config['gpiopin']['value'] - - return new_handle - - -Plugin Shutdown ---------------- - -The plugin shutdown method is called as part of the shutdown sequence of the service that loaded the plugin. It gives the plugin the opportunity to do any cleanup operations before terminating. As with all calls it is passed the handle of our plugin instance. Plugins can not prevent the shutdown and do not have to implement any actions. In our simple sensor example there is nothing to do in order to shutdown the plugin. - - -South Plugins -============= - -South plugins are used to communicate with sensors and actuators, there are two modes of plugin operation; *asyncio* and *polled*. - - -Polled Mode ------------ - -Polled mode is the simplest form of South plugin that can be written, a poll routine is called at an interval defined in the plugin configuration. The South service determines the type of the plugin by examining at the mode property in the information the plugin returns from the *plugin_info* call. - - -Plugin Poll -~~~~~~~~~~~ - -The plugin *poll* method is called periodically to collect the readings from a poll mode sensor. As with all other calls the argument passed to the method is the handle returned by the initialization call, the return of the method should be the JSON payload of the readings to return. - -The JSON payload returned, as a Python dictionary, should contain the properties; asset, timestamp, key and readings. - -+-----------+-------------------------------------------------------+ -| Property | Description | -+===========+=======================================================+ -| asset | The asset key of the sensor device that is being read | -+-----------+-------------------------------------------------------+ -| timestamp | A timestamp for the reading data | -+-----------+-------------------------------------------------------+ -| key | A UUID which is the unique key of this reading | -+-----------+-------------------------------------------------------+ -| readings | The reading data itself as a JSON object | -+-----------+-------------------------------------------------------+ - -It is important that the *poll* method does not block as this will prevent the proper operation of the South microservice. -Using the example of our simple DHT11 device attached to a GPIO pin, the *poll* routine could be: - -.. code-block:: python - - def plugin_poll(handle): - """ Extracts data from the sensor and returns it in a JSON document as a Python dict. - - Available for poll mode only. - - Args: - handle: handle returned by the plugin initialisation call - Returns: - returns a sensor reading in a JSON document, as a Python dict, if it is available - None - If no reading is available - Raises: - DataRetrievalError - """ - - try: - humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, handle) - if humidity is not None and temperature is not None: - time_stamp = str(datetime.now(tz=timezone.utc)) - readings = { 'temperature': temperature , 'humidity' : humidity } - wrapper = { - 'asset': 'dht11', - 'timestamp': time_stamp, - 'key': str(uuid.uuid4()), - 'readings': readings - } - return wrapper - else: - return None - - except Exception as ex: - raise exceptions.DataRetrievalError(ex) - - return None - - -Async IO Mode -------------- - -In asyncio mode the plugin inserts itself into the event processing loop of the South server itself. This is a more complex mechanism and is intended for plugins that need to block or listen for incoming data via a network. - - -Plugin Start -~~~~~~~~~~~~ - -The *plugin_start* method, as with other plugin calls, is called with the plugin handle data that was returned from the *plugin_init* call. The *plugin_start* call will only be called once for a plugin, it is the responsibility of *plugin_start* to install the plugin code into the python event handling system for asyncIO. Assuming an example whereby the interface to a sensor is via HTTP and the sensor will make HTTP POST calls to our plugin in order to send data into FogLAMP, a *plugin_start* for this scenario would create a web application endpoint for reception of the POST command. - -.. code-block:: python - - loop = asyncio.get_event_loop() - - app = web.Application( middlewares=[middleware.error_middleware] ) - app.router.add_route( 'POST', '/', SensorPhoneIngest.render_post ) - handler = app.make_handler() - coro = loop.create_server( handler, host, port ) - server = asyncio.ensure_future( coro ) - -This code first gets the event loop for this Python execution, it then creates the web application and adds a route for the POST request. In this case it is calling the *render_post* method of the object *SensorPhone*. It then goes on to create the handler and install the web server instance into the event system. - - -Async Handler -~~~~~~~~~~~~~ - -The async handler is defined for incoming message has the responsibility of taking the sensor data and ingesting that into FogLAMP. Unlike the poll mechanism, this is done from within the handler rather than by passing the data back to the South service itself. A convenient method exists for ingesting readings, *Ingest.add_readings*. This call is passed an asset, timestamp, key and readings document for the asset and will do everything else required to make sure the readings are stored in the FogLAMP buffer. |br| In the case of our HTTP based example above, the code would create the items needed to generate the arguments to the *Ingest.add_readings* call, by creating data items and retrieving them from the payload sent by the sensor. - -.. code-block:: python - - try: - if not Ingest.is_available(): - increment_discarded_counter = True - message = {'busy': True} - else: - payload = await request.json() - - asset = 'SensorPhone' - timestamp = str(datetime.now(tz=timezone.utc)) - messages = payload.get('messages') - - if not isinstance(messages, list): - raise ValueError('messages must be a list') - - for readings in messages: - key = str(uuid.uuid4()) - await Ingest.add_readings(asset=asset, timestamp=timestamp, key=key, readings=readings) - - except ... - -It would then respond to the HTTP request and return. Since the handler is embedded in the event loop this will happen in the context of a coroutine and would happen each time a new POST request is received. - -.. code-block:: python - - message['status'] = code - return web.json_response(message) - - -A South Plugin Example: the DHT11 Sensor -======================================== - -Let's try to put all the information together and write a plugin. We can continue to use the example of an inexpensive sensor, the DHT11, used to measure temperature and humidity, directly wired to a Raspberry PI. This plugin is also available in the FogLAMP project on GitHub, in the *contrib* folder. - -First, here is a set of links where you can find more information regarding this sensor: - -- |DHT Description| -- |DHT Manual| -- |ADAFruit| - - -The Hardware ------------- - -The DHT sensor is directly connected to a Raspberry PI 2 or 3. You may decide to buy a sensor and a resistor and solder them yourself, or you can buy a ready-made circuit that provides the correct output to wire to the Raspberry PI. |DHT Resistor| shows a DHT11 with resistor that you can buy online. - -The sensor can be directly connected to the Raspberry PI GPIO (General Purpose Input/Output). An introduction to the GPIO and the pinset is available |GPIO|. In our case, you must connect the sensor on these pins: - -- **VCC** is connected to PIN #2 (5v Power) -- **GND** is connected to PIN #6 (Ground) -- **DATA** is connected to PIN #7 (BCM 4 - GPCLK0) - -|DHT Wired| shows the sensor wired to the Raspberry PI and |DHT Pins| is a zoom into the wires used. - - -The Software ------------- - -For this plugin we use the ADAFruit Python Library (links to the GitHub repository are above). First, you must install the library (in future versions the library will be provided in a ready-made package): - -.. code-block:: console - - $ git clone https://github.com/adafruit/Adafruit_Python_DHT.git - Cloning into 'Adafruit_Python_DHT'... - remote: Counting objects: 249, done. - remote: Total 249 (delta 0), reused 0 (delta 0), pack-reused 249 - Receiving objects: 100% (249/249), 77.00 KiB | 0 bytes/s, done. - Resolving deltas: 100% (142/142), done. - $ cd Adafruit_Python_DHT - $ sudo apt-get install build-essential python-dev - Reading package lists... Done - Building dependency tree - Reading state information... Done - The following NEW packages will be installed: - build-essential python-dev - ... - $ sudo python3 setup.py install - running install - running bdist_egg - running egg_info - creating Adafruit_DHT.egg-info - ... - $ - - -The Plugin ----------- - -This is the code for the plugin: - -.. code-block:: python - - """ Plugin for a DHT11 temperature and humidity sensor attached directly - to the GPIO pins of a Raspberry Pi - - This plugin uses the Adafruit DHT library, to install this perform - the following steps: - - git clone https://github.com/adafruit/Adafruit_Python_DHT.git - cd Adafruit_Python_DHT - sudo apt-get install build-essential python-dev - sudo python setup.py install - - To access the GPIO pins foglamp must be able to access /dev/gpiomem, - the default access for this is owner and group read/write. Either - FogLAMP must be added to the group or the permissions altered to - allow FogLAMP access to the device. - """ - - from datetime import datetime, timezone - import Adafruit_DHT - import uuid - import copy - - from foglamp.common import logger - from foglamp.services.south import exceptions - - __author__ = "Mark Riddoch" - __copyright__ = "Copyright (c) 2017 OSIsoft, LLC" - __license__ = "Apache 2.0" - __version__ = "${VERSION}" - - _DEFAULT_CONFIG = { - 'plugin': { - 'description': 'Python module name of the plugin to load', - 'type': 'string', - 'default': 'dht11pi' - }, - 'pollInterval': { - 'description': 'The interval between poll calls to the device poll routine expressed in milliseconds.', - 'type': 'integer', - 'default': '1000' - }, - 'gpiopin': { - 'description': 'The GPIO pin into which the DHT11 data pin is connected', - 'type': 'integer', - 'default': '4' - } - - } - - _LOGGER = logger.setup(__name__) - """ Setup the access to the logging system of FogLAMP """ - - def plugin_info(): - """ Returns information about the plugin. - - Args: - Returns: - dict: plugin information - Raises: - """ - - return { - 'name': 'DHT11 GPIO', - 'version': '1.0', - 'mode': 'poll', - 'type': 'south', - 'interface': '1.0', - 'config': _DEFAULT_CONFIG - } - - - def plugin_init(config): - """ Initialise the plugin. - - Args: - config: JSON configuration document for the device configuration category - Returns: - handle: JSON object to be used in future calls to the plugin - Raises: - """ - - handle = config['gpiopin']['value'] - return handle - - - def plugin_poll(handle): - """ Extracts data from the sensor and returns it in a JSON document as a Python dict. - - Available for poll mode only. - - Args: - handle: handle returned by the plugin initialisation call - Returns: - returns a sensor reading in a JSON document, as a Python dict, if it is available - None - If no reading is available - Raises: - DataRetrievalError - """ - - try: - humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, handle) - if humidity is not None and temperature is not None: - time_stamp = str(datetime.now(tz=timezone.utc)) - readings = { 'temperature': temperature , 'humidity' : humidity } - wrapper = { - 'asset': 'dht11', - 'timestamp': time_stamp, - 'key': str(uuid.uuid4()), - 'readings': readings - } - return wrapper - else: - return None - - except Exception as ex: - raise exceptions.DataRetrievalError(ex) - - return None - - - def plugin_reconfigure(handle, new_config): - """ Reconfigures the plugin, it should be called when the configuration of the plugin is changed during the - operation of the device service. - The new configuration category should be passed. - - Args: - handle: handle returned by the plugin initialisation call - new_config: JSON object representing the new configuration category for the category - Returns: - new_handle: new handle to be used in the future calls - Raises: - """ - - new_handle = new_config['gpiopin']['value'] - return new_handle - - - def plugin_shutdown(handle): - """ Shutdowns the plugin doing required cleanup, to be called prior to the device service being shut down. - - Args: - handle: handle returned by the plugin initialisation call - Returns: - Raises: - """ - - -The configuration ------------------ - -Since the plugin is still experimental, it works only in a build environment, the snap version will be available in the next release. - -The configuration must be set manually in the FogLAMP metadata. in the repository, the file *cmds.sql* in the *contrib/plugins/south/dht11pi* folder must be executed with *psql* (or another PostgreSQL client) to add the configuration to the FogLAMP metadata. - -Let's see the SQL commands: - -.. code-block:: sql - - --- Create the South service instannce - INSERT INTO foglamp.scheduled_processes ( name, script ) - VALUES ( 'dht11pi', '["services/south"]'); - - --- Add the schedule to start the service at system startup - INSERT INTO foglamp.schedules ( id, schedule_name, process_name, schedule_type,schedule_interval, exclusive ) - VALUES ( '543a59ce-a9ca-11e7-abc4-cec278b6b11a', 'device', 'dht11pi', 1, '0:0', true ); - - --- Insert the config needed to load the plugin - INSERT INTO foglamp.configuration ( key, description, value ) - VALUES ( 'dht11pi', 'DHT11 on Raspberry Pi Configuration', - '{"plugin" : { "type" : "string", "value" : "dht11pi", "default" : "dht11pi", "description" : "Plugin to load" } }' ); - - -Building FogLAMP and Adding the Plugin --------------------------------------- - -If you have not built FogLAMP yet, follow the steps described |Getting Started|. After the build, you can optionally install FogLAMP following `these steps`_. - -Once the Storage database has been setup, let's update the configurarion to include the new plugin: - -.. code-block:: console - - $ psql -d foglamp -f cmds.sql - INSERT 0 1 - INSERT 0 1 - INSERT 0 1 - $ - - -Now it is time to apply a workaround and include our new plugin. - -- If you intend to start and execute FogLAMP from the build folder: copy the structure of the *contrib* folder into the *python* folder: - -.. code-block:: console - - $ cd ~/FogLAMP - $ cp -R contrib/plugins python/foglamp/. - $ - -- If you have installed FogLAMP by executing ``sudo make install``, copy the structure of the *contrib* folder into the installed *python* folder: - -.. code-block:: console - - $ cd ~/FogLAMP - $ sudo cp -R contrib/plugins /usr/local/FogLAMP/python/foglamp/. - $ - -.. note:: If you have installed FogLAMP using an alternative *DESTDIR*, remember to add the path to the destination directory to the ``cp`` command. - - -Using the Plugin ----------------- - -Now you are ready to use the DHT11 plugin. If stop and restart FogLAMP if it is already running, or start it now. - -- Starting FogLAMP from the build folder: - -.. code-block:: console - - $ cd ~/FogLAMP - $ export FOGLAMP_ROOT=$HOME/FogLAMP - $ scripts/foglamp start - Starting FogLAMP................ - FogLAMP started. - $ - - -- Starting FogLAMP from the installed folder: - -.. code-block:: console - - $ cd /usr/local/FogLAMP - $ bin/foglamp start - Starting FogLAMP................ - FogLAMP started. - $ - - -Let's see what we have collected so far: - -.. code-block:: console - - $ curl -s http://localhost:8081/foglamp/asset | jq - [ - { - "count": 158, - "asset_code": "dht11" - } - ] - $ - -Finally, let's extract some values: - -.. code-block:: console - - $ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq - [ - { - "timestamp": "2017-12-30 14:41:39.672", - "reading": { - "temperature": 19, - "humidity": 62 - } - }, - { - "timestamp": "2017-12-30 14:41:35.615", - "reading": { - "temperature": 19, - "humidity": 63 - } - }, - { - "timestamp": "2017-12-30 14:41:34.087", - "reading": { - "temperature": 19, - "humidity": 62 - } - }, - { - "timestamp": "2017-12-30 14:41:32.557", - "reading": { - "temperature": 19, - "humidity": 63 - } - }, - { - "timestamp": "2017-12-30 14:41:31.028", - "reading": { - "temperature": 19, - "humidity": 63 - } - } - ] - $ - - -Clearly we will not see many changes in temperature or humidity, unless we place our thumb on the sensor or we blow warm breathe on it :-) - -.. code-block:: console - - $ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq - [ - { - "timestamp": "2017-12-30 14:43:16.787", - "reading": { - "temperature": 25, - "humidity": 95 - } - }, - { - "timestamp": "2017-12-30 14:43:15.258", - "reading": { - "temperature": 25, - "humidity": 95 - } - }, - { - "timestamp": "2017-12-30 14:43:13.729", - "reading": { - "temperature": 24, - "humidity": 95 - } - }, - { - "timestamp": "2017-12-30 14:43:12.201", - "reading": { - "temperature": 24, - "humidity": 95 - } - }, - { - "timestamp": "2017-12-30 14:43:05.616", - "reading": { - "temperature": 22, - "humidity": 95 - } - } - ] - $ - -Needless to say, the North plugin will send the buffered data to the PI system using the PI Connector Relay OMF. Do not forget to set the correct IP address for the PI Connector Relay, as it is described `here`_. - -|DHT11 in PI| - - diff --git a/docs/06_plugins/01_FogLAMP_plugins.rst b/docs/06_plugins/01_FogLAMP_plugins.rst new file mode 100644 index 0000000000..c5ec0e8cd5 --- /dev/null +++ b/docs/06_plugins/01_FogLAMP_plugins.rst @@ -0,0 +1,94 @@ +.. FogLAMP Plugins + +.. |br| raw:: html + +
+ +.. Images + +.. Links + +.. Links in new tabs + +.. |here BT| raw:: html + + here + + +.. ============================================= + + +FogLAMP makes extensive use of plugin components to extend the base functionality of the platform. In particular, plugins are used to extend the set of sensors and actuators that FogLAMP supports, the set of services to which FogLAMP will push accumulated data gathered from those sensors and the mechanism by which FogLAMP buffers data internally. + +This chapter presents the plugins available in FogLAMP, how to write and use new plugins to support different sensors, protocols, historians and storage devices. It will guide you through the process and entry points that are required for the various different type of plugin. + + +FogLAMP Plugins +=============== + +In this version of FogLAMP you have three types of plugins: + +- **South Plugins** - They are responsible for communication between FogLAMP and the sensors and actuators they support. Each instance of a FogLAMP South microservice will use a plugin for the actual communication to the sensors or actuators that that instance of the South microservice supports. +- **North Plugins** - They are responsible for taking reading data passed to them from the South bound task and doing any necessary conversion to the data and providing the protocol to send that converted data to a north-side service. +- **Storage Plugins** - They sit between the Storage microservice and the physical data storage mechanism that stores the FogLAMP configuration and readings data. Storage plugins differ from other plugins in that they interface to a storage system which is written in C/C++ rather than Python, however they share the same common attributes and entry points that the Python based plugins must support. + + +Plugins in this version of FogLAMP +---------------------------------- + +This version of FogLAMP provides the following plugins in the main repository: + ++---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ +| Type | Name | Initial | Description | Availability | Notes | +| | | |br| Status| | | | ++=========+============+============+=============================+============================+========================================+ +| Storage | SQLite | Enabled | SQLite storage |br| | Ubuntu: x86 |br| | | +| | | | for data and metadata | Ubuntu Core: x86, ARM |br| | | +| | | | | Raspbian | | ++---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ +| Storage | Postgres | Disabled | PostgreSQL storage |br| | Ubuntu: x86 |br| | | +| | | | for data and metadata | Ubuntu Core: x86, ARM |br| | | +| | | | | Raspbian | | ++---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ +| South | COAP | Enabled | CoAP Listener | Ubuntu: x86 |br| | | +| | | | | Ubuntu Core: x86, ARM |br| | | +| | | | | Raspbian | | ++---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ +| South | CC2650POLL | Disabled | TI SensorTag CC2650 |br| | Ubuntu: x86 |br| | It requires BLE support. |br| | +| | | | in polling mode | Ubuntu Core: x86, ARM |br| | There are issues with Ubuntu Core |br| | +| | | | | Raspbian | on ARM, reported |here BT| | ++---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ +| South | CC2650ASYN | Disabled | TI SensorTag CC2650 |br| | Ubuntu: x86 |br| | It requires BLE support. |br| | +| | | | asynchronous |br| | Ubuntu Core: x86, ARM |br| | There are issues with Ubuntu Core |br| | +| | | | (listening) mode | Raspbian | on ARM, reported |here BT|. | ++---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ +| South | HTTP_SOUTH | Enabled | HTTP Listener | Ubuntu: x86 |br| | | +| | | | | Ubuntu Core: x86, ARM |br| | | +| | | | | Raspbian | | ++---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ +| North | OMF | Disabled | OSIsoft Message Format |br| | Ubuntu: x86 |br| | It works with PI Connector |br| | +| | | | sender to PI Connector |br| | Ubuntu Core: x86, ARM |br| | Relay OMF 1.2.X and 2.2 | +| | | | Relay OMF | Raspbian | | ++---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ +| North | OCS | Disabled | OSIsoft Message Format |br| | Ubuntu: x86 |br| | | +| | | | sender to the OSIsoft |br| | Ubuntu Core: x86, ARM |br| | | +| | | | Cloud Service | Raspbian | | ++---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ + + +In addition to the plugins in the main repository, these plugins are also available: + ++-------+-------------+---------------------------+---------------------------------------+--------------+ +| Type | Name | Repository | Description | Availability | ++=======+=============+===========================+=======================================+==============+ +| South | dht11pi | foglamp-south-dht11 | Wired DHT11 Sensor in polling mode | Respbian | ++-------+-------------+---------------------------+---------------------------------------+--------------+ +| South | envirophat | foglamp-south-envirophat | Enviro pHAT sensor set | Raspbian | ++-------+-------------+---------------------------+---------------------------------------+--------------+ +| South | openweather | foglamp-south-openweather | Data pull from the OpenWeatherMap API | Raspbian | ++-------+-------------+---------------------------+---------------------------------------+--------------+ +| South | pt100 | foglamp-south-pt100 | Wired PT100 temperature sensor | Raspbian | ++-------+-------------+---------------------------+---------------------------------------+--------------+ + +|br| + diff --git a/docs/06_plugins/02_writing_plugins.rst b/docs/06_plugins/02_writing_plugins.rst new file mode 100644 index 0000000000..a83e4f296f --- /dev/null +++ b/docs/06_plugins/02_writing_plugins.rst @@ -0,0 +1,178 @@ +.. Writing and Using Plugins describes how to implement a plugin for FogLAMP and how to use it +.. https://docs.google.com/document/d/1IKGXLWbyN6a7vx8UO3uDbq5Df0VvE4oCQIULgZVZbjM + +.. |br| raw:: html + +
+ +.. Images + +.. |DHT11 in PI| image:: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg + :target: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg + +.. Links +.. _here: 05_testing.html#setting-the-omf-translator-plugin +.. _these steps: 04_installation.html + +.. |Getting Started| raw:: html + + here + +.. Links in new tabs + +.. |ADAFruit| raw:: html + + ADAFruit DHT Library + +.. |here BT| raw:: html + + here + +.. |DHT Description| raw:: html + + DHT11 Product Description + +.. |DHT Manual| raw:: html + + DHT11 Product Manual + +.. |DHT Resistor| raw:: html + + This picture + +.. |DHT Wired| raw:: html + + This picture + +.. |DHT Pins| raw:: html + + this + +.. |GPIO| raw:: html + + here + + +.. ============================================= + + +Writing and Using Plugins +========================= + +Common FogLAMP Plugin API +------------------------- + +Every plugin provides at least one common API entry point, the *plugin_info* entry point. It is used to obtain information about a plugin before it is initialised and used. It allows FogLAMP to determine what type of plugin it is, e.g. a South bound plugin or a North bound plugin, obtain default configuration information for the plugin and determine version information. + + +Plugin Information +~~~~~~~~~~~~~~~~~~ + +The information entry point is implemented as a call, *plugin_info*, that takes no arguments. Data is returned from this API call as a JSON document with certain well known properties. + +A typical Python implementation of this would simply return a fixed dictionary object that encodes the required properties. + +.. code-block:: python + + def plugin_info(): + """ Returns information about the plugin. + + Args: + Returns: + dict: plugin information + Raises: + """ + + return { + 'name': 'DHT11 GPIO', + 'version': '1.0', + 'mode': 'poll', + 'type': 'south', + 'interface': '1.0', + 'config': _DEFAULT_CONFIG + } + +These are the properties returned by the JSON document: + +- **Name** - A textual name that will be used for reporting purposes for this plugin. +- **Version** - This property allows the version of the plugin to be communicated to the plugin loader. This is used for reporting purposes only and has no effect on the way FogLAMP interacts with the plugin. +- **Type** - The type of the plugin, used by the plugin loader to determine if the plugin is being used correctly. The type is a simple string and may be South, North or Storage. + +.. note:: If you browse the FogLAMP code you may find old plugins with type *device*: this was the type used to indicate a South plugin and it is now deprecated. + +- **Interface** - This property reports the version of the plugin API to which this plugin was written. It allows FogLAMP to support upgrades of the API whilst being able to recognise the version that a particular plugin is compliant with. Currently all interfaces are version 1.0. +- **Configuration** - This allows the plugin to return a JSON document which contains the default configuration of the plugin. This is in line with the extensible plugin mechanism of FogLAMP, each plugin will return a set of configuration items that it wishes to use, this will then be used to extend the set of FogLAMP configuration items. This structure, a JSON document, includes default values but no actual values for each configuration option. The first time FogLAMP’s configuration manager sees a category it will register the category and create values for each item using the default value in the configuration document. On subsequent calls the value already in the configuration manager will be used. |br| This mechanism allows the plugin to extend the set of configuration variables whilst giving the user the opportunity to modify the value of these configuration items. It also allow new versions of plugins to add new configuration items whilst retaining the values of previous items. And new items will automatically be assigned the default value for that item. |br| As an example, a plugin that wishes to maintain two configuration variables, say a GPIO pin to use and a polling interval, would return a configuration document that looks as follows: + +.. code-block:: console + + { + 'pollInterval': { + 'description': 'The interval between poll calls to the device poll routine expressed in milliseconds.', + 'type': 'integer', + 'default': '1000' + }, + 'gpiopin': { + 'description': 'The GPIO pin into which the DHT11 data pin is connected', + 'type': 'integer', + 'default': '4' + } + } + + +Plugin Initialization +--------------------- + +The plugin initialization is called after the service that has loaded the plugin has collected the plugin information and resolved the configuration of the plugin but before any other calls will be made to the plugin. The initialization routine is called with the resolved configuration of the plugin, this includes values as opposed to the defaults that were returned in the *plugin_info* call. + +This call is used by the plugin to do any initialization or state creation it needs to do. The call returns a handle which will be passed into each subsequent call of the plugin. The handle allows the plugin to have state information that is maintained and passed to it whilst allowing for multiple instances of the same plugin to be loaded by a service if desired. It is equivalent to a this or self pointer for the plugin, although the plugin is not defined as a class. + +In a simple example of a sensor that reads a GPIO pin for data, we might choose to use that configured GPIO pin as the handle we pass to other calls. + +.. code-block:: python + + def plugin_init(config): + """ Initialise the plugin. + + Args: + config: JSON configuration document for the device configuration category + Returns: + handle: JSON object to be used in future calls to the plugin + Raises: + """ + + handle = config['gpiopin']['value'] + return handle + + +Plugin Reconfigure +------------------ + +The plugin reconfigure method is called whenever the configuration of the plugin is changed. It allows for the dynamic reconfiguration of the plugin whilst it is running. The method is called with the handle of the plugin and the updated configuration document. The plugin should take whatever action it needs to and return a new or updated copy of the handle that will be passed to future calls. + +Using a simple example of our sensor reading a GPIO pin, we extract the new pin number from the new configuration data and return that as the new handle for the plugin instance. + +.. code-block:: python + + def plugin_reconfigure(handle, new_config): + """ Reconfigures the plugin, it should be called when the configuration of the plugin is changed during the + operation of the device service. + The new configuration category should be passed. + + Args: + handle: handle returned by the plugin initialisation call + new_config: JSON object representing the new configuration category for the category + Returns: + new_handle: new handle to be used in the future calls + Raises: + """ + + new_handle = new_config['gpiopin']['value'] + + return new_handle + + +Plugin Shutdown +--------------- + +The plugin shutdown method is called as part of the shutdown sequence of the service that loaded the plugin. It gives the plugin the opportunity to do any cleanup operations before terminating. As with all calls it is passed the handle of our plugin instance. Plugins can not prevent the shutdown and do not have to implement any actions. In our simple sensor example there is nothing to do in order to shutdown the plugin. + diff --git a/docs/06_plugins/03_south_plugins.rst b/docs/06_plugins/03_south_plugins.rst new file mode 100644 index 0000000000..f2cc21b37a --- /dev/null +++ b/docs/06_plugins/03_south_plugins.rst @@ -0,0 +1,188 @@ +.. Writing and Using Plugins describes how to implement a plugin for FogLAMP and how to use it +.. https://docs.google.com/document/d/1IKGXLWbyN6a7vx8UO3uDbq5Df0VvE4oCQIULgZVZbjM + +.. |br| raw:: html + +
+ +.. Images + +.. |DHT11 in PI| image:: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg + :target: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg + +.. Links +.. _here: 05_testing.html#setting-the-omf-translator-plugin +.. _these steps: 04_installation.html + +.. |Getting Started| raw:: html + + here + +.. Links in new tabs + +.. |ADAFruit| raw:: html + + ADAFruit DHT Library + +.. |here BT| raw:: html + + here + +.. |DHT Description| raw:: html + + DHT11 Product Description + +.. |DHT Manual| raw:: html + + DHT11 Product Manual + +.. |DHT Resistor| raw:: html + + This picture + +.. |DHT Wired| raw:: html + + This picture + +.. |DHT Pins| raw:: html + + this + +.. |GPIO| raw:: html + + here + + +.. ============================================= + + +South Plugins +============= + +South plugins are used to communicate with sensors and actuators, there are two modes of plugin operation; *asyncio* and *polled*. + + +Polled Mode +----------- + +Polled mode is the simplest form of South plugin that can be written, a poll routine is called at an interval defined in the plugin configuration. The South service determines the type of the plugin by examining at the mode property in the information the plugin returns from the *plugin_info* call. + + +Plugin Poll +~~~~~~~~~~~ + +The plugin *poll* method is called periodically to collect the readings from a poll mode sensor. As with all other calls the argument passed to the method is the handle returned by the initialization call, the return of the method should be the JSON payload of the readings to return. + +The JSON payload returned, as a Python dictionary, should contain the properties; asset, timestamp, key and readings. + ++-----------+-------------------------------------------------------+ +| Property | Description | ++===========+=======================================================+ +| asset | The asset key of the sensor device that is being read | ++-----------+-------------------------------------------------------+ +| timestamp | A timestamp for the reading data | ++-----------+-------------------------------------------------------+ +| key | A UUID which is the unique key of this reading | ++-----------+-------------------------------------------------------+ +| readings | The reading data itself as a JSON object | ++-----------+-------------------------------------------------------+ + +It is important that the *poll* method does not block as this will prevent the proper operation of the South microservice. +Using the example of our simple DHT11 device attached to a GPIO pin, the *poll* routine could be: + +.. code-block:: python + + def plugin_poll(handle): + """ Extracts data from the sensor and returns it in a JSON document as a Python dict. + + Available for poll mode only. + + Args: + handle: handle returned by the plugin initialisation call + Returns: + returns a sensor reading in a JSON document, as a Python dict, if it is available + None - If no reading is available + Raises: + DataRetrievalError + """ + + try: + humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, handle) + if humidity is not None and temperature is not None: + time_stamp = str(datetime.now(tz=timezone.utc)) + readings = { 'temperature': temperature , 'humidity' : humidity } + wrapper = { + 'asset': 'dht11', + 'timestamp': time_stamp, + 'key': str(uuid.uuid4()), + 'readings': readings + } + return wrapper + else: + return None + + except Exception as ex: + raise exceptions.DataRetrievalError(ex) + + return None + + +Async IO Mode +------------- + +In asyncio mode the plugin inserts itself into the event processing loop of the South server itself. This is a more complex mechanism and is intended for plugins that need to block or listen for incoming data via a network. + + +Plugin Start +~~~~~~~~~~~~ + +The *plugin_start* method, as with other plugin calls, is called with the plugin handle data that was returned from the *plugin_init* call. The *plugin_start* call will only be called once for a plugin, it is the responsibility of *plugin_start* to install the plugin code into the python event handling system for asyncIO. Assuming an example whereby the interface to a sensor is via HTTP and the sensor will make HTTP POST calls to our plugin in order to send data into FogLAMP, a *plugin_start* for this scenario would create a web application endpoint for reception of the POST command. + +.. code-block:: python + + loop = asyncio.get_event_loop() + + app = web.Application( middlewares=[middleware.error_middleware] ) + app.router.add_route( 'POST', '/', SensorPhoneIngest.render_post ) + handler = app.make_handler() + coro = loop.create_server( handler, host, port ) + server = asyncio.ensure_future( coro ) + +This code first gets the event loop for this Python execution, it then creates the web application and adds a route for the POST request. In this case it is calling the *render_post* method of the object *SensorPhone*. It then goes on to create the handler and install the web server instance into the event system. + + +Async Handler +~~~~~~~~~~~~~ + +The async handler is defined for incoming message has the responsibility of taking the sensor data and ingesting that into FogLAMP. Unlike the poll mechanism, this is done from within the handler rather than by passing the data back to the South service itself. A convenient method exists for ingesting readings, *Ingest.add_readings*. This call is passed an asset, timestamp, key and readings document for the asset and will do everything else required to make sure the readings are stored in the FogLAMP buffer. |br| In the case of our HTTP based example above, the code would create the items needed to generate the arguments to the *Ingest.add_readings* call, by creating data items and retrieving them from the payload sent by the sensor. + +.. code-block:: python + + try: + if not Ingest.is_available(): + increment_discarded_counter = True + message = {'busy': True} + else: + payload = await request.json() + + asset = 'SensorPhone' + timestamp = str(datetime.now(tz=timezone.utc)) + messages = payload.get('messages') + + if not isinstance(messages, list): + raise ValueError('messages must be a list') + + for readings in messages: + key = str(uuid.uuid4()) + await Ingest.add_readings(asset=asset, timestamp=timestamp, key=key, readings=readings) + + except ... + +It would then respond to the HTTP request and return. Since the handler is embedded in the event loop this will happen in the context of a coroutine and would happen each time a new POST request is received. + +.. code-block:: python + + message['status'] = code + return web.json_response(message) + + diff --git a/docs/06_plugins/04_north_plugins.rst b/docs/06_plugins/04_north_plugins.rst new file mode 100644 index 0000000000..25e8d03e83 --- /dev/null +++ b/docs/06_plugins/04_north_plugins.rst @@ -0,0 +1,465 @@ +.. Writing and Using Plugins describes how to implement a plugin for FogLAMP and how to use it +.. https://docs.google.com/document/d/1IKGXLWbyN6a7vx8UO3uDbq5Df0VvE4oCQIULgZVZbjM + +.. |br| raw:: html + +
+ +.. Images + +.. |DHT11 in PI| image:: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg + :target: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg + +.. Links +.. _here: 05_testing.html#setting-the-omf-translator-plugin +.. _these steps: 04_installation.html + +.. |Getting Started| raw:: html + + here + +.. Links in new tabs + +.. |ADAFruit| raw:: html + + ADAFruit DHT Library + +.. |here BT| raw:: html + + here + +.. |DHT Description| raw:: html + + DHT11 Product Description + +.. |DHT Manual| raw:: html + + DHT11 Product Manual + +.. |DHT Resistor| raw:: html + + This picture + +.. |DHT Wired| raw:: html + + This picture + +.. |DHT Pins| raw:: html + + this + +.. |GPIO| raw:: html + + here + + +.. ============================================= + + +A South Plugin Example: the DHT11 Sensor +======================================== + +Let's try to put all the information together and write a plugin. We can continue to use the example of an inexpensive sensor, the DHT11, used to measure temperature and humidity, directly wired to a Raspberry PI. This plugin is also available in the FogLAMP project on GitHub, in the *contrib* folder. + +First, here is a set of links where you can find more information regarding this sensor: + +- |DHT Description| +- |DHT Manual| +- |ADAFruit| + + +The Hardware +------------ + +The DHT sensor is directly connected to a Raspberry PI 2 or 3. You may decide to buy a sensor and a resistor and solder them yourself, or you can buy a ready-made circuit that provides the correct output to wire to the Raspberry PI. |DHT Resistor| shows a DHT11 with resistor that you can buy online. + +The sensor can be directly connected to the Raspberry PI GPIO (General Purpose Input/Output). An introduction to the GPIO and the pinset is available |GPIO|. In our case, you must connect the sensor on these pins: + +- **VCC** is connected to PIN #2 (5v Power) +- **GND** is connected to PIN #6 (Ground) +- **DATA** is connected to PIN #7 (BCM 4 - GPCLK0) + +|DHT Wired| shows the sensor wired to the Raspberry PI and |DHT Pins| is a zoom into the wires used. + + +The Software +------------ + +For this plugin we use the ADAFruit Python Library (links to the GitHub repository are above). First, you must install the library (in future versions the library will be provided in a ready-made package): + +.. code-block:: console + + $ git clone https://github.com/adafruit/Adafruit_Python_DHT.git + Cloning into 'Adafruit_Python_DHT'... + remote: Counting objects: 249, done. + remote: Total 249 (delta 0), reused 0 (delta 0), pack-reused 249 + Receiving objects: 100% (249/249), 77.00 KiB | 0 bytes/s, done. + Resolving deltas: 100% (142/142), done. + $ cd Adafruit_Python_DHT + $ sudo apt-get install build-essential python-dev + Reading package lists... Done + Building dependency tree + Reading state information... Done + The following NEW packages will be installed: + build-essential python-dev + ... + $ sudo python3 setup.py install + running install + running bdist_egg + running egg_info + creating Adafruit_DHT.egg-info + ... + $ + + +The Plugin +---------- + +This is the code for the plugin: + +.. code-block:: python + + """ Plugin for a DHT11 temperature and humidity sensor attached directly + to the GPIO pins of a Raspberry Pi + + This plugin uses the Adafruit DHT library, to install this perform + the following steps: + + git clone https://github.com/adafruit/Adafruit_Python_DHT.git + cd Adafruit_Python_DHT + sudo apt-get install build-essential python-dev + sudo python setup.py install + + To access the GPIO pins foglamp must be able to access /dev/gpiomem, + the default access for this is owner and group read/write. Either + FogLAMP must be added to the group or the permissions altered to + allow FogLAMP access to the device. + """ + + from datetime import datetime, timezone + import Adafruit_DHT + import uuid + import copy + + from foglamp.common import logger + from foglamp.services.south import exceptions + + __author__ = "Mark Riddoch" + __copyright__ = "Copyright (c) 2017 OSIsoft, LLC" + __license__ = "Apache 2.0" + __version__ = "${VERSION}" + + _DEFAULT_CONFIG = { + 'plugin': { + 'description': 'Python module name of the plugin to load', + 'type': 'string', + 'default': 'dht11pi' + }, + 'pollInterval': { + 'description': 'The interval between poll calls to the device poll routine expressed in milliseconds.', + 'type': 'integer', + 'default': '1000' + }, + 'gpiopin': { + 'description': 'The GPIO pin into which the DHT11 data pin is connected', + 'type': 'integer', + 'default': '4' + } + + } + + _LOGGER = logger.setup(__name__) + """ Setup the access to the logging system of FogLAMP """ + + def plugin_info(): + """ Returns information about the plugin. + + Args: + Returns: + dict: plugin information + Raises: + """ + + return { + 'name': 'DHT11 GPIO', + 'version': '1.0', + 'mode': 'poll', + 'type': 'south', + 'interface': '1.0', + 'config': _DEFAULT_CONFIG + } + + + def plugin_init(config): + """ Initialise the plugin. + + Args: + config: JSON configuration document for the device configuration category + Returns: + handle: JSON object to be used in future calls to the plugin + Raises: + """ + + handle = config['gpiopin']['value'] + return handle + + + def plugin_poll(handle): + """ Extracts data from the sensor and returns it in a JSON document as a Python dict. + + Available for poll mode only. + + Args: + handle: handle returned by the plugin initialisation call + Returns: + returns a sensor reading in a JSON document, as a Python dict, if it is available + None - If no reading is available + Raises: + DataRetrievalError + """ + + try: + humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, handle) + if humidity is not None and temperature is not None: + time_stamp = str(datetime.now(tz=timezone.utc)) + readings = { 'temperature': temperature , 'humidity' : humidity } + wrapper = { + 'asset': 'dht11', + 'timestamp': time_stamp, + 'key': str(uuid.uuid4()), + 'readings': readings + } + return wrapper + else: + return None + + except Exception as ex: + raise exceptions.DataRetrievalError(ex) + + return None + + + def plugin_reconfigure(handle, new_config): + """ Reconfigures the plugin, it should be called when the configuration of the plugin is changed during the + operation of the device service. + The new configuration category should be passed. + + Args: + handle: handle returned by the plugin initialisation call + new_config: JSON object representing the new configuration category for the category + Returns: + new_handle: new handle to be used in the future calls + Raises: + """ + + new_handle = new_config['gpiopin']['value'] + return new_handle + + + def plugin_shutdown(handle): + """ Shutdowns the plugin doing required cleanup, to be called prior to the device service being shut down. + + Args: + handle: handle returned by the plugin initialisation call + Returns: + Raises: + """ + + +The configuration +----------------- + +Since the plugin is still experimental, it works only in a build environment, the snap version will be available in the next release. + +The configuration must be set manually in the FogLAMP metadata. in the repository, the file *cmds.sql* in the *contrib/plugins/south/dht11pi* folder must be executed with *psql* (or another PostgreSQL client) to add the configuration to the FogLAMP metadata. + +Let's see the SQL commands: + +.. code-block:: sql + + --- Create the South service instannce + INSERT INTO foglamp.scheduled_processes ( name, script ) + VALUES ( 'dht11pi', '["services/south"]'); + + --- Add the schedule to start the service at system startup + INSERT INTO foglamp.schedules ( id, schedule_name, process_name, schedule_type,schedule_interval, exclusive ) + VALUES ( '543a59ce-a9ca-11e7-abc4-cec278b6b11a', 'device', 'dht11pi', 1, '0:0', true ); + + --- Insert the config needed to load the plugin + INSERT INTO foglamp.configuration ( key, description, value ) + VALUES ( 'dht11pi', 'DHT11 on Raspberry Pi Configuration', + '{"plugin" : { "type" : "string", "value" : "dht11pi", "default" : "dht11pi", "description" : "Plugin to load" } }' ); + + +Building FogLAMP and Adding the Plugin +-------------------------------------- + +If you have not built FogLAMP yet, follow the steps described |Getting Started|. After the build, you can optionally install FogLAMP following `these steps`_. + +Once the Storage database has been setup, let's update the configurarion to include the new plugin: + +.. code-block:: console + + $ psql -d foglamp -f cmds.sql + INSERT 0 1 + INSERT 0 1 + INSERT 0 1 + $ + + +Now it is time to apply a workaround and include our new plugin. + +- If you intend to start and execute FogLAMP from the build folder: copy the structure of the *contrib* folder into the *python* folder: + +.. code-block:: console + + $ cd ~/FogLAMP + $ cp -R contrib/plugins python/foglamp/. + $ + +- If you have installed FogLAMP by executing ``sudo make install``, copy the structure of the *contrib* folder into the installed *python* folder: + +.. code-block:: console + + $ cd ~/FogLAMP + $ sudo cp -R contrib/plugins /usr/local/FogLAMP/python/foglamp/. + $ + +.. note:: If you have installed FogLAMP using an alternative *DESTDIR*, remember to add the path to the destination directory to the ``cp`` command. + + +Using the Plugin +---------------- + +Now you are ready to use the DHT11 plugin. If stop and restart FogLAMP if it is already running, or start it now. + +- Starting FogLAMP from the build folder: + +.. code-block:: console + + $ cd ~/FogLAMP + $ export FOGLAMP_ROOT=$HOME/FogLAMP + $ scripts/foglamp start + Starting FogLAMP................ + FogLAMP started. + $ + + +- Starting FogLAMP from the installed folder: + +.. code-block:: console + + $ cd /usr/local/FogLAMP + $ bin/foglamp start + Starting FogLAMP................ + FogLAMP started. + $ + + +Let's see what we have collected so far: + +.. code-block:: console + + $ curl -s http://localhost:8081/foglamp/asset | jq + [ + { + "count": 158, + "asset_code": "dht11" + } + ] + $ + +Finally, let's extract some values: + +.. code-block:: console + + $ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq + [ + { + "timestamp": "2017-12-30 14:41:39.672", + "reading": { + "temperature": 19, + "humidity": 62 + } + }, + { + "timestamp": "2017-12-30 14:41:35.615", + "reading": { + "temperature": 19, + "humidity": 63 + } + }, + { + "timestamp": "2017-12-30 14:41:34.087", + "reading": { + "temperature": 19, + "humidity": 62 + } + }, + { + "timestamp": "2017-12-30 14:41:32.557", + "reading": { + "temperature": 19, + "humidity": 63 + } + }, + { + "timestamp": "2017-12-30 14:41:31.028", + "reading": { + "temperature": 19, + "humidity": 63 + } + } + ] + $ + + +Clearly we will not see many changes in temperature or humidity, unless we place our thumb on the sensor or we blow warm breathe on it :-) + +.. code-block:: console + + $ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq + [ + { + "timestamp": "2017-12-30 14:43:16.787", + "reading": { + "temperature": 25, + "humidity": 95 + } + }, + { + "timestamp": "2017-12-30 14:43:15.258", + "reading": { + "temperature": 25, + "humidity": 95 + } + }, + { + "timestamp": "2017-12-30 14:43:13.729", + "reading": { + "temperature": 24, + "humidity": 95 + } + }, + { + "timestamp": "2017-12-30 14:43:12.201", + "reading": { + "temperature": 24, + "humidity": 95 + } + }, + { + "timestamp": "2017-12-30 14:43:05.616", + "reading": { + "temperature": 22, + "humidity": 95 + } + } + ] + $ + +Needless to say, the North plugin will send the buffered data to the PI system using the PI Connector Relay OMF. Do not forget to set the correct IP address for the PI Connector Relay, as it is described `here`_. + +|DHT11 in PI| + + diff --git a/docs/06_plugins/index.rst b/docs/06_plugins/index.rst new file mode 100644 index 0000000000..1588fc485c --- /dev/null +++ b/docs/06_plugins/index.rst @@ -0,0 +1,13 @@ +.. Plugins + +******* +Plugins +******* + +.. toctree:: + + 01_FogLAMP_plugins + 02_writing_plugins + 03_south_plugins + 04_north_plugins + diff --git a/docs/index.rst b/docs/index.rst index fed7693e31..4e7b6b3eb3 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -14,7 +14,7 @@ Welcome to FogLAMP's documentation! 03_getting_started 04_installation 05_testing - 06_plugins + 06_plugins/index 07_admin/index 08_developer/index 91_version_history From 47c62cab4180e755b969283e1d04d25bc87a0a8a Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Sat, 12 May 2018 00:13:50 +0100 Subject: [PATCH 03/12] north plugins --- docs/06_plugins/02_writing_plugins.rst | 37 +- docs/06_plugins/03_01_DHT11.rst | 465 +++++++++++++++++++ docs/06_plugins/03_south_plugins.rst | 1 + docs/06_plugins/04_north_plugins.rst | 619 ++++++++----------------- 4 files changed, 675 insertions(+), 447 deletions(-) create mode 100644 docs/06_plugins/03_01_DHT11.rst diff --git a/docs/06_plugins/02_writing_plugins.rst b/docs/06_plugins/02_writing_plugins.rst index a83e4f296f..cd2895f2db 100644 --- a/docs/06_plugins/02_writing_plugins.rst +++ b/docs/06_plugins/02_writing_plugins.rst @@ -7,9 +7,6 @@ .. Images -.. |DHT11 in PI| image:: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg - :target: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg - .. Links .. _here: 05_testing.html#setting-the-omf-translator-plugin .. _these steps: 04_installation.html @@ -20,34 +17,6 @@ .. Links in new tabs -.. |ADAFruit| raw:: html - - ADAFruit DHT Library - -.. |here BT| raw:: html - - here - -.. |DHT Description| raw:: html - - DHT11 Product Description - -.. |DHT Manual| raw:: html - - DHT11 Product Manual - -.. |DHT Resistor| raw:: html - - This picture - -.. |DHT Wired| raw:: html - - This picture - -.. |DHT Pins| raw:: html - - this - .. |GPIO| raw:: html here @@ -118,6 +87,8 @@ These are the properties returned by the JSON document: } } +|br| + Plugin Initialization --------------------- @@ -143,6 +114,8 @@ In a simple example of a sensor that reads a GPIO pin for data, we might choose handle = config['gpiopin']['value'] return handle +|br| + Plugin Reconfigure ------------------ @@ -170,6 +143,8 @@ Using a simple example of our sensor reading a GPIO pin, we extract the new pin return new_handle +|br| + Plugin Shutdown --------------- diff --git a/docs/06_plugins/03_01_DHT11.rst b/docs/06_plugins/03_01_DHT11.rst new file mode 100644 index 0000000000..89e74722c8 --- /dev/null +++ b/docs/06_plugins/03_01_DHT11.rst @@ -0,0 +1,465 @@ +.. Writing and Using Plugins describes how to implement a plugin for FogLAMP and how to use it +.. https://docs.google.com/document/d/1IKGXLWbyN6a7vx8UO3uDbq5Df0VvE4oCQIULgZVZbjM + +.. |br| raw:: html + +
+ +.. Images + +.. |DHT11 in PI| image:: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg + :target: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg + +.. Links +.. _here: 05_testing.html#setting-the-omf-translator-plugin +.. _these steps: 04_installation.html + +.. |Getting Started| raw:: html + + here + +.. Links in new tabs + +.. |ADAFruit| raw:: html + + ADAFruit DHT Library + +.. |here BT| raw:: html + + here + +.. |DHT Description| raw:: html + + DHT11 Product Description + +.. |DHT Manual| raw:: html + + DHT11 Product Manual + +.. |DHT Resistor| raw:: html + + This picture + +.. |DHT Wired| raw:: html + + This picture + +.. |DHT Pins| raw:: html + + this + +.. |GPIO| raw:: html + + here + + +.. ============================================= + + +A South Plugin Example: the DHT11 Sensor +---------------------------------------- + +Let's try to put all the information together and write a plugin. We can continue to use the example of an inexpensive sensor, the DHT11, used to measure temperature and humidity, directly wired to a Raspberry PI. This plugin is also available in the FogLAMP project on GitHub, in the *contrib* folder. + +First, here is a set of links where you can find more information regarding this sensor: + +- |DHT Description| +- |DHT Manual| +- |ADAFruit| + + +The Hardware +~~~~~~~~~~~~ + +The DHT sensor is directly connected to a Raspberry PI 2 or 3. You may decide to buy a sensor and a resistor and solder them yourself, or you can buy a ready-made circuit that provides the correct output to wire to the Raspberry PI. |DHT Resistor| shows a DHT11 with resistor that you can buy online. + +The sensor can be directly connected to the Raspberry PI GPIO (General Purpose Input/Output). An introduction to the GPIO and the pinset is available |GPIO|. In our case, you must connect the sensor on these pins: + +- **VCC** is connected to PIN #2 (5v Power) +- **GND** is connected to PIN #6 (Ground) +- **DATA** is connected to PIN #7 (BCM 4 - GPCLK0) + +|DHT Wired| shows the sensor wired to the Raspberry PI and |DHT Pins| is a zoom into the wires used. + + +The Software +~~~~~~~~~~~~ + +For this plugin we use the ADAFruit Python Library (links to the GitHub repository are above). First, you must install the library (in future versions the library will be provided in a ready-made package): + +.. code-block:: console + + $ git clone https://github.com/adafruit/Adafruit_Python_DHT.git + Cloning into 'Adafruit_Python_DHT'... + remote: Counting objects: 249, done. + remote: Total 249 (delta 0), reused 0 (delta 0), pack-reused 249 + Receiving objects: 100% (249/249), 77.00 KiB | 0 bytes/s, done. + Resolving deltas: 100% (142/142), done. + $ cd Adafruit_Python_DHT + $ sudo apt-get install build-essential python-dev + Reading package lists... Done + Building dependency tree + Reading state information... Done + The following NEW packages will be installed: + build-essential python-dev + ... + $ sudo python3 setup.py install + running install + running bdist_egg + running egg_info + creating Adafruit_DHT.egg-info + ... + $ + + +The Plugin +~~~~~~~~~~ + +This is the code for the plugin: + +.. code-block:: python + + """ Plugin for a DHT11 temperature and humidity sensor attached directly + to the GPIO pins of a Raspberry Pi + + This plugin uses the Adafruit DHT library, to install this perform + the following steps: + + git clone https://github.com/adafruit/Adafruit_Python_DHT.git + cd Adafruit_Python_DHT + sudo apt-get install build-essential python-dev + sudo python setup.py install + + To access the GPIO pins foglamp must be able to access /dev/gpiomem, + the default access for this is owner and group read/write. Either + FogLAMP must be added to the group or the permissions altered to + allow FogLAMP access to the device. + """ + + from datetime import datetime, timezone + import Adafruit_DHT + import uuid + import copy + + from foglamp.common import logger + from foglamp.services.south import exceptions + + __author__ = "Mark Riddoch" + __copyright__ = "Copyright (c) 2017 OSIsoft, LLC" + __license__ = "Apache 2.0" + __version__ = "${VERSION}" + + _DEFAULT_CONFIG = { + 'plugin': { + 'description': 'Python module name of the plugin to load', + 'type': 'string', + 'default': 'dht11pi' + }, + 'pollInterval': { + 'description': 'The interval between poll calls to the device poll routine expressed in milliseconds.', + 'type': 'integer', + 'default': '1000' + }, + 'gpiopin': { + 'description': 'The GPIO pin into which the DHT11 data pin is connected', + 'type': 'integer', + 'default': '4' + } + + } + + _LOGGER = logger.setup(__name__) + """ Setup the access to the logging system of FogLAMP """ + + def plugin_info(): + """ Returns information about the plugin. + + Args: + Returns: + dict: plugin information + Raises: + """ + + return { + 'name': 'DHT11 GPIO', + 'version': '1.0', + 'mode': 'poll', + 'type': 'south', + 'interface': '1.0', + 'config': _DEFAULT_CONFIG + } + + + def plugin_init(config): + """ Initialise the plugin. + + Args: + config: JSON configuration document for the device configuration category + Returns: + handle: JSON object to be used in future calls to the plugin + Raises: + """ + + handle = config['gpiopin']['value'] + return handle + + + def plugin_poll(handle): + """ Extracts data from the sensor and returns it in a JSON document as a Python dict. + + Available for poll mode only. + + Args: + handle: handle returned by the plugin initialisation call + Returns: + returns a sensor reading in a JSON document, as a Python dict, if it is available + None - If no reading is available + Raises: + DataRetrievalError + """ + + try: + humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, handle) + if humidity is not None and temperature is not None: + time_stamp = str(datetime.now(tz=timezone.utc)) + readings = { 'temperature': temperature , 'humidity' : humidity } + wrapper = { + 'asset': 'dht11', + 'timestamp': time_stamp, + 'key': str(uuid.uuid4()), + 'readings': readings + } + return wrapper + else: + return None + + except Exception as ex: + raise exceptions.DataRetrievalError(ex) + + return None + + + def plugin_reconfigure(handle, new_config): + """ Reconfigures the plugin, it should be called when the configuration of the plugin is changed during the + operation of the device service. + The new configuration category should be passed. + + Args: + handle: handle returned by the plugin initialisation call + new_config: JSON object representing the new configuration category for the category + Returns: + new_handle: new handle to be used in the future calls + Raises: + """ + + new_handle = new_config['gpiopin']['value'] + return new_handle + + + def plugin_shutdown(handle): + """ Shutdowns the plugin doing required cleanup, to be called prior to the device service being shut down. + + Args: + handle: handle returned by the plugin initialisation call + Returns: + Raises: + """ + + +The configuration +~~~~~~~~~~~~~~~~~ + +Since the plugin is still experimental, it works only in a build environment, the snap version will be available in the next release. + +The configuration must be set manually in the FogLAMP metadata. in the repository, the file *cmds.sql* in the *contrib/plugins/south/dht11pi* folder must be executed with *psql* (or another PostgreSQL client) to add the configuration to the FogLAMP metadata. + +Let's see the SQL commands: + +.. code-block:: sql + + --- Create the South service instannce + INSERT INTO foglamp.scheduled_processes ( name, script ) + VALUES ( 'dht11pi', '["services/south"]'); + + --- Add the schedule to start the service at system startup + INSERT INTO foglamp.schedules ( id, schedule_name, process_name, schedule_type,schedule_interval, exclusive ) + VALUES ( '543a59ce-a9ca-11e7-abc4-cec278b6b11a', 'device', 'dht11pi', 1, '0:0', true ); + + --- Insert the config needed to load the plugin + INSERT INTO foglamp.configuration ( key, description, value ) + VALUES ( 'dht11pi', 'DHT11 on Raspberry Pi Configuration', + '{"plugin" : { "type" : "string", "value" : "dht11pi", "default" : "dht11pi", "description" : "Plugin to load" } }' ); + + +Building FogLAMP and Adding the Plugin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you have not built FogLAMP yet, follow the steps described |Getting Started|. After the build, you can optionally install FogLAMP following `these steps`_. + +Once the Storage database has been setup, let's update the configurarion to include the new plugin: + +.. code-block:: console + + $ psql -d foglamp -f cmds.sql + INSERT 0 1 + INSERT 0 1 + INSERT 0 1 + $ + + +Now it is time to apply a workaround and include our new plugin. + +- If you intend to start and execute FogLAMP from the build folder: copy the structure of the *contrib* folder into the *python* folder: + +.. code-block:: console + + $ cd ~/FogLAMP + $ cp -R contrib/plugins python/foglamp/. + $ + +- If you have installed FogLAMP by executing ``sudo make install``, copy the structure of the *contrib* folder into the installed *python* folder: + +.. code-block:: console + + $ cd ~/FogLAMP + $ sudo cp -R contrib/plugins /usr/local/FogLAMP/python/foglamp/. + $ + +.. note:: If you have installed FogLAMP using an alternative *DESTDIR*, remember to add the path to the destination directory to the ``cp`` command. + + +Using the Plugin +~~~~~~~~~~~~~~~~ + +Now you are ready to use the DHT11 plugin. If stop and restart FogLAMP if it is already running, or start it now. + +- Starting FogLAMP from the build folder: + +.. code-block:: console + + $ cd ~/FogLAMP + $ export FOGLAMP_ROOT=$HOME/FogLAMP + $ scripts/foglamp start + Starting FogLAMP................ + FogLAMP started. + $ + + +- Starting FogLAMP from the installed folder: + +.. code-block:: console + + $ cd /usr/local/FogLAMP + $ bin/foglamp start + Starting FogLAMP................ + FogLAMP started. + $ + + +Let's see what we have collected so far: + +.. code-block:: console + + $ curl -s http://localhost:8081/foglamp/asset | jq + [ + { + "count": 158, + "asset_code": "dht11" + } + ] + $ + +Finally, let's extract some values: + +.. code-block:: console + + $ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq + [ + { + "timestamp": "2017-12-30 14:41:39.672", + "reading": { + "temperature": 19, + "humidity": 62 + } + }, + { + "timestamp": "2017-12-30 14:41:35.615", + "reading": { + "temperature": 19, + "humidity": 63 + } + }, + { + "timestamp": "2017-12-30 14:41:34.087", + "reading": { + "temperature": 19, + "humidity": 62 + } + }, + { + "timestamp": "2017-12-30 14:41:32.557", + "reading": { + "temperature": 19, + "humidity": 63 + } + }, + { + "timestamp": "2017-12-30 14:41:31.028", + "reading": { + "temperature": 19, + "humidity": 63 + } + } + ] + $ + + +Clearly we will not see many changes in temperature or humidity, unless we place our thumb on the sensor or we blow warm breathe on it :-) + +.. code-block:: console + + $ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq + [ + { + "timestamp": "2017-12-30 14:43:16.787", + "reading": { + "temperature": 25, + "humidity": 95 + } + }, + { + "timestamp": "2017-12-30 14:43:15.258", + "reading": { + "temperature": 25, + "humidity": 95 + } + }, + { + "timestamp": "2017-12-30 14:43:13.729", + "reading": { + "temperature": 24, + "humidity": 95 + } + }, + { + "timestamp": "2017-12-30 14:43:12.201", + "reading": { + "temperature": 24, + "humidity": 95 + } + }, + { + "timestamp": "2017-12-30 14:43:05.616", + "reading": { + "temperature": 22, + "humidity": 95 + } + } + ] + $ + +Needless to say, the North plugin will send the buffered data to the PI system using the PI Connector Relay OMF. Do not forget to set the correct IP address for the PI Connector Relay, as it is described `here`_. + +|DHT11 in PI| + + diff --git a/docs/06_plugins/03_south_plugins.rst b/docs/06_plugins/03_south_plugins.rst index f2cc21b37a..24699ada50 100644 --- a/docs/06_plugins/03_south_plugins.rst +++ b/docs/06_plugins/03_south_plugins.rst @@ -186,3 +186,4 @@ It would then respond to the HTTP request and return. Since the handler is embed return web.json_response(message) +.. include:: 03_01_DHT11.rst diff --git a/docs/06_plugins/04_north_plugins.rst b/docs/06_plugins/04_north_plugins.rst index 25e8d03e83..6fc697ff7e 100644 --- a/docs/06_plugins/04_north_plugins.rst +++ b/docs/06_plugins/04_north_plugins.rst @@ -1,5 +1,4 @@ -.. Writing and Using Plugins describes how to implement a plugin for FogLAMP and how to use it -.. https://docs.google.com/document/d/1IKGXLWbyN6a7vx8UO3uDbq5Df0VvE4oCQIULgZVZbjM +.. North Plugins .. |br| raw:: html @@ -7,459 +6,247 @@ .. Images -.. |DHT11 in PI| image:: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg - :target: https://s3.amazonaws.com/foglamp/readthedocs/images/06_dht11_tags_in_PI.jpg - .. Links -.. _here: 05_testing.html#setting-the-omf-translator-plugin -.. _these steps: 04_installation.html .. |Getting Started| raw:: html - here - .. Links in new tabs -.. |ADAFruit| raw:: html - - ADAFruit DHT Library - -.. |here BT| raw:: html - - here - -.. |DHT Description| raw:: html - - DHT11 Product Description - -.. |DHT Manual| raw:: html - - DHT11 Product Manual - -.. |DHT Resistor| raw:: html - - This picture - -.. |DHT Wired| raw:: html - - This picture - -.. |DHT Pins| raw:: html - - this - -.. |GPIO| raw:: html - - here - - .. ============================================= -A South Plugin Example: the DHT11 Sensor -======================================== +North Plugins +============= -Let's try to put all the information together and write a plugin. We can continue to use the example of an inexpensive sensor, the DHT11, used to measure temperature and humidity, directly wired to a Raspberry PI. This plugin is also available in the FogLAMP project on GitHub, in the *contrib* folder. +North plugins are used in North tasks and microservices to extract data buffered in FogLAMP and send it Northbound, i.e. to a server or a service in the Cloud or in an Enterprise data center. We currently have two North plugins, one to send data to an OSIsoft PI Server and one to the OSIsoft Cloud Service. -First, here is a set of links where you can find more information regarding this sensor: -- |DHT Description| -- |DHT Manual| -- |ADAFruit| +The OMF Plugin +-------------- +The OMF Plugin is used by a North task to send data to an OSIsoft PI server via a PI Connector Relay OMF. *OMF* stands for OSIsoft Message Format, it is the JSON format defined by OSIsoft to send IoT data to a PI server via a Connector Relay server. -The Hardware ------------- +The plugin is designed to send two streams of data: -The DHT sensor is directly connected to a Raspberry PI 2 or 3. You may decide to buy a sensor and a resistor and solder them yourself, or you can buy a ready-made circuit that provides the correct output to wire to the Raspberry PI. |DHT Resistor| shows a DHT11 with resistor that you can buy online. +- The data collected by South microservices and buffered into FogLAMP +- The statistics generated by FogLAMP -The sensor can be directly connected to the Raspberry PI GPIO (General Purpose Input/Output). An introduction to the GPIO and the pinset is available |GPIO|. In our case, you must connect the sensor on these pins: - -- **VCC** is connected to PIN #2 (5v Power) -- **GND** is connected to PIN #6 (Ground) -- **DATA** is connected to PIN #7 (BCM 4 - GPCLK0) - -|DHT Wired| shows the sensor wired to the Raspberry PI and |DHT Pins| is a zoom into the wires used. - - -The Software ------------- - -For this plugin we use the ADAFruit Python Library (links to the GitHub repository are above). First, you must install the library (in future versions the library will be provided in a ready-made package): +The streams are managed by two different North tasks using the same plugin, but with a different configuration. The two tasks are registered in the list of scheduled jobs and they can be identified using the ``schedule`` API call: .. code-block:: console - - $ git clone https://github.com/adafruit/Adafruit_Python_DHT.git - Cloning into 'Adafruit_Python_DHT'... - remote: Counting objects: 249, done. - remote: Total 249 (delta 0), reused 0 (delta 0), pack-reused 249 - Receiving objects: 100% (249/249), 77.00 KiB | 0 bytes/s, done. - Resolving deltas: 100% (142/142), done. - $ cd Adafruit_Python_DHT - $ sudo apt-get install build-essential python-dev - Reading package lists... Done - Building dependency tree - Reading state information... Done - The following NEW packages will be installed: - build-essential python-dev - ... - $ sudo python3 setup.py install - running install - running bdist_egg - running egg_info - creating Adafruit_DHT.egg-info - ... - $ - - -The Plugin ----------- - -This is the code for the plugin: - -.. code-block:: python - - """ Plugin for a DHT11 temperature and humidity sensor attached directly - to the GPIO pins of a Raspberry Pi - - This plugin uses the Adafruit DHT library, to install this perform - the following steps: - - git clone https://github.com/adafruit/Adafruit_Python_DHT.git - cd Adafruit_Python_DHT - sudo apt-get install build-essential python-dev - sudo python setup.py install - - To access the GPIO pins foglamp must be able to access /dev/gpiomem, - the default access for this is owner and group read/write. Either - FogLAMP must be added to the group or the permissions altered to - allow FogLAMP access to the device. - """ - - from datetime import datetime, timezone - import Adafruit_DHT - import uuid - import copy - - from foglamp.common import logger - from foglamp.services.south import exceptions - - __author__ = "Mark Riddoch" - __copyright__ = "Copyright (c) 2017 OSIsoft, LLC" - __license__ = "Apache 2.0" - __version__ = "${VERSION}" - - _DEFAULT_CONFIG = { - 'plugin': { - 'description': 'Python module name of the plugin to load', - 'type': 'string', - 'default': 'dht11pi' - }, - 'pollInterval': { - 'description': 'The interval between poll calls to the device poll routine expressed in milliseconds.', - 'type': 'integer', - 'default': '1000' - }, - 'gpiopin': { - 'description': 'The GPIO pin into which the DHT11 data pin is connected', - 'type': 'integer', - 'default': '4' - } + $ curl -sX GET http://locahost:8081/foglamp/schedule + { + "schedules": [ + { "day": null, + "time": 0, + "name": "OMF to PI north", + "exclusive": true, + "processName": "North Readings to PI", + "enabled": false, + "type": "INTERVAL", + "repeat": 30, + "id": "2b614d26-760f-11e7-b5a5-be2e44b06b34" }, + { "day": null, + "time": 0, + "name": "Stats OMF to PI north", + "exclusive": true, + "processName": "North Statistics to PI", + "enabled": false, + "type": "INTERVAL", + "repeat": 30, + "id": "1d7c327e-7dae-11e7-bb31-be2e44b06b34" }, + ... + ] } - _LOGGER = logger.setup(__name__) - """ Setup the access to the logging system of FogLAMP """ - - def plugin_info(): - """ Returns information about the plugin. - - Args: - Returns: - dict: plugin information - Raises: - """ - - return { - 'name': 'DHT11 GPIO', - 'version': '1.0', - 'mode': 'poll', - 'type': 'south', - 'interface': '1.0', - 'config': _DEFAULT_CONFIG - } - - - def plugin_init(config): - """ Initialise the plugin. - - Args: - config: JSON configuration document for the device configuration category - Returns: - handle: JSON object to be used in future calls to the plugin - Raises: - """ - - handle = config['gpiopin']['value'] - return handle - - - def plugin_poll(handle): - """ Extracts data from the sensor and returns it in a JSON document as a Python dict. - - Available for poll mode only. - - Args: - handle: handle returned by the plugin initialisation call - Returns: - returns a sensor reading in a JSON document, as a Python dict, if it is available - None - If no reading is available - Raises: - DataRetrievalError - """ - - try: - humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.DHT11, handle) - if humidity is not None and temperature is not None: - time_stamp = str(datetime.now(tz=timezone.utc)) - readings = { 'temperature': temperature , 'humidity' : humidity } - wrapper = { - 'asset': 'dht11', - 'timestamp': time_stamp, - 'key': str(uuid.uuid4()), - 'readings': readings - } - return wrapper - else: - return None - - except Exception as ex: - raise exceptions.DataRetrievalError(ex) - - return None - - - def plugin_reconfigure(handle, new_config): - """ Reconfigures the plugin, it should be called when the configuration of the plugin is changed during the - operation of the device service. - The new configuration category should be passed. - - Args: - handle: handle returned by the plugin initialisation call - new_config: JSON object representing the new configuration category for the category - Returns: - new_handle: new handle to be used in the future calls - Raises: - """ - - new_handle = new_config['gpiopin']['value'] - return new_handle - - - def plugin_shutdown(handle): - """ Shutdowns the plugin doing required cleanup, to be called prior to the device service being shut down. - - Args: - handle: handle returned by the plugin initialisation call - Returns: - Raises: - """ - - -The configuration ------------------ - -Since the plugin is still experimental, it works only in a build environment, the snap version will be available in the next release. - -The configuration must be set manually in the FogLAMP metadata. in the repository, the file *cmds.sql* in the *contrib/plugins/south/dht11pi* folder must be executed with *psql* (or another PostgreSQL client) to add the configuration to the FogLAMP metadata. +The output of API call above shows three interesting tasks: the two tasks associated to the OMF plugin, the one to send data (*OMF to PI north*) and the one to send statistics (*North Statistics to PI*). -Let's see the SQL commands: - -.. code-block:: sql - - --- Create the South service instannce - INSERT INTO foglamp.scheduled_processes ( name, script ) - VALUES ( 'dht11pi', '["services/south"]'); - - --- Add the schedule to start the service at system startup - INSERT INTO foglamp.schedules ( id, schedule_name, process_name, schedule_type,schedule_interval, exclusive ) - VALUES ( '543a59ce-a9ca-11e7-abc4-cec278b6b11a', 'device', 'dht11pi', 1, '0:0', true ); - - --- Insert the config needed to load the plugin - INSERT INTO foglamp.configuration ( key, description, value ) - VALUES ( 'dht11pi', 'DHT11 on Raspberry Pi Configuration', - '{"plugin" : { "type" : "string", "value" : "dht11pi", "default" : "dht11pi", "description" : "Plugin to load" } }' ); - - -Building FogLAMP and Adding the Plugin --------------------------------------- - -If you have not built FogLAMP yet, follow the steps described |Getting Started|. After the build, you can optionally install FogLAMP following `these steps`_. - -Once the Storage database has been setup, let's update the configurarion to include the new plugin: - -.. code-block:: console - - $ psql -d foglamp -f cmds.sql - INSERT 0 1 - INSERT 0 1 - INSERT 0 1 - $ - - -Now it is time to apply a workaround and include our new plugin. - -- If you intend to start and execute FogLAMP from the build folder: copy the structure of the *contrib* folder into the *python* folder: - -.. code-block:: console - - $ cd ~/FogLAMP - $ cp -R contrib/plugins python/foglamp/. - $ - -- If you have installed FogLAMP by executing ``sudo make install``, copy the structure of the *contrib* folder into the installed *python* folder: - -.. code-block:: console - - $ cd ~/FogLAMP - $ sudo cp -R contrib/plugins /usr/local/FogLAMP/python/foglamp/. - $ - -.. note:: If you have installed FogLAMP using an alternative *DESTDIR*, remember to add the path to the destination directory to the ``cp`` command. - - -Using the Plugin ----------------- - -Now you are ready to use the DHT11 plugin. If stop and restart FogLAMP if it is already running, or start it now. - -- Starting FogLAMP from the build folder: +The two scheduled tasks are associated to two configuration items that can be retrieved using the ``category`` API call. The the items are named ``SEND_PR_1`` and ``SEND_PR_2``. .. code-block:: console - $ cd ~/FogLAMP - $ export FOGLAMP_ROOT=$HOME/FogLAMP - $ scripts/foglamp start - Starting FogLAMP................ - FogLAMP started. + $ curl -sX GET http://localhost:8081/foglamp/category/SEND_PR_1 + { "plugin": { "type": "string", + "default": "omf", + "value": "omf", + "description": "Python module name of the plugin to load" } + } + $ curl -sX GET http://localhost:8081/foglamp/category/SEND_PR_2 + { "plugin": { "type": "string", + "default": "omf", + "value": "omf", + "description": "Python module name of the plugin to load" } + } $ - -- Starting FogLAMP from the installed folder: +In order to activate the tasks, you must change their status. First you must collect their id (from the GET method of the ``schedule`` API call), then you must use the IDs with the PUT method of the same call: .. code-block:: console - $ cd /usr/local/FogLAMP - $ bin/foglamp start - Starting FogLAMP................ - FogLAMP started. + $ curl -sX PUT http://vbox-dev:8081/foglamp/schedule/2b614d26-760f-11e7-b5a5-be2e44b06b34 -d '{ "enabled" : true}' + { "schedule": { "day": null, + "time": 0, + "name": "OMF to PI north", + "exclusive": true, + "processName": "North Readings to PI", + "enabled": true, + "type": "INTERVAL", + "repeat": 30.0, + "id": "2b614d26-760f-11e7-b5a5-be2e44b06b34" } + } + $ curl -sX PUT http://vbox-dev:8081/foglamp/schedule/1d7c327e-7dae-11e7-bb31-be2e44b06b34 -d '{ "enabled" : true}' + { "schedule": { "day": null, + "time": 0, + "name": "Stats OMF to PI north", + "exclusive": true, + "processName": "North Statistics to PI", + "enabled": true, + "type": "INTERVAL", + "repeat": 30.0, + "id": "1d7c327e-7dae-11e7-bb31-be2e44b06b34" } + } $ -Let's see what we have collected so far: - -.. code-block:: console - - $ curl -s http://localhost:8081/foglamp/asset | jq - [ - { - "count": 158, - "asset_code": "dht11" - } - ] - $ - -Finally, let's extract some values: +At this point, the configuration has been enriched with default values of the tasks: .. code-block:: console - $ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq - [ - { - "timestamp": "2017-12-30 14:41:39.672", - "reading": { - "temperature": 19, - "humidity": 62 - } - }, - { - "timestamp": "2017-12-30 14:41:35.615", - "reading": { - "temperature": 19, - "humidity": 63 - } - }, - { - "timestamp": "2017-12-30 14:41:34.087", - "reading": { - "temperature": 19, - "humidity": 62 - } - }, - { - "timestamp": "2017-12-30 14:41:32.557", - "reading": { - "temperature": 19, - "humidity": 63 - } - }, - { - "timestamp": "2017-12-30 14:41:31.028", - "reading": { - "temperature": 19, - "humidity": 63 - } - } - ] + $ curl -sX GET http://vbox-dev:8081/foglamp/category/SEND_PR_1 + { "filterRule": { + "type": "string", + "default": ".[]", + "description": "JQ formatted filter to apply (applicable if applyFilter is True)", + "value": ".[]" }, + "plugin": { "type": "string", + "default": "omf", + "description": "OMF North Plugin", + "value": "omf" }, + ... + } $ - - -Clearly we will not see many changes in temperature or humidity, unless we place our thumb on the sensor or we blow warm breathe on it :-) - -.. code-block:: console - - $ curl -s http://localhost:8081/foglamp/asset/dht11?limit=5 | jq - [ - { - "timestamp": "2017-12-30 14:43:16.787", - "reading": { - "temperature": 25, - "humidity": 95 - } - }, - { - "timestamp": "2017-12-30 14:43:15.258", - "reading": { - "temperature": 25, - "humidity": 95 - } - }, - { - "timestamp": "2017-12-30 14:43:13.729", - "reading": { - "temperature": 24, - "humidity": 95 - } - }, - { - "timestamp": "2017-12-30 14:43:12.201", - "reading": { - "temperature": 24, - "humidity": 95 - } - }, - { - "timestamp": "2017-12-30 14:43:05.616", - "reading": { - "temperature": 22, - "humidity": 95 - } - } - ] + $ curl -sX GET http://vbox-dev:8081/foglamp/category/SEND_PR_2 + { "URL": { + "type": "string", + "default": "https://pi-server:5460/ingress/messages", + "value": "https://pi-server:5460/ingress/messages", + "description": "The URL of the PI Connector to send data to" }, + "filterRule": { + "type": "string", + "default": ".[]", + "value": ".[]", + "description": "JQ formatted filter to apply (applicable if applyFilter is True)" }, + ... $ -Needless to say, the North plugin will send the buffered data to the PI system using the PI Connector Relay OMF. Do not forget to set the correct IP address for the PI Connector Relay, as it is described `here`_. -|DHT11 in PI| +OMF Plugin Configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +The following table presents the list of configuration options available for the task that sends data to OMF (category *SEND_PR_1*): + ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| Item | Type | Default | Description | ++===================+==========+=========================================+========================================================+ +| applyFilter | boolean | False | Whether to apply filter before processing the data | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| blockSize | integer | 500 | The size of a block of readings |br| | +| | | | to send in each transmission. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| duration | integer | 60 | How long the sending process should run |br| | +| | | | (in seconds) before stopping | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| enable | boolean | True | A switch that can be used to enable or disable |br| | +| | | | execution of the sending process. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| filterRule | string | .[] | JQ formatted filter to apply |br| | +| | | | (applicable if applyFilter is True) | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| north | string | omf | The name of the north to use to translate the |br| | +| | | | readings into the output format and send them | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| OMFHttpTimeout | interger | 10 | Timeout in seconds for the HTTP operations |br| | +| | | | with the OMF PI Connector Relay | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| OMFMaxRetry | interger | 3 | Max number of retries for the communication |br| | +| | | | with the OMF PI Connector Relay | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| OMFRetrySleepTime | interger | 10 | Seconds between each retry for the communication |br| | +| | | | with the OMF PI Connector Relay, |br| | +| | | | NOTE : the time is doubled at each attempt. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| plugin | string | omf | OMF North Plugin name | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| producerToken | string | omf_north_0001 | The producer token that represents this FogLAMP stream | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| sleepInterval | integer | 5 | A period of time, expressed in seconds, to wait |br| | +| | | | between attempts to send readings when there are |br| | +| | | | no readings to be sent. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| source | string | readings | Defines the source of the data to be sent |br| | +| | | | the stream, this may be one of either |br| | +| | | | readings, statistics or audit. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| StaticData | JSON | { "Location" : "Palo Alto", | Static data to include in each sensor reading |br| | +| | | "Company" : "Dianomic" } | sent to OMF. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| stream_id | integer | 1 | Stream ID | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| URL | string | https://pi-server:5460/ingress/messages | The URL of the PI Connector to send data to | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ + + +The following table presents the list of configuration options available for the task that sends statistics to OMF (category *SEND_PR_2*): + ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| Item | Type | Default | Description | ++===================+==========+=========================================+========================================================+ +| applyFilter | boolean | False | Whether to apply filter before processing the data | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| blockSize | integer | 500 | The size of a block of readings |br| | +| | | | to send in each transmission. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| duration | integer | 60 | How long the sending process should run |br| | +| | | | (in seconds) before stopping | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| enable | boolean | True | A switch that can be used to enable or disable |br| | +| | | | execution of the sending process. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| filterRule | string | .[] | JQ formatted filter to apply |br| | +| | | | (applicable if applyFilter is True) | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| north | string | omf | The name of the north to use to translate the |br| | +| | | | readings into the output format and send them | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| OMFHttpTimeout | interger | 10 | Timeout in seconds for the HTTP operations |br| | +| | | | with the OMF PI Connector Relay | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| OMFMaxRetry | interger | 3 | Max number of retries for the communication |br| | +| | | | with the OMF PI Connector Relay | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| OMFRetrySleepTime | interger | 10 | Seconds between each retry for the communication |br| | +| | | | with the OMF PI Connector Relay, |br| | +| | | | NOTE : the time is doubled at each attempt. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| plugin | string | omf | OMF North Plugin name | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| producerToken | string | omf_north_0001 | The producer token that represents this FogLAMP stream | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| sleepInterval | integer | 5 | A period of time, expressed in seconds, to wait |br| | +| | | | between attempts to send readings when there are |br| | +| | | | no readings to be sent. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| source | string | readings | Defines the source of the data to be sent |br| | +| | | | the stream, this may be one of either |br| | +| | | | readings, statistics or audit. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| StaticData | JSON | { "Location" : "Palo Alto", | Static data to include in each sensor reading |br| | +| | | "Company" : "Dianomic" } | sent to OMF. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| stream_id | integer | 2 | Stream ID | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| URL | string | https://pi-server:5460/ingress/messages | The URL of the PI Connector to send data to | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ From df5dd2f1236b7a2c94549faad81f6243f5943651 Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Sat, 12 May 2018 01:11:40 +0100 Subject: [PATCH 04/12] 06 plugins north --- docs/06_plugins/04_north_plugins.rst | 41 +++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/docs/06_plugins/04_north_plugins.rst b/docs/06_plugins/04_north_plugins.rst index 6fc697ff7e..c798a9e597 100644 --- a/docs/06_plugins/04_north_plugins.rst +++ b/docs/06_plugins/04_north_plugins.rst @@ -10,6 +10,8 @@ .. |Getting Started| raw:: html + here + .. Links in new tabs .. ============================================= @@ -188,8 +190,8 @@ The following table presents the list of configuration options available for the | | | | the stream, this may be one of either |br| | | | | | readings, statistics or audit. | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ -| StaticData | JSON | { "Location" : "Palo Alto", | Static data to include in each sensor reading |br| | -| | | "Company" : "Dianomic" } | sent to OMF. | +| StaticData | JSON | ``{ "Location" : "Palo Alto",`` |br| | Static data to include in each sensor reading |br| | +| | | ``"Company" : "Dianomic" }`` | sent to OMF. | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ | stream_id | integer | 1 | Stream ID | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ @@ -241,8 +243,8 @@ The following table presents the list of configuration options available for the | | | | the stream, this may be one of either |br| | | | | | readings, statistics or audit. | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ -| StaticData | JSON | { "Location" : "Palo Alto", | Static data to include in each sensor reading |br| | -| | | "Company" : "Dianomic" } | sent to OMF. | +| StaticData | JSON | ``{ "Location" : "Palo Alto",`` |br| | Static data to include in each sensor reading |br| | +| | | ``"Company" : "Dianomic" }`` | sent to OMF. | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ | stream_id | integer | 2 | Stream ID | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ @@ -250,3 +252,34 @@ The following table presents the list of configuration options available for the +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +The last parameter to review is the *OMF Type*. The call is the GET method ``foglamp/category/OMF_TYPES``, which returns an integer value that identifies the measurement type: + + +.. code-block:: console + + $ curl -sX GET http://vbox-dev:8081/foglamp/category/OMF_TYPES + { + "type-id": { + "description": "Identify sensor and measurement types", + "type": "integer", + "default": "0001", + "value": "0001" + } + } + $ + +If you change the value, you can easily identify the set of data sent to and then stored into PI. + + + +Changing the OMF Plugin Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before you send date to the PI server, it is extremely likey that you need to apply more changes to the configuration. The most important items to change are: + +- **URL** : the URL to the PI Connector Relay OMF. It is usually composed by the name or address of the Windows server where the Connector Relay service is running, the port associated to the service and the ingress/messages API call. The communication is via HTTPS protocol. +- **producerToken** : the token provided by the Data Collection Manager when the PI administrator sets the use of FogLAMP. +- **type-id** : the measurement type for the stream of data. + + + From 8ea4c14e1e9c18b1efc9e809fa45025a91428892 Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Mon, 14 May 2018 19:28:11 +0100 Subject: [PATCH 05/12] fixed wrong value in table --- docs/06_plugins/04_north_plugins.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/06_plugins/04_north_plugins.rst b/docs/06_plugins/04_north_plugins.rst index c798a9e597..c292f74c19 100644 --- a/docs/06_plugins/04_north_plugins.rst +++ b/docs/06_plugins/04_north_plugins.rst @@ -174,7 +174,7 @@ The following table presents the list of configuration options available for the | OMFMaxRetry | interger | 3 | Max number of retries for the communication |br| | | | | | with the OMF PI Connector Relay | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ -| OMFRetrySleepTime | interger | 10 | Seconds between each retry for the communication |br| | +| OMFRetrySleepTime | interger | 1 | Seconds between each retry for the communication |br| | | | | | with the OMF PI Connector Relay, |br| | | | | | NOTE : the time is doubled at each attempt. | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ @@ -227,7 +227,7 @@ The following table presents the list of configuration options available for the | OMFMaxRetry | interger | 3 | Max number of retries for the communication |br| | | | | | with the OMF PI Connector Relay | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ -| OMFRetrySleepTime | interger | 10 | Seconds between each retry for the communication |br| | +| OMFRetrySleepTime | interger | 1 | Seconds between each retry for the communication |br| | | | | | with the OMF PI Connector Relay, |br| | | | | | NOTE : the time is doubled at each attempt. | +-------------------+----------+-----------------------------------------+--------------------------------------------------------+ From 9a3aea985e91af28110142e42ecf65d500a3519d Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Mon, 14 May 2018 19:45:29 +0100 Subject: [PATCH 06/12] Fixed some missed links --- docs/03_getting_started.rst | 3 ++- docs/06_plugins/04_north_plugins.rst | 8 ++------ 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/docs/03_getting_started.rst b/docs/03_getting_started.rst index b8f24e5e7e..732cb67325 100644 --- a/docs/03_getting_started.rst +++ b/docs/03_getting_started.rst @@ -62,7 +62,8 @@ You may also want to install some utilities to make your life easier when you us Building FogLAMP ================ -In this section we will describe how to build FogLAMP on Ubuntu 16.04.3 LTS (Server or Desktop). Other Linux distributions, Debian or Red-Hat based, or even other versions of Ubuntu may differ. If you are not familiar with Linux and you do not want to build FogLAMP from the source code, you can download a snap package from |snappy|. +In this section we will describe how to build FogLAMP on Ubuntu 16.04.3 LTS (Server or Desktop). Other Linux distributions, Debian or Red-Hat based, or even other versions of Ubuntu may differ. If you are not familiar with Linux and you do not want to build FogLAMP from the source code, you can download a ready-made Debian package (the list of packages is `here <92_downloads.html>`_). + Build Pre-Requisites -------------------- diff --git a/docs/06_plugins/04_north_plugins.rst b/docs/06_plugins/04_north_plugins.rst index c292f74c19..5eaf56a34b 100644 --- a/docs/06_plugins/04_north_plugins.rst +++ b/docs/06_plugins/04_north_plugins.rst @@ -8,10 +8,6 @@ .. Links -.. |Getting Started| raw:: html - - here - .. Links in new tabs .. ============================================= @@ -275,11 +271,11 @@ If you change the value, you can easily identify the set of data sent to and the Changing the OMF Plugin Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Before you send date to the PI server, it is extremely likey that you need to apply more changes to the configuration. The most important items to change are: +Before you send data to the PI server, it is likey that you need to apply more changes to the configuration. The most important items to change are: - **URL** : the URL to the PI Connector Relay OMF. It is usually composed by the name or address of the Windows server where the Connector Relay service is running, the port associated to the service and the ingress/messages API call. The communication is via HTTPS protocol. - **producerToken** : the token provided by the Data Collection Manager when the PI administrator sets the use of FogLAMP. - **type-id** : the measurement type for the stream of data. - +An example of the changes to apply to the plugins to send data to the PI system is available here `here <../05_testing.html#sending-greetings-to-the-northern-hemisphere>`_. From 0a1f7dd304da7a2d242359cd9dafce24083b397b Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Mon, 14 May 2018 23:36:30 +0100 Subject: [PATCH 07/12] OMF Communication --- docs/06_plugins/04_north_plugins.rst | 54 +++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/docs/06_plugins/04_north_plugins.rst b/docs/06_plugins/04_north_plugins.rst index 5eaf56a34b..efb6230b24 100644 --- a/docs/06_plugins/04_north_plugins.rst +++ b/docs/06_plugins/04_north_plugins.rst @@ -267,7 +267,6 @@ The last parameter to review is the *OMF Type*. The call is the GET method ``fog If you change the value, you can easily identify the set of data sent to and then stored into PI. - Changing the OMF Plugin Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -279,3 +278,56 @@ Before you send data to the PI server, it is likey that you need to apply more c An example of the changes to apply to the plugins to send data to the PI system is available here `here <../05_testing.html#sending-greetings-to-the-northern-hemisphere>`_. + +Data in the PI System +~~~~~~~~~~~~~~~~~~~~~ + +Once the North plugins have been set properly, you should expect to see data automatically sent and stored in the PI Server. More specifically, the process of the plugin is the following: + +- **Assets** buffered in FogLAMP are stored as *elements* in the PI System. + - *PI Asset Framework* is automatically update with the new assets. + - JSON objects captured as part of the *reading* in FogLAMP become *attributes* in the PI Data Archive +- The **Producer Token** is used to authenticate and create the hierarchy of elements in the *PI Asset Framework* +- The configuration object named as **Static Data** is added as a set of *attributes* in the PI Data Archive + ++-----------+--------------------+--------------------------------------------------------------------------+ +| System | Object | Value | ++===========+====================+==========================================================================+ +| FogLAMP | Producer Token | readings_001 | +| +--------------------+--------------------------------------------------------------------------+ +| | OMF Type | 0001 | +| +--------------------+--------------------------------------------------------------------------+ +| | Static Data | { "Company" : "Dianomic", "Location" : "Palo Alto" } | +| +--------------------+--------------------------------------------------------------------------+ +| | Asset | fogbench/accelerometer | +| +--------------------+--------------------------------------------------------------------------+ +| | Reading | [{"reading":{"y":1,"z":1,"x":-1}, "timestamp":"2018-05-14 19:27:06.788}] | ++-----------+--------------------+--------------------------------------------------------------------------+ +| PI System | Element Template | [OMF.readings_001 Connector.0001_fogbench/accelerometer_typename_sensor] | +| +--------------------+----------+---------------------------------------------------------------+ +| | Attribute Template | Company | Configuration Item, Excluded, String | +| | +----------+---------------------------------------------------------------+ +| | | Location | Configuration Item, Excluded, String | +| | +----------+---------------------------------------------------------------+ +| | | x | Excluded, Int64 | +| | +----------+---------------------------------------------------------------+ +| | | y | Excluded, Int64 | +| | +----------+---------------------------------------------------------------+ +| | | z | Excluded, Int64 | +| +--------------------+----------+---------------------------------------------------------------+ +| | Element | foglamp > readings_001 > fogbench/accelerometer | +| +--------------------+----------+---------------+-----------------------------------------------+ +| | Attributes | **Name** | **Value** | **Timestamp** | +| | +----------+---------------+-----------------------------------------------+ +| | | Company | Dianomic | 1970-01-01 00:00:00 | ++ | +----------+---------------+-----------------------------------------------+ +| | | Location | Palo Alto | 1970-01-01 00:00:00 | ++ | +----------+---------------+-----------------------------------------------+ +| | | x | -1 | 2018-05-14 19:27:06.788 | ++ | +----------+---------------+-----------------------------------------------+ +| | | y | 1 | 2018-05-14 19:27:06.788 | ++ | +----------+---------------+-----------------------------------------------+ +| | | z | 1 | 2018-05-14 19:27:06.788 | ++-----------+--------------------+----------+---------------+-----------------------------------------------+ + + From 9b37333659315c300b0e126eea9692c376ed25f6 Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Tue, 15 May 2018 00:46:16 +0100 Subject: [PATCH 08/12] Storage Plugins --- docs/02_foglamp_at_a_glance.rst | 6 +-- docs/06_plugins/05_storage_plugins.rst | 56 ++++++++++++++++++++++++++ docs/06_plugins/index.rst | 1 + 3 files changed, 60 insertions(+), 3 deletions(-) create mode 100644 docs/06_plugins/05_storage_plugins.rst diff --git a/docs/02_foglamp_at_a_glance.rst b/docs/02_foglamp_at_a_glance.rst index f93e34b7cc..8232f35784 100644 --- a/docs/02_foglamp_at_a_glance.rst +++ b/docs/02_foglamp_at_a_glance.rst @@ -52,7 +52,7 @@ Details of the Architecture - **Multi-tenancy** for external entities (applications, user sessions, devices) and internal entities (scheduled tasks, running services). Entities are identified as *tenants*. FogLAMP can serve multiple tenants at the same time. Tenants have access to a defined set of resources (data and operations) that can be shared or exclusive. - **Federation** of local and distributed data and metadata for tenants. Tenants may have access to a whole set of data or only part of it. - **Provisioning & Updates** for FogLAMP, operated automatically, unattended and securely. Updates can be applied to a running FogLAMP without service disruption and in case of issues they can be reverted. In a cluster installation, updates are performed without downtime. - - **Security** applied as plugins, handled centrally as a service for all the components that require secure communication and authentication for data in motion and at rest. |br|:w! + - **Security** applied as plugins, handled centrally as a service for all the components that require secure communication and authentication for data in motion and at rest. |br| The figure below shows where the security plugins can operate. |foglamp_security| |br| |br| - **Data Transformation**: it is a set of multiple plugins that can be loaded/unloaded, activated/deactivated and scheduled to transform stored data (data at rest), based on functions that are executed as external modules. Transformation plugins can also be used to filter or transform data and metadata received or to be sent (data in motion). |br| The figure below shows where the transformation plugins can operate. |foglamp_transformation| |br| |br| - **Monitoring**: it is a set of multiple plugins that can be loaded/unloaded, activated/deactivated and scheduled to monitor: @@ -92,8 +92,8 @@ Details of the Architecture |br| |br| - **Storage Layer** - A microservice that offers storage, either transient or permanent, optionally resilient and/or transactional. The type of storage is pluggable and extendible, i.e. the model can be extended to provide specific features used by other plugins. For example, in installations with a small footprint, a plugin for SQLite may be chosen, in installations with a high number of concurrent requests and larger footprint a plugin for more feature-rich databases may be suitable. In micro installations, for example on Edge devices, an in-memory temporary storage may be the best option. A *pass-through* option can also reduce the latency of data transiting in FogLAMP, especially when the northbound or east/westbound destination is reachable via a stable network. Data and Metadata may be handled in different ways, for example when metadata is persistent and data only transient. |br| |br| -- **Northbound microservice** - A microservice that offers bi-directional communication with data and metadata exchange between the platform and larger systems and databases in the Cloud or in data centers. Larger systems may be private and public Cloud data services, proprietary solutions or FogLAMP instances with larger footprint, optionally distributed on multiple servers, geographically or in the data center. |br| |br| -- **Southbound microservice** - A microservice that offers bi-directional communication with data and metadata exchange between the platform and Edge devices, such as sensors, actuators, PLCs or other FogLAMP installations. Smaller systems may have this service installed on board Edge devices. +- **Northbound Microservice** - A microservice that offers bi-directional communication with data and metadata exchange between the platform and larger systems and databases in the Cloud or in data centers. Larger systems may be private and public Cloud data services, proprietary solutions or FogLAMP instances with larger footprint, optionally distributed on multiple servers, geographically or in the data center. |br| |br| +- **Southbound Microservice** - A microservice that offers bi-directional communication with data and metadata exchange between the platform and Edge devices, such as sensors, actuators, PLCs or other FogLAMP installations. Smaller systems may have this service installed on board Edge devices. To Know More About the Archicture diff --git a/docs/06_plugins/05_storage_plugins.rst b/docs/06_plugins/05_storage_plugins.rst new file mode 100644 index 0000000000..67c397e62f --- /dev/null +++ b/docs/06_plugins/05_storage_plugins.rst @@ -0,0 +1,56 @@ +.. Storage Plugins + +.. |br| raw:: html + +
+ +.. Images + +.. Links + +.. Links in new tabs + +.. ============================================= + + +Storage Plugins +=============== + +Storage plugins are used to interact with the Storage Microservice and provide the persistent storage of information for FogLAMP. + +The current version of FogLAMP comes with two storage plugins: + +- The **SQLite plugin**: this is the default plugin and it is used for general purpose storage on constrained devices. +- The **PostgreSQL plugin**: this plugin can be set on request (or it can be built as a default plugin from source) and it is used for a more significant demand of storage on relatively larger systems. + + +Data and Metadata +----------------- + +Persistency is split in two blocks: + +- **Metadata persistency**: it refers to the storage of metadata for FogLAMP, such as the configuration of the plugins, the scheduling of jobs and tasks and the the storage of statistical information. +- **Data persistency**: it refers to the storage of data collected from sensors and devices by the South microservices. + +In the current implementation of FogLAMP, metadata and data use the same Storage plugin. In future implementations, administrators will be able to select different plugins. + + +Common Elements for Storage Plugins +----------------------------------- + +In designing the Storage API and plugins, we have first of all considered that there may be a large number of use cases for data and metadata persistence, therefore we have designed a flexible architecture that poses very few limitations. In practice, this means that developers can build their own Storage plugin and they can rely on anything they want to use as persistent storage. They can use a memory structure, or even a pass-through library, a file, a message queue system, a time series database, a relational database, NoSQL or something else. + +After having praised the flexibility of the Storage plugins, let's provide guidelines about the basic functionality they should provide, bearing in mind that such functionality may not be relevant for some use cases. + +- **Metadata persistency**: As mentioned before, one of the main reasons to use a Storage plugin is to safely store the configuration of the FogLAMP components. Since the configuration must survive to a system crash or reboot, it is fair to say that such information should be stored in one or more files or in a database system. +- **Data buffering**: The second most important feature of a Storage plugin is the ability to buffer (or store) data coming from the outside world, tipically from the South microservices. In some cases this feature may not be necessary, since administrators may want to send data to other systems as soon as possible, using a North task of microservice. Even in situations where data can be sent up North instantaneously, you should consider these scenarios: + + - FogLAMP may be installed in areas where the network is unreliable. The North plugins will provide the logic of retrying to gain connectivity and resending data when the connection has been lost in the middle of the transfer operations. + - North services may rely on the use of networks that provide time windows to operate. + - Historians and other systems may work better when data is transferred in blocks instead of a constant streaming. + +- **Data purging**: Data may persist for the time needed by any specific use case, but it is pretty common that after a while (it can be seconds or minutes, but also day or months) data is no longer needed in FogLAMP. For this reason, the Storage plugin is able to purge data. Purging may be by time or by space usage, in conjuction with the fact that data may have been already transferred to other systems. + +- **Data backup/restore**: Data, but especially metadata (i.e. configuration), can be backed up and stored safely on other systems. In case of crash and recovery, the same data may be restored into FogLAMP. FogLAMP provides a set of generic API to execute backup and restore operations. + + diff --git a/docs/06_plugins/index.rst b/docs/06_plugins/index.rst index 1588fc485c..17e7b66e18 100644 --- a/docs/06_plugins/index.rst +++ b/docs/06_plugins/index.rst @@ -10,4 +10,5 @@ Plugins 02_writing_plugins 03_south_plugins 04_north_plugins + 05_storage_plugins From 06cae2701a76a94012bfa3b74f50d0fdfd304559 Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Tue, 15 May 2018 20:01:19 +0100 Subject: [PATCH 09/12] Fixed North Plugin issue for statistics --- docs/06_plugins/04_north_plugins.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/06_plugins/04_north_plugins.rst b/docs/06_plugins/04_north_plugins.rst index efb6230b24..b5f01f6190 100644 --- a/docs/06_plugins/04_north_plugins.rst +++ b/docs/06_plugins/04_north_plugins.rst @@ -275,6 +275,7 @@ Before you send data to the PI server, it is likey that you need to apply more c - **URL** : the URL to the PI Connector Relay OMF. It is usually composed by the name or address of the Windows server where the Connector Relay service is running, the port associated to the service and the ingress/messages API call. The communication is via HTTPS protocol. - **producerToken** : the token provided by the Data Collection Manager when the PI administrator sets the use of FogLAMP. - **type-id** : the measurement type for the stream of data. +- **source** : this parameter should be set to *readings* (default) when the plugin is used to send data collected by South microservices, and to *statistics* when the plugin is used to send FogLAMP statistics to the PI system. An example of the changes to apply to the plugins to send data to the PI system is available here `here <../05_testing.html#sending-greetings-to-the-northern-hemisphere>`_. From 28e1c9c1203c99d11d2a0f853376714206a6ff7b Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Tue, 15 May 2018 21:01:04 +0100 Subject: [PATCH 10/12] data purge --- docs/07_admin/05_tasks.rst | 141 +++++++++++++++++++++++++++++++++++++ docs/07_admin/index.rst | 1 + 2 files changed, 142 insertions(+) create mode 100644 docs/07_admin/05_tasks.rst diff --git a/docs/07_admin/05_tasks.rst b/docs/07_admin/05_tasks.rst new file mode 100644 index 0000000000..1b1520ebf7 --- /dev/null +++ b/docs/07_admin/05_tasks.rst @@ -0,0 +1,141 @@ +.. Tasks + +.. |br| raw:: html + +
+ +.. Images + +.. Links + +.. Links in new tabs + +.. ============================================= + + +************* +FogLAMP Tasks +************* + +Tasks are part of the FogLAMP IoT platform. They are like services, but with a clear distinction: + +- *services* are started at a certain point (usually at startup) and they are likely to continue to work until FogLAMP stops. +- *tasks* are started when required, they execute a job and then they terminate. + +In simple terms, a service is meant to always listen and react to requests, while a task is triggered by an event and then when job is terminated, the tasks ends. + +That said, tasks and services shared these same features: + +- They are both started by the FogLAMP scheduler. It is likely that services are started at startup, while tasks can start at a given time or interval. +- They both use the internal API to communicate with other services. +- They both use the same pluggable architecture to separate a common logic, usually associated to the internal features of FogLAMP, from a more generic logic, usually closer to the type of operations that must be performed. + +In this chapter we present a set of tasks that are commonly available in FogLAMP. + + +Purge +===== + +The *Purge* task is triggered by the scheduler to purge old data that is still stored (buffered) in FogLAMP. The logic applied to the task is relatively simple: + +- The task is called exclusively (i.e. there cannot be more than one *Purge* task running at any given time) by the FogLAMP scheduler every hour (by default). +- Data that is older than a certain date/time is removed. +- Optionally, data is removed if the total size of the stored objects is bigger than 1GByte (default) +- Optionally, data is not removed if it has not been extracted and used by any North task or service yet. +- All purge operations are stored in the audit log. + + +Purge Schedule +-------------- + +*Purge* is one of the tasks launched by the FogLAMP scheduler. You can retrieve information about the scheduling by calling the *GET* method of the *schedule* call. The name and the process name of the task are both *purge*: + +.. code-block:: console + + $ curl -sX GET http://localhost:8081/foglamp/schedule + ... + { "id" : "cea17db8-6ccc-11e7-907b-a6006ad3dba0", + "name" : "purge", + "time" : 0, + "enabled" : true, + "repeat" : 3600, + "type" : "INTERVAL", + "exclusive" : true, + "processName" : "purge", + "day" : null }, + ... + $ + +As you can see from the JSON output, the task is scheduled to be executed every hour (3,600 seconds). In order to change the interval between *Purge* tasks, you can call the *PUT* method of the *schedule* call by passing the associated *id*. For example, in order to change the task to be executed any 5 minutes (i.e. 300 seconds) you should call: + +.. code-block:: console + + $ curl -sX PUT http://localhost:8081/foglamp/schedule/cea17db8-6ccc-11e7-907b-a6006ad3dba0 -d '{"repeat": 300}' + { "schedule": { "id": "cea17db8-6ccc-11e7-907b-a6006ad3dba0", + "name" : "purge", + "time" : 0, + "enabled" : true, + "repeat" : 300, + "type" : "INTERVAL", + "exclusive" : true, + "processName" : "purge", + "day" : null } + } + $ + + +Purge Configuration +------------------- + +The configuration of the *Purge* task is stored in the metadata structures of FogLAMP and it can be retrieve using the *GET* method of the *category/PURGE_READ* call. This is the command used to retrieve the configuration in JSON format: + +.. code-block:: console + + $ curl -sX GET http://localhost:8081/foglamp/category/PURGE_READ + { "retainUnsent" : { "type": "boolean", + "default": "False", + "description": "Retain data that has not been sent to any historian yet.", + "value": "False" }, + "age" : { "type": "integer", + "default": "72", + "description": "Age of data to be retained, all data that is older than this value will be removed,unless retained. (in Hours)", + "value": "72" }, + "size" : { "type": "integer", + "default": "1000000", + "description": "Maximum size of data to be retained, the oldest data will be removed to keep below this size, unless retained. (in Kbytes)", + "value": "1000000" } } + $ + + +Changes can be applied using the *PUT* method for each parameter call. For example, in order to change the retention policy for data that has not been sent to historians yet, you can use this call: + +.. code-block:: console + + $ curl -sX PUT http://locahost:8081/foglamp/category/PURGE_READ/retainUnsent -d '{"value": "True"}' + { "type": "boolean", + "default": "False", + "description": "Retain data that has not been sent to any historian yet.", + "value": "True" } + $ + +The following table shows the list of parameters that can be changed in the *Purge* task: + ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| Item | Type | Default | Description | ++===================+==========+=========================================+========================================================+ +| retainUnsent | boolean | False | Retain data that has not been sent to "North" yet |br| | +| | | | When *True*, data that has not yet been retrieved |br| | +| | | | by any North service or task, will not be purged. |br| | +| | | | When *False*, data is purged withouth checking |br| | +| | | | whether it has been sent to a North destination |br| | +| | | | yet or not. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| age | integer | 72 | Age in hours of the data to be retained. Data |br| | +| | | | that is older than this value, will be purged. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ +| size | integer | 1000000 | Size in KBytes of data that will be retained in |br| | +| | | | FogLAMP. Older data will be removed to keep the |br| | +| | | | data stored in FogLAMP below this size. | ++-------------------+----------+-----------------------------------------+--------------------------------------------------------+ + + diff --git a/docs/07_admin/index.rst b/docs/07_admin/index.rst index 8cf82f0704..8b1f6e5e82 100644 --- a/docs/07_admin/index.rst +++ b/docs/07_admin/index.rst @@ -10,3 +10,4 @@ Administrators' and Users' Guide 02_RESTadmin 03_RESTuser 04_utilities + 05_tasks From 3051910bd1ae9f764f9c7a455595bc8df0d0d10f Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Tue, 15 May 2018 22:49:48 +0100 Subject: [PATCH 11/12] statistics --- docs/07_admin/02_RESTadmin.rst | 160 +++++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) diff --git a/docs/07_admin/02_RESTadmin.rst b/docs/07_admin/02_RESTadmin.rst index a215e3b3da..bd7db8f949 100644 --- a/docs/07_admin/02_RESTadmin.rst +++ b/docs/07_admin/02_RESTadmin.rst @@ -894,4 +894,164 @@ The response payload is some basic health information in a JSON object. "uptime": 2113.076449394226 } $ + +statistics +---------- + +The *statistics* interface allows the retrieval of live statistics and statistical history for the FogLAMP device. + + +GET statistics +~~~~~~~~~~~~~~ + +``GET /foglamp/statistics`` - return a general set of statistics + + +**Response Payload** + +The response payload is a JSON document with statistical information (all numerical), these statistics are absolute counts since FogLAMP started. + ++------------------------+-----------------------------------------------------------------------------+ +| Key | Description | ++========================+=============================================================================+ +| BUFFERED | The number of readings currently in the FogLAMP buffer | ++------------------------+-----------------------------------------------------------------------------+ +| DISCARDED | The number of readings discarded at the input side by FogLAMP, |br| | +| | i.e. discarded before being placed in the buffer. This may be due |br| | +| | to some error in the readings themselves. | ++------------------------+-----------------------------------------------------------------------------+ +| PURGED | The number of readings removed from the buffer by the *Purge* task | ++------------------------+-----------------------------------------------------------------------------+ +| READINGS | The number of readings received by FogLAMP since startup | ++------------------------+-----------------------------------------------------------------------------+ +| SENT_1 | The number of readings sent to the PI system via the OMF plugin | ++------------------------+-----------------------------------------------------------------------------+ +| SENT_2 | The number of statistics sent to the PI system via the OMF plugin | ++------------------------+-----------------------------------------------------------------------------+ +| SENT_3 | The number of readings sent to another system via the HTTP plugin | ++------------------------+-----------------------------------------------------------------------------+ +| SENT_4 | The number of readings sent to the OSIsoft Cloud Service via the OCS plugin | ++------------------------+-----------------------------------------------------------------------------+ +| UNSENT | The number of readings filtered out in the send process | ++------------------------+-----------------------------------------------------------------------------+ +| UNSNPURGED | The number of readings that were purged from the buffer before being sent | ++------------------------+-----------------------------------------------------------------------------+ +| *ASSET-CODE* | The number of readings received by FogLAMP since startup |br| | +| | with name *asset-code* | ++------------------------+-----------------------------------------------------------------------------+ + + +**Example** + +.. code-block:: console + + $ curl -s http://localhost:8081/foglamp/statistics + [ { "description" : "The number of readings currently in the FogLAMP buffer", + "key" : "BUFFERED", + "value" : 0 }, + ... + { "description" : "The number of readings received by FogLAMP since startup for sensor FOGBENCH/ACCELEROMETER", + "key" : "FOGBENCH/ACCELEROMETER", + "value" : 2 }, + ... ] + $ + + +GET statistics/history +~~~~~~~~~~~~~~~~~~~~~~ + +``GET /foglamp/statistics/history`` - return a historical set of statistics. This interface is normally used to check if a set of sensors or devices are sending data to FogLAMP, by comparing the recent statistics and the number of readings received for an asset. + + +**Reguest Parameters** + +- **limit** - limit the result set to the *N* most recent entries. + + +**Response Payload** + +A JSON document containing an array of statistical information, these statistics are delta counts since the previous entry in the array. The time interval between values is a constant defined that runs the gathering process which populates the history statistics in the storage layer. + ++---------------------------+-----------------------------------------------------------------------------+ +| Key | Description | ++===========================+=============================================================================+ +| interval | The interval in seconds between successive statistics values | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].BUFFERED | The number of readings currently in the FogLAMP buffer | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].DISCARDED | The number of readings discarded at the input side by FogLAMP, |br| | +| | i.e. discarded before being placed in the buffer. This may be due |br| | +| | to some error in the readings themselves. | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].PURGED | The number of readings removed from the buffer by the *Purge* task | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].READINGS | The number of readings received by FogLAMP since startup | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].SENT_1 | The number of readings sent to the PI system via the OMF plugin | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].SENT_2 | The number of statistics sent to the PI system via the OMF plugin | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].SENT_3 | The number of readings sent to another system via the HTTP plugin | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].SENT_4 | The number of readings sent to the OSIsoft Cloud Service via the OCS plugin | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].UNSENT | The number of readings filtered out in the send process | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].UNSNPURGED | The number of readings that were purged from the buffer before being sent | ++---------------------------+-----------------------------------------------------------------------------+ +| statistics[].*ASSET-CODE* | The number of readings received by FogLAMP since startup |br| | +| | with name *asset-code* | ++---------------------------+-----------------------------------------------------------------------------+ + + +**Example** + +.. code-block:: console + + $ curl -s http://localhost:8081/foglamp/statistics/history?limit=2 + { "interval" : 15, + "statistics" : [ { "READINGS": 0, + "FOGBENCH/LUXOMETER": 0, + "DISCARDED": 0, + "FOGBENCH/HUMIDITY": 0, + "FOGBENCH/ACCELEROMETER": 0, + "UNSENT": 0, + "SENT_2": 0, + "SENT_4": 0, + "FOGBENCH/TEMPERATURE": 0, + "FOGBENCH/GYROSCOPE": 0, + "UNSNPURGED": 0, + "BUFFERED": 0, + "FOGBENCH/MOUSE": 0, + "FOGBENCH/MAGNETOMETER": 0, + "PURGED": 0, + "FOGBENCH/WALL CLOCK": 0, + "SENT_1": 0, + "FOGBENCH/PRESSURE": 0, + "SENT_3": 0, + "FOGBENCH/SWITCH": 0, + "history_ts": "2018-05-15 22:39:10.374" }, + { "READINGS": 0, + "FOGBENCH/LUXOMETER": 0, + "DISCARDED": 0, + "FOGBENCH/HUMIDITY": 0, + "FOGBENCH/ACCELEROMETER": 0, + "UNSENT": 0, + "SENT_2": 0, + "SENT_4": 0, + "FOGBENCH/TEMPERATURE": 0, + "FOGBENCH/GYROSCOPE": 0, + "UNSNPURGED": 0, + "BUFFERED": 0, + "FOGBENCH/MOUSE": 0, + "FOGBENCH/MAGNETOMETER": 0, + "PURGED": 0, + "FOGBENCH/WALL CLOCK": 0, + "SENT_1": 0, + "FOGBENCH/PRESSURE": 0, + "SENT_3": 0, + "FOGBENCH/SWITCH": 0, + "history_ts": "2018-05-15 22:38:55.653" } ] + $ + From e5f570e1c368b81fd169968f967eb9df37186040 Mon Sep 17 00:00:00 2001 From: Ivan Zoratti Date: Wed, 16 May 2018 02:12:25 +0100 Subject: [PATCH 12/12] Installing Plugins --- docs/06_plugins/01_FogLAMP_plugins.rst | 125 ++++++++++++++++++++----- 1 file changed, 104 insertions(+), 21 deletions(-) diff --git a/docs/06_plugins/01_FogLAMP_plugins.rst b/docs/06_plugins/01_FogLAMP_plugins.rst index c5ec0e8cd5..6f32cfecfb 100644 --- a/docs/06_plugins/01_FogLAMP_plugins.rst +++ b/docs/06_plugins/01_FogLAMP_plugins.rst @@ -42,35 +42,35 @@ This version of FogLAMP provides the following plugins in the main repository: | Type | Name | Initial | Description | Availability | Notes | | | | |br| Status| | | | +=========+============+============+=============================+============================+========================================+ -| Storage | SQLite | Enabled | SQLite storage |br| | Ubuntu: x86 |br| | | +| Storage | SQLite | Enabled | SQLite storage |br| | Ubuntu: x86_64 |br| | | | | | | for data and metadata | Ubuntu Core: x86, ARM |br| | | | | | | | Raspbian | | +---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| Storage | Postgres | Disabled | PostgreSQL storage |br| | Ubuntu: x86 |br| | | +| Storage | Postgres | Disabled | PostgreSQL storage |br| | Ubuntu: x86_64 |br| | | | | | | for data and metadata | Ubuntu Core: x86, ARM |br| | | | | | | | Raspbian | | +---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| South | COAP | Enabled | CoAP Listener | Ubuntu: x86 |br| | | +| South | COAP | Enabled | CoAP Listener | Ubuntu: x86_64 |br| | | | | | | | Ubuntu Core: x86, ARM |br| | | | | | | | Raspbian | | +---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| South | CC2650POLL | Disabled | TI SensorTag CC2650 |br| | Ubuntu: x86 |br| | It requires BLE support. |br| | +| South | CC2650POLL | Disabled | TI SensorTag CC2650 |br| | Ubuntu: x86_64 |br| | It requires BLE support. |br| | | | | | in polling mode | Ubuntu Core: x86, ARM |br| | There are issues with Ubuntu Core |br| | | | | | | Raspbian | on ARM, reported |here BT| | +---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| South | CC2650ASYN | Disabled | TI SensorTag CC2650 |br| | Ubuntu: x86 |br| | It requires BLE support. |br| | +| South | CC2650ASYN | Disabled | TI SensorTag CC2650 |br| | Ubuntu: x86_64 |br| | It requires BLE support. |br| | | | | | asynchronous |br| | Ubuntu Core: x86, ARM |br| | There are issues with Ubuntu Core |br| | | | | | (listening) mode | Raspbian | on ARM, reported |here BT|. | +---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| South | HTTP_SOUTH | Enabled | HTTP Listener | Ubuntu: x86 |br| | | +| South | HTTP_SOUTH | Enabled | HTTP Listener | Ubuntu: x86_64 |br| | | | | | | | Ubuntu Core: x86, ARM |br| | | | | | | | Raspbian | | +---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| North | OMF | Disabled | OSIsoft Message Format |br| | Ubuntu: x86 |br| | It works with PI Connector |br| | +| North | OMF | Disabled | OSIsoft Message Format |br| | Ubuntu: x86_64 |br| | It works with PI Connector |br| | | | | | sender to PI Connector |br| | Ubuntu Core: x86, ARM |br| | Relay OMF 1.2.X and 2.2 | | | | | Relay OMF | Raspbian | | +---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ -| North | OCS | Disabled | OSIsoft Message Format |br| | Ubuntu: x86 |br| | | +| North | OCS | Disabled | OSIsoft Message Format |br| | Ubuntu: x86_64 |br| | | | | | | sender to the OSIsoft |br| | Ubuntu Core: x86, ARM |br| | | | | | | Cloud Service | Raspbian | | +---------+------------+------------+-----------------------------+----------------------------+----------------------------------------+ @@ -78,17 +78,100 @@ This version of FogLAMP provides the following plugins in the main repository: In addition to the plugins in the main repository, these plugins are also available: -+-------+-------------+---------------------------+---------------------------------------+--------------+ -| Type | Name | Repository | Description | Availability | -+=======+=============+===========================+=======================================+==============+ -| South | dht11pi | foglamp-south-dht11 | Wired DHT11 Sensor in polling mode | Respbian | -+-------+-------------+---------------------------+---------------------------------------+--------------+ -| South | envirophat | foglamp-south-envirophat | Enviro pHAT sensor set | Raspbian | -+-------+-------------+---------------------------+---------------------------------------+--------------+ -| South | openweather | foglamp-south-openweather | Data pull from the OpenWeatherMap API | Raspbian | -+-------+-------------+---------------------------+---------------------------------------+--------------+ -| South | pt100 | foglamp-south-pt100 | Wired PT100 temperature sensor | Raspbian | -+-------+-------------+---------------------------+---------------------------------------+--------------+ - -|br| ++-------+----------------+------------------------------+---------------------------------------+---------------+ +| Type | Name | Repository | Description | Availability | ++=======+================+==============================+=======================================+===============+ +| South | dht11pi | foglamp-south-dht11 | Wired DHT11 Sensor in polling mode | Respbian | ++-------+----------------+------------------------------+---------------------------------------+---------------+ +| South | envirophat | foglamp-south-envirophat | Enviro pHAT sensor set | Raspbian | ++-------+----------------+------------------------------+---------------------------------------+---------------+ +| South | openweathermap | foglamp-south-openweathermap | Data pull from the OpenWeatherMap API | Ubuntu x86_64 | +| | | | | Raspbian | ++-------+----------------+------------------------------+---------------------------------------+---------------+ +| South | pt100 | foglamp-south-pt100 | Wired PT100 temperature sensor | Raspbian | ++-------+----------------+------------------------------+---------------------------------------+---------------+ + + +Installing New Plugins +---------------------- + +As a general rule and unless the documentation states otherwise, plugins should be installed in two ways: + +- When the plugin is available as **source code**, it should be installed when **FogLAMP is not running**. |br| This is the recommended method because you may want to manually move the plugin code into the right location where FogLAMP is installed, add pre-requisites and execute the REST commands necessary to start the plugin. +- When the plugin is available as **package**, it should be installed when **FogLAMP is running**. |br| This is the required method because the package executed pre and post-installtion tasks that require FogLAMP to run. + +In general, FogLAMP must be restarted when a new plugin has been installed. + +For example, this is the command to use to install the *OpenWeather* South plugin: + +.. code-block:: console + + $ sudo systemctl status foglamp.service + ● foglamp.service - LSB: FogLAMP + Loaded: loaded (/etc/init.d/foglamp; bad; vendor preset: enabled) + Active: active (running) since Wed 2018-05-16 01:32:25 BST; 4min 1s ago + Docs: man:systemd-sysv-generator(8) + CGroup: /system.slice/foglamp.service + ├─13741 python3 -m foglamp.services.core + ├─13746 /usr/local/foglamp/services/storage --address=0.0.0.0 --port=40138 + ├─13785 /bin/sh services/south --port=40138 --address=127.0.0.1 --name=COAP + ├─13786 python3 -m foglamp.services.south --port=40138 --address=127.0.0.1 --name=COAP + ├─13787 /bin/sh services/south --port=40138 --address=127.0.0.1 --name=HTTP_SOUTH + └─13788 python3 -m foglamp.services.south --port=40138 --address=127.0.0.1 --name=HTTP_SOUTH + + May 16 01:36:09 ubuntu python3[13741]: FogLAMP[13741] INFO: scheduler: foglamp.services.core.scheduler.scheduler: Process started: Schedule 'stats collection' process 'stats coll + ['tasks/statistics', '--port=40138', '--address=127.0.0.1', '--name=stats collector'] + May 16 01:36:09 ubuntu python3[13741]: FogLAMP[13741] INFO: scheduler: foglamp.services.core.scheduler.scheduler: Sleeping for 1730.0983202457428 seconds + May 16 01:36:10 ubuntu python3[13741]: FogLAMP[13741] INFO: scheduler: foglamp.services.core.scheduler.scheduler: Process terminated: Schedule 'stats collection' process 'stats c + ['tasks/statistics'] + FogLAMP v1.2 running. + FogLAMP Uptime: 266 seconds. + FogLAMP records: 0 read, 0 sent, 0 purged. + FogLAMP does not require authentication. + === FogLAMP services: + foglamp.services.core + foglamp.services.south --port=40138 --address=127.0.0.1 --name=COAP + foglamp.services.south --port=40138 --address=127.0.0.1 --name=HTTP_SOUTH + === FogLAMP tasks: + $ + $ sudo cp foglamp-south-openweathermap-1.2-x86_64.deb /var/cache/apt/archives/. + $ sudo apt install /var/cache/apt/archives/foglamp-south-openweathermap-1.2-x86_64.deb + Reading package lists... Done + Building dependency tree + Reading state information... Done + Note, selecting 'foglamp-south-openweathermap' instead of '/var/cache/apt/archives/foglamp-south-openweathermap-1.2-x86_64.deb' + The following packages were automatically installed and are no longer required: + linux-headers-4.4.0-109 linux-headers-4.4.0-109-generic linux-headers-4.4.0-119 linux-headers-4.4.0-119-generic linux-headers-4.4.0-121 linux-headers-4.4.0-121-generic + linux-image-4.4.0-109-generic linux-image-4.4.0-119-generic linux-image-4.4.0-121-generic linux-image-extra-4.4.0-109-generic linux-image-extra-4.4.0-119-generic + linux-image-extra-4.4.0-121-generic + Use 'sudo apt autoremove' to remove them. + The following NEW packages will be installed + foglamp-south-openweathermap + 0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade. + Need to get 0 B/3,404 B of archives. + After this operation, 0 B of additional disk space will be used. + Selecting previously unselected package foglamp-south-openweathermap. + (Reading database ... 211747 files and directories currently installed.) + Preparing to unpack .../foglamp-south-openweathermap-1.2-x86_64.deb ... + Unpacking foglamp-south-openweathermap (1.2) ... + Setting up foglamp-south-openweathermap (1.2) ... + openweathermap plugin installed. Restart FogLAMP to enable the plugin. + $ + $ sudo systemctl stop foglamp.service + $ sudo systemctl start foglamp.service + $ + $ foglamp status + FogLAMP v1.2 running. + FogLAMP Uptime: 271 seconds. + FogLAMP records: 36 read, 0 sent, 0 purged. + FogLAMP does not require authentication. + === FogLAMP services: + foglamp.services.core + foglamp.services.south --port=42066 --address=127.0.0.1 --name=openweathermap + foglamp.services.south --port=42066 --address=127.0.0.1 --name=COAP + foglamp.services.south --port=42066 --address=127.0.0.1 --name=HTTP_SOUTH + === FogLAMP tasks: + $ + +