diff --git a/README.md b/README.md index 4c663787c..49e006401 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,7 @@ This project is part of [FIWARE](https://www.fiware.org/). For more information [IoT Agents](https://github.com/Fiware/catalogue/tree/master/iot-agents). | :books: [Documentation](https://iotagent-node-lib.rtfd.io) | :mortar_board: [Academy](https://fiware-academy.readthedocs.io/en/latest/iot-agents/idas) | :dart: [Roadmap](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/roadmap.md) | -| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | - +| ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ## Index @@ -47,8 +46,6 @@ functions. communications are left to the library. - Standardized OAuth2-based security is available to enable each IoT Agent to connect to several common Identity Managers (e.g. Keystone and Keyrock) so that communications can be restricted to trusted components. -- A series of additional plugins are offered where necessary to allow for expression parsing, attribute aliasing and - the processing of timestamp metadata. Each individual IoT Agent offers is driven by a `config.js` configuration file contains explicit custom settings based on the protocol and payload the IoT Agent is translating. It will also contain some common flags for common @@ -76,16 +73,18 @@ const iotagentLib = require('iotagent-node-lib'); ``` Information about how to configure the Library can be found at the corresponding section of the -[Installation & Administration Guide](doc/installationguide.md). +[Installation & Administration Guide](doc/admin.md). ## Usage -This library has no packaging or build processes. The [Getting Started](doc/getting-started.md) is a good place to -start. Usage of the library is explained in the [User & Programmers Manual](doc/usermanual.md). +This library has no packaging or build processes. The [Getting Started](./doc/getting-started.md) is a good place to +start. You can also review the [API documentation](./doc/api.md) for a full list of the available functions. + +If you plan to use the library in your own IoT Agent, you should read the [Developer Guide](./doc/devel/development.md). +You can also review the [Architecture](./doc/devel/architecture.md) documentation and +[Northbound API](./doc/devel/northbound-api.md). -- Details of the architecture of an IoT Agent be found [here](doc/architecture.md). -- Further Advanced topics can be found [here](doc/advanced-topics.md). -- The following features are listed as [deprecated](doc/deprecated.md). +The following features are listed as [deprecated](doc/deprecated.md). ## API @@ -94,7 +93,7 @@ decommission devices. [API](doc/api.md). ## Testing -Contributions to development can be found [here](doc/development.md) - additional contributions are welcome. +Contributions to development can be found [here](doc/devel/development.md) - additional contributions are welcome. ### Agent Console diff --git a/doc/README.md b/doc/README.md new file mode 100644 index 000000000..96d1b448e --- /dev/null +++ b/doc/README.md @@ -0,0 +1,16 @@ +# IoT Agent documentation + +## User documentation + +- [Getting started](getting-started.md) +- [IoT Agent API](api.md) +- [Installation and administration manual](admin.md) +- [Deprecated features](deprecated.md) +- [Roadmap](roadmap.md) + +## Development documentation + +- [Development manual](devel/development.md) +- [Contributing guide](devel/contribution-guidelines.md) +- [Architecture](devel/architecture.md) +- [North Port - NGSI Interactions](devel/northboundinteractions.md) diff --git a/doc/admin.md b/doc/admin.md new file mode 100644 index 000000000..9d5ca99f1 --- /dev/null +++ b/doc/admin.md @@ -0,0 +1,565 @@ +# Administration + +- [Configuration](#configuration) + - [Configuration parameters:](#configuration-parameters) + - [loglevel](#loglevel) + - [contextBroker](#contextbroker) + - [server](#server) + - [stats](#stats) + - [authentication](#authentication) + - [deviceRegistry](#deviceregistry) + - [mongodb](#mongodb) + - [iotManager](#iotmanager) + - [types](#types) + - [service](#service) + - [subservice](#subservice) + - [providerUrl](#providerurl) + - [iotaVersion](#iotaversion) + - [dieOnUnexpectedError](#dieonunexpectederror) + - [singleConfigurationMode](#singleconfigurationmode) + - [timestamp](#timestamp) + - [defaultResource](#defaultresource) + - [defaultKey](#defaultkey) + - [componentName](#componentname) + - [pollingExpiration](#pollingexpiration) + - [pollingDaemonFrequency](#pollingdaemonfrequency) + - [autocast](#autocast) + - [multiCore](#multicore) + - [fallbackTenant](#fallbacktenant) + - [fallbackPath](#fallbackpath) + - [explicitAttrs](#explicitattrs) + - [defaultEntityNameConjunction](#defaultentitynameconjunction) + - [relaxTemplateValidation](#relaxtemplatevalidation) + - [Configuration using environment variables](#configuration-using-environment-variables) +- [Logs](#logs) + - [Errors](#errors) +- [Alarms](#alarms) + +The **IoT Agent node library** is not a standalone product and should be added as a dependency to `package.json` of the +project is going to be used in. The library is published in the npm repository, so it can be added to the project by +adding the following line to the `package.json` file: + +```json +... +"dependencies": { + "iotagent-node-lib": "x.y.z", +} +``` + +wher `x.y.z` is an actual version number. + +As alternative, you can use the master branch as dependency. In this case, you will be using the latest version of the +code but note that some instability could exist (as the code in master is work in progress until the next version is +closed). + +```json +... +"dependencies": { + "iotagent-node-lib": "https://github.com/telefonicaid/iotagent-node-lib.git#master", +} +``` + +In order to use the library within your own IoT Agent, you must first you require it before use: + +```javascript +const iotagentLib = require('iotagent-node-lib'); +``` + +## Configuration + +The `activate()` function that starts the IoT Agent receives as single parameter with the configuration for the IoT +Agent. The Agent Console reads the same configuration from the `config.js` file or from the environment variables. The +configuration parameters work in the same way in both cases but the environment variables have precedence over the +`config.js` file. The following sections describe the configuration parameters that can be used in the IoT Agent. + +### Configuration parameters: + +These are the parameters that can be configured in the global section: + +#### `loglevel` + +It is the minimum log level to log. May take one of the following values: DEBUG, INFO, ERROR, FATAL. E.g.: 'DEBUG'. + +#### `contextBroker` + +It configures the connection parameters to stablish a connection to the Context Broker (host and port). E.g.: + +```javascript +{ + host: '192.168.56.101', + port: '1026' +} +``` + +- If you want to use **NGSI v2**: + +```javascript +{ + host: '192.168.56.101', + port: '1026', + ngsiVersion: 'v2' +} +``` + +- If you want to use **NGSI-LD** (experimental): + +```javascript +{ + host: '192.168.56.101', + port: '1026', + ngsiVersion: 'ld', + jsonLdContext: 'http://context.json-ld' // or ['http://context1.json-ld','http://context2.json-ld'] if you need more than one +} +``` + +Where `http://context.json-ld` is the location of the NGSI-LD `@context` element which provides additional information +allowing the computer to interpret the rest of the data with more clarity and depth. Read the +[JSON-LD specification](https://w3c.github.io/json-ld-syntax/#the-context) for more information. + +- If you want to support a "mixed" mode with both **NGSI-v2** and **NGSI-LD** (experimental): + +```javascript +{ + host: '192.168.56.101', + port: '1026', + ngsiVersion: 'mixed', + jsonLdContext: 'http://context.json-ld' // or ['http://context1.json-ld','http://context2.json-ld'] if you need more than one +} +``` + +Under mixed mode, **NGSI v2** payloads are used for context broker communications by default, but this payload may also +be switched to **NGSI LD** at service group or device provisioning time using the `ngsiVersion` field in the +provisioning API. The `ngsiVersion` field switch may be added at either group or device level, with the device level +overriding the group setting. + +#### `server` + +This parameter is used to create the Context Server (port where the IoT Agent will be listening as a Context Provider +and base root to prefix all the paths). The `port` attribute is required. If no `baseRoot` attribute is used, '/' is +used by default. E.g.: + +```javascript +{ + baseRoot: '/', + port: 4041 +} +``` + +When connected to an **NGSI-LD** context broker, an IoT Agent is able to indicate whether it is willing to accept `null` +values and also whether it is able to process the **NGSI-LD** `datasetId` metadata element. Setting these values to +`false` will cause the IoT Agent to return a 400 **Bad Request** HTTP status code explaining that the IoT Agent does not +support nulls or multi-attribute requests if they are encountered. + +```javascript +{ + baseRoot: '/', + port: 4041, + ldSupport : { + null: true, + datasetId: true + } +} +``` + +#### `stats` + +It configures the periodic collection of statistics. Use `interval` in milliseconds to set the time between stats +writings. + +```javascript +stats: { + interval: 100; +} +``` + +#### `authentication` + +Stores the authentication data, for use in retrieving tokens for devices with a trust token (required in scenarios with +security enabled in the Context Broker side). Currently, two authentication provider are supported: `keystone` and +`oauth2`. Authentication need to be enabled by setting the field `enabled` to `true`. In `keystone` based +authentication, the `trust` associated to the `device` or `deviceGroup` is a token representing a specific user and his +rights on a given domain (i.e. combination of `fiware-service` and `fiware-servicepath`). The authentication process use +the trust delegation workflow to check if the trust provided is valid, in which case return a `x-subject-token` that can +be used to authenticate the request to the Context Broker. Required parameters are: the `url` of the keystone to be used +(alternatively `host` and `port` but if you use this combination, the IoT Agent will assume that the protocol is HTTP), +the `user` and `password` to which it is delegated the `trust` verification. E.g.: + +```javascript +{ + enabled: true, + url: 'https://localhost:5000', + type: 'keystone', + user: 'iotagent', + password: 'iotagent' +} +``` + +In `oauth2` based authentication, two types of tokens can be used depending on the availability in the IDM to be used. +On one hand, the `trust` associated to the `device` or `deviceGroup` is a `refresh_token` issued by a specific user for +the Context Broker client. The authentication process uses the +[`refresh_token` grant type](https://tools.ietf.org/html/rfc6749#section-1.5) to obtain an `access_token` that can be +used to authenticate the request to the Context Broker. At the time being the assumption is that the `refresh_token` is +a not expiring `offline_token` (we believe this is the best solution in the case of IoT Devices, since injecting a +refresh token look may slow down communication. Still, the developer would be able to invalidate the refresh token on +the provider side in case of security issues connected to a token). The code was tested using +[Keycloak](http://www.keycloak.org), [Auth0](https://auth0.com) and [FIWARE Keyrock](https://github.com/ging/fiware-idm) +(it may require customisation for other providers - while OAuth2 is a standard, not all implementations behave in the +same way, especially as regards status codes and error messages). Required parameters are: the `url` of the OAuth 2 +provider to be used (alternatively `host` and `port` but if you use this combination, the IoT Agent will assume that the +protocol is HTTP), the `tokenPath` to which the validation request should be sent +(`/auth/realms/default/protocol/openid-connect/token` for Keycloak and Auth0, `/oauth2/token` for Keyrock), the +`clientId` and `clientSecret` that identify the Context Broker, and the `header` field that should be used to send the +authentication request (that will be sent in the form `Authorization: Bearer `). E.g.: + +```javascript +{ + enabled: true, + type: 'oauth2', + url: 'http://localhost:3000', + header: 'Authorization', + clientId: 'context-broker', + clientSecret: 'c8d58d16-0a42-400e-9765-f32e154a5a9e', + tokenPath: '/auth/realms/default/protocol/openid-connect/token' +} +``` + +Nevertheless, this kind of authentication relying on `refresh_token` grant type implies that when the acces_token +expires, it is needed to request a new one from the IDM, causing some overhead in the communication with the Context +Broker. To mitigate this issue, FIWARE KeyRock IDM implements `permanent tokens` that can be retrieved using +`scope=permanent`. With this approach, the IOTA does not need to interact with the IDM and directly include the +`permanent token` in the header. In order to use this type of token, an additional parameter `permanentToken` must be +set to `true` in the `authentication` configuration. An environment variable `IOTA_AUTH_PERMANENT_TOKEN` can be also +used for the same purpose. For instance: + +```javascript +{ + type: 'oauth2', + url: 'http://localhost:3000', + header: 'Authorization', + clientId: 'context-broker', + clientSecret: '0c2492e1-3ce3-4cca-9723-e6075b89c244', + tokenPath: '/oauth2/token', + enabled: true, + permanentToken: true +} +``` + +#### `deviceRegistry` + +Stores type of Device Registry to create. Currently, two values are supported: `memory` and `mongodb`. If the former is +configured, a transient memory-based device registry will be used to register all the devices. This registry will be +emptied whenever the process is restarted. If the latter is selected, a MongoDB database will be used to store all the +device information, so it will be persistent from one execution to the other. Mongodb databases must be configured in +the `mongob` section (as described bellow). E.g.: + +```javascript +{ + type: 'mongodb'; +} +``` + +#### `mongodb` + +It configures the MongoDB driver for those repositories with 'mongodb' type. If the `host` parameter is a list of +comma-separated IPs, they will be considered to be part of a Replica Set. In that case, the optional property +`replicaSet` should contain the Replica Set name. If the database requires authentication, username (`username`), +password (`password`) and authSource (`authSource`) can be set. If the database requires TLS/SSL connection but any +validation of the certificate chain is not mandatory, all you need is to set the ssl (`ssl`) option as `true` to connect +the database. If you need to add more complex option(s) such as `retryWrites=true` or `w=majority` when connection +database, extraArgs (`extraArgs`) can be used to perform it. For The MongoBD driver will retry the connection at startup +time `retries` times, waiting `retryTime` seconds between attempts, if those attributes are present (default values are +5 and 5 respectively). E.g.: + +```javascript +{ + host: 'localhost', + port: '27017', + db: 'iotagent', + retries: 5, + retryTime: 5 +} +``` + +```javascript +{ + host: 'mongodb-0,mongodb-1,mongodb-2', + port: '27017', + db: 'iotagent', + replicaSet: 'rs0', + user: 'rootuser', + password: 'password', + authSource: 'admin', + ssl: true, + extraArgs: { + retryWrites: true, + readPreference: 'nearest', + w: 'majority' + }, + retries: 5, + retryTime: 5 +} +``` + +#### `iotManager` + +This parameter configures all the information needed to register the IoT Agent in the IoTManager. If this section is +present, the IoTA will try to register to a IoTAM in the `host`, `port` and `path` indicated, with the information +configured in the object. The IoTAgent URL that will be reported will be the `providedUrl` (described below) with the +added `agentPath`: + +```javascript +{ + host: 'mockediotam.com', + port: 9876, + path: '/protocols', + protocol: 'GENERIC_PROTOCOL', + description: 'A generic protocol', + agentPath: '/iot' +} +``` + +#### `types` + +See **Type Configuration** in the [Configuration API](#configurationapi) section below. + +#### `service` + +Default service for the IoT Agent. If a device is being registered, and no service information comes with the device +data, and no service information is configured for the given type, the default IoT agent service will be used instead. +E.g.: 'smartGondor'. + +#### `subservice` + +default subservice for the IoT Agent. If a device is being registered, and no subservice information comes with the +device data, and no subservice information is configured for the given type, the default IoT agent subservice will be +used instead. E.g.: '/gardens'. + +#### `providerUrl` + +URL to send in the Context Provider registration requests. Should represent the external IP of the deployed IoT Agent +(the IP where the Context Broker will redirect the NGSI requests). E.g.: 'http://192.168.56.1:4041'. + +#### `iotaVersion` + +indicates the version of the IoTA that will be displayed in the about method (it should be filled automatically by each +IoTA). + +#### `dieOnUnexpectedError` + +if this flag is activated, the IoTAgent will not capture global exception, thus dying upon any unexpected error. + +#### `singleConfigurationMode` + +enables the Single Configuration mode for backwards compatibility (see description in the Overview). Default to false. + +#### `timestamp` + +if this flag is activated: + +- For NGSI-v2, the IoT Agent will add a `TimeInstant` metadata attribute to all the attributes updated from device + information. This flag is overwritten by `timestamp` flag in group or device +- With NGSI-LD, the standard `observedAt` property-of-a-property is created instead. + +#### `defaultResource` + +default string to use as resource for the registration of new Configurations (if no resource is provided). + +#### `defaultKey` + +default string to use as API Key for devices that do not belong to a particular Configuration. + +#### `componentName` + +default string identifying the component name for this IoT Agent in the logs. + +#### `pollingExpiration` + +expiration time for commands waiting in the polling queue in miliseconds. If a command has been in the queue for this +amount of time without being collected by the device, the expiration daemon will reclaim it. This attribute is optional +(if it doesn't exist, commands won't expire). + +#### `pollingDaemonFrequency` + +time between collection of expired commands in milliseconds. This attribute is optional (if this parameter doesn't exist +the polling daemon won't be started). + +#### `autocast` + +When enabled, the IoT Agents will try to cast attribute's values considering the JSON native type (only for NGSI v2). + +#### `multiCore` + +When enabled, the IoT Agents runs in multi-thread environment to take advantage of multi-core systems. It allows two +values `true` or `false`. This attribute is optional with default to false, which means that the IoTAgent runs in a +single thread. For more details about multi-core functionality, please refer to the +[Cluster](https://nodejs.org/api/cluster.html) module in Node.js and +[this section](devel/development.md#iot-agent-in-multi-thread-mode) of the library documentation. + +#### `fallbackTenant` + +For Linked Data Context Brokers which do not support multi-tenancy, this provides an alternative mechanism for supplying +the `NGSILD-Tenant` header. Note that NGSILD-Tenant has not yet been included in the NGSI-LD standard (it has been +proposed for the next update of the standard, but the final decision has yet been confirmed), take into account it could +change. Note that for backwards compatibility with NGSI v2, the `fiware-service` header is already used as alternative +if the `NGSILD-Tenant` header is not supplied. + +#### `fallbackPath` + +For Linked Data Context Brokers which do not support a service path, this provides an alternative mechanism for suppling +the `NGSILD-Path` header. Note that for backwards compatibility with NGSI v2, the `fiware-servicepath` header is already +used as alternative if the `NGSILD-Path` header is not supplied. Note that NGSILD-Path has not yet been included in the +NGSI-LD standard (it has been proposed for the next update of the standard, but the final decision has yet been +confirmed), take into account it could change + +#### `explicitAttrs` + +if this flag is activated, only provisioned attributes will be processed to Context Broker. This flag is overwritten by +`explicitAttrs` flag in group or device provision. Additionally `explicitAttrs` can be used to define which meassures +defined in JSON/JEXL array will be propagated to NGSI interface. + +#### `defaultEntityNameConjunction` + +the default conjunction string used to compose a default `entity_name` when is not provided at device provisioning time; +in that case `entity_name` is composed by `type` + `:` + `device_id`. Default value is `:`. This value is overwritten by +`defaultEntityNameConjunction` in group provision. + +#### `relaxTemplateValidation` + +if this flag is activated, `objectId` attributes for incoming devices are not validated, and may exceptionally include +characters (such as semi-colons) which are +[forbidden](https://fiware-orion.readthedocs.io/en/master/user/forbidden_characters/index.html) according to the NGSI +specification. When provisioning devices, it is necessary that the developer provides valid `objectId`-`name` mappings +whenever relaxed mode is used, to prevent the consumption of forbidden characters. + +### Configuration using environment variables + +Some of the configuration parameters can be overriden with environment variables, to ease the use of those parameters +with container-based technologies, like Docker, Heroku, etc... + +The following table shows the accepted environment variables, as well as the configuration parameter the variable +overrides. + +| Environment variable | Configuration attribute | +| :----------------------------------- | :------------------------------ | +| IOTA_CB_URL | `contextBroker.url` | +| IOTA_CB_HOST | `contextBroker.host` | +| IOTA_CB_PORT | `contextBroker.port` | +| IOTA_CB_NGSI_VERSION | `contextBroker.ngsiVersion` | +| IOTA_NORTH_HOST | `server.host` | +| IOTA_NORTH_PORT | `server.port` | +| IOTA_LD_SUPPORT_NULL | `server.ldSupport.null` | +| IOTA_LD_SUPPORT_DATASET_ID | `server.ldSupport.datasetId` | +| IOTA_PROVIDER_URL | `providerUrl` | +| IOTA_AUTH_ENABLED | `authentication.enabled` | +| IOTA_AUTH_TYPE | `authentication.type` | +| IOTA_AUTH_HEADER | `authentication.header` | +| IOTA_AUTH_URL | `authentication.url` | +| IOTA_AUTH_HOST | `authentication.host` | +| IOTA_AUTH_PORT | `authentication.port` | +| IOTA_AUTH_USER | `authentication.user` | +| IOTA_AUTH_PASSWORD | `authentication.password` | +| IOTA_AUTH_CLIENT_ID | `authentication.clientId` | +| IOTA_AUTH_CLIENT_SECRET | `authentication.clientSecret` | +| IOTA_AUTH_TOKEN_PATH | `authentication.tokenPath` | +| IOTA_AUTH_PERMANENT_TOKEN | `authentication.permanentToken` | +| IOTA_REGISTRY_TYPE | `deviceRegistry.type` | +| IOTA_LOG_LEVEL | `logLevel` | +| IOTA_TIMESTAMP | `timestamp` | +| IOTA_IOTAM_URL | `iotManager.url` | +| IOTA_IOTAM_HOST | `iotManager.host` | +| IOTA_IOTAM_PORT | `iotManager.port` | +| IOTA_IOTAM_PATH | `iotManager.path` | +| IOTA_IOTAM_AGENTPATH | `iotManager.agentPath` | +| IOTA_IOTAM_PROTOCOL | `iotManager.protocol` | +| IOTA_IOTAM_DESCRIPTION | `iotManager.description` | +| IOTA_MONGO_HOST | `mongodb.host` | +| IOTA_MONGO_PORT | `mongodb.port` | +| IOTA_MONGO_DB | `mongodb.db` | +| IOTA_MONGO_REPLICASET | `mongodb.replicaSet` | +| IOTA_MONGO_USER | `mongodb.user` | +| IOTA_MONGO_PASSWORD | `mongodb.password` | +| IOTA_MONGO_AUTH_SOURCE | `mongodb.authSource` | +| IOTA_MONGO_RETRIES | `mongodb.retries` | +| IOTA_MONGO_RETRY_TIME | `mongodb.retryTime` | +| IOTA_MONGO_SSL | `mongodb.ssl` | +| IOTA_MONGO_EXTRAARGS | `mongodb.extraArgs` | +| IOTA_SINGLE_MODE | `singleConfigurationMode` | +| IOTA_POLLING_EXPIRATION | `pollingExpiration` | +| IOTA_POLLING_DAEMON_FREQ | `pollingDaemonFrequency` | +| IOTA_AUTOCAST | `autocast` | +| IOTA_MULTI_CORE | `multiCore` | +| IOTA_JSON_LD_CONTEXT | `jsonLdContext` | +| IOTA_FALLBACK_TENANT | `fallbackTenant` | +| IOTA_FALLBACK_PATH | `fallbackPath` | +| IOTA_EXPLICIT_ATTRS | `explicitAttrs` | +| IOTA_DEFAULT_ENTITY_NAME_CONJUNCTION | `defaultEntityNameConjunction` | +| IOTA_RELAX_TEMPLATE_VALIDATION | `relaxTemplateValidation` | + +Note: + +- If you need to pass more than one JSON-LD context, you can define the IOTA_JSON_LD_CONTEXT environment variable as a + comma separated list of contexts (e.g. `'http://context1.json-ld,http://context2.json-ld'`) + +## Logs + +This section describes the logs that can be generated by the IoT Agent library. The IoT Agent library uses the following +log levels: + +| Level | Description | +| :------ | :-------------------------------------------------------------------------------- | +| `DEBUG` | Used to log information useful for debugging. | +| `INFO` | Used to log information about the normal operation of the IoT Agent library. | +| `ERROR` | Used to log information about errors that may affect the IoT Agent library. | +| `FATAL` | Used to log information about fatal errors that may affect the IoT Agent library. | + +Additionally, every error log has an associated error code that can be used to identify the error. The error codes are +composed by a prefix and a number. The following table shows the prefixes used in the IoT Agent library: + +| Prefix | Type of operation | +| :----------------- | :--------------------------------------------------------- | +| `MONGODB` | Errors related with the MongoDB repository | +| `IOTAM` | Errors related with the IoTA Manager | +| `KEYSTONE` | Errors related with trust token retrieval | +| `ORION` | Errors in Context Broker access | +| `VALIDATION-FATAL` | Errors related with management of the Validation templates | + +### Errors + +| Error code | Error name | Description | +| :--------------------- | :----------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `GENERAL-001` | Couldn't find callback in listDevices call. | Implies that the callback function was not found in the listDevices call. This error is thrown when the callback function is not provided in the listDevices call. | +| `MONGODB-001` | Error trying to connect to MongoDB: %s | Implies there has been an error connecting with the DB. The component will automatically retry from this error, but it may be a sign of connectivity problems between the DB and the component. If the connection cannot be restablished from this error, a MONGODB-002 error will be raised. | +| `MONGODB-002` | Error found after [%d] attempts: %s | Indicates that it was impossible to establish a connection to the MongoDB cluster, even after retrying N times. This could be caused by a connectivity problem with the MongoDB machine, a problem in the MongoDB cluster, or a misconfiguration of the IoTA Manager. Check the conectivity, the state of the MongoDB cluster and the Mongo configuration data. | +| `MONGODB-003` | No host found for MongoDB driver. | This error will thrown if MongoDB is selected as the configured repository for data but some information is missing in the configuration file. Check the configuration file and add all the required information. | +| `MONGODB-004` | MongoDB connection was lost. | Indicates that it was impossible to reestablish the connection with the MongoDB server after retrying N times. This could be caused by a connectivity problem with the MongoDB machine or by changes on the configuration of the MongoDB server done while the IoT Agent was running. This error is only thrown when using a single MongoDB instance or when using sharding but just a single mongos proxy. When using MongoDB instances using replica sets or multiple mongos servers, the IoT Agent will retry connecting forever alternating between the different nodes. | +| `IOTAM-001` | Error updating information in the IOTAM. Status Code [%d] | The IoT Agent could not contact the IoT Agent manager to update its information. This condition may indicate a lack of connectivity between machines or a problem in the IoT Agent Manager. The IoT Agent information in the IoT Agent Manager will be out-of-date until this problem is solved. | +| `KEYSTONE-001` | Error retrieving token from Keystone: %s | There was connection error connecting with Keystone to retrieve a token. This condition may indicate a lack of connectivity between both machines or a problem with Keystone. | +| `KEYSTONE-002` | Unexpected status code: %d | There was a problem retrieving a token from keystone that was not caused by connectivity errors. Check the Keystone log for errors and the security configuration in the IoTAgent. This may also be caused by a wrong trust token used by the user. | +| `KEYSTONE-003` | Token missing in the response headers. | Authentication flow worked correctly, but the response headers did not include the expected header `x-subject-token`. Check the Keystone logs and configuration. | +| `OAUTH2-001` | Error retrieving token from OAuth2 provider: %s | There was connection error connecting with OAuth2 provider to retrieve a token. This condition may indicate a lack of connectivity between both machines or a problem with OAuth2 provider. | +| `OAUTH2-002` | Unexpected status code: %d | There was a problem retrieving a token from OAuth2 provider that was not caused by connectivity errors. Check the OAuth2 provider log for errors and the security configuration in the IoTAgent. This may also be caused by an invalid `refresh_token` used by the user. | +| `OAUTH2-003` | Token missing in the response body | The JSON response body returned by the OAuth2 provider does not include a field `access_token`. Check the OAuth2 logs and configuration. | +| `ORION-001` | Connection error creating initial entity in the Context Broker: %s | There was a connectivity error accessing Context Broker to create an initial entity (or the Context Broker was down). Check connectivity between the machines, the status of the remote Context Broker and the configuration of the IoTAgent. | +| `ORION-002` | Connection error sending registrations to the Context Broker: %s | There was a connectivity error accessing Context Broker to register the IoTA as a Context Provider (or the Context Broker was down). Check connectivity between the machines, the status of the remote Context Broker and the configuration of the IoT Agent. | +| `VALIDATION-FATAL-001` | Validation Request templates not found | Validation templates were not found. Check all the validation templates are properly located in the IoTAgent Library folder and that the file permissions are correct. | + +## Alarms + +The following table shows the alarms that can be raised in the IoTAgent library. All the alarms are signaled by a error +log starting with the prefix "Raising [%s]:" (where %s is the alarm name). All the alarms are released by an info log +with the prefix "Releasing [%s]". These texts appear in the `msg=` field of the generic log record format. + +| Alarm name | Severity | Description | +| :--------------- | :----------- | :------------------------------------------------------- | +| `MONGO-ALARM_XX` | **Critical** | Indicates an error in the MongoDB connectivity | +| `ORION-ALARM` | **Critical** | Indicates a persistent error accesing the Context Broker | +| `IOTAM-ALARM` | **Critical** | Indicates a persistent error accessing the IoTAM | + +while the 'Severity' criterium is as follows: + +- **Critical** - The system is not working +- **Major** - The system has a problem that degrades the service and must be addressed +- **Warning** - It is happening something that must be notified + +In order to identify the internal flow which origins a mongo alarm, there is a suffix `_XX` which identifies from `01` +to `11` each flow. diff --git a/doc/api.md b/doc/api.md index 80ea35a9e..73d388200 100644 --- a/doc/api.md +++ b/doc/api.md @@ -183,7 +183,7 @@ All of them have the same syntax, a list of objects with the following attribute - **type** (mandatory): name of the type of the attribute in the target entity. - **metadata** (optional): additional static metadata for the attribute in the target entity. (e.g. `unitCode`) -Some transformation plugins also allow the use of the following optional fields: +Some advanced features also allow the use of the following optional fields: - **expression**: indicates that the value of the target attribute will not be the plain value or the measurement, but an expression based on a combination of the reported values. See the @@ -751,7 +751,7 @@ following to CB: ### Multientity measurement transformation support (`object_id`) -To allow support for measurement transformation in combination with multi entity plugin, where the same attribute is +To allow support for measurement transformation in combination with multi entity feature, where the same attribute is generated for different entities out of different incoming attribute values (i.e. `object_id`), we introduced support for `object_id` in the expression context. @@ -845,7 +845,7 @@ it in queries (and viceversa, receive the extended one in queries and return it ## Timestamp Processing -The IOTA processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSI v2, the plugin +The IOTA processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSI v2, then it adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the Standard `observedAt` property-of-a-property is used instead. diff --git a/doc/config-basic-example.js b/doc/config-basic-example.js deleted file mode 100644 index 61e81f236..000000000 --- a/doc/config-basic-example.js +++ /dev/null @@ -1,20 +0,0 @@ -var config = { - logLevel: 'DEBUG', - contextBroker: { - host: 'localhost', - port: '1026' - }, - server: { - port: 4041 - }, - deviceRegistry: { - type: 'memory' - }, - types: {}, - service: 'howtoService', - subservice: '/howto', - providerUrl: 'http://localhost:4041', - defaultType: 'Thing' -}; - -module.exports = config; diff --git a/doc/deprecated.md b/doc/deprecated.md index 2a3f8ce36..d73d2d930 100644 --- a/doc/deprecated.md +++ b/doc/deprecated.md @@ -44,16 +44,16 @@ information in the case you want to use old versions: The following table provides information about the last iotagent-node-lib version supporting currently removed features: -| **Removed feature** | **Last iotagent-node-lib version supporting feature** | **That version release date** | -| ---------------------- | ----------------------------------------------------- | ----------------------------- | -| NGSI v1 API | 2.17.0 | August 30th, 2021 | -| Support to Node.js v4 | 2.8.1 | December 19th, 2018 | -| Support to Node.js v6 | 2.9.0 | May 22nd, 2019 | -| Support to Node.js v8 | 2.12.0 | April 7th, 2020 | -| Support to Node.js v10 | 2.15.0 | February 18th, 2021 | -| Support to Node.js v12 | 2.24.0 | September 2nd, 2022 | -| Support to NGSI-LD 1.3 | 2.25.0 | January 24th, 2023 | -| eventType configuration | 2.26.0 | March 15th, 2023 | -| Support to Legacy Expressions | 3.1.0 | April 25th, 2023 | -| bidirectional plugin | 3.3.0 | August 24th, 2023 | -| appendMode configuration (`IOTA_APPEND_MODE` env var) | 3.3.0 | August 24th, 2023 | +| **Removed feature** | **Last iotagent-node-lib version supporting feature** | **That version release date** | +| ----------------------------------------------------- | ----------------------------------------------------- | ----------------------------- | +| NGSI v1 API | 2.17.0 | August 30th, 2021 | +| Support to Node.js v4 | 2.8.1 | December 19th, 2018 | +| Support to Node.js v6 | 2.9.0 | May 22nd, 2019 | +| Support to Node.js v8 | 2.12.0 | April 7th, 2020 | +| Support to Node.js v10 | 2.15.0 | February 18th, 2021 | +| Support to Node.js v12 | 2.24.0 | September 2nd, 2022 | +| Support to NGSI-LD 1.3 | 2.25.0 | January 24th, 2023 | +| eventType configuration | 2.26.0 | March 15th, 2023 | +| Support to Legacy Expressions | 3.1.0 | April 25th, 2023 | +| bidirectional plugin | 3.3.0 | August 24th, 2023 | +| appendMode configuration (`IOTA_APPEND_MODE` env var) | 3.3.0 | August 24th, 2023 | diff --git a/doc/NorthboundInteractions.postman_collection b/doc/devel/NorthboundInteractions.postman_collection similarity index 100% rename from doc/NorthboundInteractions.postman_collection rename to doc/devel/NorthboundInteractions.postman_collection diff --git a/doc/architecture.md b/doc/devel/architecture.md similarity index 98% rename from doc/architecture.md rename to doc/devel/architecture.md index cbc2b5620..6b9a14fc8 100644 --- a/doc/architecture.md +++ b/doc/devel/architecture.md @@ -1,6 +1,6 @@ ## Architecture -The following section defines the archtecture and message flow which is common to all IoT Agents which use the library. +The following section defines the architecture and message flow which is common to all IoT Agents which use the library. ### Device to NGSI Mapping @@ -32,7 +32,7 @@ basis preprovisioning the devices). Device measures can have three different beh The following sequence diagram shows the different NGSI interactions an IoT Agent makes with the Context Broker, explained in the following subsections (using the example of a OMA Lightweight M2M device). -![General ](./img/ngsiInteractions.png "NGSI Interactions") +![General ](../img/ngsiInteractions.png 'NGSI Interactions') Be aware that the IoT Agents are only required to support NGSI10 operations `updateContext` and `queryContext` in their standard formats (currently in JSON format; XML deprecated) but will not answer to NGSI9 operations (or NGSI convenience @@ -254,7 +254,7 @@ the concrete IoT Agent implementations will be to map between the native device The following figure offers a graphical example of how a COAP IoT Agent work, ordered from the registration of the device to a command update to the device. -![General ](./img/iotAgentLib.png "Architecture Overview") +![General ](../img/iotAgentLib.png 'Architecture Overview') ### The `TimeInstant` element diff --git a/doc/Contribution.md b/doc/devel/contribution-guidelines.md similarity index 70% rename from doc/Contribution.md rename to doc/devel/contribution-guidelines.md index bbd921fde..cfce3a297 100644 --- a/doc/Contribution.md +++ b/doc/devel/contribution-guidelines.md @@ -4,21 +4,22 @@ Before we get started, here are a few things we expect from you (and that you should expect from others): -* Be kind and thoughtful in your conversations around this project. We all come from different backgrounds and - projects, which means we likely have different perspectives on "how open source is done." Try to listen to others - rather than convince them that your way is correct. -* Please ensure that your contribution passes all tests. If there are test failures, you will need to address them - before we can merge your contribution. -* When adding content, please consider if it is widely valuable. Please don't add references or links to things you or - your employer have created as others will do so if they appreciate it. -* When reporting a vulnerability on the software, please, put in contact with IoT Agent Node Lib repository maintainers in order to discuss it - in a private way. +- Be kind and thoughtful in your conversations around this project. We all come from different backgrounds and + projects, which means we likely have different perspectives on "how open source is done." Try to listen to others + rather than convince them that your way is correct. +- Please ensure that your contribution passes all tests. If there are test failures, you will need to address them + before we can merge your contribution. +- When adding content, please consider if it is widely valuable. Please don't add references or links to things you or + your employer have created as others will do so if they appreciate it. +- When reporting a vulnerability on the software, please, put in contact with IoT Agent Node Lib repository + maintainers in order to discuss it in a private way. ## How to contribute -If you'd like to contribute, start by searching through the [issues](https://github.com/telefonicaid/iotagent-node-lib/issues) and -[pull requests](https://github.com/telefonicaid/iotagent-node-lib/pulls) to see whether someone else has raised a similar idea or -question. +If you'd like to contribute, start by searching through the +[issues](https://github.com/telefonicaid/iotagent-node-lib/issues) and +[pull requests](https://github.com/telefonicaid/iotagent-node-lib/pulls) to see whether someone else has raised a +similar idea or question. If you don't see your idea listed, and you think it fits into the goals of this guide, do one of the following: @@ -28,36 +29,43 @@ If you don't see your idea listed, and you think it fits into the goals of this ### Pull Request protocol -As explained in ([FIWARE Contribution Requirements](https://fiware-requirements.readthedocs.io/en/latest)) -contributions are done using a pull request (PR). The detailed "protocol" used in such PR is described below: - -* Direct commits to master branch (even single-line modifications) are not allowed. Every modification has to come as a PR -* In case the PR is implementing/fixing a numbered issue, the issue number has to be referenced in the body of the PR at creation time -* Anybody is welcome to provide comments to the PR (either direct comments or using the review feature offered by GitHub) -* Use *code line comments* instead of *general comments*, for traceability reasons (see comments lifecycle below) -* Comments lifecycle - * Comment is created, initiating a *comment thread* - * New comments can be added as responses to the original one, starting a discussion - * After discussion, the comment thread ends in one of the following ways: - * `Fixed in ` in case the discussion involves a fix in the PR branch (which commit hash is - included as reference) - * `NTC`, if finally nothing needs to be done (NTC = Nothing To Change) - * PR can be merged when the following conditions are met: - * All comment threads are closed - * All the participants in the discussion have provided a `LGTM` general comment (LGTM = Looks good to me) - * Self-merging is not allowed (except in rare and justified circumstances) +As explained in ([FIWARE Contribution Requirements](https://fiware-requirements.readthedocs.io/en/latest)) contributions +are done using a pull request (PR). The detailed "protocol" used in such PR is described below: + +- Direct commits to master branch (even single-line modifications) are not allowed. Every modification has to come as + a PR +- In case the PR is implementing/fixing a numbered issue, the issue number has to be referenced in the body of the PR + at creation time +- Anybody is welcome to provide comments to the PR (either direct comments or using the review feature offered by + GitHub) +- Use _code line comments_ instead of _general comments_, for traceability reasons (see comments lifecycle below) +- Comments lifecycle + - Comment is created, initiating a _comment thread_ + - New comments can be added as responses to the original one, starting a discussion + - After discussion, the comment thread ends in one of the following ways: + - `Fixed in ` in case the discussion involves a fix in the PR branch (which commit hash is + included as reference) + - `NTC`, if finally nothing needs to be done (NTC = Nothing To Change) +- PR can be merged when the following conditions are met: + - All comment threads are closed + - All the participants in the discussion have provided a `LGTM` general comment (LGTM = Looks good to me) +- Self-merging is not allowed (except in rare and justified circumstances) Some additional remarks to take into account when contributing with new PRs: -* PR must include not only code contributions, but their corresponding pieces of documentation (new or modifications to existing one) and tests -* PR modifications must pass full regression based on existing test (unit, functional, memory, e2e) in addition to whichever new test added due to the new functionality -* PR should be of an appropriated size that makes review achievable. Too large PRs could be closed with a "please, redo the work in smaller pieces" without any further discussing +- PR must include not only code contributions, but their corresponding pieces of documentation (new or modifications + to existing one) and tests +- PR modifications must pass full regression based on existing test (unit, functional, memory, e2e) in addition to + whichever new test added due to the new functionality +- PR should be of an appropriated size that makes review achievable. Too large PRs could be closed with a "please, + redo the work in smaller pieces" without any further discussing ## Community Discussions about the Open Source Guides take place on this repository's -[Issues](https://github.com/telefonicaid/iotagent-node-lib/issues) and [Pull Requests](https://github.com/telefonicaid/iotagent-node-lib/pulls) -sections. Anybody is welcome to join these conversations. +[Issues](https://github.com/telefonicaid/iotagent-node-lib/issues) and +[Pull Requests](https://github.com/telefonicaid/iotagent-node-lib/pulls) sections. Anybody is welcome to join these +conversations. Wherever possible, do not take these conversations to private channels, including contacting the maintainers directly. diff --git a/doc/devel/development.md b/doc/devel/development.md new file mode 100644 index 000000000..1bf2c1a29 --- /dev/null +++ b/doc/devel/development.md @@ -0,0 +1,1879 @@ +# Development documentation + +- [Preface](#preface) +- [Contributing](#contributing) +- [Project management](#project-management) + - [Installing dependencies](#installing-dependencies) + - [Project build](#project-build) + - [Testing](#testing) + - [Test requirements](#test-requirements) + - [Debug Test](#debug-test) + - [Continuous testing](#continuous-testing) + - [Code Coverage](#code-coverage) + - [Clean](#clean) + - [Checking code style](#checking-code-style) + - [Source code style validation - ESLint](#source-code-style-validation---eslint) + - [Documentation Markdown validation](#documentation-markdown-validation) + - [Documentation Spell-checking](#documentation-spell-checking) + - [Prettify Code](#prettify-code) +- [Library functions and modules](#library-functions-and-modules) + - [Stats Registry](#stats-registry) + - [Alarm module](#alarm-module) + - [Transactions](#transactions) + - [Library overview](#library-overview) + - [Function reference](#function-reference) + - [Generic middlewares](#generic-middlewares) +- [DB Models from API document](#db-models-from-api-document) + - [Service group model](#service-group-model) + - [Device model](#device-model) +- [Developing a new IoT Agent](#developing-a-new-iot-agent) + - [Protocol](#protocol) + - [Requirements](#requirements) + - [Basic IOTA](#basic-iot-agent) + - [IOTA With Active attributes](#iot-agent-with-active-attributes) + - [IOTA With Lazy attributes](#iota-with-lazy-attributes) + - [Previous considerations](#previous-considerations) + - [Implementation](#implementation) + - [IoT Agent in multi-thread mode](#iot-agent-in-multi-thread-mode) + - [Configuration management](#configuration-management) + - [Provisioning handlers](#provisioning-handlers) + +## Preface + +The **IoT Agent node library** as the name suggests is a library that provides a set of functions that can be used by +IoT Agents to implement the northbound interface. The library is used by several FIWARE IoT Agents, such as: + +a standalone library that can be used by any IoT Agent to implement the northbound interface, + +is not a standalone product and should be added as a dependency to `package.json` of the IoT Agent. + +```json +... +"dependencies": { + "iotagent-node-lib": "*", +} +``` + +In order to use the library within your own IoT Agent, you must first you require it before use: + +```javascript +const iotagentLib = require('iotagent-node-lib'); +``` + +This file contains the documentation for developers who wish to contribute to the **IoT Agent node library** project and +also for those who wish to use the library within their own IoT Agent project. + +## Contributing + +Contributions to this project are welcome. Developers planning to contribute should follow the +[Contribution Guidelines](contribution-guidelines.md) + +## Project management + +The **IoT Agent node library** project is managed using [npm](https://www.npmjs.com/). The following sections show the +available options in detail: + +### Installing dependencies + +This is the first step to be executed after cloning the project. To install them, type the following command: + +```bash +npm install +``` + +### Project build + +The project is managed using npm. + +For a list of available task, type + +```bash +npm run +``` + +The following sections show the available options in detail. + +### Testing + +[Mocha](https://mochajs.org/) Test Runner + [Should.js](https://shouldjs.github.io/) Assertion Library. + +The test environment is preconfigured to run BDD testing style. + +Module mocking during testing can be done with [proxyquire](https://github.com/thlorenz/proxyquire) + +To run tests, type + +```bash +npm test +``` + +There are additional targets starting with `test:` prefix to run specific test subsets isolated. For instance, the +`test:expressions` target runs the subset of tests related with expression language feature: + +```bash +npm run test:expressions +``` + +#### Test requirements + +A [MongoDB](https://www.mongodb.com/) 3.2+ instance is required to run tests. You can deploy one by using the commodity +`docker-compose-dev.yml`: + +``` +docker-compose -f docker-compose-dev.yml up -d +``` + +To run docker compose you will need [docker](https://docs.docker.com/get-docker/) and +[docker-compose](https://docs.docker.com/compose/install/). + +#### Debug Test + +To debug the code while running run tests, type + +```bash +npm run test:debug +``` + +In the console the link to the debugger will be provided. You can connect to it via Chrome, for example, by opening the +following url: `chrome://inspect`. + +Additional debug clients are listed on [node.js](https://nodejs.org/en/docs/guides/debugging-getting-started/). + +#### Continuous testing + +Support for continuous testing by modifying a src file or a test. For continuous testing, type + +```bash +npm run test:watch +``` + +If you want to continuously check also source code style, use instead: + +```bash +npm run watch +``` + +#### Code Coverage + +Istanbul + +Analyze the code coverage of your tests. + +To generate an HTML coverage report under `site/coverage/` and to print out a summary, type + +```bash +# Use git-bash on Windows +npm run test:coverage +``` + +### Clean + +Removes `node_modules` and `coverage` folders, and `package-lock.json` file so that a fresh copy of the project is +restored. + +```bash +# Use git-bash on Windows +npm run clean +``` + +### Checking code style + +#### Source code style validation - ESLint + +Uses the provided `.eslintrc.json` flag file. To check source code style, type + +```bash +npm run lint +``` + +#### Documentation Markdown validation + +Checks the Markdown documentation for consistency + +```bash +# Use git-bash on Windows +npm run lint:md +``` + +#### Documentation Spell-checking + +Uses the provided `.textlintrc` flag file. To check the Markdown documentation for spelling and grammar errors, dead +links & etc. + +```bash +# Use git-bash on Windows +npm run lint:text +``` + +#### Prettify Code + +Runs the [prettier](https://prettier.io) code formatter to ensure consistent code style (whitespacing, parameter +placement and breakup of long lines etc.) within the codebase. + +```bash +# Use git-bash on Windows +npm run prettier +``` + +To ensure consistent Markdown formatting run the following: + +```bash +# Use git-bash on Windows +npm run prettier:text +``` + +## Library functions and modules + +### Stats Registry + +The library provides a mechanism for the periodic reporting of stats related to the library's work. In order to activate +the use of the periodic stats, it must be configured in the config file, as described in the +[Configuration](../admin.md#configuration) section. + +The Stats Registry holds two dictionaries, with the same set of stats. For each stat, one of the dictionaries holds the +historical global value and the other one stores the value since the last value reporting (or current value). + +The stats library currently stores only the following values: + +- **deviceCreationRequests**: number of Device Creation Requests that arrived to the API (no matter the result). +- **deviceRemovalRequests**: number of Removal Device Requests that arrived to the API (no matter the result). +- **measureRequests**: number of times the ngsiService.update() function has been invoked (no matter the result). + +More values will be added in the future to the library. The applications using the library can add values to the Stats +Registry just by using the following function: + +```javascript +iotagentLib.statsRegistry.add('statName', statIncrementalValue, callback); +``` + +The first time this function is invoked, it will add the new stat to the registry. Subsequent calls will add the value +to the specified stat both to the current and global measures. The stat will be cleared in each interval as usual. + +### Alarm module + +The library provide an alarm module that can be used to track through the logs alarms raised in the IoTAgent. This +module provides: + +- Two functions to raise and release and alarm (`raise()` and `release()`): every alarm is identified by a name and a + description. When the alarm is raised, an error with the text `Raising [%s]` is logged. When the alarm is released, + the corresponding text, `Releasing [%s]` is logged. If an alarm is raised multiple times, it is only logged once. If + its released multiple times it is only released once. Releasing a non-existing alarm has no effect. + +- Functions to list all the raised alarms and clean all the alarms (`list()` and `clean()`). + +- A function to instrument other functions, so when one of that functions return an error, an alarm is raised, and + when it returns a success an alarm is ceased (`intercept()`). + +All this functions can be accessed through the `.alarms` attribute of the library. + +### Transactions + +The library implements a concept of transactions, in order to follow the execution flow the library follows when +treating requests entering both from the North and the South ports of the IoT Agent. + +To follow the transactions, a new Domain is created for each incoming request; in the case of requests received on the +North Port of the IoT Agent, this domain is automatically created by a Express middleware, and no further action is +needed from the user. For the case of requests received on the South Port of the IoT Agent, the user is responsible of +creating an stopping the transaction, using the `ensureSouthboundDomain` and `finishSouthBoundTransaction`. In this +case, the transaction will last from the invocation to the former to the invocation of the latter. + +The Transaction Correlator is used along all the IoT Platform to follow the trace of a transaction between multiple +components. To do so, in all the HTTP requests sent to other components of the platform, a custom header named +`Fiware-Correlator` is sent with the correlator of the transaction that generated the request. If a component of the +platform receives a request containing this header that starts a transaction, the component will create the transaction +with the received correlator, instead of creating a new one. If the header is not present or the transaction originates +in the component, the transaction ID in this component will be used as the correlator. + +During the duration of a transaction, all the log entries created by the code will write the current Transaction ID and +correlator for the operation being executed. + +### Library overview + +In order to use the library, add the following dependency to your package.json file: + +```json +"iotagent-node-lib": "*" +``` + +In order to use this library, first you must require it: + +```javascript +var iotagentLib = require('iotagent-node-lib'); +``` + +The library supports four groups of features, one for each direction of the communication: client-to-server and +server-to-client (and each flow both for the client and the server). Each feature set is defined in the following +sections. + +### Function reference + +> **WARNING** This section is outdated. Functions described here may be outdated and not reflect the current +> implementation of the IoT Agent Library. You could have a look to [iotagentLib.js](./lib/iotagentLib.js) file to see +> the current detail of functions implemented. + +The following fucntions are available in the library: + +- [iotagentLib.activate()](#iotagentlibactivate) +- [iotagentLib.deactivate()](#iotagentlibdeactivate) +- [iotagentLib.register()](#iotagentlibregister) +- [iotagentLib.unregister()](#iotagentlibunregister) +- [iotagentLib.update()](#iotagentlibupdate) +- [iotagentLib.setCommandResult()](#iotagentlibsetcommandresult) +- [iotagentLib.listDevices()](#iotagentliblistdevices) +- [iotagentLib.setDataUpdateHandler()](#iotagentlibsetdataupdatehandler) +- [iotagentLib.setDataQueryHandler()](#iotagentlibsetdataqueryhandler) +- [iotagentLib.setNotificationHandler()](#iotagentlibsetnotificationhandler) +- [iotagentLib.setCommandHandler()](#iotagentlibsetcommandhandler) +- [iotagentLib.setMergePatchHandler()](#iotagentlibsetmergepatchhandler) +- [iotagentLib.setProvisioningHandler()](#iotagentlibsetprovisioninghandler) +- [iotagentLib.setRemoveDeviceHandler()](#iotagentlibsetremovedevicehandler) +- [iotagentLib.setConfigurationHandler()](#iotagentlibsetconfigurationhandler) +- [iotagentLib.setRemoveConfigurationHandler()](#iotagentlibsetremoveconfigurationhandler) +- [iotagentLib.getDevice()](#iotagentlibgetdevice) +- [iotagentLib.getDeviceByName()](#iotagentlibgetdevicebyname) +- [iotagentLib.getDevicesByAttribute()](#iotagentlibgetdevicesbyattribute) +- [iotagentLib.retrieveDevice()](#iotagentlibretrievedevice) +- [iotagentLib.mergeDeviceWithConfiguration()](#iotagentlibmergedevicewithconfiguration) +- [iotagentLib.getConfiguration()](#iotagentlibgetconfiguration) +- [iotagentLib.findConfiguration()](#iotagentlibfindconfiguration) +- [iotagentLib.getEffectiveApiKey()](#iotagentlibgeteffectiveapikey) +- [iotagentLib.subscribe()](#iotagentlibsubscribe) +- [iotagentLib.unsubscribe()](#iotagentlibunsubscribe) +- [iotagentLib.ensureSouthboundDomain()](#iotagentlibensuresouthbounddomain) +- [iotagentLib.finishSouthBoundTransaction()](#iotagentlibfinishsouthboundtransaction) +- [iotagentLib.startServer()](#iotagentlibstartserver) +- [iotagentLib.request()](#iotagentlibrequest) + +##### iotagentLib.activate() + +###### Signature + +```javascript +function activate(newConfig, callback) +``` + +###### Description + +Activates the IoT Agent to start listening for NGSI Calls (acting as a Context Provider). It also creates the device +registry for the IoT Agent (based on the deviceRegistry.type configuration option). + +###### Params + +- newConfig: Configuration of the Context Server (described in the [Configuration](../admin.md#configuration) + section). + +##### iotagentLib.deactivate() + +###### Signature + +```javascript +function deactivate(callback) +``` + +###### Description + +Stops the HTTP server. + +###### Params + +##### iotagentLib.register() + +###### Signature + +```javascript +function registerDevice(deviceObj, callback) +``` + +###### Description + +Register a new device in the IoT Agent. This registration will also trigger a Context Provider registration in the +Context Broker for all its lazy attributes. + +The device Object can have the following attributes: + +- `id`: Device ID of the device. +- `type`: type to be assigned to the device. +- `name`: name that will be used for the Entity representing the device in the Context Broker. +- `service`: name of the service associated with the device. +- `subservice`: name of the subservice associated with th device. +- `lazy`: list of lazy attributes with their types. +- `active`: list of active attributes with their types. +- `staticAttributes`: list of NGSI attributes to add to the device entity 'as is' in updates, queries and + registrations. +- `internalAttributes`: optional section with free format, to allow specific IoT Agents to store information along + with the devices in the Device Registry. + +The device `id` and `type` are required fields for any registration. The rest of the attributes are optional, but, if +they are not present in the function call arguments, the type must be registered in the configuration, so the service +can infer their default values from the configured type. If an optional attribute is not given in the parameter list and +there isn't a default configuration for the given type, a TypeNotFound error is raised. + +If the device has been previously preprovisioned, the missing data will be completed with the values from the registered +device. + +###### Params + +- deviceObj: object containing all the information about the device to be registered (mandatory). + +##### iotagentLib.unregister() + +###### Signature + +```javascript +function unregisterDevice(id, service, subservice, callback) +``` + +###### Description + +Unregister a device from the Context broker and the internal registry. + +###### Params + +- id: Device ID of the device to register. +- service: Service of the device to unregister. +- subservice: Subservice inside the service for the unregistered device. + +##### iotagentLib.update() + +###### Signature + +```javascript +function update(entityName, attributes, typeInformation, token, callback) +``` + +###### Description + +Makes an update in the Device's entity in the context broker, with the values given in the 'attributes' array. This +array should comply to the NGSI's attribute format. + +###### Params + +- entityName: Name of the entity to register. +- attributes: Attribute array containing the values to update. +- typeInformation: Configuration information for the device. +- token: User token to identify against the PEP Proxies (optional). + +##### iotagentLib.setCommandResult() + +###### Signature + +```javascript +function setCommandResult(entityName, resource, apikey, commandName, commandResult, status, deviceInformation, callback) +``` + +###### Description + +Update the result of a command in the Context Broker. The result of the command has two components: the result of the +command itself will be represented with the suffix `_info` in the entity while the status is updated in the attribute +with the `_status` suffix. + +###### Params + +- entityName: Name of the entity holding the command. +- resource: Resource name of the endpoint the device is calling. +- apikey: Apikey the device is using to send the values (can be the empty string if none is needed). +- commandName: Name of the command whose result is being updated. +- commandResult: Result of the command in string format. +- deviceInformation: Device information, including security and service information. (optional). + +##### iotagentLib.listDevices() + +###### Signature + +```javascript +function listDevices(callback) +function listDevices(limit, offset, callback) +function listDevices(service, subservice, limit, offset, callback) +``` + +###### Description + +Return a list of all the devices registered in the specified service and subservice. This function can be invoked in +three different ways: + +- with just one parameter (the callback) +- with three parameters (service, subservice and callback) +- or with five parameters (including limit and offset). + +###### Params + +- service: service from where the devices will be retrieved. +- subservice: subservice from where the devices will be retrieved. +- limit: maximum number of results to retrieve (optional). +- offset: number of results to skip from the listing (optional). + +##### iotagentLib.setDataUpdateHandler() + +###### Signature + +```javascript +function setDataUpdateHandler(newHandler) +``` + +###### Description + +Sets the new user handler for Entity update requests. This handler will be called whenever an update request arrives +with the following parameters: (`id`, `type`, `service`, `subservice`, `attributes`, `callback`). Every object within of +the `attributes` array contains `name`, `type` and `value` attributes, and may also include additional attributes for +`metadata` and `datasetId`. The handler is in charge of updating the corresponding values in the devices with the +appropriate protocol. + +Once all the updates have taken place, the callback must be invoked with the updated Context Element. E.g.: + +```javascript +callback(null, { + type: 'TheType', + isPattern: false, + id: 'EntityID', + attributes: [ + { + name: 'lumniscence', + type: 'Lumens', + value: '432' + } + ] +}); +``` + +In the case of NGSI requests affecting multiple entities, this handler will be called multiple times, one for each +entity, and all the results will be combined into a single response. + +###### Params + +- newHandler: User handler for update requests + +##### iotagentLib.setDataQueryHandler() + +###### Signature + +```javascript +function setDataQueryHandler(newHandler) +``` + +###### Description + +Sets the new user handler for Entity query requests. This handler will be called whenever a query request arrives, with +the following parameters: (`id`, `type`, `service`, `subservice`, `attributes`, `callback`). The handler must retrieve +all the corresponding information from the devices and return a NGSI entity with the requested values. + +The callback must be invoked with the updated Context Element, using the information retrieved from the devices. E.g.: + +```javascript +callback(null, { + type: 'TheType', + isPattern: false, + id: 'EntityID', + attributes: [ + { + name: 'lumniscence', + type: 'Lumens', + value: '432' + } + ] +}); +``` + +In the case of NGSI requests affecting multiple entities, this handler will be called multiple times, one for each +entity, and all the results will be combined into a single response. + +###### Params + +- newHandler: User handler for query requests. + +##### iotagentLib.setNotificationHandler() + +###### Signature + +```javascript +function setNotificationHandler(newHandler) +``` + +###### Description + +Sets the new handler for incoming notifications. The notifications are sent by the Context Broker based on the IoT Agent +subscriptions created with the `subscribe()` function. + +The handler must adhere to the following signature: + +```javascript +function mockedHandler(device, data, callback) +``` + +The `device` parameter contains the device object corresponding to the entity whose changes were notified with the +incoming notification. Take into account that multiple entities may be modified with each single notification. The +handler will be called once for each one of those entities. + +The `data` parameter is an array with all the attributes that were requested in the subscription and its respective +values. + +The handler is expected to call its callback once with no parameters (failing to do so may cause unexpected behaviors in +the IoT Agent). + +##### iotagentLib.setCommandHandler() + +###### Signature + +```javascript +function setCommandHandler(newHandler) +``` + +###### Description + +Sets the new user handler for registered entity commands. This handler will be called whenever a command request +arrives, with the following parameters: (`id`, `type`, `service`, `subservice`, `attributes`, `callback`). The handler +must retrieve all the corresponding information from the devices and return a NGSI entity with the requested values. + +The callback must be invoked with the updated Context Element, using the information retrieved from the devices. E.g.: + +```javascript +callback(null, { + type: 'TheType', + isPattern: false, + id: 'EntityID', + attributes: [ + { + name: 'lumniscence', + type: 'Lumens', + value: '432' + } + ] +}); +``` + +In the case of NGSI requests affecting multiple entities, this handler will be called multiple times, one for each +entity, and all the results will be combined into a single response. Only IoT Agents which deal with actuator devices +will include a handler for commands. + +###### Params + +- newHandler: User handler for command requests. + +##### iotagentLib.setMergePatchHandler() + +###### Signature + +```javascript +function setMergePatchHandler(newHandler) +``` + +###### Description + +Sets the new user handler for NGSI-LD Entity [merge-patch](https://datatracker.ietf.org/doc/html/rfc7386) requests. This +handler will be called whenever a merge-patch request arrives, with the following parameters: (`id`, `type`, `service`, +`subservice`, `attributes`, `callback`). The handler must retrieve all the corresponding information from the devices +and return a NGSI entity with the requested values. + +The callback must be invoked with the updated Context Element, using the information retrieved from the devices. E.g.: + +```javascript +callback(null, { + type: 'TheType', + isPattern: false, + id: 'EntityID', + attributes: [ + { + name: 'lumniscence', + type: 'Lumens', + value: '432' + } + ] +}); +``` + +In the case of NGSI-LD requests affecting multiple entities, this handler will be called multiple times. Since +merge-patch is an advanced function, not all IoT Agents will include a handler for merge-patch. + +###### Params + +- newHandler: User handler for merge-patch requests. + +##### iotagentLib.setProvisioningHandler() + +###### Signature + +```javascript +function setProvisioningHandler (newHandler) +``` + +###### Description + +Sets the new user handler for the provisioning of devices. This handler will be called every time a new device is +created. + +The handler must adhere to the following signature: + +```javascript +function(newDevice, callback) +``` + +The `newDevice` parameter will contain the newly created device. The handler is expected to call its callback with no +parameters (this handler should only be used for reconfiguration purposes of the IoT Agent). + +##### iotagentLib.setRemoveDeviceHandler() + +###### Signature + +```javascript +function setRemoveDeviceHandler(newHandler) +``` + +###### Description + +Sets the new user handler for the removal of a device. This handler will be called every time a device is removed. + +The handler must adhere to the following signature: + +```javascript +function(deviceToDelete, callback) +``` + +The `deviceToDelete` parameter will contain the device to be deleted. The handler is expected to call its callback with +no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent). + +##### iotagentLib.setConfigurationHandler() + +###### Signature + +```javascript +function setConfigurationHandler(newHandler) +``` + +###### Description + +Sets the new user handler for the configuration updates. This handler will be called every time a new configuration is +created or an old configuration is updated. + +The handler must adhere to the following signature: + +```javascript +function(newConfiguration, callback) +``` + +The `newConfiguration` parameter will contain the newly created configuration. The handler is expected to call its +callback with no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent). + +For the cases of multiple updates (a single Device Configuration POST that will create several device groups), the +handler will be called once for each of the configurations (both in the case of the creations and the updates). + +The handler will be also called in the case of updates related to configurations. In that situation, the +`newConfiguration` parameter contains also the fields needed to identify the configuration to be updated, i.e., +`service`, `subservice`, `resource` and `apikey`. + +##### iotagentLib.setRemoveConfigurationHandler() + +###### Signature + +```javascript +function setRemoveConfigurationHandler(newHandler) +``` + +###### Description + +Sets the new user handler for the removal of configuratios. This handler will be called every time a configuration is +removed. + +The handler must adhere to the following signature: + +```javascript +function(configurationToDelete, callback) +``` + +The `configurationToDelete` parameter will contain the configuration to be deleted. The handler is expected to call its +callback with no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent). + +##### iotagentLib.getDevice() + +###### Signature + +```javascript +function getDevice(deviceId, service, subservice, callback) +``` + +###### Description + +Retrieve all the information about a device from the device registry. + +###### Params + +- deviceId: ID of the device to be found. +- service: Service for which the requested device. +- subservice: Subservice inside the service for which the device is requested. + +##### iotagentLib.getDeviceByName() + +###### Signature + +```javascript +function getDeviceByName(deviceName, service, subservice, callback) +``` + +###### Description + +Retrieve a device from the registry based on its entity name. + +###### Params + +- deviceName: Name of the entity associated to a device. +- service: Service the device belongs to. +- subservice: Division inside the service. + +##### iotagentLib.getDevicesByAttribute() + +###### Signature + +```javascript +function getDevicesByAttribute(attributeName, attributeValue, service, subservice, callback) +``` + +###### Description + +Retrieve all the devices having an attribute named `name` with value `value`. + +###### Params + +- name: name of the attribute to match. +- value: value to match in the attribute. +- service: Service the device belongs to. +- subservice: Division inside the service. + +##### iotagentLib.retrieveDevice() + +###### Signature + +```javascript +function retrieveDevice(deviceId, apiKey, callback) +``` + +###### Description + +Retrieve a device from the device repository based on the given APIKey and DeviceID, creating one if none is found for +the given data. + +###### Params + +- deviceId: Device ID of the device that wants to be retrieved or created. +- apiKey: APIKey of the Device Group (or default APIKey). + +##### iotagentLib.mergeDeviceWithConfiguration() + +###### Signature + +```javascript +function mergeDeviceWithConfiguration(fields, defaults, deviceData, configuration, callback) +``` + +###### Description + +Complete the information of the device with the information in the configuration group (with precedence of the device). +The first argument indicates what fields would be merged. + +###### Params + +- fields: Fields that will be merged. +- defaults: Default values fot each of the fields. +- deviceData: Device data. +- configuration: Configuration data. + +##### iotagentLib.getConfiguration() + +###### Signature + +```javascript +function getConfiguration(resource, apikey, callback) +``` + +###### Description + +Gets the device group identified by the given (`resource`, `apikey`) pair. + +###### Params + +- resource: representation of the configuration in the IoT Agent (dependent on the protocol) . +- apikey: special key the devices will present to prove they belong to a particular configuration. + +##### iotagentLib.findConfiguration() + +###### Signature + +```javascript +function findConfiguration(service, subservice, callback) +``` + +###### Description + +Find a device group based on its service and subservice. + +###### Params + +- service: name of the service of the configuration. +- subservice: name of the subservice of the configuration. + +##### iotagentLib.getEffectiveApiKey() + +###### Signature + +```javascript +function getEffectiveApiKey(service, subservice, type, callback) +``` + +###### Description + +Get the API Key for the selected service if there is any, or the default API Key if a specific one does not exist. + +###### Params + +- service: Name of the service whose API Key we are retrieving. +- subservice: Name of the subservice whose API Key we are retrieving. +- type: Type of the device. + +##### iotagentLib.subscribe() + +###### Signature + +```javascript +function subscribe(device, triggers, content, callback) +``` + +###### Description + +Creates a subscription for the IoTA to the entity representing the selected device. + +###### Params + +- device: Object containing all the information about a particular device. +- triggers: Array with the names of the attributes that would trigger the subscription +- content: Array with the names of the attributes to retrieve in the notification. + +##### iotagentLib.unsubscribe() + +###### Signature + +```javascript +function unsubscribe(device, id, callback) +``` + +###### Description + +Removes a single subscription from the selected device, identified by its ID. + +###### Params + +- `device`: Object containing all the information about a particular device. +- `id`: ID of the subscription to remove. + +##### iotagentLib.ensureSouthboundDomain() + +###### Signature + +```javascript +function ensureSouthboundTransaction(context, callback) +``` + +###### Description + +Ensures that the current operation is executed inside a transaction with all the information needed for the appropriate +platform logging: start date, transaction ID and correlator in case one is needed. If the function is executed in the +context of a previous transaction, just the context is changed (and the Transaction ID and start time are kept). + +###### Params + +- context: New context data for the transaction. + +##### iotagentLib.finishSouthBoundTransaction() + +###### Signature + +```javascript +function finishSouthboundTransaction(callback) +``` + +###### Description + +Terminates the current transaction, if there is any, cleaning its context. + +##### iotagentLib.startServer() + +###### Signature + +```javascript +function startServer(newConfig, iotAgent, callback) +``` + +###### Description + +Start the HTTP server either in single-thread or multi-thread (multi-core) based on the value of _multiCore_ variable +(described in the [Configuration](../admin.md#configuration) section). If the value is `False` (either was directly +specified `False` in the `config.js` or it was not specified and by default is assigned `False`), it is a normal +(single-thread) behaviour. Nevertheless, if _multiCore_ is `True`, the IoTAgent is executed in multi-thread environment. + +The number of parallel processes is calculated based on the number of available CPUs. In case of some of the process +unexpectedly dead, a new process is created automatically to keep always the maximum of them working in parallel. + +> Note: `startServer()` initializes the server but it does not activate the library. The function in the Node Lib will +> call the `iotAgent.start()` in order to complete the activation of the library. Therefore, it is expected that the IoT +> Agent implement the `iotAgent.start()` function with the proper invocation to the `iotAgentLib.activate()`. + +###### Params + +- newConfig: Configuration of the Context Server (described in the [Configuration](../admin.md#configuration) + section). +- iotAgent: The IoT Agent Objects, used to start the agent. +- callback: The callback function. + +##### iotagentLib.request() + +###### Signature + +```javascript +function request(options, callback) +``` + +###### Description + +Make a direct HTTP request using the underlying request library (currently [got](https://github.com/sindresorhus/got)), +this is useful when creating agents which use an HTTP transport for their southbound commands, and removes the need for +the custom IoT Agent to import its own additional request library + +###### Params + +- options: definition of the request (see + [got options](https://github.com/sindresorhus/got/blob/main/documentation/2-options.md) for more details). The + following attributes are currently exposed. + - `method` - HTTP Method + - `searchParams` - query string params + - `qs` - alias for query string params + - `headers` + - `responseType` - either `text` or `json`. `json` is the default + - `json` - a supplied JSON object as the request body + - `body` - any ASCII text as the request body. It takes precedence over `json` if both are provided at the same + time (not recommended). + - `url` - the request URL + - `uri` - alternative alias for the request URL. +- callback: The callback currently returns an `error` Object, the `response` and `body`. The `body` is parsed to a + JSON object if the `responseType` is JSON. + +#### Generic middlewares + +This collection of utility middlewares is aimed to be used to north of the IoT Agent Library, as well as in other +HTTP-based APIs of the IoT Agents. All the middlewares follow the Express convention of `(req, res, next)` objects, so +this information will not be repeated in the descriptions for the middleware functions. All the middlewares can be added +to the servers using the standard Express mechanisms. + +##### iotagentLib.middlewares.handleError() + +###### Signature + +```javascript +function handleError(error, req, res, next) +``` + +###### Description + +Express middleware for handling errors in the IoTAs. It extracts the code information to return from the error itself +returning 500 when no error code has been found. + +##### iotagentLib.middlewares.traceRequest() + +###### Signature + +```javascript +function traceRequest(req, res, next) +``` + +###### Description + +Express middleware for tracing the complete request arriving to the IoTA in debug mode. + +##### iotagentLib.middlewares.changeLogLevel() + +###### Signature + +```javascript +function changeLogLevel(req, res, next) +``` + +###### Description + +Changes the log level to the one specified in the request. + +##### iotagentLib.middlewares.ensureType() + +###### Signature + +```javascript +function ensureType(req, res, next) +``` + +###### Description + +Ensures the request type is one of the supported ones. + +##### iotagentLib.middlewares.validateJson() + +###### Signature + +```javascript +function validateJson(template) +``` + +###### Description + +Generates a Middleware that validates incoming requests based on the JSON Schema template passed as a parameter. + +Returns an Express middleware used in request validation with the given template. + +###### Params + +- _template_: JSON Schema template to validate the request. + +##### iotagentLib.middlewares.retrieveVersion() + +###### Signature + +```javascript +function retrieveVersion(req, res, next) +``` + +###### Description + +Middleware that returns all the IoTA information stored in the module. + +##### iotagentLib.middlewares.setIotaInformation() + +###### Signature + +```javascript +function setIotaInformation(newIoTAInfo) +``` + +###### Description + +Stores the information about the IoTAgent for further use in the `retrieveVersion()` middleware. + +###### Params + +- _newIoTAInfo_: Object containing all the IoTA Information. + +## DB Models (from API document) + +> **WARNING** This section is outdated. DB fields described here may be outdated and not reflect the current +> implementation of the IoT Agent Library. + +The following sections describe the models used in the database to store the information about the devices and the +config groups. + +### Config group model + +The table below shows the information held in the Config group provisioning resource and the correspondence between the +API resource fields and the same fields in the database model. + +You can find the description of the fields in the config group datamodel of the +[API document](../api.md#config-group-datamodel). + +| Payload Field | DB Field | Note | +| ------------------------------ | ------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------ | +| `service` | `service` | | +| `subservice` | `subservice` | | +| `resource` | `resource` | | +| `apikey` | `apikey` | | +| `timestamp` | `timestamp` | | +| `entity_type` | `entity_type` | | +| `trust` | `trust` | | +| `cbHost` | `cbHost` | | +| `lazy` | `lazy` | | +| `commands` | `commands` | | +| `attributes` | `attributes` | | +| `static_attributes` | `staticAttributes` | | +| `internal_attributes` | `internalAttributes` | | +| `explicitAttrs` | `explicitAttrs` | | +| `entityNameExp` | `entityNameExp` | | +| `ngsiVersion` | `ngsiVersion` | | +| `defaultEntityNameConjunction` | `defaultEntityNameConjunction` | optional string value to set default conjunction string used to compose a default `entity_name` when is not provided at device provisioning time. | +| `autoprovision` | `autoprovision` | | + +### Device model + +The table below shows the information held in the Device resource. The table also contains the correspondence between +the API resource fields and the same fields in the database model. + +You can find the description of the fields in the config group datamodel of the +[API document](../api.md#device-datamodel). + +| Payload Field | DB Field | Note | +| --------------------- | -------------------- | :----------------------------------------------------------------------------------------------------------------------------------- | +| `device_id` | `id` | | +| `service` | `service` | | +| `service_path` | `subservice` | | +| `entity_name` | `name` | | +| `entity_type` | `type` | | +| `timezone` | `timezone` | | +| `timestamp` | `timestamp` | | +| `apikey` | `apikey` | | +| `endpoint` | `endpoint` | | +| `protocol` | `protocol` | Name of the device protocol, for its use with an IoT Manager. IE: IoTA-UL | +| `transport` | `transport` | | +| `attributes` | `active` | | +| `lazy` | `lazy` | | +| `commands` | `commands` | | +| `internal_attributes` | `internalAttributes` | List of internal attributes with free format for specific IoT Agent configuration. I.E:LWM2M mappings from object URIs to attributes | +| `static_attributes` | `staticAttributes` | | +| `explicitAttrs` | `explicitAttrs` | | +| `ngsiVersion` | `ngsiVersion` | | + +## Developing a new IoT Agent + +> **WARNING** This section is outdated. Methods and steps described here may be outdated and not reflect the current +> implementation of the IoT Agent Library. You could have a look to other IoT Agents developed using the IoT Agent +> Library to get a better idea of how to use it, like the +> [IoT Agent JSON](http://www.github.com/telefonicaid/iotagent-json) + +This section's goal is to show how to develop a new IoT Agent step by step. To do so, a simple invented HTTP protocol +will be used, so it can be tested with simple command-line instructions as `curl` and `nc`. + +### Protocol + +The invented protocol will be freely adapted from +[Ultralight 2.0](https://github.com/telefonicaid/fiware-IoTAgent-Cplusplus/blob/develop/doc/modules.md#ultra-light-agent). +Whenever a device wants to send an update, it will send a request as the following: + +```bash +curl -X GET 'http://127.0.0.1:8080/iot/d?i=ULSensor&k=abc&d=t|15,l|19.6' -i +``` + +Where: + +- **i**: is the device ID. +- **k**: the API Key for the device's service. +- **d**: the data payload, consisting of key-value pairs separated by a pipe (`|`), with each pair separated by comma + (`,`); + +### Requirements + +This tutorial expects a Node.js v8 (at least) installed and working on your machine. It also expects you to have access +to a Context Broker (without any security proxies). + +### Basic IoT Agent + +In this first chapter, we will just develop an IoT Agent with a fully connected North Port. This will send and receive +NGSI traffic and can be administered using the IoT Agent's Device Provisioning API. The South Port will remain +unconnected and no native protocol traffic will be sent to the devices. This may seem useless (and indeed it is) but it +will serve us well on showing the basic steps in the creation of an IoT Agent. + +First of all, we have to create the Node project. Create a folder to hold your project and type the following +instruction: + +```bash +npm init +``` + +This will create the `package.json` file for our project. Now, add the following lines to your project file: + +```json + "dependencies": { + "iotagent-node-lib": "*" + }, + +``` + +And install the dependencies, executing, as usual: + +```bash +npm install +``` + +The first step is to write a configuration file, that will be used to tune the behavior of our IOTA. The contents can be +copied from the following example: + +```javascript +var config = { + logLevel: 'DEBUG', + contextBroker: { + host: 'localhost', + port: '1026' + }, + server: { + port: 4041 + }, + deviceRegistry: { + type: 'memory' + }, + types: {}, + service: 'howtoService', + subservice: '/howto', + providerUrl: 'http://localhost:4041', + defaultType: 'Thing' +}; + +module.exports = config; +``` + +Create a `config.js` file with it in the root folder of your project. Remember to change the Context Broker IP to your +local Context Broker. + +Now we can begin with the code of our IoT Agent. The very minimum code we need to start an IoT Agent is the following: + +```javascript +var iotAgentLib = require('iotagent-node-lib'), + config = require('./config'); + +iotAgentLib.activate(config, function (error) { + if (error) { + console.log('There was an error activating the IOTA'); + process.exit(1); + } +}); +``` + +The IoT Agent is now ready to be used. Execute it with the following command: + +```bash +node index.js +``` + +The North Port interface should now be fully functional, i.e.: management of device registrations and configurations. + +### IoT Agent With Active attributes + +In the previous section we created an IoT Agent that exposed just the North Port interface, but that was pretty useless +(aside from its didactic use). In this section we are going to create a simple South Port interface. It's important to +remark that the nature of the traffic South of the IoT Agent itself has nothing to do with the creation process of an +IoT Agent. Each device protocol will use its own mechanisms and it is up to the IoT Agent developer to find any +libraries that would help him in its development. In this example, we will use Express as such library. + +In order to add the Express dependency to your project, add the following line to the `dependencies` section of the +`package.json`: + +```json + "express": "*", +``` + +The require section would end up like this (the standard `http` module is also needed): + +```javascript +var iotAgentLib = require('iotagent-node-lib'), + http = require('http'), + express = require('express'), + config = require('./config'); +``` + +And install the dependencies as usual with `npm install`. You will have to require both `express` and `http` in your +code as well. + +Now, in order to accept connections in our code, we have to start express first. With this purpose in mind, we will +create a new function `initSouthbound()`, that will be called from the initialization code of our IoT Agent: + +```javascript +function initSouthbound(callback) { + southboundServer = { + server: null, + app: express(), + router: express.Router() + }; + + southboundServer.app.set('port', 8080); + southboundServer.app.set('host', '0.0.0.0'); + + southboundServer.router.get('/iot/d', manageULRequest); + southboundServer.server = http.createServer(southboundServer.app); + southboundServer.app.use('/', southboundServer.router); + southboundServer.server.listen(southboundServer.app.get('port'), southboundServer.app.get('host'), callback); +} +``` + +This Express code sets up a HTTP server, listening in the 8080 port, that will handle incoming requests targeting path +`/iot/d` using the middleware `manageULRequest()`. This middleware will contain all the logic south of the IoT Agent, +and the library methods we need in order to progress the information to the Context Broker. The code of this middleware +would be as follows: + +```javascript +function manageULRequest(req, res, next) { + var values; + + iotAgentLib.retrieveDevice(req.query.i, req.query.k, function (error, device) { + if (error) { + res.status(404).send({ + message: "Couldn't find the device: " + JSON.stringify(error) + }); + } else { + values = parseUl(req.query.d, device); + iotAgentLib.update(device.name, device.type, '', values, device, function (error) { + if (error) { + res.status(500).send({ + message: 'Error updating the device' + }); + } else { + res.status(200).send({ + message: 'Device successfully updated' + }); + } + }); + } + }); +} +``` + +For this middleware we have made use of a function `parseUl()` that parses the data payload and transforms it in the +data object expected by the update function (i.e.: an attribute array with NGSI syntax): + +```javascript +function parseUl(data, device) { + function findType(name) { + for (var i = 0; i < device.active.length; i++) { + if (device.active[i].name === name) { + return device.active[i].type; + } + } + + return null; + } + + function createAttribute(element) { + var pair = element.split('|'), + attribute = { + name: pair[0], + value: pair[1], + type: findType(pair[0]) + }; + + return attribute; + } + + return data.split(',').map(createAttribute); +} +``` + +Here as an example of the output of the function return for the UL payload `t|15,l|19.6`: + +```json +[ + { + "name": "t", + "type": "celsius", + "value": "15" + }, + { + "name": "l", + "type": "meters", + "value": "19.6" + } +] +``` + +The last thing to do is to invoke the initialization function inside the IoT Agent startup function. The next excerpt +show the modifications in the `activate()` function: + +```javascript +iotAgentLib.activate(config, function (error) { + if (error) { + console.log('There was an error activating the IOTA'); + process.exit(1); + } else { + initSouthbound(function (error) { + if (error) { + console.log('Could not initialize South bound API due to the following error: %s', error); + } else { + console.log('Both APIs started successfully'); + } + }); + } +}); +``` + +Some logs were added in this piece of code to help debugging. + +Once the IOTA is finished the last thing to do is to test it. To do so, launch the IoT Agent and provision a new device +(an example for provisioning can be found in the `examples/howtoProvisioning1.json` file). Once the device is +provisioned, send a new measure by using the example command: + +```bash +curl -X GET 'http://127.0.0.1:8080/iot/d?i=ULSensor&k=abc&d=t|15,l|19.6' -i +``` + +Now you should be able to see the measures in the Context Broker entity of the device. + +### IOTA With Lazy attributes + +#### Previous considerations + +The IoT Agents also give the possibility for the device to be asked about the value of one of its measures, instead of +reporting it. In order to do so, the device must be capable of receiving messages of some kind. In this case, we are +going to simulate an HTTP server with `nc` in order to see the values sent by the IOTA. We also have to decide a syntax +for the protocol request for asking the device about a measure. For clarity, we will use the same HTTP GET request we +used to report a measure, but indicating the attribute to ask instead of the data payload. Something like: + +```bash +curl -X GET 'http://127.0.0.1:9999/iot/d?i=ULSensor&k=abc&q=t,l' -i +``` + +In a real implementation, the server will need to know the URL and port where the devices are listening, in order to +send the request to the appropriate device. For this example, we will assume that the device is listening in port 9999 +in localhost. For more complex cases, the mechanism to bind devices to addresses would be IoT-Agent-specific (e.g.: the +OMA Lightweight M2M IoT Agent captures the address of the device in the device registration, and stores the +device-specific information in a MongoDB document). + +Being lazy attributes of a read/write nature, another syntax has to be declared for updating. This syntax will mimic the +one used for updating the server: + +``` +curl -X GET 'http://127.0.0.1:9999/iot/d?i=ULSensor&k=abc&d=t|15,l|19.6' -i +``` + +Both types of calls to the device will be distinguished by the presence or absence of the `d` and `q` attributes. + +A HTTP request library will be needed in order to make those calls. To this extent, `mikeal/request` library will be +used. In order to do so, add the following require statement to the initialization code: + +```javascript +request = require('request'); +``` + +and add the `request` dependency to the `package.json` file: + +```json + "dependencies": [ + [...] + + "request": "*", + + ] +``` + +The require section should now look like this: + +```javascript +var iotAgentLib = require('iotagent-node-lib'), + http = require('http'), + express = require('express'), + request = require('request'), + config = require('./config'); +``` + +#### Implementation + +##### QueryContext implementation + +The main step to complete in order to implement the Lazy attributes mechanism in the IoT Agent is to provide handlers +for the context provisioning requests. At this point, we should provide two handlers: the `/v2/op/update` and the +`/v2/op/query` handlers. To do so, we must first define the handlers themselves: + +```javascript +function queryContextHandler(id, type, service, subservice, attributes, callback) { + var options = { + url: 'http://127.0.0.1:9999/iot/d', + method: 'GET', + qs: { + q: attributes.join() + } + }; + + request(options, function (error, response, body) { + if (error) { + callback(error); + } else { + callback(null, createResponse(id, type, attributes, body)); + } + }); +} +``` + +The queryContext handler is called whenever a `/v2/op/query` request arrives at the North port of the IoT Agent. It is +invoked once for each entity requested, passing the entity ID and Type as the parameters, as well as a list of the +attributes that are requested. In our case, the handler uses this parameters to compose a request to the device. Once +the results of the device are returned, the values are returned to the caller, in the NGSI attribute format. + +In order to format the response from the device in a readable way, we created a `createResponse()` function that maps +the values to its correspondent attributes. This function assumes the type of all the attributes is "string" (this will +not be the case in a real scenario, where the IoT Agent should retrieve the associated device to guess the type of its +attributes). Here is the code for the `createResponse()` function: + +```javascript +function createResponse(id, type, attributes, body) { + var values = body.split(','), + responses = []; + + for (var i = 0; i < attributes.length; i++) { + responses.push({ + name: attributes[i], + type: 'string', + value: values[i] + }); + } + + return { + id: id, + type: type, + attributes: responses + }; +} +``` + +##### UpdateContext implementation + +```javascript +function updateContextHandler(id, type, service, subservice, attributes, callback) { + var options = { + url: 'http://127.0.0.1:9999/iot/d', + method: 'GET', + qs: { + d: createQueryFromAttributes(attributes) + } + }; + + request(options, function (error, response, body) { + if (error) { + callback(error); + } else { + callback(null, { + id: id, + type: type, + attributes: attributes + }); + } + }); +} +``` + +The updateContext handler deals with the modification requests that arrive at the North Port of the IoT Agent via +`/v2/op/update`. It is invoked once for each entity requested (note that a single request can contain multiple entity +updates), with the same parameters used in the queryContext handler. The only difference is the value of the attributes +array, now containing a list of attribute objects, each containing name, type and value. The handler must also make use +of the callback to return a list of updated attributes. + +For this handler we have used a helper function called `createQueryFromAttributes()`, that transforms the NGSI +representation of the attributes to the UL type expected by the device: + +```javascript +function createQueryFromAttributes(attributes) { + var query = ''; + + for (var i in attributes) { + query += attributes[i].name + '|' + attributes[i].value; + + if (i != attributes.length - 1) { + query += ','; + } + } + + return query; +} +``` + +##### Handler registration + +Once both handlers have been defined, they have to be registered in the IoT Agent, adding the following code to the +setup function: + +```javascript +iotAgentLib.setDataUpdateHandler(updateContextHandler); +iotAgentLib.setDataQueryHandler(queryContextHandler); +``` + +Where necessary, additional handlers to deal with command actuations and merge-patch operations may also be added when +necessary. + +```javascript +iotAgentLib.setCommandHandler(commandHandler); +iotAgentLib.setMergePatchHandler(mergePatchHandler); +``` + +##### IOTA Testing + +In order to test it, we need to create an HTTP server simulating the device. The quickest way to do that may be using +netcat. In order to start it just run the following command from the command-line (Linux and Mac only): + +```bash +nc -l 9999 +``` + +This will open a simple TCP server listening on port `9999`, where the requests from the IoT Agent will be printed. In +order for the complete workflow to work (and to receive the response in the application side), the HTTP response has to +be written in the `nc` console (although for testing purposes this is not needed). + +While netcat is great to test simple connectivity, you will need something just a bit more complex to get the complete +scenario working (at least without the need to be incredibly fast sending your response). In order to do so, a simple +echo server was created, that answers 42 to any query to its `/iot/d` path. You can use it to test your attributes one +by one (or you can modify it to accept more requests and give more complex responses). Copy the +[Echo Server script](echo.js) to the same folder of your IoTAgent (as it uses the same dependencies). In order to run +the echo server, just execute the following command: + +```bash +node echo.js +``` + +Once the mock server has been started (either `nc` or the `echo` server), proceed with the following steps to test your +implementation: + +1. Provision a device with two lazy attributes. The following request can be used as an example: + +```text +POST /iot/devices HTTP/1.1 +Host: localhost:4041 +Content-Type: application/json +fiware-service: howtoserv +fiware-servicepath: /test +Cache-Control: no-cache +Postman-Token: 993ac66b-72da-9e96-ab46-779677a5896a + +{ + "devices": [ + { + "device_id": "ULSensor", + "entity_name": "Sensor01", + "entity_type": "BasicULSensor", + "lazy": [ + { + "name": "t", + "type": "celsius" + }, + { + "name": "l", + "type": "meters" + } + ], + "attributes": [ + ] + } + ] +} +``` + +2. Execute a `/v2/op/query` or `/v2/op/update` against one of the entity attributes (use a NGSI client of curl command). + +```text +POST /v2/op/query HTTP/1.1 +Host: localhost:1026 +Content-Type: application/json +Accept: application/json +Fiware-Service: howtoserv +Fiware-ServicePath: /test +Cache-Control: no-cache + +{ + entities: [ + { + id: 'Light:light1' + } + ], + attrs: ['dimming'] +} +``` + +3. Check the received request in the nc console is the expected one. + +4. (In case you use netcat). Answer the request with an appropriate HTTP response and check the result of the + `/v2/op/query` or `/v2/op/update` request is the expected one. An example of HTTP response, for a query to the `t` + and `l` attributes would be: + +```text +HTTP/1.0 200 OK +Content-Type: text/plain +Content-Length: 3 + +5,6 +``` + +This same response can be used both for updates and queries for testing purposes (even though in the former the body +won't be read). + +### IoT Agent in multi-thread mode + +It is possible that an IoT Agent can be executed in multi-thread approach, which will increase the number of +request/seconds that can be manage by the server. It's important to remark that the nature of this functionality in +included in the IoT Agent Node Lib but it is not mandatory that you activate this functionality. In this example, we +will see how to use this functionality to deploy an IoT Agent in multi-thread environment. + +In order to activate the functionality, you have two options, configure the `config.js` file to add the following line: + +```javascript +/** + * flag indicating whether the node server will be executed in multi-core option (true) or it will be a + * single-thread one (false). + */ +config.multiCore = true; +``` + +or you can define the proper IOTA_MULTI_CORE environment variable. By default, the first choice is the environment +variable and afterward the value of the multiCore in the `config.js` file. The require section would end up like this +(the standard `http` module is also needed): + +```javascript +var iotAgent = require('../lib/iotagent-implementation'), + iotAgentLib = require('iotagent-node-lib'), + config = require('./config'); +``` + +It is important to mention the purpose of the `iotAgent` variable. It is the proper implementation of the IoT Agent +based on the IoT Agent Node Lib. We will need this variable just to make a callback to the corresponding `start()` +process from the library. The variable `config` is used to get details of the configuration file and send that +information to the Node Lib. The Node Lib will take the decision of single-thread or multi-thread execution base on the +value of `config.multiCore` attribute. + +Finally, we can call the corresponding [iotagentLib.startServer()](#iotagentlibstartserver) like the following code with +a callback function to show details about any error during the execution or just print the message about starting the +IoTAgent: + +```javascript +iotAgentLib.startServer(config, iotAgent, function (error) { + if (error) { + console.log(context, 'Error starting IoT Agent: [%s] Exiting process', error); + } else { + console.log(context, 'IoT Agent started'); + } +}); +``` + +> Note: `startServer()` initializes the server but it does not activate the library. The function in the Node Lib will +> call the `iotAgent.start()` in order to complete the activation of the library. Therefore, it is expected that the IoT +> Agent implement the `iotAgent.start()` function with the proper invocation to the `iotAgentLib.activate()`. + +### Configuration management + +For some IoT Agents, it will be useful to know what devices or configurations were registered in the Agent, or to do +some actions whenever a new device is registered. All this configuration and provisioning actions can be performed using +two mechanisms: the provisioning handlers and the provisioning API. + +#### Provisioning handlers + +The handlers provide a way for the IoT Agent to act whenever a new device, or configuration is provisioned. This can be +used for registering the device in external services, for storing important information about the device, or to listen +in new ports in the case of new configuration. For the simple example we are developing, we will just print the +information we are receiving whenever a new device or configuration is provisioned. + +We need to complete two further steps to have a working set of provisioning handlers. First of all, defining the +handlers themselves. Here we can see the definition of the configuration handler: + +```javascript +function configurationHandler(configuration, callback) { + console.log('\n\n* REGISTERING A NEW CONFIGURATION:\n%s\n\n', JSON.stringify(configuration, null, 4)); + callback(null, configuration); +} +``` + +As we can see, the handlers receive the device or configuration that is being provisioned, as well as a callback. The +handler MUST call the callback once in order for the IOTA to work properly. If an error is passed as a parameter to the +callback, the provisioning will be aborted. If no error is passed, the provisioning process will continue. This +mechanism can be used to implement security mechanisms or to filter the provisioning of devices to the IoT Agent. + +Note also that the same `device` or `configuration` object is passed along to the callback. This lets the IoT Agent +change some of the values provisioned by the user, to add or restrict information in the provisioning. To test this +feature, let's use the provisioning handler to change the value of the type of the provisioning device to +`CertifiedType` (reflecting some validation process performed on the provisioning): + +```javascript +function provisioningHandler(device, callback) { + console.log('\n\n* REGISTERING A NEW DEVICE:\n%s\n\n', JSON.stringify(device, null, 4)); + device.type = 'CertifiedType'; + callback(null, device); +} +``` + +Once the handlers are defined, the new set of handlers has to be registered into the IoT Agent: + +```javascript +iotAgentLib.setConfigurationHandler(configurationHandler); +iotAgentLib.setProvisioningHandler(provisioningHandler); +``` + +Now we can test our implementation by sending provisioning requests to the North Port of the IoT Agent. If we provision +a new device into the platform, and then we ask for the list of provisioned devices, we shall see the type of the +provisioned device has changed to `CertifiedType`. diff --git a/doc/echo.js b/doc/devel/echo.js similarity index 100% rename from doc/echo.js rename to doc/devel/echo.js diff --git a/doc/finalResult.js b/doc/devel/finalResult.js similarity index 100% rename from doc/finalResult.js rename to doc/devel/finalResult.js diff --git a/doc/northboundinteractions.md b/doc/devel/northboundinteractions.md similarity index 96% rename from doc/northboundinteractions.md rename to doc/devel/northboundinteractions.md index ff209c357..0554dbae4 100644 --- a/doc/northboundinteractions.md +++ b/doc/devel/northboundinteractions.md @@ -145,7 +145,7 @@ This **NGSI-v2** payload is associated to an update operation (POST `/v2/op/upda } } ], - "actionType": "update" + "actionType": "append" } ``` @@ -155,9 +155,8 @@ As it can be seen in the example, the payload is a JSON Object with the followin with the information needed to identify the target entity `id` and `type` attributes. The `entities` attribute is an array, so a single update context batch operation can be used to update multiple devices -- An `actionType` indicating the type of update: if this attribute has the value `"append"` the appropriate entity and - attributes will be created if the don't exist; if the value is `"update"`, an error will be thrown if the target - resources don't exist. +- An `actionType` indicating the type of update. It has the value `"append"` the appropriate entity and attributes + will be created if the don't exist. The equivalent **NGSI-LD** payload is associated to an update operation (PATCH `/ngsi-ld/v1/entities//attrs/`). @@ -427,7 +426,7 @@ Be sure to understand how each scenario works (as shown in the theory section) b Along this document, IP addresses and passwords will be concealed. Substitute the concealed passwords by your own. A postman collection is available alongside this document to help in reproducing this examples. It can be found -[here](./doc/NorthboundInteractions.postman_collection). +[here](NorthboundInteractions.postman_collection). ### Retrieving a token @@ -507,8 +506,8 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" - } } ], - "actionType": "update" -} ' "https://:10027/v2/op/update" + "actionType": "append" +} ' "https://:1026/v2/op/update" ``` If the request is correct, the Context Broker will reply with the following R1 response (200 OK): @@ -554,7 +553,7 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" - } ], "attrs": ["temperature","pressure"] -}' "https://:10027/v2/op/query" +}' "https://:1026/v2/op/query" ``` The Context Broker will reply with the updated data values in R2 format (200 OK): @@ -600,20 +599,6 @@ It is worth mentioning that the Context Broker will reply with a 200 OK status c refer to transport protocol level errors, while the status codes inside of a payload give information about the application level protocol. -The example shows an error updating an non-existent attribute (due to the use of UPDATE instead of APPEND). - -The following error payload is also valid in standard NGSI: - -```json -{ - "error": "NotFound", - "description": "The requested entity has not been found. Check type and id" -} -``` - -Different kinds of errors can return their information in different formats, so NGSI implementations should check for -the existence of both. - ### Scenario 2: lazy attributes (happy path) Scenario 2 relies on the Context Provider mechanism of the Context Broker. For this scenario to work, the IoTAgent must @@ -639,7 +624,7 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" - } } } -' "https://:10027/v2/registrations" +' "https://:1026/v2/registrations" ``` If everything has gone OK, the Context Broker will return the following payload: @@ -669,7 +654,7 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" - } ], "attrs": ["batteryLevel"] -}' "https://:10027/v2/op/query" +}' "https://:1026/v2/op/query" ``` The Context Broker receives this request and detects that it can be served by a Context Provider (the IoT Agent), so it @@ -772,7 +757,7 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" - } ], "duration": "P1M" -}' "https://:10027/v2/registrations" +}' "https://:1026/v2/registrations" ``` If everything has gone OK, the Context Broker will return the following payload: @@ -806,8 +791,8 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" - } } ], - "updateAction": "update" -} ' "https://:10027/v2/op/update" + "updateAction": "append" +} ' "https://:1026/v2/op/update" ``` The Context Broker receives this command and detects that it can be served by a Context Provider (the IoT Agent), so it @@ -836,7 +821,7 @@ Fiware-Correlator: 9cae9496-8ec7-11e6-80fc-fa163e734aab } } ], - "updateAction" : "update" + "updateAction" : "append" } ``` @@ -900,8 +885,8 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" - } } ], - "actionType": "update" -} ' "https://:10027/v2/op/update" + "actionType": "append" +} ' "https://:1026/v2/op/update" ``` This update does not modify the original command attribute, but two auxiliary attributes, that are not provided by the @@ -948,7 +933,7 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" - "switch_info", "switch_status" ] -}' "https://:10027/v2/op/query" +}' "https://:1026/v2/op/query" ``` The Context Broker replies with all the desired data, in R2 format (200 OK): @@ -994,8 +979,8 @@ curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" - } } ], - "actionType": "update" -} ' "https://:10027/v2/op/update" + "actionType": "append" +} ' "https://:1026/v2/op/update" ``` In this case, the Context Broker reply with the following response (200 OK): diff --git a/doc/development.md b/doc/development.md deleted file mode 100644 index 1fe830637..000000000 --- a/doc/development.md +++ /dev/null @@ -1,285 +0,0 @@ -## Development documentation - -- [Contributions](#contributions) -- [Project build](#project-build) -- [Testing](#testing) -- [Coding guidelines](#coding-guidelines) -- [Continuous testing](#continuous-testing) -- [Code Coverage](#code-coverage) -- [Clean](#clean) -- [Data mapping plugins](#data-mapping-plugins) - - [Development](#development) - - [Provided plugins](#provided-plugins) - -### Contributions - -All contributions to this project are welcome. Developers planning to contribute should follow the -[Contribution Guidelines](Contribution.md) - -### Project build - -The project is managed using npm. - -For a list of available task, type - -```bash -npm run -``` - -The following sections show the available options in detail. - -### Environment requirements - -A [MongoDB](https://www.mongodb.com/) 3.2+ instance is required to run tests. You can deploy one by using the commodity -`docker-compose-dev.yml`: - -``` -docker-compose -f docker-compose-dev.yml up -d -``` - -To run docker compose you will need [docker](https://docs.docker.com/get-docker/) and -[docker-compose](https://docs.docker.com/compose/install/). - -### Testing - -[Mocha](https://mochajs.org/) Test Runner + [Should.js](https://shouldjs.github.io/) Assertion Library. - -The test environment is preconfigured to run BDD testing style. - -Module mocking during testing can be done with [proxyquire](https://github.com/thlorenz/proxyquire) - -To run tests, type - -```bash -npm test -``` - -There are additional targets starting with `test:` prefix to run specific test subsets isolatedly. For instance, the -`test:expressions` target runs the subset of tests related with expression language feature: - -```bash -npm run test:expressions -``` - -### Debug Test - -To debug the code while running run tests, type - -```bash -npm run test:debug -``` - -In the console the link to the debugger will be provided. You can connect to it via Chrome, for example, by opening the -following url: `chrome://inspect`. - -Additional debug clients are listed on [node.js](https://nodejs.org/en/docs/guides/debugging-getting-started/). - -### Coding guidelines - -ESLint - -Uses the provided `.eslintrc.json` flag file. To check source code style, type - -```bash -npm run lint -``` - -### Continuous testing - -Support for continuous testing by modifying a src file or a test. For continuous testing, type - -```bash -npm run test:watch -``` - -If you want to continuously check also source code style, use instead: - -```bash -npm run watch -``` - -### Code Coverage - -Istanbul - -Analyze the code coverage of your tests. - -To generate an HTML coverage report under `site/coverage/` and to print out a summary, type - -```bash -# Use git-bash on Windows -npm run test:coverage -``` - -### Clean - -Removes `node_modules` and `coverage` folders, and `package-lock.json` file so that a fresh copy of the project is -restored. - -```bash -# Use git-bash on Windows -npm run clean -``` - -### Documentation Markdown validation - -Checks the Markdown documentation for consistency - -```bash -# Use git-bash on Windows -npm run lint:md -``` - -### Documentation Spell-checking - -Uses the provided `.textlintrc` flag file. To check the Markdown documentation for spelling and grammar errors, dead -links & etc. - -```bash -# Use git-bash on Windows -npm run lint:text -``` - -### Clean - -Removes `node_modules` and `coverage` folders, and `package-lock.json` file so that a fresh copy of the project is -restored. - -```bash -# Use git-bash on Windows -npm run clean -``` - -### Prettify Code - -Runs the [prettier](https://prettier.io) code formatter to ensure consistent code style (whitespacing, parameter -placement and breakup of long lines etc.) within the codebase. - -```bash -# Use git-bash on Windows -npm run prettier -``` - -To ensure consistent Markdown formatting run the following: - -```bash -# Use git-bash on Windows -npm run prettier:text -``` - -# DB Models (from API document) - -## Service group model - -The table below shows the information held in the service group provisioning resource. The table also contains the -correspondence between the API resource fields and the same fields in the database model. - -| Payload Field | DB Field | Definition | -| ------------------------------ | ------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---- | -| `service` | `service` | Service of the devices of this type | -| `subservice` | `subservice` | Subservice of the devices of this type. | -| `resource` | `resource` | string representing the Southbound resource that will be used to assign a type to a device (e.g.: pathname in the southbound port). | -| `apikey` | `apikey` | API Key string. | -| `timestamp` | `timestamp` | Optional flag about whether or not to add the `TimeInstant` attribute to the device entity created, as well as a `TimeInstant` metadata to each attribute, with the current timestamp. With NGSI-LD, the Standard `observedAt` property-of-a-property is created instead. | true | -| `entity_type` | `entity_type` | name of the Entity `type` to assign to the group. | -| `trust` | `trust` | trust token to use for secured access to the Context Broker for this type of devices (optional; only needed for secured scenarios). | -| `cbHost` | `cbHost` | Context Broker connection information. This options can be used to override the global ones for specific types of devices. | -| `lazy` | `lazy` | list of common lazy attributes of the device. For each attribute, its `name` and `type` must be provided. | -| `commands` | `commands` | list of common commands attributes of the device. For each attribute, its `name` and `type` must be provided, additional `metadata` is optional. | -| `attributes` | `attributes` | list of common active attributes of the device. For each attribute, its `name` and `type` must be provided, additional `metadata` is optional. | -| `static_attributes` | `staticAttributes` | this attributes will be added to all the entities of this group 'as is', additional `metadata` is optional. | -| `internal_attributes` | `internalAttributes` | optional section with free format, to allow specific IoT Agents to store information along with the devices in the Device Registry. | -| `expressionLanguage` | `expresionLanguage` | optional boolean value, to set expression language used to compute expressions, possible values are: legacy or jexl. When not set or wrongly set, `legacy` is used as default value. | -| `explicitAttrs` | `explicitAttrs` | optional field to support selective ignore of measures so that IOTA doesn’t progress. See details in [specific section](advanced-topics.md#explicitly-defined-attributes-explicitattrs) | -| `entityNameExp` | `entityNameExp` | optional field to allow use expressions to define entity name, instead default `id` and `type` | -| `ngsiVersion` | `ngsiVersion` | optional string value used in mixed mode to switch between **NGSI-v2** and **NGSI-LD** payloads. Possible values are: `v2` or `ld`. The default is `v2`. When not running in mixed mode, this field is ignored. | -| `defaultEntityNameConjunction` | `defaultEntityNameConjunction` | optional string value to set default conjunction string used to compose a default `entity_name` when is not provided at device provisioning time. | -| `autoprovision` | `autoprovision` | optional boolean: If `false`, autoprovisioned devices (i.e. devices that are not created with an explicit provision operation but when the first measure arrives) are not allowed in this group. Default (in the case of omitting the field) is `true`. | - -## Device model - -The table below shows the information held in the Device resource. The table also contains the correspondence between -the API resource fields and the same fields in the database model. - -| Payload Field | DB Field | Definition | Example of value | -| --------------------- | -------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :-------------------------------------------- | -| `device_id` | `id` | Device ID that will be used to identify the device. | UO834IO | -| `service` | `service` | Name of the service the device belongs to (will be used in the fiware-service header). | smartGondor | -| `service_path` | `subservice` | Name of the subservice the device belongs to (used in the fiware-servicepath header). | /gardens | -| `entity_name` | `name` | Name of the entity representing the device in the Context Broker | ParkLamplight12 | -| `entity_type` | `type` | Type of the entity in the Context Broker | Lamplights | -| `timezone` | `timezone` | Time zone of the sensor if it has any | America/Santiago | -| `timestamp` | `timestamp` | Optional flag about whether or not to add the `TimeInstant` attribute to the device entity created, as well as a `TimeInstant` metadata to each attribute, with the current timestamp. With NGSI-LD, the Standard `observedAt` property-of-a-property is created instead. | true | -| `apikey` | `apikey` | Optional Apikey key string to use instead of group apikey | 9n4hb1vpwbjozzmw9f0flf9c2 | -| `endpoint` | `endpoint` | Endpoint where the device is going to receive commands, if any. | http://theDeviceUrl:1234/commands | -| `protocol` | `protocol` | Name of the device protocol, for its use with an IoT Manager. | IoTA-UL | -| `transport` | `transport` | Name of the device transport protocol, for the IoT Agents with multiple transport protocols. | MQTT | -| `attributes` | `active` | List of active attributes of the device | `[ { "name": "attr_name", "type": "Text" } ]` | -| `lazy` | `lazy` | List of lazy attributes of the device | `[ { "name": "attr_name", "type": "Text" } ]` | -| `commands` | `commands` | List of commands of the device | `[ { "name": "attr_name", "type": "Text" } ]` | -| `internal_attributes` | `internalAttributes` | List of internal attributes with free format for specific IoT Agent configuration | LWM2M mappings from object URIs to attributes | -| `static_attributes` | `staticAttributes` | List of static attributes to append to the entity. All the updateContext requests to the CB will have this set of attributes appended. | `[ { "name": "attr_name", "type": "Text" } ]` | -| `expressionLanguage` | `expresionLanguage` | optional boolean value, to set expression language used to compute expressions, possible values are: legacy or jexl. When not set or wrongly set, legacy is used as default value. | -| `explicitAttrs` | `explicitAttrs` | optional field to support selective ignore of measures so that IOTA doesn’t progress. See details in [specific section](advanced-topics.md#explicitly-defined-attributes-explicitattrs) | (see details in specific section) | -| `ngsiVersion` | `ngsiVersion` | optional string value used in mixed mode to switch between **NGSI-v2** and **NGSI-LD** payloads. The default is `v2`. When not running in mixed mode, this field is ignored. | `v2/ld` | - -## Data mapping plugins - -The IoT Agent Library provides a plugin mechanism in order to facilitate reusing code that makes small transformations -on incoming data (both from the device and from the context consumers). This mechanism is based in the use of -middlewares, i.e.: small pieces of code that receive and return an `entity`, making as many changes as they need, but -taking care of returning a valid entity, that can be used as the input for other middlewares; this way, all those pieces -of code can be chained together in order to make all the needed transformations in the target entity. - -There are two kinds of middlewares: updateContext middlewares and queryContext middlewares. The updateContext -middlewares are applied before the information is sent to the Context Broker, modifiying the entity before it is sent to -Orion. The queryContext middlewares are applied on the received data, whenever the IoT Agent queries the Context Broker -for information. I.e.: both middlewares will be automatically applied whenever the `update()` or `query()` functions are -called in the library. - -All the middlewares have the opportunity to break the chain of middleware applications by calling the `callback()` with -an error object (the usual convention). If any of the updateContext middlewares raise an error, no request will be sent -to the Context Broker. On the other hand, the queryContext request is always performed, but the call to the `query()` -function will end up in an error if any of the queryContext middlewares report an error. - -### Development - -All the middlewares have the same signature: - -```javascript -function middlewareName(entity, typeInformation, callback) {} -``` - -The arguments for any middleware are the NGSI data over which it can operate: - -- An updateContext payload in the case of an updateContext middleware and a queryContext payload otherwise; -- a typeInformation object containing all the information about the device stored during registration. -- and the customary `callback` parameter, with the usual meaning. It's really important for the library user to call - this callback, as failing to do so may hang the IoT Agent completely. The callback must be called with the an - optional error in the first argument and the same arguments received (potentially modified) as the following. - -In order to manage the middlewares to the system, the following functions can be used: - -- `addUpdateMiddleware`: adds an updateContext middleware to the stack of middlewares. All the middlewares will be - applied to every call to the `update()` function. The final payload of the updateContext request will be the result - of applying all this middlewares in the order they have been defined. - -- `addQueryMiddleware`: adds a queryContext middleware to the stack of middlewares. All the middlewares will be - applied to every call to the `query()` function. - -- `resetMiddlewares`: remove all the middlewares from the system. - -Usually, the full list of middlewares an IoT Agent will use would be added in the IoTAgent start sequence, so they -should not change a lot during the IoT lifetime. - -### Provided plugins - -The library provides some plugins out of the box, in the `dataPlugins` collection. In order to load any of them, just -use the `addQueryMiddleware` and `addUpdateMiddleware` functions with the selected plugin, as in the example: - -```javascript -var iotaLib = require('iotagent-node-lib'); - -iotaLib.addUpdateMiddleware(iotaLib.dataPlugins.compressTimestamp.update); -iotaLib.addQueryMiddleware(iotaLib.dataPlugins.compressTimestamp.query); -``` diff --git a/doc/howto.md b/doc/howto.md deleted file mode 100644 index 081bf55f7..000000000 --- a/doc/howto.md +++ /dev/null @@ -1,641 +0,0 @@ -# How to develop a new IOTAgent - -- [Overview](#overview) - - [Protocol](#protocol) - - [Requirements](#requirements) -- [Basic IOTA](#basic-iot-agent) -- [IOTA With Active attributes](#iot-agent-with-active-attributes) -- [IOTA With Lazy attributes](#iota-with-lazy-attributes) - - [Previous considerations](#previous-considerations) - - [Implementation](#implementation) -- [IoT Agent in multi-thread mode](#iot-agent-in-multi-thread-mode) -- [Configuration management](#configuration-management) - - [Provisioning handlers](#provisioning-handlers) - -## Overview - -This document's goal is to show how to develop a new IoT Agent step by step. To do so, a simple invented HTTP protocol -will be used, so it can be tested with simple command-line instructions as `curl` and `nc`. - -### Protocol - -The invented protocol will be freely adapted from -[Ultralight 2.0](https://github.com/telefonicaid/fiware-IoTAgent-Cplusplus/blob/develop/doc/modules.md#ultra-light-agent). -Whenever a device wants to send an update, it will send a request as the following: - -```bash -curl -X GET 'http://127.0.0.1:8080/iot/d?i=ULSensor&k=abc&d=t|15,l|19.6' -i -``` - -Where: - -- **i**: is the device ID. -- **k**: the API Key for the device's service. -- **d**: the data payload, consisting of key-value pairs separated by a pipe (`|`), with each pair separated by comma - (`,`); - -### Requirements - -This tutorial expects a Node.js v8 (at least) installed and working on your machine. It also expects you to have access -to a Context Broker (without any security proxies). - -## Basic IoT Agent - -In this first chapter, we will just develop an IoT Agent with a fully connected North Port. This will send and receive -NGSI traffic and can be administered using the IoT Agent's Device Provisioning API. The South Port will remain -unconnected and no native protocol traffic will be sent to the devices. This may seem useless (and indeed it is) but it -will serve us well on showing the basic steps in the creation of an IoT Agent. - -First of all, we have to create the Node project. Create a folder to hold your project and type the following -instruction: - -```bash -npm init -``` - -This will create the `package.json` file for our project. Now, add the following lines to your project file: - -```json - "dependencies": { - "iotagent-node-lib": "*" - }, - -``` - -And install the dependencies, executing, as usual: - -```bash -npm install -``` - -The first step is to write a configuration file, that will be used to tune the behavior of our IOTA. The contents can be -copied from the `config-basic-example.js` file, in this same folder. Create a `config.js` file with it in the root -folder of your project. Remember to change the Context Broker IP to your local Context Broker. - -Now we can begin with the code of our IoT Agent. The very minimum code we need to start an IoT Agent is the following: - -```javascript -var iotAgentLib = require('iotagent-node-lib'), - config = require('./config'); - -iotAgentLib.activate(config, function (error) { - if (error) { - console.log('There was an error activating the IOTA'); - process.exit(1); - } -}); -``` - -The IoT Agent is now ready to be used. Execute it with the following command: - -```bash -node index.js -``` - -The North Port interface should now be fully functional, i.e.: management of device registrations and configurations. - -## IoT Agent With Active attributes - -In the previous section we created an IoT Agent that exposed just the North Port interface, but that was pretty useless -(aside from its didactic use). In this section we are going to create a simple South Port interface. It's important to -remark that the nature of the traffic South of the IoT Agent itself has nothing to do with the creation process of an -IoT Agent. Each device protocol will use its own mechanisms and it is up to the IoT Agent developer to find any -libraries that would help him in its development. In this example, we will use Express as such library. - -In order to add the Express dependency to your project, add the following line to the `dependencies` section of the -`package.json`: - -```json - "express": "*", -``` - -The require section would end up like this (the standard `http` module is also needed): - -```javascript -var iotAgentLib = require('iotagent-node-lib'), - http = require('http'), - express = require('express'), - config = require('./config'); -``` - -And install the dependencies as usual with `npm install`. You will have to require both `express` and `http` in your -code as well. - -Now, in order to accept connections in our code, we have to start express first. With this purpose in mind, we will -create a new function `initSouthbound()`, that will be called from the initialization code of our IoT Agent: - -```javascript -function initSouthbound(callback) { - southboundServer = { - server: null, - app: express(), - router: express.Router() - }; - - southboundServer.app.set('port', 8080); - southboundServer.app.set('host', '0.0.0.0'); - - southboundServer.router.get('/iot/d', manageULRequest); - southboundServer.server = http.createServer(southboundServer.app); - southboundServer.app.use('/', southboundServer.router); - southboundServer.server.listen(southboundServer.app.get('port'), southboundServer.app.get('host'), callback); -} -``` - -This Express code sets up a HTTP server, listening in the 8080 port, that will handle incoming requests targeting path -`/iot/d` using the middleware `manageULRequest()`. This middleware will contain all the logic south of the IoT Agent, -and the library methods we need in order to progress the information to the Context Broker. The code of this middleware -would be as follows: - -```javascript -function manageULRequest(req, res, next) { - var values; - - iotAgentLib.retrieveDevice(req.query.i, req.query.k, function (error, device) { - if (error) { - res.status(404).send({ - message: "Couldn't find the device: " + JSON.stringify(error) - }); - } else { - values = parseUl(req.query.d, device); - iotAgentLib.update(device.name, device.type, '', values, device, function (error) { - if (error) { - res.status(500).send({ - message: 'Error updating the device' - }); - } else { - res.status(200).send({ - message: 'Device successfully updated' - }); - } - }); - } - }); -} -``` - -For this middleware we have made use of a function `parseUl()` that parses the data payload and transforms it in the -data object expected by the update function (i.e.: an attribute array with NGSI syntax): - -```javascript -function parseUl(data, device) { - function findType(name) { - for (var i = 0; i < device.active.length; i++) { - if (device.active[i].name === name) { - return device.active[i].type; - } - } - - return null; - } - - function createAttribute(element) { - var pair = element.split('|'), - attribute = { - name: pair[0], - value: pair[1], - type: findType(pair[0]) - }; - - return attribute; - } - - return data.split(',').map(createAttribute); -} -``` - -Here as an example of the output of the function return for the UL payload `t|15,l|19.6`: - -```json -[ - { - "name": "t", - "type": "celsius", - "value": "15" - }, - { - "name": "l", - "type": "meters", - "value": "19.6" - } -] -``` - -The last thing to do is to invoke the initialization function inside the IoT Agent startup function. The next excerpt -show the modifications in the `activate()` function: - -```javascript -iotAgentLib.activate(config, function (error) { - if (error) { - console.log('There was an error activating the IOTA'); - process.exit(1); - } else { - initSouthbound(function (error) { - if (error) { - console.log('Could not initialize South bound API due to the following error: %s', error); - } else { - console.log('Both APIs started successfully'); - } - }); - } -}); -``` - -Some logs were added in this piece of code to help debugging. - -Once the IOTA is finished the last thing to do is to test it. To do so, launch the IoT Agent and provision a new device -(an example for provisioning can be found in the `examples/howtoProvisioning1.json` file). Once the device is -provisioned, send a new measure by using the example command: - -```bash -curl -X GET 'http://127.0.0.1:8080/iot/d?i=ULSensor&k=abc&d=t|15,l|19.6' -i -``` - -Now you should be able to see the measures in the Context Broker entity of the device. - -## IOTA With Lazy attributes - -### Previous considerations - -The IoT Agents also give the possibility for the device to be asked about the value of one of its measures, instead of -reporting it. In order to do so, the device must be capable of receiving messages of some kind. In this case, we are -going to simulate an HTTP server with `nc` in order to see the values sent by the IOTA. We also have to decide a syntax -for the protocol request for asking the device about a measure. For clarity, we will use the same HTTP GET request we -used to report a measure, but indicating the attribute to ask instead of the data payload. Something like: - -```bash -curl -X GET 'http://127.0.0.1:9999/iot/d?i=ULSensor&k=abc&q=t,l' -i -``` - -In a real implementation, the server will need to know the URL and port where the devices are listening, in order to -send the request to the appropriate device. For this example, we will assume that the device is listening in port 9999 -in localhost. For more complex cases, the mechanism to bind devices to addresses would be IoT-Agent-specific (e.g.: the -OMA Lightweight M2M IoT Agent captures the address of the device in the device registration, and stores the -device-specific information in a MongoDB document). - -Being lazy attributes of a read/write nature, another syntax has to be declared for updating. This syntax will mimic the -one used for updating the server: - -``` -curl -X GET 'http://127.0.0.1:9999/iot/d?i=ULSensor&k=abc&d=t|15,l|19.6' -i -``` - -Both types of calls to the device will be distinguished by the presence or absence of the `d` and `q` attributes. - -A HTTP request library will be needed in order to make those calls. To this extent, `mikeal/request` library will be -used. In order to do so, add the following require statement to the initialization code: - -```javascript -request = require('request'); -``` - -and add the `request` dependency to the `package.json` file: - -```json - "dependencies": [ - [...] - - "request": "*", - - ] -``` - -The require section should now look like this: - -```javascript -var iotAgentLib = require('iotagent-node-lib'), - http = require('http'), - express = require('express'), - request = require('request'), - config = require('./config'); -``` - -### Implementation - -#### QueryContext implementation - -The main step to complete in order to implement the Lazy attributes mechanism in the IoT Agent is to provide handlers -for the context provisioning requests. At this point, we should provide two handlers: the `/v2/op/update` and the -`/v2/op/query` handlers. To do so, we must first define the handlers themselves: - -```javascript -function queryContextHandler(id, type, service, subservice, attributes, callback) { - var options = { - url: 'http://127.0.0.1:9999/iot/d', - method: 'GET', - qs: { - q: attributes.join() - } - }; - - request(options, function (error, response, body) { - if (error) { - callback(error); - } else { - callback(null, createResponse(id, type, attributes, body)); - } - }); -} -``` - -The queryContext handler is called whenever a `/v2/op/query` request arrives at the North port of the IoT Agent. It is -invoked once for each entity requested, passing the entity ID and Type as the parameters, as well as a list of the -attributes that are requested. In our case, the handler uses this parameters to compose a request to the device. Once -the results of the device are returned, the values are returned to the caller, in the NGSI attribute format. - -In order to format the response from the device in a readable way, we created a `createResponse()` function that maps -the values to its correspondent attributes. This function assumes the type of all the attributes is "string" (this will -not be the case in a real scenario, where the IoT Agent should retrieve the associated device to guess the type of its -attributes). Here is the code for the `createResponse()` function: - -```javascript -function createResponse(id, type, attributes, body) { - var values = body.split(','), - responses = []; - - for (var i = 0; i < attributes.length; i++) { - responses.push({ - name: attributes[i], - type: 'string', - value: values[i] - }); - } - - return { - id: id, - type: type, - attributes: responses - }; -} -``` - -#### UpdateContext implementation - -```javascript -function updateContextHandler(id, type, service, subservice, attributes, callback) { - var options = { - url: 'http://127.0.0.1:9999/iot/d', - method: 'GET', - qs: { - d: createQueryFromAttributes(attributes) - } - }; - - request(options, function (error, response, body) { - if (error) { - callback(error); - } else { - callback(null, { - id: id, - type: type, - attributes: attributes - }); - } - }); -} -``` - -The updateContext handler deals with the modification requests that arrive at the North Port of the IoT Agent via -`/v2/op/update`. It is invoked once for each entity requested (note that a single request can contain multiple entity -updates), with the same parameters used in the queryContext handler. The only difference is the value of the attributes -array, now containing a list of attribute objects, each containing name, type and value. The handler must also make use -of the callback to return a list of updated attributes. - -For this handler we have used a helper function called `createQueryFromAttributes()`, that transforms the NGSI -representation of the attributes to the UL type expected by the device: - -```javascript -function createQueryFromAttributes(attributes) { - var query = ''; - - for (var i in attributes) { - query += attributes[i].name + '|' + attributes[i].value; - - if (i != attributes.length - 1) { - query += ','; - } - } - - return query; -} -``` - -#### Handler registration - -Once both handlers have been defined, they have to be registered in the IoT Agent, adding the following code to the -setup function: - -```javascript -iotAgentLib.setDataUpdateHandler(updateContextHandler); -iotAgentLib.setDataQueryHandler(queryContextHandler); -``` - -Where necessary, additional handlers to deal with command actuations and merge-patch operations may also be added when -necessary. - -```javascript -iotAgentLib.setCommandHandler(commandHandler); -iotAgentLib.setMergePatchHandler(mergePatchHandler); -``` - -#### IOTA Testing - -In order to test it, we need to create an HTTP server simulating the device. The quickest way to do that may be using -netcat. In order to start it just run the following command from the command-line (Linux and Mac only): - -```bash -nc -l 9999 -``` - -This will open a simple TCP server listening on port `9999`, where the requests from the IoT Agent will be printed. In -order for the complete workflow to work (and to receive the response in the application side), the HTTP response has to -be written in the `nc` console (although for testing purposes this is not needed). - -While netcat is great to test simple connectivity, you will need something just a bit more complex to get the complete -scenario working (at least without the need to be incredibly fast sending your response). In order to do so, a simple -echo server was created, that answers 42 to any query to its `/iot/d` path. You can use it to test your attributes one -by one (or you can modify it to accept more requests and give more complex responses). Copy the -[Echo Server script](echo.js) to the same folder of your IoTAgent (as it uses the same dependencies). In order to run -the echo server, just execute the following command: - -```bash -node echo.js -``` - -Once the mock server has been started (either `nc` or the `echo` server), proceed with the following steps to test your -implementation: - -1. Provision a device with two lazy attributes. The following request can be used as an example: - -```text -POST /iot/devices HTTP/1.1 -Host: localhost:4041 -Content-Type: application/json -fiware-service: howtoserv -fiware-servicepath: /test -Cache-Control: no-cache -Postman-Token: 993ac66b-72da-9e96-ab46-779677a5896a - -{ - "devices": [ - { - "device_id": "ULSensor", - "entity_name": "Sensor01", - "entity_type": "BasicULSensor", - "lazy": [ - { - "name": "t", - "type": "celsius" - }, - { - "name": "l", - "type": "meters" - } - ], - "attributes": [ - ] - } - ] -} -``` - -2. Execute a `/v2/op/query` or `/v2/op/update` against one of the entity attributes (use a NGSI client of curl command). - -```text -POST /v2/op/query HTTP/1.1 -Host: localhost:1026 -Content-Type: application/json -Accept: application/json -Fiware-Service: howtoserv -Fiware-ServicePath: /test -Cache-Control: no-cache - -{ - entities: [ - { - id: 'Light:light1' - } - ], - attrs: ['dimming'] -} -``` - -3. Check the received request in the nc console is the expected one. - -4. (In case you use netcat). Answer the request with an appropriate HTTP response and check the result of the - `/v2/op/query` or `/v2/op/update` request is the expected one. An example of HTTP response, for a query to the `t` - and `l` attributes would be: - -```text -HTTP/1.0 200 OK -Content-Type: text/plain -Content-Length: 3 - -5,6 -``` - -This same response can be used both for updates and queries for testing purposes (even though in the former the body -won't be read). - -## IoT Agent in multi-thread mode - -It is possible that an IoT Agent can be executed in multi-thread approach, which will increase the number of -request/seconds that can be manage by the server. It's important to remark that the nature of this functionality in -included in the IoT Agent Node Lib but it is not mandatory that you activate this functionality. In this example, we -will see how to use this functionality to deploy an IoT Agent in multi-thread environment. - -In order to activate the functionality, you have two options, configure the `config.js` file to add the following line: - -```javascript -/** - * flag indicating whether the node server will be executed in multi-core option (true) or it will be a - * single-thread one (false). - */ -config.multiCore = true; -``` - -or you can define the proper IOTA_MULTI_CORE environment variable. By default, the first choice is the environment -variable and afterward the value of the multiCore in the `config.js` file. The require section would end up like this -(the standard `http` module is also needed): - -```javascript -var iotAgent = require('../lib/iotagent-implementation'), - iotAgentLib = require('iotagent-node-lib'), - config = require('./config'); -``` - -It is important to mention the purpose of the `iotAgent` variable. It is the proper implementation of the IoT Agent -based on the IoT Agent Node Lib. We will need this variable just to make a callback to the corresponding `start()` -process from the library. The variable `config` is used to get details of the configuration file and send that -information to the Node Lib. The Node Lib will take the decision of single-thread or multi-thread execution base on the -value of `config.multiCore` attribute. - -Finally, we can call the corresponding [iotagentLib.startServer()](usermanual.md#iotagentlibstartserver) like the -following code with a callback function to show details about any error during the execution or just print the message -about starting the IoTAgent: - -```javascript -iotAgentLib.startServer(config, iotAgent, function (error) { - if (error) { - console.log(context, 'Error starting IoT Agent: [%s] Exiting process', error); - } else { - console.log(context, 'IoT Agent started'); - } -}); -``` - -> Note: `startServer()` initializes the server but it does not activate the library. The function in the Node Lib will -> call the `iotAgent.start()` in order to complete the activation of the library. Therefore, it is expected that the IoT -> Agent implement the `iotAgent.start()` function with the proper invocation to the `iotAgentLib.activate()`. - -## Configuration management - -For some IoT Agents, it will be useful to know what devices or configurations were registered in the Agent, or to do -some actions whenever a new device is registered. All this configuration and provisioning actions can be performed using -two mechanisms: the provisioning handlers and the provisioning API. - -### Provisioning handlers - -The handlers provide a way for the IoT Agent to act whenever a new device, or configuration is provisioned. This can be -used for registering the device in external services, for storing important information about the device, or to listen -in new ports in the case of new configuration. For the simple example we are developing, we will just print the -information we are receiving whenever a new device or configuration is provisioned. - -We need to complete two further steps to have a working set of provisioning handlers. First of all, defining the -handlers themselves. Here we can see the definition of the configuration handler: - -```javascript -function configurationHandler(configuration, callback) { - console.log('\n\n* REGISTERING A NEW CONFIGURATION:\n%s\n\n', JSON.stringify(configuration, null, 4)); - callback(null, configuration); -} -``` - -As we can see, the handlers receive the device or configuration that is being provisioned, as well as a callback. The -handler MUST call the callback once in order for the IOTA to work properly. If an error is passed as a parameter to the -callback, the provisioning will be aborted. If no error is passed, the provisioning process will continue. This -mechanism can be used to implement security mechanisms or to filter the provisioning of devices to the IoT Agent. - -Note also that the same `device` or `configuration` object is passed along to the callback. This lets the IoT Agent -change some of the values provisioned by the user, to add or restrict information in the provisioning. To test this -feature, let's use the provisioning handler to change the value of the type of the provisioning device to -`CertifiedType` (reflecting some validation process performed on the provisioning): - -```javascript -function provisioningHandler(device, callback) { - console.log('\n\n* REGISTERING A NEW DEVICE:\n%s\n\n', JSON.stringify(device, null, 4)); - device.type = 'CertifiedType'; - callback(null, device); -} -``` - -Once the handlers are defined, the new set of handlers has to be registered into the IoT Agent: - -```javascript -iotAgentLib.setConfigurationHandler(configurationHandler); -iotAgentLib.setProvisioningHandler(provisioningHandler); -``` - -Now we can test our implementation by sending provisioning requests to the North Port of the IoT Agent. If we provision -a new device into the platform, and then we ask for the list of provisioned devices, we shall see the type of the -provisioned device has changed to `CertifiedType`. diff --git a/doc/index.md b/doc/index.md index f32f7a6c1..13112780b 100644 --- a/doc/index.md +++ b/doc/index.md @@ -11,8 +11,8 @@ Broker using their own native protocols. IoT Agents should also be able to deal platform (authentication and authorization of the channel) and provide other common services to the device programmer. Github's [README.md](https://github.com/telefonicaid/iotagent-node-lib/blob/master/README.md) provides a good -documentation summary. The [User Manual](usermanual.md) and the [Admin Guide](installationguide.md) cover more advanced -topics. +documentation summary. The [API reference](doc/api.md) and the [Development documentation](devel/development.md) cover +more advanced topics. ## Background @@ -26,8 +26,6 @@ functions. communications are left to the library. - Standardized OAuth2-based security is available to enable each IoT Agent to connect to several common Identity Managers (e.g. Keystone and Keyrock) so that communications can be restricted to trusted components. -- A series of additional plugins are offered where necessary to allow for expression parsing, attribute aliasing and - the processing of timestamp metadata. Each individual IoT Agent offers is driven by a `config.js` configuration file contains explicit custom settings based on the protocol and payload the IoT Agent is translating. It will also contain some common flags for common @@ -51,5 +49,5 @@ IoT Agent In order to use the library within your own IoT Agent, you must first you require it before use: ```javascript -const iotagentLib = require("iotagent-node-lib"); +const iotagentLib = require('iotagent-node-lib'); ``` diff --git a/doc/installationguide.md b/doc/installationguide.md deleted file mode 100644 index 5d054f999..000000000 --- a/doc/installationguide.md +++ /dev/null @@ -1,365 +0,0 @@ -# Installation & Administration Guide - -## Configuration - -The `activate()` function that starts the IoT Agent receives as single parameter with the configuration for the IoT -Agent. The Agent Console reads the same configuration from the `config.js` file. - -### Global Configuration - -These are the parameters that can be configured in the global section: - -- **logLevel**: minimum log level to log. May take one of the following values: DEBUG, INFO, ERROR, FATAL. E.g.: - 'DEBUG'. -- **contextBroker**: connection data to the Context Broker (host and port). E.g.: - -```javascript -{ - host: '192.168.56.101', - port: '1026' -} -``` - -- If you want to use **NGSI v2**: - -```javascript -{ - host: '192.168.56.101', - port: '1026', - ngsiVersion: 'v2' -} -``` - -- If you want to use **NGSI-LD** (experimental): - -```javascript -{ - host: '192.168.56.101', - port: '1026', - ngsiVersion: 'ld', - jsonLdContext: 'http://context.json-ld' // or ['http://context1.json-ld','http://context2.json-ld'] if you need more than one -} -``` - -Where `http://context.json-ld` is the location of the NGSI-LD `@context` element which provides additional information -allowing the computer to interpret the rest of the data with more clarity and depth. Read the -[JSON-LD specification](https://w3c.github.io/json-ld-syntax/#the-context) for more information. - -- If you want to support a "mixed" mode with both **NGSI-v2** and **NGSI-LD** (experimental): - -```javascript -{ - host: '192.168.56.101', - port: '1026', - ngsiVersion: 'mixed', - jsonLdContext: 'http://context.json-ld' // or ['http://context1.json-ld','http://context2.json-ld'] if you need more than one -} -``` - -Under mixed mode, **NGSI v2** payloads are used for context broker communications by default, but this payload may also -be switched to **NGSI LD** at service group or device provisioning time using the `ngsiVersion` field in the -provisioning API. The `ngsiVersion` field switch may be added at either group or device level, with the device level -overriding the group setting. - -- **server**: configuration used to create the Context Server (port where the IoT Agent will be listening as a Context - Provider and base root to prefix all the paths). The `port` attribute is required. If no `baseRoot` attribute is - used, '/' is used by default. E.g.: - -```javascript -{ - baseRoot: '/', - port: 4041 -} -``` - -When connected to an **NGSI-LD** context broker, an IoT Agent is able to indicate whether it is willing to accept `null` -values and also whether it is able to process the **NGSI-LD** `datasetId` metadata element. Setting these values to -`false` will cause the IoT Agent to return a 400 **Bad Request** HTTP status code explaining that the IoT Agent does not -support nulls or multi-attribute requests if they are encountered. - -```javascript -{ - baseRoot: '/', - port: 4041, - ldSupport : { - null: true, - datasetId: true - } -} -``` - -- **stats**: configure the periodic collection of statistics. Use `interval` in milliseconds to set the time between - stats writings. - -```javascript -stats: { - interval: 100; -} -``` - -- **authentication**: authentication data, for use in retrieving tokens for devices with a trust token (required in - scenarios with security enabled in the Context Broker side). Currently, two authentication provider are supported: - `keystone` and `oauth2`. Authentication need to be enabled by setting the field `enabled` to `true`. In `keystone` - based authentication, the `trust` associated to the `device` or `deviceGroup` is a token representing a specific - user and his rights on a given domain (i.e. combination of `fiware-service` and `fiware-servicepath`). The - authentication process use the trust delegation workflow to check if the trust provided is valid, in which case - return a `x-subject-token` that can be used to authenticate the request to the Context Broker. Required parameters - are: the `url` of the keystone to be used (alternatively `host` and `port` but if you use this combination, the IoT - Agent will assume that the protocol is HTTP), the `user` and `password` to which it is delegated the `trust` - verification. E.g.: - -```javascript -{ - enabled: true, - url: 'https://localhost:5000', - type: 'keystone', - user: 'iotagent', - password: 'iotagent' -} -``` - -In `oauth2` based authentication, two types of tokens can be used depending on the availability in the IDM to be used. -On one hand, the `trust` associated to the `device` or `deviceGroup` is a `refresh_token` issued by a specific user for -the Context Broker client. The authentication process uses the -[`refresh_token` grant type](https://tools.ietf.org/html/rfc6749#section-1.5) to obtain an `access_token` that can be -used to authenticate the request to the Context Broker. At the time being the assumption is that the `refresh_token` is -a not expiring `offline_token` (we believe this is the best solution in the case of IoT Devices, since injecting a -refresh token look may slow down communication. Still, the developer would be able to invalidate the refresh token on -the provider side in case of security issues connected to a token). The code was tested using -[Keycloak](http://www.keycloak.org), [Auth0](https://auth0.com) and [FIWARE Keyrock](https://github.com/ging/fiware-idm) -(it may require customisation for other providers - while OAuth2 is a standard, not all implementations behave in the -same way, especially as regards status codes and error messages). Required parameters are: the `url` of the OAuth 2 -provider to be used (alternatively `host` and `port` but if you use this combination, the IoT Agent will assume that the -protocol is HTTP), the `tokenPath` to which the validation request should be sent -(`/auth/realms/default/protocol/openid-connect/token` for Keycloak and Auth0, `/oauth2/token` for Keyrock), the -`clientId` and `clientSecret` that identify the Context Broker, and the `header` field that should be used to send the -authentication request (that will be sent in the form `Authorization: Bearer `). E.g.: - -```javascript -{ - enabled: true, - type: 'oauth2', - url: 'http://localhost:3000', - header: 'Authorization', - clientId: 'context-broker', - clientSecret: 'c8d58d16-0a42-400e-9765-f32e154a5a9e', - tokenPath: '/auth/realms/default/protocol/openid-connect/token' -} -``` - -Nevertheless, this kind of authentication relying on `refresh_token` grant type implies that when the acces_token -expires, it is needed to request a new one from the IDM, causing some overhead in the communication with the Context -Broker. To mitigate this issue, FIWARE KeyRock IDM implements `permanent tokens` that can be retrieved using -`scope=permanent`. With this approach, the IOTA does not need to interact with the IDM and directly include the -`permanent token` in the header. In order to use this type of token, an additional parameter `permanentToken` must be -set to `true` in the `authentication` configuration. An environment variable `IOTA_AUTH_PERMANENT_TOKEN` can be also -used for the same purpose. For instance: - -```javascript -{ - type: 'oauth2', - url: 'http://localhost:3000', - header: 'Authorization', - clientId: 'context-broker', - clientSecret: '0c2492e1-3ce3-4cca-9723-e6075b89c244', - tokenPath: '/oauth2/token', - enabled: true, - permanentToken: true -} -``` - -- **deviceRegistry**: type of Device Registry to create. Currently, two values are supported: `memory` and `mongodb`. - If the former is configured, a transient memory-based device registry will be used to register all the devices. This - registry will be emptied whenever the process is restarted. If the latter is selected, a MongoDB database will be - used to store all the device information, so it will be persistent from one execution to the other. Mongodb - databases must be configured in the `mongob` section (as described bellow). E.g.: - -```javascript -{ - type: 'mongodb'; -} -``` - -- **mongodb**: configures the MongoDB driver for those repositories with 'mongodb' type. If the `host` parameter is a - list of comma-separated IPs, they will be considered to be part of a Replica Set. In that case, the optional - property `replicaSet` should contain the Replica Set name. If the database requires authentication, username - (`username`), password (`password`) and authSource (`authSource`) can be set. If the database requires TLS/SSL - connection but any validation of the certificate chain is not mandatory, all you need is to set the ssl (`ssl`) - option as `true` to connect the database. If you need to add more complex option(s) such as `retryWrites=true` or - `w=majority` when connection database, extraArgs (`extraArgs`) can be used to perform it. For The MongoBD driver - will retry the connection at startup time `retries` times, waiting `retryTime` seconds between attempts, if those - attributes are present (default values are 5 and 5 respectively). E.g.: - -```javascript -{ - host: 'localhost', - port: '27017', - db: 'iotagent', - retries: 5, - retryTime: 5 -} -``` - -```javascript -{ - host: 'mongodb-0,mongodb-1,mongodb-2', - port: '27017', - db: 'iotagent', - replicaSet: 'rs0', - user: 'rootuser', - password: 'password', - authSource: 'admin', - ssl: true, - extraArgs: { - retryWrites: true, - readPreference: 'nearest', - w: 'majority' - }, - retries: 5, - retryTime: 5 -} -``` - -- **iotManager**: configures all the information needed to register the IoT Agent in the IoTManager. If this section - is present, the IoTA will try to register to a IoTAM in the `host`, `port` and `path` indicated, with the - information configured in the object. The IoTAgent URL that will be reported will be the `providedUrl` (described - below) with the added `agentPath`: - -```javascript -{ - host: 'mockediotam.com', - port: 9876, - path: '/protocols', - protocol: 'GENERIC_PROTOCOL', - description: 'A generic protocol', - agentPath: '/iot' -} -``` - -- **types**: See **Type Configuration** in the [Configuration API](#configurationapi) section below. -- **eventType**: Default type for the Events (useful only with the `addEvents` plugin). -- **service**: default service for the IoT Agent. If a device is being registered, and no service information comes - with the device data, and no service information is configured for the given type, the default IoT agent service - will be used instead. E.g.: 'smartGondor'. -- **subservice**: default subservice for the IoT Agent. If a device is being registered, and no subservice information - comes with the device data, and no subservice information is configured for the given type, the default IoT agent - subservice will be used instead. E.g.: '/gardens'. -- **providerUrl**: URL to send in the Context Provider registration requests. Should represent the external IP of the - deployed IoT Agent (the IP where the Context Broker will redirect the NGSI requests). E.g.: - 'http://192.168.56.1:4041'. -- **iotaVersion**: indicates the version of the IoTA that will be displayed in the about method (it should be filled - automatically by each IoTA). -- **dieOnUnexpectedError**: if this flag is activated, the IoTAgent will not capture global exception, thus dying upon - any unexpected error. -- **singleConfigurationMode**: enables the Single Configuration mode for backwards compatibility (see description in - the Overview). Default to false. -- **timestamp**: if this flag is activated: - - For NGSI-v2, the IoT Agent will add a `TimeInstant` metadata attribute to all the attributes updated from device - information. This flag is overwritten by `timestamp` flag in group or device - - With NGSI-LD, the standard `observedAt` property-of-a-property is created instead. -- **defaultResource**: default string to use as resource for the registration of new Configurations (if no resource is - provided). -- **defaultKey**: default string to use as API Key for devices that do not belong to a particular Configuration. -- **componentName**: default string identifying the component name for this IoT Agent in the logs. -- **pollingExpiration**: expiration time for commands waiting in the polling queue in miliseconds. If a command has - been in the queue for this amount of time without being collected by the device, the expiration daemon will reclaim - it. This attribute is optional (if it doesn't exist, commands won't expire). -- **pollingDaemonFrequency**: time between collection of expired commands in milliseconds. This attribute is optional - (if this parameter doesn't exist the polling daemon won't be started). -- **autocast**: When enabled, the IoT Agents will try to cast attribute's values considering the JSON native type - (only for NGSI v2). -- **multiCore**: When enabled, the IoT Agents runs in multi-thread environment to take advantage of multi-core - systems. It allows two values `true` or `false`. This attribute is optional with default to false, which means that - the IoTAgent runs in a single thread. For more details about multi-core functionality, please refer to the - [Cluster](https://nodejs.org/api/cluster.html) module in Node.js and - [this section](howto.md#iot-agent-in-multi-thread-mode) of the library documentation. -- **fallbackTenant** - For Linked Data Context Brokers which do not support multi-tenancy, this provides an - alternative mechanism for supplying the `NGSILD-Tenant` header. Note that NGSILD-Tenant has not yet been included in - the NGSI-LD standard (it has been proposed for the next update of the standard, but the final decision has yet been - confirmed), take into account it could change. Note that for backwards compatibility with NGSI v2, the - `fiware-service` header is already used as alternative if the `NGSILD-Tenant` header is not supplied. -- **fallbackPath** - For Linked Data Context Brokers which do not support a service path, this provides an alternative - mechanism for suppling the `NGSILD-Path` header. Note that for backwards compatibility with NGSI v2, the - `fiware-servicepath` header is already used as alternative if the `NGSILD-Path` header is not supplied. Note that - NGSILD-Path has not yet been included in the NGSI-LD standard (it has been proposed for the next update of the - standard, but the final decision has yet been confirmed), take into account it could change -- **explicitAttrs**: if this flag is activated, only provisioned attributes will be processed to Context Broker. This - flag is overwritten by `explicitAttrs` flag in group or device provision. Additionally `explicitAttrs` can be used - to define which meassures defined in JSON/JEXL array will be propagated to NGSI interface. -- **defaultEntityNameConjunction**: the default conjunction string used to compose a default `entity_name` when is not - provided at device provisioning time; in that case `entity_name` is composed by `type` + `:` + `device_id`. Default - value is `:`. This value is overwritten by `defaultEntityNameConjunction` in group provision. -- **relaxTemplateValidation**: if this flag is activated, `objectId` attributes for incoming devices are not - validated, and may exceptionally include characters (such as semi-colons) which are - [forbidden](https://fiware-orion.readthedocs.io/en/master/user/forbidden_characters/index.html) according to the - NGSI specification. When provisioning devices, it is necessary that the developer provides valid `objectId`-`name` - mappings whenever relaxed mode is used, to prevent the consumption of forbidden characters. - -### Configuration using environment variables - -Some of the configuration parameters can be overriden with environment variables, to ease the use of those parameters -with container-based technologies, like Docker, Heroku, etc... - -The following table shows the accepted environment variables, as well as the configuration parameter the variable -overrides. - -| Environment variable | Configuration attribute | -| :----------------------------------- | :------------------------------ | -| IOTA_CB_URL | `contextBroker.url` | -| IOTA_CB_HOST | `contextBroker.host` | -| IOTA_CB_PORT | `contextBroker.port` | -| IOTA_CB_NGSI_VERSION | `contextBroker.ngsiVersion` | -| IOTA_NORTH_HOST | `server.host` | -| IOTA_NORTH_PORT | `server.port` | -| IOTA_LD_SUPPORT_NULL | `server.ldSupport.null` | -| IOTA_LD_SUPPORT_DATASET_ID | `server.ldSupport.datasetId` | -| IOTA_PROVIDER_URL | `providerUrl` | -| IOTA_AUTH_ENABLED | `authentication.enabled` | -| IOTA_AUTH_TYPE | `authentication.type` | -| IOTA_AUTH_HEADER | `authentication.header` | -| IOTA_AUTH_URL | `authentication.url` | -| IOTA_AUTH_HOST | `authentication.host` | -| IOTA_AUTH_PORT | `authentication.port` | -| IOTA_AUTH_USER | `authentication.user` | -| IOTA_AUTH_PASSWORD | `authentication.password` | -| IOTA_AUTH_CLIENT_ID | `authentication.clientId` | -| IOTA_AUTH_CLIENT_SECRET | `authentication.clientSecret` | -| IOTA_AUTH_TOKEN_PATH | `authentication.tokenPath` | -| IOTA_AUTH_PERMANENT_TOKEN | `authentication.permanentToken` | -| IOTA_REGISTRY_TYPE | `deviceRegistry.type` | -| IOTA_LOG_LEVEL | `logLevel` | -| IOTA_TIMESTAMP | `timestamp` | -| IOTA_IOTAM_URL | `iotManager.url` | -| IOTA_IOTAM_HOST | `iotManager.host` | -| IOTA_IOTAM_PORT | `iotManager.port` | -| IOTA_IOTAM_PATH | `iotManager.path` | -| IOTA_IOTAM_AGENTPATH | `iotManager.agentPath` | -| IOTA_IOTAM_PROTOCOL | `iotManager.protocol` | -| IOTA_IOTAM_DESCRIPTION | `iotManager.description` | -| IOTA_MONGO_HOST | `mongodb.host` | -| IOTA_MONGO_PORT | `mongodb.port` | -| IOTA_MONGO_DB | `mongodb.db` | -| IOTA_MONGO_REPLICASET | `mongodb.replicaSet` | -| IOTA_MONGO_USER | `mongodb.user` | -| IOTA_MONGO_PASSWORD | `mongodb.password` | -| IOTA_MONGO_AUTH_SOURCE | `mongodb.authSource` | -| IOTA_MONGO_RETRIES | `mongodb.retries` | -| IOTA_MONGO_RETRY_TIME | `mongodb.retryTime` | -| IOTA_MONGO_SSL | `mongodb.ssl` | -| IOTA_MONGO_EXTRAARGS | `mongodb.extraArgs` | -| IOTA_SINGLE_MODE | `singleConfigurationMode` | -| IOTA_POLLING_EXPIRATION | `pollingExpiration` | -| IOTA_POLLING_DAEMON_FREQ | `pollingDaemonFrequency` | -| IOTA_AUTOCAST | `autocast` | -| IOTA_MULTI_CORE | `multiCore` | -| IOTA_JSON_LD_CONTEXT | `jsonLdContext` | -| IOTA_FALLBACK_TENANT | `fallbackTenant` | -| IOTA_FALLBACK_PATH | `fallbackPath` | -| IOTA_EXPLICIT_ATTRS | `explicitAttrs` | -| IOTA_DEFAULT_ENTITY_NAME_CONJUNCTION | `defaultEntityNameConjunction` | -| IOTA_RELAX_TEMPLATE_VALIDATION | `relaxTemplateValidation` | - -Note: - -- If you need to pass more than one JSON-LD context, you can define the IOTA_JSON_LD_CONTEXT environment variable as a - comma separated list of contexts (e.g. `'http://context1.json-ld,http://context2.json-ld'`) diff --git a/doc/operations.md b/doc/operations.md deleted file mode 100644 index 103902657..000000000 --- a/doc/operations.md +++ /dev/null @@ -1,127 +0,0 @@ -# Operations Manual: logs and alarms - -- [Logs](#logs) -- [Alarms](#alarms) -- [Error naming code](#error-naming-code) - -The following document shows all the errors that can appear in an IoT Agent using this library, and gives a brief idea -of the severity and how to react to those errors. - -## Logs - -The following section contains the error log entries that can appear in the IoT Agents logs, grouped by category. - -##### GENERAL-001 Couldn't find callback in listDevices() call. - -##### MONGODB-001: Error trying to connect to MongoDB: %s - -Implies there has been an error connecting with the DB. The component will automatically retry from this error, but it -may be a sign of connectivity problems between the DB and the component. If the connection cannot be restablished from -this error, a MONGODB-002 error will be raised. - -##### MONGODB-002: Error found after [%d] attempts: %s - -Indicates that it was impossible to establish a connection to the MongoDB cluster, even after retrying N times. This -could be caused by a connectivity problem with the MongoDB machine, a problem in the MongoDB cluster, or a -misconfiguration of the IoTA Manager. Check the conectivity, the state of the MongoDB cluster and the Mongo -configuration data. - -##### MONGODB-003: No host found for MongoDB driver. - -This error will thrown if MongoDB is selected as the configured repository for data but some information is missing in -the configuration file. Check the configuration file and add all the required information. - -##### MONGODB-004: MongoDB connection was lost. - -Indicates that it was impossible to reestablish the connection with the MongoDB server after retrying N times. This -could be caused by a connectivity problem with the MongoDB machine or by changes on the configuration of the MongoDB -server done while the IoT Agent was running. This error is only thrown when using a single MongoDB instance or when -using sharding but just a single mongos proxy. When using MongoDB instances using replica sets or multiple mongos -servers, the IoT Agent will retry connecting forever alternating between the different nodes. - -##### IOTAM-001: Error updating information in the IOTAM. Status Code [%d] - -The IoT Agent could not contact the IoT Agent manager to update its information. This condition may indicate a lack of -connectivity between machines or a problem in the IoT Agent Manager. The IoT Agent information in the IoT Agent Manager -will be out-of-date until this problem is solved. - -##### KEYSTONE-001: Error retrieving token from Keystone: %s - -There was connection error connecting with Keystone to retrieve a token. This condition may indicate a lack of -connectivity between both machines or a problem with Keystone. - -##### KEYSTONE-002: Unexpected status code: %d - -There was a problem retrieving a token from keystone that was not caused by connectivity errors. Check the Keystone log -for errors and the security configuration in the IoTAgent. This may also be caused by a wrong trust token used by the -user. - -##### KEYSTONE-003: Token missing in the response headers. - -Authentication flow worked correctly, but the response headers did not include the expected header `x-subject-token`. -Check the Keystone logs and configuration. - -##### OAUTH2-001: Error retrieving token from OAuth2 provider: %s - -There was connection error connecting with OAuth2 provider to retrieve a token. This condition may indicate a lack of -connectivity between both machines or a problem with OAuth2 provider. - -##### OAUTH2-002: Unexpected status code: %d - -There was a problem retrieving a token from OAuth2 provider that was not caused by connectivity errors. Check the OAuth2 -provider log for errors and the security configuration in the IoTAgent. This may also be caused by an invalid -`refresh_token` used by the user. - -##### OAUTH2-003: Token missing in the response body - -The JSON response body returned by the OAuth2 provider does not include a field `access_token`. Check the OAuth2 logs -and configuration. - -##### ORION-001: Connection error creating initial entity in the Context Broker: %s - -There was a connectivity error accessing Context Broker to create an initial entity (or the Context Broker was down). -Check connectivity between the machines, the status of the remote Context Broker and the configuration of the IoTAgent. - -##### ORION-002: Connection error sending registrations to the Context Broker: %s - -There was a connectivity error accessing Context Broker to register the IoTA as a Context Provider (or the Context -Broker was down). Check connectivity between the machines, the status of the remote Context Broker and the configuration -of the IoT Agent. - -##### VALIDATION-FATAL-001: Validation Request templates not found - -Validation templates were not found. Check all the validation templates are properly located in the IoTAgent Library -folder and that the file permissions are correct. - -## Alarms - -The following table shows the alarms that can be raised in the IoTAgent library. All the alarms are signaled by a error -log starting with the prefix "Raising [%s]:" (where %s is the alarm name). All the alarms are released by an info log -with the prefix "Releasing [%s]". These texts appear in the `msg=` field of the generic log record format. - -| Alarm name | Severity | Description | -| :------------- | :----------- | :------------------------------------------------------- | -| MONGO-ALARM_XX | **Critical** | Indicates an error in the MongoDB connectivity | -| ORION-ALARM | **Critical** | Indicates a persistent error accesing the Context Broker | -| IOTAM-ALARM | **Critical** | Indicates a persistent error accessing the IoTAM | - -while the 'Severity' criterium is as follows: - -- **Critical** - The system is not working -- **Major** - The system has a problem that degrades the service and must be addressed -- **Warning** - It is happening something that must be notified - -In order to identify the internal flow which origins a mongo alarm, there is a suffix `_XX` which identifies from `01` to -`11` each flow. - -## Error naming code - -Every error has a code composed of a prefix and an ID, codified with the following table: - -| Prefix | Type of operation | -| :--------------- | :--------------------------------------------------------- | -| MONGODB | Errors related with the MongoDB repository | -| IOTAM | Errors related with the IoTA Manager | -| KEYSTONE | Errors related with trust token retrieval | -| ORION | Errors in Context Broker access | -| VALIDATION-FATAL | Errors related with management of the Validation templates | diff --git a/doc/usermanual.md b/doc/usermanual.md deleted file mode 100644 index 843f6f77a..000000000 --- a/doc/usermanual.md +++ /dev/null @@ -1,900 +0,0 @@ -# Library Functions - -- [Stats Registry](#stats-registry) -- [Alarm module](#alarm-module) -- [Transactions](#transactions) -- [Library overview](#library-overview) -- [Function reference](#function-reference) - -### Stats Registry - -The library provides a mechanism for the periodic reporting of stats related to the library's work. In order to activate -the use of the periodic stats, it must be configured in the config file, as described in the -[Configuration](installationguide.md#configuration) section. - -The Stats Registry holds two dictionaries, with the same set of stats. For each stat, one of the dictionaries holds the -historical global value and the other one stores the value since the last value reporting (or current value). - -The stats library currently stores only the following values: - -- **deviceCreationRequests**: number of Device Creation Requests that arrived to the API (no matter the result). -- **deviceRemovalRequests**: number of Removal Device Requests that arrived to the API (no matter the result). -- **measureRequests**: number of times the ngsiService.update() function has been invoked (no matter the result). - -More values will be added in the future to the library. The applications using the library can add values to the Stats -Registry just by using the following function: - -```javascript -iotagentLib.statsRegistry.add("statName", statIncrementalValue, callback); -``` - -The first time this function is invoked, it will add the new stat to the registry. Subsequent calls will add the value -to the specified stat both to the current and global measures. The stat will be cleared in each interval as usual. - -### Alarm module - -The library provide an alarm module that can be used to track through the logs alarms raised in the IoTAgent. This -module provides: - -- Two functions to raise and release and alarm (`raise()` and `release()`): every alarm is identified by a name and a - description. When the alarm is raised, an error with the text `Raising [%s]` is logged. When the alarm is released, - the corresponding text, `Releasing [%s]` is logged. If an alarm is raised multiple times, it is only logged once. If - its released multiple times it is only released once. Releasing a non-existing alarm has no effect. - -- Functions to list all the raised alarms and clean all the alarms (`list()` and `clean()`). - -- A function to instrument other functions, so when one of that functions return an error, an alarm is raised, and - when it returns a success an alarm is ceased (`intercept()`). - -All this functions can be accessed through the `.alarms` attribute of the library. - -### Transactions - -The library implements a concept of transactions, in order to follow the execution flow the library follows when -treating requests entering both from the North and the South ports of the IoT Agent. - -To follow the transactions, a new Domain is created for each incoming request; in the case of requests received on the -North Port of the IoT Agent, this domain is automatically created by a Express middleware, and no further action is -needed from the user. For the case of requests received on the South Port of the IoT Agent, the user is responsible of -creating an stopping the transaction, using the `ensureSouthboundDomain` and `finishSouthBoundTransaction`. In this -case, the transaction will last from the invocation to the former to the invocation of the latter. - -The Transaction Correlator is used along all the IoT Platform to follow the trace of a transaction between multiple -components. To do so, in all the HTTP requests sent to other components of the platform, a custom header named -`Fiware-Correlator` is sent with the correlator of the transaction that generated the request. If a component of the -platform receives a request containing this header that starts a transaction, the component will create the transaction -with the received correlator, instead of creating a new one. If the header is not present or the transaction originates -in the component, the transaction ID in this component will be used as the correlator. - -During the duration of a transaction, all the log entries created by the code will write the current Transaction ID and -correlator for the operation being executed. - -### Library overview - -In order to use the library, add the following dependency to your package.json file: - -```json -"iotagent-node-lib": "*" -``` - -In order to use this library, first you must require it: - -```javascript -var iotagentLib = require("iotagent-node-lib"); -``` - -The library supports four groups of features, one for each direction of the communication: client-to-server and -server-to-client (and each flow both for the client and the server). Each feature set is defined in the following -sections. - -### Function reference - -##### iotagentLib.activate() - -###### Signature - -```javascript -function activate(newConfig, callback) -``` - -###### Description - -Activates the IoT Agent to start listening for NGSI Calls (acting as a Context Provider). It also creates the device -registry for the IoT Agent (based on the deviceRegistry.type configuration option). - -###### Params - -- newConfig: Configuration of the Context Server (described in the [Configuration](installationguide.md#configuration) - section). - -##### iotagentLib.deactivate() - -###### Signature - -```javascript -function deactivate(callback) -``` - -###### Description - -Stops the HTTP server. - -###### Params - -##### iotagentLib.register() - -###### Signature - -```javascript -function registerDevice(deviceObj, callback) -``` - -###### Description - -Register a new device in the IoT Agent. This registration will also trigger a Context Provider registration in the -Context Broker for all its lazy attributes. - -The device Object can have the following attributes: - -- `id`: Device ID of the device. -- `type`: type to be assigned to the device. -- `name`: name that will be used for the Entity representing the device in the Context Broker. -- `service`: name of the service associated with the device. -- `subservice`: name of the subservice associated with th device. -- `lazy`: list of lazy attributes with their types. -- `active`: list of active attributes with their types. -- `staticAttributes`: list of NGSI attributes to add to the device entity 'as is' in updates, queries and - registrations. -- `internalAttributes`: optional section with free format, to allow specific IoT Agents to store information along - with the devices in the Device Registry. - -The device `id` and `type` are required fields for any registration. The rest of the attributes are optional, but, if -they are not present in the function call arguments, the type must be registered in the configuration, so the service -can infer their default values from the configured type. If an optional attribute is not given in the parameter list and -there isn't a default configuration for the given type, a TypeNotFound error is raised. - -If the device has been previously preprovisioned, the missing data will be completed with the values from the registered -device. - -###### Params - -- deviceObj: object containing all the information about the device to be registered (mandatory). - -##### iotagentLib.unregister() - -###### Signature - -```javascript -function unregisterDevice(id, service, subservice, callback) -``` - -###### Description - -Unregister a device from the Context broker and the internal registry. - -###### Params - -- id: Device ID of the device to register. -- service: Service of the device to unregister. -- subservice: Subservice inside the service for the unregistered device. - -##### iotagentLib.update() - -###### Signature - -```javascript -function update(entityName, attributes, typeInformation, token, callback) -``` - -###### Description - -Makes an update in the Device's entity in the context broker, with the values given in the 'attributes' array. This -array should comply to the NGSI's attribute format. - -###### Params - -- entityName: Name of the entity to register. -- attributes: Attribute array containing the values to update. -- typeInformation: Configuration information for the device. -- token: User token to identify against the PEP Proxies (optional). - -##### iotagentLib.setCommandResult() - -###### Signature - -```javascript -function setCommandResult(entityName, resource, apikey, commandName, commandResult, status, deviceInformation, callback) -``` - -###### Description - -Update the result of a command in the Context Broker. The result of the command has two components: the result of the -command itself will be represented with the suffix `_info` in the entity while the status is updated in the attribute -with the `_status` suffix. - -###### Params - -- entityName: Name of the entity holding the command. -- resource: Resource name of the endpoint the device is calling. -- apikey: Apikey the device is using to send the values (can be the empty string if none is needed). -- commandName: Name of the command whose result is being updated. -- commandResult: Result of the command in string format. -- deviceInformation: Device information, including security and service information. (optional). - -##### iotagentLib.listDevices() - -###### Signature - -```javascript -function listDevices(callback) -function listDevices(limit, offset, callback) -function listDevices(service, subservice, limit, offset, callback) -``` - -###### Description - -Return a list of all the devices registered in the specified service and subservice. This function can be invoked in -three different ways: - -- with just one parameter (the callback) -- with three parameters (service, subservice and callback) -- or with five parameters (including limit and offset). - -###### Params - -- service: service from where the devices will be retrieved. -- subservice: subservice from where the devices will be retrieved. -- limit: maximum number of results to retrieve (optional). -- offset: number of results to skip from the listing (optional). - -##### iotagentLib.setDataUpdateHandler() - -###### Signature - -```javascript -function setDataUpdateHandler(newHandler) -``` - -###### Description - -Sets the new user handler for Entity update requests. This handler will be called whenever an update request arrives -with the following parameters: (`id`, `type`, `service`, `subservice`, `attributes`, `callback`). Every object within of -the `attributes` array contains `name`, `type` and `value` attributes, and may also include additional attributes for -`metadata` and `datasetId`. The handler is in charge of updating the corresponding values in the devices with the -appropriate protocol. - -Once all the updates have taken place, the callback must be invoked with the updated Context Element. E.g.: - -```javascript -callback(null, { - type: "TheType", - isPattern: false, - id: "EntityID", - attributes: [ - { - name: "lumniscence", - type: "Lumens", - value: "432" - } - ] -}); -``` - -In the case of NGSI requests affecting multiple entities, this handler will be called multiple times, one for each -entity, and all the results will be combined into a single response. - -###### Params - -- newHandler: User handler for update requests - -##### iotagentLib.setDataQueryHandler() - -###### Signature - -```javascript -function setDataQueryHandler(newHandler) -``` - -###### Description - -Sets the new user handler for Entity query requests. This handler will be called whenever a query request arrives, with -the following parameters: (`id`, `type`, `service`, `subservice`, `attributes`, `callback`). The handler must retrieve -all the corresponding information from the devices and return a NGSI entity with the requested values. - -The callback must be invoked with the updated Context Element, using the information retrieved from the devices. E.g.: - -```javascript -callback(null, { - type: "TheType", - isPattern: false, - id: "EntityID", - attributes: [ - { - name: "lumniscence", - type: "Lumens", - value: "432" - } - ] -}); -``` - -In the case of NGSI requests affecting multiple entities, this handler will be called multiple times, one for each -entity, and all the results will be combined into a single response. - -###### Params - -- newHandler: User handler for query requests. - -##### iotagentLib.setNotificationHandler() - -###### Signature - -```javascript -function setNotificationHandler(newHandler) -``` - -###### Description - -Sets the new handler for incoming notifications. The notifications are sent by the Context Broker based on the IoT Agent -subscriptions created with the `subscribe()` function. - -The handler must adhere to the following signature: - -```javascript -function mockedHandler(device, data, callback) -``` - -The `device` parameter contains the device object corresponding to the entity whose changes were notified with the -incoming notification. Take into account that multiple entities may be modified with each single notification. The -handler will be called once for each one of those entities. - -The `data` parameter is an array with all the attributes that were requested in the subscription and its respective -values. - -The handler is expected to call its callback once with no parameters (failing to do so may cause unexpected behaviors in -the IoT Agent). - -##### iotagentLib.setCommandHandler() - -###### Signature - -```javascript -function setCommandHandler(newHandler) -``` - -###### Description - -Sets the new user handler for registered entity commands. This handler will be called whenever a command request arrives, with -the following parameters: (`id`, `type`, `service`, `subservice`, `attributes`, `callback`). The handler must retrieve -all the corresponding information from the devices and return a NGSI entity with the requested values. - -The callback must be invoked with the updated Context Element, using the information retrieved from the devices. E.g.: - -```javascript -callback(null, { - type: "TheType", - isPattern: false, - id: "EntityID", - attributes: [ - { - name: "lumniscence", - type: "Lumens", - value: "432" - } - ] -}); -``` - -In the case of NGSI requests affecting multiple entities, this handler will be called multiple times, one for each -entity, and all the results will be combined into a single response. Only IoT Agents which deal with actuator devices will include a handler for commands. - -###### Params - -- newHandler: User handler for command requests. - -##### iotagentLib.setMergePatchHandler() - -###### Signature - -```javascript -function setMergePatchHandler(newHandler) -``` - -###### Description - -Sets the new user handler for NGSI-LD Entity [merge-patch](https://datatracker.ietf.org/doc/html/rfc7386) requests. This handler will be called whenever a merge-patch request arrives, with -the following parameters: (`id`, `type`, `service`, `subservice`, `attributes`, `callback`). The handler must retrieve -all the corresponding information from the devices and return a NGSI entity with the requested values. - -The callback must be invoked with the updated Context Element, using the information retrieved from the devices. E.g.: - -```javascript -callback(null, { - type: "TheType", - isPattern: false, - id: "EntityID", - attributes: [ - { - name: "lumniscence", - type: "Lumens", - value: "432" - } - ] -}); -``` - -In the case of NGSI-LD requests affecting multiple entities, this handler will be -called multiple times. Since merge-patch is an advanced function, not all IoT Agents -will include a handler for merge-patch. - -###### Params - -- newHandler: User handler for merge-patch requests. - -##### iotagentLib.setProvisioningHandler() - -###### Signature - -```javascript -function setProvisioningHandler (newHandler) -``` - -###### Description - -Sets the new user handler for the provisioning of devices. This handler will be called every time a new device is -created. - -The handler must adhere to the following signature: - -```javascript -function(newDevice, callback) -``` - -The `newDevice` parameter will contain the newly created device. The handler is expected to call its callback with no -parameters (this handler should only be used for reconfiguration purposes of the IoT Agent). - -##### iotagentLib.setRemoveDeviceHandler() - -###### Signature - -```javascript -function setRemoveDeviceHandler(newHandler) -``` - -###### Description - -Sets the new user handler for the removal of a device. This handler will be called every time a device is removed. - -The handler must adhere to the following signature: - -```javascript -function(deviceToDelete, callback) -``` - -The `deviceToDelete` parameter will contain the device to be deleted. The handler is expected to call its callback with -no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent). - -##### iotagentLib.setConfigurationHandler() - -###### Signature - -```javascript -function setConfigurationHandler(newHandler) -``` - -###### Description - -Sets the new user handler for the configuration updates. This handler will be called every time a new configuration is -created or an old configuration is updated. - -The handler must adhere to the following signature: - -```javascript -function(newConfiguration, callback) -``` - -The `newConfiguration` parameter will contain the newly created configuration. The handler is expected to call its -callback with no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent). - -For the cases of multiple updates (a single Device Configuration POST that will create several device groups), the -handler will be called once for each of the configurations (both in the case of the creations and the updates). - -The handler will be also called in the case of updates related to configurations. In that situation, the -`newConfiguration` parameter contains also the fields needed to identify the configuration to be updated, i.e., -`service`, `subservice`, `resource` and `apikey`. - -##### iotagentLib.setRemoveConfigurationHandler() - -###### Signature - -```javascript -function setRemoveConfigurationHandler(newHandler) -``` - -###### Description - -Sets the new user handler for the removal of configuratios. This handler will be called every time a configuration is -removed. - -The handler must adhere to the following signature: - -```javascript -function(configurationToDelete, callback) -``` - -The `configurationToDelete` parameter will contain the configuration to be deleted. The handler is expected to call its -callback with no parameters (this handler should only be used for reconfiguration purposes of the IoT Agent). - -##### iotagentLib.getDevice() - -###### Signature - -```javascript -function getDevice(deviceId, service, subservice, callback) -``` - -###### Description - -Retrieve all the information about a device from the device registry. - -###### Params - -- deviceId: ID of the device to be found. -- service: Service for which the requested device. -- subservice: Subservice inside the service for which the device is requested. - -##### iotagentLib.getDeviceByName() - -###### Signature - -```javascript -function getDeviceByName(deviceName, service, subservice, callback) -``` - -###### Description - -Retrieve a device from the registry based on its entity name. - -###### Params - -- deviceName: Name of the entity associated to a device. -- service: Service the device belongs to. -- subservice: Division inside the service. - -##### iotagentLib.getDevicesByAttribute() - -###### Signature - -```javascript -function getDevicesByAttribute(attributeName, attributeValue, service, subservice, callback) -``` - -###### Description - -Retrieve all the devices having an attribute named `name` with value `value`. - -###### Params - -- name: name of the attribute to match. -- value: value to match in the attribute. -- service: Service the device belongs to. -- subservice: Division inside the service. - -##### iotagentLib.retrieveDevice() - -###### Signature - -```javascript -function retrieveDevice(deviceId, apiKey, callback) -``` - -###### Description - -Retrieve a device from the device repository based on the given APIKey and DeviceID, creating one if none is found for -the given data. - -###### Params - -- deviceId: Device ID of the device that wants to be retrieved or created. -- apiKey: APIKey of the Device Group (or default APIKey). - -##### iotagentLib.mergeDeviceWithConfiguration() - -###### Signature - -```javascript -function mergeDeviceWithConfiguration(fields, defaults, deviceData, configuration, callback) -``` - -###### Description - -Complete the information of the device with the information in the configuration group (with precedence of the device). -The first argument indicates what fields would be merged. - -###### Params - -- fields: Fields that will be merged. -- defaults: Default values fot each of the fields. -- deviceData: Device data. -- configuration: Configuration data. - -##### iotagentLib.getConfiguration() - -###### Signature - -```javascript -function getConfiguration(resource, apikey, callback) -``` - -###### Description - -Gets the device group identified by the given (`resource`, `apikey`) pair. - -###### Params - -- resource: representation of the configuration in the IoT Agent (dependent on the protocol) . -- apikey: special key the devices will present to prove they belong to a particular configuration. - -##### iotagentLib.findConfiguration() - -###### Signature - -```javascript -function findConfiguration(service, subservice, callback) -``` - -###### Description - -Find a device group based on its service and subservice. - -###### Params - -- service: name of the service of the configuration. -- subservice: name of the subservice of the configuration. - -##### iotagentLib.getEffectiveApiKey() - -###### Signature - -```javascript -function getEffectiveApiKey(service, subservice, type, callback) -``` - -###### Description - -Get the API Key for the selected service if there is any, or the default API Key if a specific one does not exist. - -###### Params - -- service: Name of the service whose API Key we are retrieving. -- subservice: Name of the subservice whose API Key we are retrieving. -- type: Type of the device. - -##### iotagentLib.subscribe() - -###### Signature - -```javascript -function subscribe(device, triggers, content, callback) -``` - -###### Description - -Creates a subscription for the IoTA to the entity representing the selected device. - -###### Params - -- device: Object containing all the information about a particular device. -- triggers: Array with the names of the attributes that would trigger the subscription -- content: Array with the names of the attributes to retrieve in the notification. - -##### iotagentLib.unsubscribe() - -###### Signature - -```javascript -function unsubscribe(device, id, callback) -``` - -###### Description - -Removes a single subscription from the selected device, identified by its ID. - -###### Params - -- `device`: Object containing all the information about a particular device. -- `id`: ID of the subscription to remove. - -##### iotagentLib.ensureSouthboundDomain() - -###### Signature - -```javascript -function ensureSouthboundTransaction(context, callback) -``` - -###### Description - -Ensures that the current operation is executed inside a transaction with all the information needed for the appropriate -platform logging: start date, transaction ID and correlator in case one is needed. If the function is executed in the -context of a previous transaction, just the context is changed (and the Transaction ID and start time are kept). - -###### Params - -- context: New context data for the transaction. - -##### iotagentLib.finishSouthBoundTransaction() - -###### Signature - -```javascript -function finishSouthboundTransaction(callback) -``` - -###### Description - -Terminates the current transaction, if there is any, cleaning its context. - -##### iotagentLib.startServer() - -###### Signature - -```javascript -function startServer(newConfig, iotAgent, callback) -``` - -###### Description - -Start the HTTP server either in single-thread or multi-thread (multi-core) based on the value of _multiCore_ variable -(described in the [Configuration](installationguide.md#configuration) section). If the value is `False` (either was -directly specified `False` in the `config.js` or it was not specified and by default is assigned `False`), it is a -normal (single-thread) behaviour. Nevertheless, if _multiCore_ is `True`, the IoTAgent is executed in multi-thread -environment. - -The number of parallel processes is calculated based on the number of available CPUs. In case of some of the process -unexpectedly dead, a new process is created automatically to keep always the maximum of them working in parallel. - -> Note: `startServer()` initializes the server but it does not activate the library. The function in the Node Lib will -> call the `iotAgent.start()` in order to complete the activation of the library. Therefore, it is expected that the IoT -> Agent implement the `iotAgent.start()` function with the proper invocation to the `iotAgentLib.activate()`. - -###### Params - -- newConfig: Configuration of the Context Server (described in the [Configuration](installationguide.md#configuration) - section). -- iotAgent: The IoT Agent Objects, used to start the agent. -- callback: The callback function. - -##### iotagentLib.request() - -###### Signature - -```javascript -function request(options, callback) -``` - -###### Description - -Make a direct HTTP request using the underlying request library (currently [got](https://github.com/sindresorhus/got)), -this is useful when creating agents which use an HTTP transport for their southbound commands, and removes the need for -the custom IoT Agent to import its own additional request library - -###### Params - -- options: definition of the request (see - [got options](https://github.com/sindresorhus/got/blob/main/documentation/2-options.md) for more details). The - following attributes are currently exposed. - - `method` - HTTP Method - - `searchParams` - query string params - - `qs` - alias for query string params - - `headers` - - `responseType` - either `text` or `json`. `json` is the default - - `json` - a supplied JSON object as the request body - - `body` - any ASCII text as the request body. It takes precedence over `json` if both are provided at the same - time (not recommended). - - `url` - the request URL - - `uri` - alternative alias for the request URL. -- callback: The callback currently returns an `error` Object, the `response` and `body`. The `body` is parsed to a - JSON object if the `responseType` is JSON. - -#### Generic middlewares - -This collection of utility middlewares is aimed to be used to north of the IoT Agent Library, as well as in other -HTTP-based APIs of the IoT Agents. All the middlewares follow the Express convention of `(req, res, next)` objects, so -this information will not be repeated in the descriptions for the middleware functions. All the middlewares can be added -to the servers using the standard Express mechanisms. - -##### iotagentLib.middlewares.handleError() - -###### Signature - -```javascript -function handleError(error, req, res, next) -``` - -###### Description - -Express middleware for handling errors in the IoTAs. It extracts the code information to return from the error itself -returning 500 when no error code has been found. - -##### iotagentLib.middlewares.traceRequest() - -###### Signature - -```javascript -function traceRequest(req, res, next) -``` - -###### Description - -Express middleware for tracing the complete request arriving to the IoTA in debug mode. - -##### iotagentLib.middlewares.changeLogLevel() - -###### Signature - -```javascript -function changeLogLevel(req, res, next) -``` - -###### Description - -Changes the log level to the one specified in the request. - -##### iotagentLib.middlewares.ensureType() - -###### Signature - -```javascript -function ensureType(req, res, next) -``` - -###### Description - -Ensures the request type is one of the supported ones. - -##### iotagentLib.middlewares.validateJson() - -###### Signature - -```javascript -function validateJson(template) -``` - -###### Description - -Generates a Middleware that validates incoming requests based on the JSON Schema template passed as a parameter. - -Returns an Express middleware used in request validation with the given template. - -###### Params - -- _template_: JSON Schema template to validate the request. - -##### iotagentLib.middlewares.retrieveVersion() - -###### Signature - -```javascript -function retrieveVersion(req, res, next) -``` - -###### Description - -Middleware that returns all the IoTA information stored in the module. - -##### iotagentLib.middlewares.setIotaInformation() - -###### Signature - -```javascript -function setIotaInformation(newIoTAInfo) -``` - -###### Description - -Stores the information about the IoTAgent for further use in the `retrieveVersion()` middleware. - -###### Params - -- _newIoTAInfo_: Object containing all the IoTA Information. diff --git a/lib/jexlTranformsMap.js b/lib/jexlTranformsMap.js index 4c263d5be..b6031c035 100644 --- a/lib/jexlTranformsMap.js +++ b/lib/jexlTranformsMap.js @@ -79,7 +79,8 @@ const map = { // https://developer.mozilla.org/es/docs/Web/JavaScript/Reference/Global_Objects/Date/toLocaleString localestring: (d, timezone, options) => new Date(d).toLocaleString(timezone, options), now: () => Date.now(), - hextostring: (val) => new TextDecoder().decode(new Uint8Array(val.match(/.{1,2}/g).map(byte => parseInt(byte, 16)))) + hextostring: (val) => + new TextDecoder().decode(new Uint8Array(val.match(/.{1,2}/g).map((byte) => parseInt(byte, 16)))) }; exports.map = map; diff --git a/lib/request-shim.js b/lib/request-shim.js index f1aab860e..fe2694e0a 100644 --- a/lib/request-shim.js +++ b/lib/request-shim.js @@ -59,7 +59,7 @@ function getOptions(options) { // got library is not properly documented, so it is not clear which takes precedence // among body, json and form (see https://stackoverflow.com/q/70754880/1485926). // Thus, we are enforcing our own precedence with the "else if" chain below. - // Behaviour is consistent with the one described at usermanual.md#iotagentlibrequest + // Behaviour is consistent with the one described at development.md#iotagentlibrequest if (options.method === 'GET' || options.method === 'HEAD' || options.method === 'OPTIONS') { // Do nothing - Never add a body @@ -70,7 +70,7 @@ function getOptions(options) { // json takes precedence over form httpOptions.json = options.json; } else if (options.form) { - // Note that we don't consider 'form' part of the function API (check usermanual.md#iotagentlibrequest) + // Note that we don't consider 'form' part of the function API (check development.md#iotagentlibrequest) // but we are preparing the code anyway as a safe measure httpOptions.form = options.form; } diff --git a/mkdocs.yml b/mkdocs.yml index 94795c5a5..090c06b71 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -13,15 +13,10 @@ pages: - Home: 'index.md' - 'Getting Started' : 'getting-started.md' - 'User & Programmers Manual': - - 'Architecture' : 'architecture.md' - 'IoT Agent API' : 'api.md' - - 'Advanced Topics' : 'advanced-topics.md' - - 'Library Functions': 'usermanual.md' - - 'Measurement Transformation Expression Language': 'expressionLanguage.md' - - 'How to develop a new IoT Agent': 'howto.md' - - 'North Port - NGSI Interactions': 'northboundinteractions.md' - - 'Development Documentation': development.md - - 'Installation & Administration Manual': - - 'Installation Guide': 'installationguide.md' - - 'Operations (logs & alarms)': 'operations.md' - + - 'Installation and administration manual': 'admin.md' + - 'Development documentation': + - 'Development manual': 'devel/development.md' + - 'Contributing guide': 'devel/contribution-guidelines.md' + - 'Architecture' : 'devel/architecture.md' + - 'North Port - NGSI Interactions': 'devel/northboundinteractions.md' \ No newline at end of file diff --git a/scripts/legacy_expression_tool/README.md b/scripts/legacy_expression_tool/README.md index bc507c069..bce38c25c 100644 --- a/scripts/legacy_expression_tool/README.md +++ b/scripts/legacy_expression_tool/README.md @@ -82,34 +82,50 @@ python legacy_expression_tool.py \ The list of possible arguments that the scripts accepts are: -| Argument | Description | Default value | Mandatory | -| ---------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------------- | --------- | -| `--mongouri` | The MongoDB URI to connect to | `mongodb://localhost:27017/` | No | -| `--database` | The database name to replace the expressions | NA | Yes | -| `--collection` | The collection name to replace the expressions | NA | Yes | -| `--translation` | The translation dictionary file to replace the expressions | `translation.json` | No | -| `--debug` | Enable debug mode | `False` | No | -| `--commit` | Commit the changes to the database | `False` | No | +| Argument | Description | Default value | Mandatory | +| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | --------- | +| `--mongouri` | The MongoDB URI to connect to | `mongodb://localhost:27017/` | No | +| `--database` | The database name to replace the expressions | NA | Yes | +| `--collection` | The collection name to replace the expressions | NA | Yes | +| `--translation` | The translation dictionary file to replace the expressions | `translation.json` | No | +| `--debug` | Enable debug mode | `False` | No | +| `--commit` | Commit the changes to the database | `False` | No | | `--expressionlanguage` | What to do with the expression language field. Possibles values: `delete`, `ignore`, `jexl` or `jexlall`. More detail on this bellow. | `ignore` | No | -| `--statistics` | Print match statistics. Aggregation modes are the possible values: `service` and `subservice` | `service` | No | -| `--service` | The fiware service filter to replace the expressions | All subservices | No | -| `--service-path` | The fiware service path filter to replace the expressions | All subservices | No | -| `--deviceid` | The device id filter to replace the expressions | All devices | No | -| `--entitytype` | The entity type filter to replace the expressions | All entity types | No | -| `--regexservice` | The fiware service regex filter to replace the expressions | All subservices | No | -| `--regexservicepath` | The fiware service path regex filter to replace the expressions | All subservices | No | -| `--regexdeviceid` | The device id regex filter to replace the expressions | All devices | No | -| `--regexentitytype` | The entity type regex filter to replace the expressions | All entity types | No | +| `--statistics` | Print match statistics. Aggregation modes are the possible values: `service` and `subservice` | `service` | No | +| `--service` | The fiware service filter to replace the expressions | All subservices | No | +| `--service-path` | The fiware service path filter to replace the expressions | All subservices | No | +| `--deviceid` | The device id filter to replace the expressions | All devices | No | +| `--entitytype` | The entity type filter to replace the expressions | All entity types | No | +| `--regexservice` | The fiware service regex filter to replace the expressions | All subservices | No | +| `--regexservicepath` | The fiware service path regex filter to replace the expressions | All subservices | No | +| `--regexdeviceid` | The device id regex filter to replace the expressions | All devices | No | +| `--regexentitytype` | The entity type regex filter to replace the expressions | All entity types | No | Note that filters (`--service`, `--service-path`, `--deviceid` and `--entitytype`, and the regex versions) are interpreted in additive way (i.e. like a logical AND). With regards to `--expressionlanguage`: -* `delete`: changes expressions from legacy to JEXL equivalence in the [fields where expressions may be used](#fields-in-which-expressions-may-be-used) (based on the translation dictionary specified by `--translations`). In addition, deletes the `expressionLanguage` field (no matter its value) in the case it exists in the group/device. -* `ignore`: changes expressions from legacy to JEXL equivalence in the [fields where expressions may be used](#fields-in-which-expressions-may-be-used) (based on the translation dictionary specified by `--translations`). In addition, it leaves untouched the `expressionLanguage` field. This may cause inconsistencies, if the value of the `expressionLanguage` is `legacy`, as detailed [in this section](#replacing-expression-without-setting-jexl-at-group-or-device-level). -* `jexl`: changes expressions from legacy to JEXL equivalence in the [fields where expressions may be used](#fields-in-which-expressions-may-be-used) (based on the translation dictionary specified by `--translations`). In addition, it set `expressionLanguage` field to `jexl` (no matter if the field originally exists in the group/device or not) **if some JEXL expression were translated**. -* `jexlall`: changes expression from legacy to JEXL equivalence in the [fields where expressions may be used](#fields-in-which-expressions-may-be-used) (based on the translation dictionary specified by `--translations`). In addition, it set `expressionLanguage` field to `jexl` (no matter if the field originally exists in the group/device or not). The difference between `jexl`and `jexlall` is in the second case, the script is not only looking for documents that contains legacy expressions, it also includes all groups/devices that have `expressionLanguage` field defined. The `expressionLanguage` field on those documents (and also on the documents that contains legacy expresions) is set to `jexl`. +- `delete`: changes expressions from legacy to JEXL equivalence in the + [fields where expressions may be used](#fields-in-which-expressions-may-be-used) (based on the translation + dictionary specified by `--translations`). In addition, deletes the `expressionLanguage` field (no matter its value) + in the case it exists in the group/device. +- `ignore`: changes expressions from legacy to JEXL equivalence in the + [fields where expressions may be used](#fields-in-which-expressions-may-be-used) (based on the translation + dictionary specified by `--translations`). In addition, it leaves untouched the `expressionLanguage` field. This may + cause inconsistencies, if the value of the `expressionLanguage` is `legacy`, as detailed + [in this section](#replacing-expression-without-setting-jexl-at-group-or-device-level). +- `jexl`: changes expressions from legacy to JEXL equivalence in the + [fields where expressions may be used](#fields-in-which-expressions-may-be-used) (based on the translation + dictionary specified by `--translations`). In addition, it set `expressionLanguage` field to `jexl` (no matter if + the field originally exists in the group/device or not) **if some JEXL expression were translated**. +- `jexlall`: changes expression from legacy to JEXL equivalence in the + [fields where expressions may be used](#fields-in-which-expressions-may-be-used) (based on the translation + dictionary specified by `--translations`). In addition, it set `expressionLanguage` field to `jexl` (no matter if + the field originally exists in the group/device or not). The difference between `jexl`and `jexlall` is in the second + case, the script is not only looking for documents that contains legacy expressions, it also includes all + groups/devices that have `expressionLanguage` field defined. The `expressionLanguage` field on those documents (and + also on the documents that contains legacy expresions) is set to `jexl`. ## Usage @@ -231,32 +247,34 @@ All 1 1 2 The script implements expression detection and translation in the following fields in group/device documents at DB: -* active.expression -* active.entity_name -* active.reverse -* attributes.expression -* attributes.entity_name -* attributes.expression -* commands.expression -* endpoint -* entityNameExp -* explicitAttrs +- active.expression +- active.entity_name +- active.reverse +- attributes.expression +- attributes.entity_name +- attributes.expression +- commands.expression +- endpoint +- entityNameExp +- explicitAttrs ### Known issues #### Execution with `expressionlanguage` set to `jexlall` When executing the script with `expressionlanguage` set to `jexlall`, the script will look for all the documents -containing legacy expressions (in some of the [expression capable fields](#fields-in-which-expressions-may-be-used)) or the existence of the `expressionLanguage` field. -This would change the number of documents found, and the statistics will include extra documents. +containing legacy expressions (in some of the [expression capable fields](#fields-in-which-expressions-may-be-used)) or +the existence of the `expressionLanguage` field. This would change the number of documents found, and the statistics +will include extra documents. Running the script with the option set to `jexl` would not include such extra documents in statistics. #### Replacing expression without setting jexl at group or device level -When executing the script setting `--expressionlanguage` to `ignore` (or when `--expressionlanguage` is not used), the script will not -change the`expressionLanguage` field in the document. This means that the legacy expressions will be replaced, but the -`expressionLanguage` field will still be set to the default value or legacy. This would make expression evaluation to -fail, propagating the value of the attribute as the expression literal to the context broker. +When executing the script setting `--expressionlanguage` to `ignore` (or when `--expressionlanguage` is not used), the +script will not change the`expressionLanguage` field in the document. This means that the legacy expressions will be +replaced, but the `expressionLanguage` field will still be set to the default value or legacy. This would make +expression evaluation to fail, propagating the value of the attribute as the expression literal to the context broker. -To avoid this, it is recommended use always the `--expressionlangauge` parameter set to `jexl`, so doing it a value different from `ignore` will be used. +To avoid this, it is recommended use always the `--expressionlangauge` parameter set to `jexl`, so doing it a value +different from `ignore` will be used.