[HELP] Implementing custom connector that read data from serial port

[ktran1005]

Describe the issue
HI ThingsBoard-gateway team, I am implementing a custom connector to connect to my serial port on my device, read the data and push to the edge. I followed the document from Thingsboard-gateway document to implement it. At the moment, I just hard-coded the data that the sensor will generate. I expected I will see this data on my "edge" server but it seems like the edge server does not receive this data (it is still in inactive state). The first highlight text below is the text got generated from the gateway and I assumed that the got parsed correctly. I expected to see this number on the latest telemetry in my gateway but nothing happened.

"2024-01-31 18:04:18" - |INFO| - [custom_serial_converter.py] - custom_serial_converter - convert - 150 - Converted data: {'deviceName': 'CustomSerialDevice1', 'deviceType': 'default', 'attributes': [], 'telemetry': [{'test': 123, 'ts': 1706742258308, 'values': {'test': 123}}]}"

Configuration (Attach your configuration file)

thingsboard:
  host: localhost
  port: 8080
  remoteShell: false
  remoteConfiguration: false
  statsSendPeriodInSeconds: 3600
  minPackSendDelayMS: 0
  checkConnectorsConfigurationInSeconds: 60
  handleDeviceRenaming: true
  checkingDeviceActivity:
    checkDeviceInactivity: false
    inactivityTimeoutSeconds: 120
    inactivityCheckPeriodSeconds: 10
  security:
    accessToken: ....
  qos: 1
storage:
  type: memory
  read_records_count: 100
  max_records_count: 100000
grpc:
  enabled: false
  serverPort: 9595
  keepaliveTimeMs: 10000
  keepaliveTimeoutMs: 5000
  keepalivePermitWithoutCalls: true
  maxPingsWithoutData: 0
  minTimeBetweenPingsMs: 10000
  minPingIntervalWithoutDataMs: 5000
connectors:

  -
   name: Custom Serial Connector
   type: serial
   configuration: custom_serial.json
   class: CustomSerialConnector

Here is my custom convert method

    def convert(self, config, data: bytes):
        global byteBuffer, byteBufferLength

        # Constants
        OBJ_STRUCT_SIZE_BYTES = 12;
        BYTE_VEC_ACC_MAX_SIZE = 2**15;
        MMWDEMO_UART_MSG_DETECTED_POINTS = 1;
        MMWDEMO_UART_MSG_RANGE_PROFILE   = 2;
        maxBufferSize = 2**15;
        tlvHeaderLengthInBytes = 8;
        pointLengthInBytes = 16;
        magicWord = [2, 1, 4, 3, 6, 5, 8, 7]
        
        # Initialize variables
        magicOK = 0 # Checks if magic number has been read
        dataOK = 0 # Checks if the data has been read correctly
        frameNumber = 0
        detObj = {}

        keys = ['attributes', 'telemetry']
        for key in keys:
            self.result_dict[key] = []
            if self.__config.get(key) is not None:
                for config_object in self.__config.get(key):
                    now = int(round(time.time()*1000))
                    converted_data = {config_object['key']: 123, "ts": now, "values": {"test": 123}}
                    self.result_dict[key].append(converted_data)
        log.info("Converted data: %s", self.result_dict)
        return self.result_dict

** Here is SerialConnector configuration**

{
  "name": "Custom serial connector",
  "devices": [
    {
      "name": "CustomSerialDevice1",
      "type": "default",
      "port": "/dev/ttyACM1",
      "baudrate": 921600,
      "converter": "CustomSerialUplinkConverter",
      "telementry": [],
      "attributes": [
        {
          "key": "SerialNumber",
          "type": "string"
        }
      ],
      "attributeUpdates": []
    }
  ]
}

I look forward to hearing any suggestions from you guys. Thank you in advanced!

Versions (please complete the following information):

• OS: [e.g. Ubuntu 18.04.6]
• Thingsboard IoT Gateway version [e.g. 3.4.4]
• Python version[e.g. 3.10]

[github-actions]

Hi @ktran1005.
Thank you for your interest in ThingsBoard IoT Gateway.
Your issue was registered, please wait for response from engineer.

IOTGW-90

[imbeacon]

Hi @ktran1005,

Could you set logging level to DEBUG, instead of INFO, run the gateway and provide service.log, connector.log and converter.log file?

[ktran1005]

Hi @imbeacon, Thanks for your response. By changing the logging level to DEBUG, you mean the log statement that I am having right now in the convert method?

    def convert(self, config, data: bytes):
        global byteBuffer, byteBufferLength

        # Constants
        OBJ_STRUCT_SIZE_BYTES = 12;
        BYTE_VEC_ACC_MAX_SIZE = 2**15;
        MMWDEMO_UART_MSG_DETECTED_POINTS = 1;
        MMWDEMO_UART_MSG_RANGE_PROFILE   = 2;
        maxBufferSize = 2**15;
        tlvHeaderLengthInBytes = 8;
        pointLengthInBytes = 16;
        magicWord = [2, 1, 4, 3, 6, 5, 8, 7]
        
        # Initialize variables
        magicOK = 0 # Checks if magic number has been read
        dataOK = 0 # Checks if the data has been read correctly
        frameNumber = 0
        detObj = {}

        keys = ['attributes', 'telemetry']
        for key in keys:
            self.result_dict[key] = []
            if self.__config.get(key) is not None:
                for config_object in self.__config.get(key):
                    now = int(round(time.time()*1000))
                    converted_data = {config_object['key']: 123, "ts": now, "values": {"test": 123}}
                    self.result_dict[key].append(converted_data)
        log.debug("Converted data: %s", self.result_dict)
        return self.result_dict

and can you suggest where I can monitor the service.log, connecter.log, converter.log? This is my first device to connect to ThingsBoard-gateway so I dont have much experience. Any suggestion would be appreciated!

[ktran1005]

Hi @imbeacon,

I ran the gateway and cat the content from service.log, connector.log and converter.log:

service.log

converter.log

connector.log

[ktran1005]

Hi @imbeacon, can I get any update on this?

[imbeacon]

Hi @ktran1005,

It looks like the port connection wasn't established correctly, because device_serial_port is None, it means that connector wasn't connect to the port, please check that it was opened correctly before.

[ktran1005]

Hi @imbeacon. Thanks for your response. I re-ran the service again and it seems like the connector was able to connect to the port but the error is still there. Do you have other recommendation? I included the latest output of connector.log below

[ktran1005]

Hi @imbeacon, just want to give you an update on this. I tried to ping thingsboard.cloud and I was able to see the value on the thingsboard cloud. However, when I tried to ping the thingsboard edge that I installed from docker and running on my localhost, the data was not there.

[ktran1005]

Hi @imbeacon, I wonder if we can still use port 8080 for the custom connector in this case?

[ktran1005]

Hi @imbeacon, can I get any update on this?

[ktran1005]

Hi @imbeacon, I got it working now so I'll close this ticket. Thanks!