Errors in collector bots with file

[Qazxcvbd]

My intelmq version is 3.1.0.
When I ran collector bot with file, these error show in log:

2023-03-29 09:38:20,445 - collector-openphish - INFO - Bot will continue in 15 seconds.
2023-03-29 09:38:35,461 - collector-openphish - INFO - Processing file '/var/feed/openphish/Openphish.json'.
2023-03-29 09:38:35,461 - collector-openphish - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 319, in start
    self.process()
  File "/usr/lib/python3/dist-packages/intelmq/bots/collectors/file/collector_file.py", line 78, in process
    for report in generate_reports(template, file_handle,
  File "/usr/lib/python3/dist-packages/intelmq/lib/splitreports.py", line 163, in generate_reports
    for chunk in read_delimited_chunks(infile, chunk_size):
  File "/usr/lib/python3/dist-packages/intelmq/lib/splitreports.py", line 105, in read_delimited_chunks
    new_chunk = infile.read(chunk_size)
TypeError: argument should be integer or None, not 'str'
2023-03-29 09:38:35,462 - collector-openphish - INFO - Bot will continue in 15 seconds. 

This problem just spesific to the situation when I run openfish collector bot.

[Qazxcvbd]

Besides, the events.txt in /var/lib/intelmq/bots/file-output coudn't be successfully imported into the Kibana system.
My Kibana system is working.

[gethvi]

Can you please share your collector-openphish bot configuration and a snippet of the file you are trying to process?

[sebix]

Looks like the bot parameter chunk_size is a string instead of an integer

[Qazxcvbd]

My collector-openphish bot configuration :

The Original feed format look like this:
[{"url": "https://XXX", "brand": "X", "ip": "X.X.X.X", "asn": "X", "asn_name": "X", "country_code": "X", "country_name": "X", "tld": "X", "discover_time": "X", "family_id": "X", "host": "X", "isotime": "X", "page_language": X, "ssl_cert_issued_by": "X", "ssl_cert_issued_to": "X", "ssl_cert_serial": "X", "is_spear": X, "sector": ""}, {"url": "https://XXX", "brand": "X", "ip": "X.X.X.X", "asn": "X", "asn_name": "X", "country_code": "X", "country_name": "X", "tld": "X", "discover_time": "X", "family_id": "X", "host": "X", "isotime": "X", "page_language": X, "ssl_cert_issued_by": "X", "ssl_cert_issued_to": "X", "ssl_cert_serial": "X", "is_spear": X, "sector": ""}]

The file I modified and moved to collector feed's directory :
{"url": "https://XXX", "brand": "X", "ip": "X.X.X.X", "asn": "X", "asn_name": "X", "country_code": "X", "country_name": "X", "tld": "X", "discover_time": "X", "family_id": "X", "host": "X", "isotime": "X", "page_language": X, "ssl_cert_issued_by": "X", "ssl_cert_issued_to": "X", "ssl_cert_serial": "X", "is_spear": X, "sector": ""}
{"url": "https://XXX", "brand": "X", "ip": "X.X.X.X", "asn": "X", "asn_name": "X", "country_code": "X", "country_name": "X", "tld": "X", "discover_time": "X", "family_id": "X", "host": "X", "isotime": "X", "page_language": X, "ssl_cert_issued_by": "X", "ssl_cert_issued_to": "X", "ssl_cert_serial": "X", "is_spear": X, "sector": ""}

[Qazxcvbd]

@sebix

Looks like the bot parameter chunk_size is a string instead of an integer

Yes, chunk_size looks to be the cause of this error.

My previous version of IntelMq was 2.2.1 and the file I modified was worked in version 2.2.1, so I don't know how to solve this problem.

[gethvi]

As a workaround you can try to set the chunk_size to some very high number like 1073741824 (=1GB, should be enough). The number needs to be higher than the size of any individual file you are trying to process.

[Qazxcvbd]

@gethvi Thanks, it's works !!

Besides, my output file still couldn't import in my kibana system.

[Qazxcvbd]

After running the output bot, an events.txt file will be generated in var/lib/intelmq/bots/file-output/.
This file will automatically become events.txt.1 the next day, how can I set the file as events.txt-${ELK_INDEX_DATE}

[gethvi]

If you are trying to automatically import IntelMQ events to Elasticsearch and you are also using Logstash I suggest to follow this guide instead of using File Output Bot (I wrote it and I am still using it this way so I know it works):
https://intelmq.readthedocs.io/en/develop/user/ELK-Stack.html

If you are not using Logstash you can try to use Elasticsearch Output Bot:
https://intelmq.readthedocs.io/en/develop/user/bots.html#elasticsearch-output-bot

If you really want to use the File Output, share it's configuration please.

[sebix]

The reason for the wrong parameter is probably certtools/intelmq-manager#294

[Qazxcvbd]

Thanks! The Output file is successfully imported in Kibana !!

[sebix]

Thanks for the feedback!

I'm closing this issue in favor of certtools/intelmq-manager#294