Custom AL services

[kam193]

Hey!

As I'm maintaining my custom services for quite a few months already, I think it's time to share them a little broader - although some of you may have already seen them.

Generally, they are made for my custom use case, but I believe there are a few that may be generally useful at the moment. So, if you look to extend your AL instance, here are services I'd generally recommend:

• ASARExtractor - unpacks Electron archives, rather stable, but not battle-tested;
• ClamAV - ClamAV in the daemon mode, works very efficient. I recommend keeping one instance running all the time to avoid unnecessary re-initialising the daemon. The default configuration provides some examples of additional DB sources.
• Simple-Downloader lets you download files when you don't need the whole Chromium, can also extract URLs from some directory listings to let you download a whole open directory automatically. Although - no proxy configuration presently, but it could be easily extended.
• OOPreview - the newest one. It can be a nice support if you work with MS Office files, as it generates better previews than the Document Preview service (you can check what you get for the example file from OnlyOffice website). However, this is only the visual preview, no OCR or other heuristics - it's not intended to be the replacement for Document Preview.

There are also some services, that may be useful in some use cases:

• File Similarity is useful if you expect very similar malicious files to come back in scans, can compare files against TLSH hashes from the badlist or other sources, and mark files with a given similarity score.
• Hashlookup is especially useful if you do not have resources to keep the NSRL database (and more sources) in the local safelist. The downside: hashes are sent to an external service.
• Network Information is a simple and still a little buggy service to get information about IPs (offline MaxMind DBs) and domains (online WHOIS queries, so not always available unfortunately), you can also mark newly created domains if the creation date is available.
• PCAP Extractor - a service to extract data and files from PCAP files, also when TLS keys are included (in my experience, Suricata service wasn't able to extract files from them), and leverage Safelist to limit extracted data. Unfortunately, it has bugs and I plan to test other backends when I find time.
• Python-Magic is very specific for my current use case, but may be useful if you need to deal with Pyinstaller files or byte-compiled Python - it uses different tools than the Extract service. Especially, it can work on PYC files from newer Python versions, although - most often generates only partial or a little wrong code.

...and a few more, and when I find a new need, then probably come a new one :)

Feel free to test them if they match your use case. I'm happy for any feedback, but to be clear - I maintain them in my spare time only.

[cccs-rs]

@kam193 have you noticed any performance benefits of using OOPreview compared to DocumentPreview (namely OnlyOffice's DocBuilder vs unoconv via LibreOffice)?

[kam193]

Unfortunately, I don't process documents often enough to say if there is a real performance difference :/ OOPreview needs less resources to generate an image, but also note that I don't do the OCR processing here at the moment. There is definitely a difference in quality of the file representation in the preview

[cccs-rs]

Hmm... that might be something I can test side-by-side just from the image rendering perspective.

I notice for unoconv, it always has to start the listener to interact with LibreOffice's API and I believe it slows down the service when run at scale (not to mention I also have to install LibreOffice so the image size isn't lightweight). From the looks of OnlyOffice's DocBuilder and the OOPreview service, it doesn't require the actual editor which should make the the image smaller (and hopefully more performant).

[kam193]

You can try to install the service, it should actually give you the answer 😄

[kam193]

Out of curiosity, how significant improvement have you noticed when switching to OO?

[cccs-rs]

Well with comparing a test on our staging system vs Libre version running in production, I noticed that with OO we can achieve the same throughput with fewer containers. Which I'll take it as a win because we can allocate those resources elsewhere 😁

I would probably attribute that to the fact we don't have to start up LibreOffice (or the unoconv listener) every time a task comes in anymore.

[gdesmar]

I think those services would fit great in the community services page and community.json feed. :)

[cccs-rs]

That's definitely something we should look at automating (namely the services in the community.json are also documented in the community services page)