Swirl Metasearch 2.1

Team Swirl is announcing General Availability of Swirl Metasearch 2.1!

This version features the new Galaxy User Interface with Dark Mode, new GitHub SearchProviders, support for custom request headers, a new DateFinderResultProcessor that finds dates in the body field and copies them to date_published, plus new Result Processors for trimming and cleaning long responses. 2.1 also improves handling of recurring Microsoft 365 Calendar events and Outlook Message threads, showing only the most recent.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 Introducing the Galaxy UI:

🔹 Search public repositories with token-based GitHub SearchProviders for Code, Commits, Pull Requests, and Issues

🔹 Micrsoft 365 Calendar Events pre-configured SearchProvider with automatic hiding of recurring meetings

🔹 New date_published_display option available in SearchProvider result_mappings
You can now specify different values for date_published and date_published_display in SearchProvider configurations:

"result_mappings":  ... date_published=foo.bar.date1,date_published_display=foo.bar.date2 ...

Swirl Result objects will reflect both date values:

"date_published": "2010-01-01 00:00:00",
"date_published_display": "c2010",

And, the new Galaxy UI displays date_published_display while date-sorting by date_published:

🔹 Pass custom HTTP Header values in SearchProviders
The optional http_request_headers field is available to all SearchProviders for passing custom HTTP Header values to the source alongside the user's query. The new GitHub SearchProviders use this feature to return a more accurate result snippet from GitHub that is then mapped to Swirl's body field:

"http_request_headers": {
            "Accept": "application/vnd.github.text-match+json"
        },

"result_mappings": "title=name,body=text_matches[*].fragment, ...

🔹 New Result Processor: LenLimitingResultProcessor
This processor checks to see if the title and body responses from a source exceed a configurable length, truncates anything after that value, and adds an elipsis ("..."). If the body field has been truncated, the processor reports the entire response in a new body_full field in the Payload. The default max_length value for this processor is set in swirl_server/settings.py: SWIRL_MAX_FIELD_LEN = 512 and can be overridden for a specific SearchProvider using a new Tag value (max_length:###):

"tags": [
        "Articles",
        "Wiki",
        "YouTrack",
        "max_length:256"
    ]

The processor reports on the number of items updated as expected. It should be considered for sources that consistently return lengthy title or body fields and should follow the MappingResultProcessor. The default SearchProvider configurations for Atlassian Jira and Confluence have been updated with this processor.

"result_processors": [
            "MappingResultProcessor",
            "LenLimitingResultProcessor"
        ],

🔹 New Result Processor: CleanTextResultProcessor
This processor removes non-alphanumeric characters from the source response. It should be considered for lengthy responses where URLs or other HTML or Markdown syntax appear in results. It should be installed before the LenLimitingResultProcessor when both are used. The default SearchProvider configurations for JetBrains YouTrack have been updated with these processors.

"result_processors": [
            "MappingResultProcessor",
            "CleanTextResultProcessor",
            "LenLimitingResultProcessor"
        ],

🔹 New Result Processor: DateFinderResultProcessor
This processor looks for a date in any of the following forms in the body field of each result item. Should it find one, and the date_published for that item is 'unknown', it replaces date_published with the date extracted from the body, and notes this in the result.messages. This processor can detect the following date formats:

06/01/23
06/01/2023
06-01-23
06-01-2023
jun 1, 2023
june 1, 2023

Improvements

🔹 Updates to the genericRequestsPost connector and example SearchProvider JSON

🔹 Refined handling of threaded email messages in Outlook Messages results

🔹 Updated sorting of the search page Source dropdown and the result page Source facet in the Galaxy UI

🔹 Adjusted default Relevancy weight values to improve the relevancy ranking of shorter results

Resolved Issues

🔹 Re-running the previous search is now supported correctly in the Galaxy UI

🔹 An issue with Microsoft Authorization logout has been fixed

🔹 Clearing the search form now clears the previous Sort ("View By") selection correctly

🔹 Result Mixer choices are no long retained across searches

🔹 Result list numbering now displays correctly if Swirl's url field is not mapped

🔹 Various isssues with SearchProvider Tag searching were addressed

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.1 requires database migration. See Upgrading Swirl.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!