v2.5.0

![Swirl Metasearch Logo](https://raw.githubusercontent.com/wiki/swirlai/swirl-search/images/swirl-metasearch-logo.png)

Team Swirl is announcing General Availability of Swirl 2.5! 

The theme for this release was performance. Configured with 12 SearchProviders, Swirl 2.5 supports up to 15 queries/second on a Standard F16s v2 server (16 vcpus, 32 GiB memory) with a median response time of 3 seconds.  

### PLEASE STAR OUR REPO: [https://github.com/swirlai/swirl-search](https://github.com/swirlai/swirl-search)

<br/>

# New Features

## Performance Improvements

:small_blue_diamond: Swirl's asynchronous back-end was refactored to use the `group` method and replaced RabbitMQ with Redis

:small_blue_diamond:  Relevancy processing was split into two stages to improve performance
* The revised `CosineRelevancyPostResultProcessor` must be in the `Search.post_result_processors` list.
* Please review the JSON in the `SearchProviders/` directory and update existing configurations to match. 

## HubSpot SearchProviders

:small_blue_diamond:  Search Company, Contact, and Deal records with new token-based HubSpot SearchProviders
* HubSpot requires creation of a "Private Apps" API token by the Super Admin with [these scopes](https://raw.githubusercontent.com/wiki/swirlai/swirl-search/images/HubSpot-scopes.png)
* In `result_mappings`, the `url` mapping must contain the HubSpot Org ID

# Improvements

:small_blue_diamond:  The `DateFindingResultProcessor` was added to the Google PSE SearchProvider JSON. It finds a date in a large percentage of results that otherwise wouldn't have one, and copies the date to the `date_published` field.  Existing PSE SearchProvider configurations should be updated as follows:

```
"result_processors": [
            "MappingResultProcessor",
            "DateFinderResultProcessor",
            "CosineRelevancyResultProcessor"
        ],
```

:small_blue_diamond: Swirl's control script (`python swirl.py`) was updated

* A `default` key was added to the services list, including `debug`, in `swirl/services.py`
* The `celery-beats` service was set to `default: False`
* `swirl.py` was modified to load only services with `default: True`

To use the subscribe or expiration features, `celery-beats` must be started manually:
`python swirl.py start celery-beats`

`Celery-beats` must also be stopped separately. 

:small_blue_diamond:  Swirl now outputs a single log entry with each request at the default log level INFO

```
2023-08-02 10:49:09,466 INFO     admin search 452 FULL_RESULTS_READY 32 2.2
```

Detailed logging is available in Debug mode: restart Swirl with the `--debug` flag to enable

:small_blue_diamond:  The `install.sh` script was refactored to better handle spaCy upgrades

:small_blue_diamond:  Swirl's Docker image was updated to use the latest, stable release of Python, version 3.11.4

:small_blue_diamond:  Use of the unsafe `eval()` function was removed


# Resolved Issues

:small_blue_diamond:  Swirl's log files now persist and append new entries across restarts

:small_blue_diamond:  The SearchProvider configurations for Elastic, OpenSearch, and Microsoft Outlook Messages were updated

:small_blue_diamond:  A seesion-related issue in the connectors for Sqlite3, PostgreSQL, and Elastic was fixed

:small_blue_diamond:  Errors in the `DedupeByFieldResultProcessor` were addressed

:small_blue_diamond:  The SearchProvider `results_per_query` value is handled correctly now

:small_blue_diamond:  An issue where some searches didn't appear to reach the Swirl backend was fixed

:small_blue_diamond:  Unnecessary WARNING messages were removed from the logs

:small_blue_diamond: Result URLs on Search objects are now constructed correctly using the Request instead of loading from django settings


# Known Issues

:small_blue_diamond: Clicking a Microsoft Teams result sometimes results in this error from the MS Teams app:
```
We cannot take you to that message because it's in a chat you're not in.
```
For Teams links to resolve correctly, you must have the Microsoft Teams app open *before* clicking the result link.

:small_blue_diamond: Subscribed searches are not returning results for Microsoft sources; this will be addressed in an upcoming release.

:small_blue_diamond: [Creating searches from a browser with q=](https://github.com/swirlai/swirl-search/wiki/5.-Developer-Guide#create-a-search-object-with-the-q-url-parameter) can sometimes create two Search objects. 

This is because of browser prefetch AKA predictive service. [Turn off Chrome prediction service](https://www.ghacks.net/2019/04/23/missing-chromes-use-a-prediction-service-setting/). [Turn off Safari prefetch](https://stackoverflow.com/questions/29214246/how-to-turn-off-safaris-prefetch-feature).

Please report any issues with this to [support](#support).


# Upgrading

:warning:  Version 2.5 requires database migration. See [Upgrading Swirl](https://github.com/swirlai/swirl-search/wiki/3.-Admin-Guide#upgrading-swirl).

:warning: Version 2.5 replaces RabbitMQ with Redis.  Swirl's latest Docker image already contains this change.  See the [Quick Start - Local Installation](https://github.com/swirlai/swirl-search/wiki/1.-Quick-Start#local-installation) section for details on how to install Redis with Swirl on MacOS or Linux.

:warning: The revised `CosineRelevancyResultProcessor` must be added *last* to the `result_processors` block of all  SearchProvider configurations *except* ChatGPT.  For example:
```
"result_processors": [
            "MappingResultProcessor",
            "LenLimitingResultProcessor",
            "CosineRelevancyResultProcessor"
        ],
```

This following of Error message in the Swirl logs indicates that one or more SearchProviders have not been updated:
```
INFO     search.py: invoking processor: CosineRelevancyPostResultProcessor
2023-07-31 16:31:39,268 ERROR    CosineRelevancyPostResultProcessor_2051: Error: Dictionary of result lengths is empty. Was CosineRelevancyResultProcessor included in Search Providers Processor configuration?
```

:warning: Exisitng SearchProvider configurations for Elastic, OpenSearch, and Microsoft Outlook Messages shoudl be updated:
* Elastic: Remove `hosts=` and `http-auth=` (just that text, not the values that follow)
* OpenSearch:  Update the `query_template` syntax to the following:
```
"query_template": "{\"query\":{\"query_string\":{\"query\":\"{query_string}\",\"default_field\":\"{default_field}\",\"default_operator\":\"and\"}}}",
```
* Microsoft Outlook Messages:  Update both `result_grouping_field` and `result_processors` to the following:
```
"result_grouping_field": "conversationId",
"result_processors": [
            "MappingResultProcessor",
            "DedupeByFieldResultProcessor",
	    "CosineRelevancyResultProcessor"
],
```

# Documentation

[Overview](https://docs.swirl.today/) | [Quick Start](https://docs.swirl.today/1.-Quick-Start) | [User Guide](https://docs.swirl.today/2.-User-Guide) | [Admin Guide](https://docs.swirl.today/3.-Admin-Guide) | [M365 Guide](https://docs.swirl.today/4.-M365-Guide) | [Developer Guide](https://docs.swirl.today/5.-Developer-Guide) | [Developer Reference](https://docs.swirl.today/6.-Developer-Reference)
_(updated links on 13-October-2023)_

# Support

:small_blue_diamond: [Join the Swirl Metasearch Community on Slack!](https://join.slack.com/t/swirlmetasearch/shared_invite/zt-1qk7q02eo-kpqFAbiZJGOdqgYVvR1sfw)

:small_blue_diamond: Email: [support@swirl.today](mailto:support@swirl.today) with issues, requests, questions, etc - we'd love to hear from you!

<br/>