v3.2.0

![Swirl Logo](https://raw.githubusercontent.com/swirlai/swirl-search/main/docs/images/transparent_header_3.png)

## Team Swirl is excited to announce General Availability of Swirl 3.2!
This release includes the first version of the `AutomaticPayloadMapperResultProcessor`, `RequireQueryStringInTitleResultProcessor`, `DropIrrelevantPostResultProcessor` and `NoModQueryProcessor`, along with new SearchProviders for [OpenSanctions.org](https://www.opensanctions.org/) and [LittleSis.org](https://littlesis.org/). We've also added support for running OpenAI in Azure, storing query templates in JSON, plus updated Python, Django, and OpenAI API versions, and resolved issues found in previous releases. Finally, the Galaxy UI has been updated to :star: results above a particular relevancy score.

We're seeking anyone interested in testing new Connectors to Collibra, Thoughtspot and Pinecone DB. Please [contact Swirl via email](mailto:hello@swirl.today) or [Slack](https://join.slack.com/t/swirlmetasearch/shared_invite/zt-1qk7q02eo-kpqFAbiZJGOdqgYVvR1sfw) for access.

### PLEASE STAR OUR REPO: [https://github.com/swirlai/swirl-search](https://github.com/swirlai/swirl-search)

<br/>

# New Features

:small_blue_diamond: New `AutomaticPayloadMapperResultProcessor` which profiles response data to find good strings for Swirl's `title`, `body`, and `date_published` fields. It is intended for SearchProviders that would otherwise have few (or no) good `result_mappings` options.  It should be place after the `MappingResultProcessor`, and the `result_mappings` field should be blank.
```
 "result_processors": [
    "MappingResultProcessor",
    "AutomaticPayloadMapperResultProcessor",
    "CosineRelevancyResultProcessor"
],
...
"result_mappings": "",
```
![Big Query results](https://raw.githubusercontent.com/swirlai/swirl-search/main/docs/images/3_2_0-BigQuery.png)

:small_blue_diamond: New `NoModQueryProcessor` that only removes leading SearchProvider Tags and does not modify the query terms in any way. It is intended for repositories that allow non-search characters (such as brackets).
```
"query_processors": [
    "NoModQueryProcessor"
],
```

:small_blue_diamond: New `RequireQueryStringInTitleResultProcessor` that drops results that do not contain the `query_string_to_provider` in the result `title` field.  It should be added after the `MappingResultProcessor` and is now included by default in the "LinkedIn - Google PSE" SearchProvider.
```
"result_processors": [
    "MappingResultProcessor",
    "RequireQueryStringInTitleResultProcessor",
    "DateFinderResultProcessor",
    "CosineRelevancyResultProcessor"
],
```

:small_blue_diamond: New `DropIrrelevantPostResultProcessor`  drops results with `swirl_score < settings.MIN_SWIRL_SCORE` (set to `500` by default) and results with no `swirl_score`.
:warning: The Galaxy UI will not display the correct number of results if this ResultProcessor is deployed.  This will be addressed in a future release.

:small_blue_diamond:  Swirl now supports using Azure OpenAI models for RAG processing in addition to OpenAI's ChatGPT models.
```
2024-01-29 16:36:07,335 INFO     get model AZUREAI gpt-35-turbo
2024-01-29 16:36:10,542 INFO     HTTP Request: POST https://swirltest-openai.openai.azure.com/openai/deployments/gpt-35-turbo/chat/completions?api-version=2023-10-01-preview "HTTP/1.1 200 OK"
```

:small_blue_diamond: The Galaxy UI now highlghts results with a `swirl_score` above a configurable threshold with a star in the results list.  The `swirl_score` configuration is available in `theminimumSwirlScore` entry of `static/api/config/default`, and the default is `100`.
![Galaxy UI Stars](https://raw.githubusercontent.com/swirlai/swirl-search/main/docs/images/3_2_0-Galaxy-star.png)

:small_blue_diamond: New `SearchProvider.query_template_json` field which stores the `query_template` as JSON.  The MongoDB SearchProvider has been updated to use this feature.
```
"query_template_json": {
    "$text": {
        "$search": "{query_string}"
    }
},
```

:small_blue_diamond: Support for lowercasing of URLs is now available via a new `LC_URL` option in SearchProvider `result_mappings`.
```
"result_mappings": "title=resource.name,body=resource.description, ... LC_URL,NO_PAYLOAD",
```

:small_blue_diamond: Added support for list-of-list responses from source APIs, where the first list element is the column names.

## New SearchProviders

:small_blue_diamond:  Search the LittleSis free database of "who-knows-who at the heights of business and government" with the new LittleSis.org SearchProvider.
![LittleSis.org](https://raw.githubusercontent.com/swirlai/swirl-search/main/docs/images/3_2_0-LittleSis.png)

:small_blue_diamond:  Find sanctions targets and persons of interest with the new OpenSanctions.org SearchProvider.
![OpenSanctions.org](https://raw.githubusercontent.com/swirlai/swirl-search/main/docs/images/3_2_0-OpenSanctions.png)

:key: Note: Additional SearchProviders contributed by the Swirl Community may be found in the `SearchProviders/untested` directory.

# Improvements

:small_blue_diamond: Lists of SearchProviders can be added to the endpoint with copy/paste, and Swirl will load them all.

:small_blue_diamond: When signing in to Swirl via Microsoft OIDC, users are now automatically authorized to the M365 SearchProviders.

:small_blue_diamond: Swirl was validated on Python 3.12.1.

:small_blue_diamond: Swirl now runs on the new Django 5.0.x series Python packages.

:small_blue_diamond: Swirl now uses the latest 1.6.x OpenAI Python API.

:small_blue_diamond: The `QueryProcessors` base classes were updated.

:small_blue_diamond: The default AI Summary timeout value can now be overridden with a URL parameter in the Galaxy UI.  For example: http://localhost:8000/galaxy/?q=gig%20economics&rag=true&rag_timeout=90000

:small_blue_diamond: The Snowflake SearchProvider `query_template` was revised.

:small_blue_diamond: The BigQuery SearchProvider no longer dedupes results.

:small_blue_diamond: Improved zero results feedback and logfile messages.

# Resolved Issues

:small_blue_diamond: Removed duplicate `static/api/config/default` configuration file.

:small_blue_diamond: Removed unused timeout settings (`SWIRL_Q_WAIT` and `SWIRL_RERUN_WAIT`).

:small_blue_diamond: Fixed Elasticsearch authorization and addressed a certificate error.

:small_blue_diamond: Updated the `index_email_elastic.py` script and related documentation.

:small_blue_diamond: Stopwords are no longer highlighted in search results or AI Summaries.

:small_blue_diamond: Various small fixes to Galaxy UI behavior.

# Known Issues

:small_blue_diamond: Clicking a Microsoft Teams result sometimes results in this error from the Teams app.
```
We cannot take you to that message because it's in a chat you're not in.
```
For Teams links to resolve correctly, you must have the Microsoft Teams app open *before* clicking the result link.

:small_blue_diamond: [Creating searches from a browser with q=](https://docs.swirl.today/Developer-Guide.html#create-a-search-object-with-the-q-url-parameter) can sometimes create two Search objects. 

This is because of browser prefetch AKA predictive service. [Turn off Chrome prediction service](https://www.ghacks.net/2019/04/23/missing-chromes-use-a-prediction-service-setting/). [Turn off Safari prefetch](https://stackoverflow.com/questions/29214246/how-to-turn-off-safaris-prefetch-feature).

Please report any issues with this to [support](#support).

# Upgrading

:warning: Version 3.2.0 requires database migration. See [Upgrading Swirl](https://docs.swirl.today/Admin-Guide.html#upgrading-swirl).

# Documentation

[Overview](https://docs.swirl.today/) | [Quick Start](https://docs.swirl.today/Quick-Start) | [User Guide](https://docs.swirl.today/User-Guide) | [Admin Guide](https://docs.swirl.today/Admin-Guide) | [M365 Guide](https://docs.swirl.today/M365-Guide) | [Developer Guide](https://docs.swirl.today/Developer-Guide) | [Developer Reference](https://docs.swirl.today/Developer-Reference) | [AI Guide](https://docs.swirl.today/AI-Guide) | [Azure Marketplace Guide](https://docs.swirl.today/Azure)

# Support

:small_blue_diamond: [Join the Swirl Community on Slack!](https://join.slack.com/t/swirlmetasearch/shared_invite/zt-1qk7q02eo-kpqFAbiZJGOdqgYVvR1sfw)

:small_blue_diamond: Email [support@swirl.today](mailto:support@swirl.today) with issues, requests, questions, etc - we'd love to hear from you!