MeiliSearch for all your searching needs…

A year ago, I developed KgsApp as a dedicated platform for my family and released it on the Google Play Store. However, one critical feature was missing – a comprehensive text search that was both typo tolerant and efficient. Despite using Google Cloud Firestore as the app’s backend, its query capabilities had limitations when it came to implementing a fully-fledged, typo-tolerant search. Consequently, the search bar on the news/articles page could only perform basic searches limited to article titles, leading to dissatisfaction among users.

The Challenge: Limitations of the Basic Search

Users who utilized the search bar frequently found themselves unable to obtain the desired results. Unbeknownst to them, the search scope was confined to titles, and precise keyword matching was necessary for successful searches. Consequently, searching for a YouTube link within an article proved futile, as it was not included in any of the news/article titles.

The solution

Meilisearch is a RESTful search API. It aims to be a ready-to-go solution for everyone who wants a fast and relevant search experience for their end-users.

It’s blazing fast (response within 50 milliseconds). Priority is given to fast answers for a smooth search experience. It supported Search as you type, results are updated on each keystroke and the best part of it is its Typo tolerance – Understands typos and misspellings.

Interestingly, I wanted to host it on the raspberry pi because, well, it is fun! 😊

Installation

Installing and running Meilisearch is easy and straightforward. Let’s use the official script that will carry out the installation process. It will copy a binary of Meilisearch to your machine and enable you to use it immediately.

Once you are logged in to your machine via SSH, ensure your system and its dependencies are up-to-date (*apt update for Debian*) before proceeding with the installation.

$ curl -L https://install.meilisearch.com | sh

If you do not trust the script, here’s the script so you can have a look at it.

$ mv ./meilisearch /usr/bin/

I wanted to run it on a screen hence my Linux service looks like this:

[Unit]
Description=Meilisearch
After=systemd-user-sessions.service

[Service]
Type=forking
EnvironmentFile=/etc/env/meilienv.conf
ExecStart=/usr/bin/screen -dmS meiliSession /usr/bin/meilisearch

and the meilienv.conf:

MEILI_DB_PATH="/home/myuser/meilisearch/db/kgsapp"
MEILI_ENV="production"
MEILI_HTTP_ADDR="127.0.0.1:9001"
MEILI_NO_ANALYTICS="true"
MEILI_MASTER_KEY="anEf**NS&*srongPassPh4as3"

Note that by default, Meilisearch’s API is unprotected. This means all routes are publicly accessible and require no authorization to access. However, to run it on production, the configuration is not ideal and MEILI_MASTER_KEY is a must. This means you will need an API key to access endpoints such as POST /search and GET /documents.

After an instance is secured, only the GET /health endpoint will be publicly available. To access any other API endpoint, you must add a security key with suitable permissions to your request.

When you launch an instance for the first time, Meilisearch will automatically generate two API keys: Default Search API Key and Default Admin API Key.

As its name indicates, the Default Search API Key can only be used to access the search route. i.e, (POST /indexes/{index_uid}/search)

The second automatically-generated key, Default Admin API Key, has access to all API routes except /keys. You should avoid exposing the default admin key in publicly accessible code.

Both default API keys have access to all indexes in an instance and do not have an expiry date. They can be retrieved, modified, or deleted the same way as user-created keys. I have deleted my default keys and created new ones with a specific custom scope (news documents in my case)

E.g:

$ curl \
  -X POST 'http://127.0.0.1:9001/keys' \
  -H 'Authorization: Bearer masterKey' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "description": "Add news documents: News PI key",
    "actions": [
      "documents.add"
    ],
    "indexes": [
    "news"
    ],
    "expiresAt": "2023-02-02T00:00:00Z"
  }'

Running on Production

Here’s a sample reverse proxy setup on Nginx so that the clients on the Internet can get in touch with my Meili Instance. This assumes that you have set up the DNS etc without issues and have the SSL certificates. I have not detailed it here because it is out of scope but here’s a brief idea.

server {
    server_name search.kgs.in;
    location / {
        proxy_pass  http://127.0.0.1:9001;
    }

    listen [::]:443 ssl; 
    listen 443 ssl; 
    ssl_certificate /etc/././fullchain.pem;
    ssl_certificate_key /etc/././privkey.pem;
    include /etc/.//options-ssl-nginx.conf;
    ssl_dhparam /etc/./ssl-dhparams.pem;

}

Add existing news to Meili

python snippet:

search= 'https://search.kgs.in/'
c = 0
def addToMeili():
    news_ref = firestoredb.collection(u'NewsAndArticles')
    try:
        news_docs = news_ref.stream()
        news_doc_list = []
        news_single_dict = {}
        for doc in news_docs:
            c = c + 1
            uuid_genera = str(uuid.uuid4())
            news_single_dict = doc.to_dict()
            news_single_dict = {k: v for k, v in news_single_dict.items() if k in [
                'author', 'desc', 'title', 'pic', 'category']}
            news_single_dict.update({'newsidentifier': uuid_genera})
            news_single_dict.update({'newsidonfirestore': doc.id})
            news_doc_list.append(news_single_dict)
        print(f"Found {c} number of news docs on Firestore.")
        url = f'{search}/indexes/news/documents?PrimaryKey=newsidentifier'
        headers = {"Authorization": f"Bearer {api_key}"}
        response = requests.post(url, headers=headers,
                                 json=news_doc_list, timeout=(10, 150))
        print("Status Code", response.status_code)
        response_data = response.json()
        uid = response_data['uid']
        added = getUidStatus(uid)
        if added:
            return ("Documents added to Meili")
    except Exception as e:
        return (f"Exception {e}")

Communicating with an instance

Using CURL:

$ curl -X POST 'http://search.kgs.in/indexes/news/search' \
  -H 'Authorization: Bearer apiKey

Dart code from the android app:

Future<MeiliSearchResults> fetchMeiliSearchResults(String query) async {
  final response = await http.post(
    Uri.parse('https://search.kgs.in/indexes/news/search'),
    headers: <String, String>{
      'Authorization':
          'Bearer $newsSearchApiKey',
      'Content-Type': 'application/json; charset=utf-8'
    },
    body: jsonEncode(<String, dynamic>{'q': query, 'limit': 20}),
  );

  if (response.statusCode == 200) {
    print(response.body);
    return MeiliSearchResults.fromJson(
        json.decode(utf8.decode(response.bodyBytes)));
  } else {
    throw Exception('Failed to load MeiliSearchResults');
    
  }
}

MeiliSearchResults:

import 'package:meta/meta.dart';
import 'dart:convert';

MeiliSearchResults meiliSearchResultsFromJson(String str) => MeiliSearchResults.fromJson(json.decode(str));

String meiliSearchResultsToJson(MeiliSearchResults data) => json.encode(data.toJson());

class MeiliSearchResults {
  MeiliSearchResults({
    required this.hits,
    required this.nbHits,
    required this.exhaustiveNbHits,
    required this.query,
    required this.limit,
    required this.offset,
    required this.processingTimeMs,
  });

  List<Hit> hits;
  int nbHits;
  bool exhaustiveNbHits;
  String query;
  int limit;
  int offset;
  int processingTimeMs;

  factory MeiliSearchResults.fromJson(Map<String, dynamic> json) => MeiliSearchResults(
    hits: List<Hit>.from(json["hits"].map((x) => Hit.fromJson(x))),
    nbHits: json["nbHits"],
    exhaustiveNbHits: json["exhaustiveNbHits"],
    query: json["query"],
    limit: json["limit"],
    offset: json["offset"],
    processingTimeMs: json["processingTimeMs"],
  );

  Map<String, dynamic> toJson() => {
    "hits": List<dynamic>.from(hits.map((x) => x.toJson())),
    "nbHits": nbHits,
    "exhaustiveNbHits": exhaustiveNbHits,
    "query": query,
    "limit": limit,
    "offset": offset,
    "processingTimeMs": processingTimeMs,
  };
}

class Hit {
  Hit({
    required this.author,
    required this.shortDesc,
    required this.title,
    required this.desc,
    required this.newsidentifier,
    required this.newsidonfirestore,
    required this.authorPic,
    required this.category,
    required this.imageUrl,
  });

  String author;
  String shortDesc;
  String title;
  String desc;
  String newsidentifier;
  String newsidonfirestore;
  String authorPic;
  String category;
  String imageUrl;

  factory Hit.fromJson(Map<String, dynamic> json) => Hit(
    author: json["author"],
    shortDesc: json["shortDesc"],
    title: json["title"],
    desc: json["desc"],
    newsidentifier: json["newsidentifier"],
    newsidonfirestore: json["newsidonfirestore"],
    authorPic: json["authorPic"],
    category: json["category"],
    imageUrl: json["imageUrl"],
  );

  Map<String, dynamic> toJson() => {
    "author": author,
    "shortDesc": shortDesc,
    "title": title,
    "desc": desc,
    "newsidentifier": newsidentifier,
    "newsidonfirestore": newsidonfirestore,
    "authorPic": authorPic,
    "category": category,
    "imageUrl": imageUrl,
  };
}

This may not be the best of solutions (ahem load balancing ahem )but for now, on the UI, I have a fallback mechanism to firebase if my raspberry pi dies and Meili is unreachable:

Container(child: Obx(() => meiliSearchAvailable.value
?      FutureBuilder<MeiliSearchResults>(...
:      StreamBuilder<QuerySnapshot>(...)

Demo