When you have over a 1000 pages of documentation, it becomes a necessity to be able to search the documentation effectively to find relevant content. With this in mind, we’ve recently changed to powering LoopBack documentation search with IBM Watson Discovery instead of Google Custom Search. Read on to learn more about the new search.

Google Custom Search has a few issues – namely ads in search results and lack of context around a search (for example, searching in LoopBack 3 docs will show results for all versions of LoopBack. We looked at several other options to see what might best suit our needs.

Alternatives Considered

Lunr.js

lunr.js is a great indexing library with support for multiple languages. It was explored as an alternative but the size of LoopBack documentation - over 1000 pages - was a concern. The pre-built index was over 9MB, compressed was still over 4MB. This alone was reason enough for us to pass on this option, but there were also concerns raised in the library’s repository stating the site would become unresponsive on mobile devices if index size was over 1MB.

Elasticsearch

Another solution we considered was Elasticsearch, a full-text search engine. It’s one of the most popular solutions to consider when trying to implement full-text search. While this would have been a great solution, it also would’ve required us to maintain the Elasticsearch server. We opted to not pursue this option because of the maintenance efforts - we would rather dedicate our resources to LoopBack Development and Support.

Duck Duck Go is a privacy minded search engine alternative to Google that similarly also provides custom search. A quick test showed it would’ve still served ads and that it also lacked context aware search. As such, we dropped this as an alternative.

Watson Discovery

IBM Watson Discovery allows search queries to be filtered based on some additional metadata (in our case LoopBack version). It also doesn’t serve any ads. Watson Discovery met the criteria for our new search and since last week has been powering the LoopBack Docs search. Try it out for yourself!

Implementation

Implementing a new search presented some challenges such as keeping Watson up-to-date, making results context aware (on a static website), and creating a new search results page and a way to query the search results from Watson without credentials.

Proxy Search Function

Watson Discovery doesn’t provide readonly credentials and as such can’t be called directly from the front end. Instead, it requires the use of a proxy service. We implemented the proxy as a serverless function deployed on IBM Cloud Functions. This offers a simple solution that is easy to maintain, scale and use while hiding the Watson Discovery credentials. The proxy function code can be viewed in the new strongloop/loopback.io-search repository.

You can now search for a word from the home page and the search results will include results from all versions of LoopBack. But if you are in the documentation for a particular version of LoopBack, searching from that page will only show results relevant to that version (version context – set as the sidebar value on each page as a hidden form input).

An example is searching for the word controller:

Auto-Deploy

Whenever changes are merged to the gh-pages branch for loopback.io repository, Travis CI is used to upload the documentation to Watson Discovery into a new collection. Once the collection is ready for use, the older collection is deleted and the search proxy uses the only collection it has available (or the oldest collection if multiple exist). This automation allows the search to always be up-to-date and requires no maintenance effort from the team.

Search Results Page

New loopback.io search results page

The last part of implementing a new search was creating a new search results page as this was previously handled by the Google Custom Search. The new search results page shows 10 results with up to 3 matching sentence excerpts with the matching text bolded. The link for the result page is also shown. The new search results page also makes it possible to share search result page URLs with others. When displaying results, special care had to be taken to ensure results are escaped for any malicious code to prevent XSS attacks as the documentation is open source.

Happy Searching

Try out the new search in the docs and share your feedback with the team for further improving it.

Call for Action

LoopBack’s success counts on you! We appreciate your continuous support and engagement to make LoopBack even better and meaningful for your API creation experience. Please join us and help the project by: