Over the last two years, I’ve had the pleasure of doing and re-doing our search at work 4 times. Search and information retrieval is a deep and complex field. Nevertheless, I feel like it should be easy to add a simple search to a rails website.
Similar to a lot of Rails source, we already have something working in production @ Ollie Order. However, it can be a lot better if it was deeply integrated into rails. The recent changes that @eileencodes made to support multiple database can be key to paving support for more advanced search solution like ElasticSearch, Algolia, etc. Baking a lot of things right into ActiveRecord would be ideal.
For the first version, I’d like to focus on keeping it simple using a single database for data & searching. Most databases these days support some form of full-text search. Similar to ActiveRecord it will support the ability to use any search engine you want. Unlike ActiveRecord, it will allow users to choose the search engine per view. Different views of the same data need different search capabilities and performance (e.g. /admin/posts vs /posts will get different traffic).
I’ll be putting together all these gems in a repo and a rails guide document to experiment with what this new API might look like.
Ask
Looking to hear how others approach & solve this problem in Rails. I’m also looking for a few individuals who are equally passionate and want to help solve this problem.
IMHO search is really specific to application and it is hard to bring universal API. But what could be good start is to add optional simple search functionality to generated scaffold with minimal dependency (no additional gem used) using just AR (and maybe a little of AREL if needed).
Thanks for the reply . Haven’t tried Textacular, thought it was older and not really maintained anymore compared to pg_search .
Search is indeed very application specific and thus needs to be easy to configure. Thus, I believe a good set of defaults will get us far. Convention over configuration.
Both Elastic and Algolia have made great strides in making search far simpler. The complexity using this services come from having a separate data-store and replicating you data. Another issue with Algolia in particular is that pricing is based on an e-commerce per transaction model. This makes it less viable for back-office interfaces. However, it is my belief that back-office employees should get the same if not better UX than front-facing customer things. After all, they use it everyday and often in stressful/time-pressure situations.
I like your suggestion of keeping dependencies low. For now, I went the opposite route to start. I’ve tried to cobble together all these gems in a project: GitHub - yagudaev/active_search_demo. My hope is to find a way to create conceptual compression over time. There is value right away to others of a way to put all these different libraries together. I’ll keep an eye on dependencies and try not to get them too deeply rooted so they cannot be remove/replaced.
Do you plan to solve the interface problem or the data replication one or both?
Since ActiveRecord handles data migrations, I feel like ActiveSearch could/should handle data replication. I’m sure we’re all using similar but non-standard techniques to get data into our search infra and it would be great to either:
a.) have tooling in place to do this or
b.) have recommendations (Sync vs Async, middle tier - eg. kafka, etc).
There are so many considerations and honestly this is likely the longest tail in getting search done correctly.
Thanks Anthony. The plan is first to solve the programming interface problem. Most search adapter for Elastic, Algolia, PG Fulltext Search all use the same style of call-back based data replication.
I’ve yet to dig into the multiple-database model offered by latest AR, but this is exactly why I thought it would be interesting to integrate directly.
The simplest level of API for this is to not have any replication and keeping things in one database.
To get a good search means having all data in a search-index, so I do like the term “replication” you introduced. It sometimes takes a different form, for example, if you want to search for orders that contain a specific product. In elastic (/any document db), you would want to embed the products into the orders index. Thus, it is replication + transformation.
The extension of search is reporting, dashboarding, recommendations and machine learning. All area I think Rails can make easy to build.
For now, I started with the gems highlighted in the original post and putting them together in an example repo. I took a list of Netflix titles from Kaggle and designed a relational domain model around it. Then I built out a [plan to add features together] (Planning · GitHub)
Quick update: I’ve been working on understanding the principles of what makes good search. The best way I find to do that is to try to teach it to other. Therefore, I started on this course: https://www.searchonrails.com/rails-search. Love to hear your thoughts.
. Haven’t tried Textacular, thought it was older and not really maintained anymore compared to