Novel strategies for serverless search

A requirement I seem to meet in every substantial web application is search: somewhere in the system there is a large collection of text - documents, messages, invoices, audit logs - and a user who needs to find the records that mention a particular string. For a modest dataset this is comfortably handled by the database you already have, as a SQL LIKE query over a few thousand rows returns quickly enough that nobody minds. Once the collection grows towards a million records those queries begin to grind, and the conventional advice is to bring in a dedicated search engine such as Elasticsearch, or its AWS-managed relative OpenSearch.

That advice is perfectly sound, and for plenty of applications it's the right call. A search cluster is a standing piece of infrastructure, however, which has to be provisioned, patched, scaled and monitored, and which has to be paid for around the clock whether anyone searches or not. A great deal of my work in recent years has been serverless - Lambda functions, DynamoDB tables, S3 buckets and API Gateway, with nothing running and nothing billed while the application is quiet - and a search cluster sitting in the middle of an otherwise serverless architecture has always felt to me like a step backwards. So I've been turning a question over: what could search look like if it were built entirely from serverless parts?

This article is the first in a series exploring exactly that. Below I'll look at why the obvious managed answers are less attractive than they first appear, and then sketch out the five strategies the rest of the series will work through in detail - some of which are thoroughly practical, and some of which are honest experiments.

The trouble with the obvious answers

AWS does have an off-the-shelf way to query data in S3 without any infrastructure at all, which is Amazon Athena. Athena runs standard SQL over files sitting in S3, there is nothing to provision, and you pay only when you run a query. It's an excellent service for its intended purpose, and nothing in this series is a criticism of Athena as an analytics tool.

The difficulty comes from its pricing model, which is $5 per terabyte of data scanned, with a 10 MB minimum per query. Athena has no index, so every query has to read through the underlying files to find its answer. An analytics workload suits this very well, being a handful of substantial queries a day over well-organised, columnar data. A search workload has the opposite shape: a stream of small queries arriving all day long, each of which needs to look through the whole collection for a string that might be anywhere. If you keep 100 GB of documents in S3 as plain text, every single search scans the lot and costs around fifty cents. Put that behind a search box that gets used twenty times an hour and the arithmetic becomes uncomfortable very quickly - roughly $10 an hour, all day, and growing as your data grows.

The managed search alternative is OpenSearch Serverless, and it's a good product, though its name needs a little qualification. It relieves you of managing the servers, but it bills for a minimum number of OpenSearch Compute Units which remain allocated around the clock, so a quiet application still pays a standing charge every hour of every day. Between the two options an awkward gap opens up: Athena becomes expensive when searches are frequent, and OpenSearch Serverless stays expensive when they're not.

There used to be a third option sitting in between. S3 Select allowed you to run SQL against a single object in place, retrieving just the matching rows rather than downloading the whole file, and it quietly solved some fiddly problems on your behalf. AWS closed S3 Select to new customers in July 2024, recommending Athena or client-side filtering in its place. I mention it partly for completeness, and partly because one of its features - the ScanRange parameter, which we'll meet again in part three - is something we now have to rebuild for ourselves.

Five serverless search strategies

So what might search look like if we assembled it ourselves from the serverless primitives - S3, Lambda, DynamoDB and their newer relatives? The five strategies in this series sit on a spectrum. At one end we do careful work up front to structure the data; in the middle we skip the structure entirely and lean on raw parallelism; and at the far end we hand the decision-making over to an AI agent.

1. Creative indexing in DynamoDB

DynamoDB has no full-text search of its own, and scanning a table of a million items for a string is exactly the wrong way to use it. What it does superbly is return the items stored under a known key, quickly and at any scale, and that single strength is enough to build a search engine on. The technique is the inverted index: you store a list of documents against each word, rather than storing words inside each document, so that finding every document containing "cat" becomes a single query against the key "cat". This is the same structure Elasticsearch is built upon, laid out by hand across DynamoDB partitions, and building one teaches you a great deal about how search engines actually work. Part two covers the design in full, including tokenisation, ranking, hot partitions, and keeping the index fresh with DynamoDB Streams.

2. A Lambda fan-out over S3

The second strategy skips the index entirely. The documents live in S3 as plain text, and when a search arrives, a parent Lambda function divides the data into byte ranges and invokes a fleet of child Lambdas, each of which fetches its own range with a ranged GET request and scans it for the search string. A million records searched by a thousand parallel workers takes roughly the time one worker needs for a thousand records, and while nobody is searching, the whole arrangement costs nothing at all. There are some genuinely interesting problems along the way, such as what to do when a byte range cuts a record in half, and how a Bloom filter for each shard of the data lets you skip most of the collection on most searches. Part three works through all of it.

3. Semantic search with S3 Vectors

Both of the strategies above find the exact string the user typed, and quite often that isn't really what the user wants; they want the documents that are about the thing they typed, in whatever words those documents happen to use. Vector search answers this by comparing meanings rather than strings, and until recently it required a dedicated vector database, which brings back the standing infrastructure we were trying to avoid. S3 Vectors, which reached general availability in December 2025, builds vector storage and similarity queries directly into S3, with per-use pricing and support for up to two billion vectors in an index. Part four looks at pairing it with an embedding model in Amazon Bedrock to give a serverless application genuine semantic search.

4. Bringing your own query engine

The fourth strategy takes aim at the Athena bill directly, by running a query engine inside a Lambda function. DuckDB will happily read Parquet files straight out of S3, fetching only the byte ranges a query actually needs, and SQLite's FTS5 extension provides a proper full-text index in a single portable file which can itself live in S3. In both cases the marginal cost of a search is a few Lambda-seconds. Part five explores how far this approach can be pushed and where its practical limits are, including cold starts, package sizes, and keeping the data fresh.

5. Agentic search with Amazon Bedrock

The final article brings the strategies together. Each of the approaches above ends up wrapped in a Lambda function, and a Lambda function is precisely the shape of tool you can hand to a Bedrock agent. Given a user's question, an agent can decide for itself whether an exact-match lookup, a brute-force sweep or a semantic query is the right tool for the job, try the inexpensive option first, and fall back to a costlier one if nothing comes back. I've written before about agents earning their keep in the awkward middle ground where fixed rules run out, and deciding how to search is a neat example of exactly that.

In summary

Serverless architectures still lack a truly satisfying story for search, and the two managed options each solve half of the problem: Athena keeps you free of infrastructure but charges by the scan, while OpenSearch Serverless gives you a proper search engine with a standing bill attached. Over the coming articles I'll build out each of the five strategies above with working code and honest costings, starting with the DynamoDB inverted index in part two. If your application is outgrowing its LIKE queries and you're weighing up the options, I hope the series proves useful - and as ever, I'm happy to talk through the particulars of your own situation.