CloudNugget

Querying Amazon OpenSearch from an Application: REST API and SigV4 Signing#

When you’re building applications on AWS that need powerful search and analytics capabilities, Amazon OpenSearch becomes an invaluable tool. But there’s often a gap between knowing that OpenSearch exists and actually integrating it into your application code. The question that trips up many developers is straightforward but crucial: How do I actually query OpenSearch from my application, especially when I’m using AWS Identity and Access Management (IAM) for authentication?

This guide walks you through the practical realities of calling the OpenSearch REST API from application code. We’ll explore the query patterns you’ll use most often, dive into AWS SigV4 request signing for secure IAM-based authentication, and work through real code examples that you can adapt to your own projects. Whether you’re building a search feature for a web application, analyzing logs, or implementing real-time analytics, understanding these fundamentals will save you hours of debugging and frustration.

Understanding the OpenSearch REST API#

Amazon OpenSearch exposes a REST API that’s compatible with the Elasticsearch REST API. Every operation you perform against your OpenSearch domain—whether you’re indexing documents, searching, or managing cluster settings—goes through HTTP endpoints. The core endpoint structure looks like this:

https://<domain-endpoint>/<index-name>/<operation>

For example, to search an index called products, you’d make a request to something like:

https://my-opensearch-domain.us-east-1.es.amazonaws.com/products/_search

The underscore prefix is a convention in OpenSearch and Elasticsearch for API operations. The _search endpoint is perhaps the one you’ll interact with most frequently. But before we jump into writing queries, it’s worth understanding that OpenSearch is fundamentally a document database built on top of Lucene. You don’t query it like you’d query a relational database. Instead, you describe what you’re looking for using the Query DSL—a JSON-based language designed specifically for OpenSearch and Elasticsearch.

The Query DSL is expressive but follows consistent patterns. Let’s explore the most commonly used query types, because understanding these will cover the vast majority of your real-world search needs.

Match Queries#

The match query is your workhorse for full-text search. It analyzes the input text using the same analyzer that was applied to the field when it was indexed, then looks for documents containing those terms. Imagine you have an e-commerce index with product descriptions, and a user searches for “red running shoes.” A match query will find products that contain these terms, even if the exact phrase doesn’t appear.

{
  "query": {
    "match": {
      "description": "red running shoes"
    }
  }
}

Term Queries#

While match queries analyze text, term queries search for exact values. They’re perfect for structured fields like status codes, category IDs, or any field you’ve explicitly told OpenSearch not to analyze. If you’re searching for products with a specific category ID or looking for all orders with a particular status, term queries are your answer.

{
  "query": {
    "term": {
      "status": "active"
    }
  }
}

Range Queries#

Range queries let you find documents where a field falls within certain boundaries. They work beautifully for numeric and date fields. Want to find all orders placed in the last 30 days? Or products priced between $10 and $50? Range queries handle this elegantly.

{
  "query": {
    "range": {
      "timestamp": {
        "gte": "2024-01-01",
        "lte": "2024-12-31"
      }
    }
  }
}

Bool Queries#

As your queries become more sophisticated, you’ll need the bool query, which combines multiple queries using boolean logic. The bool query has four clauses: must (all conditions required), should (at least one condition preferred), must_not (conditions that must not match), and filter (like must, but doesn’t affect relevance scoring).

{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "laptop" } }
      ],
      "filter": [
        { "range": { "price": { "lte": 1500 } } }
      ],
      "should": [
        { "term": { "brand": "Dell" } }
      ]
    }
  }
}

This query finds documents with “laptop” in the title, priced at $1500 or less, with a slight boost if the brand is Dell. The filter clause doesn’t affect relevance scoring—it simply includes or excludes documents, making it more efficient than using must when you don’t care about scoring.

Authentication with IAM and SigV4#

Here’s where many developers hit their first real challenge. If your OpenSearch domain uses IAM authentication (which is a best practice for AWS-managed domains), you can’t simply send HTTP requests to the domain endpoint. You need to sign your requests using the AWS SigV4 signing process.

SigV4 is AWS’s request signing mechanism that proves to AWS that you are who you claim to be, that your request hasn’t been tampered with, and that you’re authorized to perform the action. When you use the AWS CLI or SDKs, this happens behind the scenes. But when you’re making direct HTTP calls to OpenSearch, you need to handle it yourself—or use a library that does it for you.

The SigV4 signing process involves creating a canonical string representation of your request, then using your AWS secret access key to create a signature. The signature gets added to a special Authorization header. AWS then verifies this signature on the server side to authenticate your request.

You have two main approaches: use an official OpenSearch client library that understands SigV4 signing, or use the AWS SDK’s request signing utilities and make HTTP calls yourself. Let’s explore both.

Python: Using opensearch-py with AWS Request Signing#

Python developers have a straightforward option with the opensearch-py library combined with the aws-requests-auth package. Here’s how to set it up and use it:

First, install the necessary packages:

pip install opensearch-py aws-requests-auth boto3

Now, here’s a complete example that creates a connection to your OpenSearch domain and performs a search:

from opensearchpy import OpenSearch, RequestsHttpConnection
from aws_requests_auth.aws_auth import AWSRequestsAuth
import boto3

# Get AWS credentials from your environment or IAM role
credentials = boto3.Session().get_credentials()

# Create the auth handler
auth = AWSRequestsAuth(
    aws_access_key=credentials.access_key,
    aws_secret_access_key=credentials.secret_key,
    aws_token=credentials.token,
    aws_host='my-opensearch-domain.us-east-1.es.amazonaws.com',
    aws_region='us-east-1',
    aws_service='es'
)

# Initialize the OpenSearch client
client = OpenSearch(
    hosts=[{'host': 'my-opensearch-domain.us-east-1.es.amazonaws.com', 'port': 443}],
    http_auth=auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

# Perform a search
search_body = {
    "query": {
        "bool": {
            "must": [
                {"match": {"title": "aws"}},
            ],
            "filter": [
                {"range": {"timestamp": {"gte": "2024-01-01"}}}
            ]
        }
    },
    "size": 20
}

try:
    response = client.search(index="documents", body=search_body)
    print(f"Found {response['hits']['total']['value']} documents")
    
    for hit in response['hits']['hits']:
        print(f"  - {hit['_source']['title']} (score: {hit['_score']})")
        
except Exception as e:
    print(f"Search failed: {e}")

The beauty of this approach is that the opensearch-py client handles connection pooling, retries, and request formatting for you. The AWSRequestsAuth handler transparently signs each request with SigV4, so you don’t need to think about it.

Notice that we’re pulling AWS credentials from a boto3 session. In production, you’d typically get these from an IAM role attached to your EC2 instance, Lambda function, or ECS task. The credentials are automatically retrieved from the instance metadata service, so you don’t hardcode them.

Node.js: AWS SDK v3 with OpenSearch#

Node.js developers can use the AWS SDK v3 with the @opensearch-project/opensearch client, or they can use the @aws-sdk/core utilities for request signing. Here’s the recommended approach:

npm install @opensearch-project/opensearch @aws-sdk/credential-provider-node @aws-sdk/core

Here’s the implementation:

import { Client } from '@opensearch-project/opensearch';
import { defaultProvider } from '@aws-sdk/credential-provider-node';
import { SignatureV4 } from '@aws-sdk/signature-v4';
import { HttpRequest } from '@aws-sdk/protocol-http';
import { Sha256 } from '@aws-crypto/sha256-js';

const client = new Client({
  nodes: ['https://my-opensearch-domain.us-east-1.es.amazonaws.com'],
  getNodeConnectionParams: async () => {
    const credentials = await defaultProvider()();
    
    return {
      headers: {
        host: 'my-opensearch-domain.us-east-1.es.amazonaws.com'
      }
    };
  }
});

// Create a custom request interceptor to sign requests
const createSignedRequest = async (request) => {
  const credentials = await defaultProvider()();
  const signer = new SignatureV4({
    credentials,
    region: 'us-east-1',
    service: 'es',
    sha256: Sha256
  });

  const signedRequest = await signer.sign(
    new HttpRequest({
      method: request.method,
      path: request.path,
      headers: request.headers,
      body: request.body,
      hostname: 'my-opensearch-domain.us-east-1.es.amazonaws.com'
    })
  );

  return {
    headers: signedRequest.headers
  };
};

// Perform a search
const searchQuery = {
  index: 'products',
  body: {
    query: {
      bool: {
        must: [
          { match: { name: 'laptop' } }
        ],
        filter: [
          { range: { price: { lte: 2000 } } }
        ]
      }
    },
    size: 20
  }
};

try {
  const results = await client.search(searchQuery);
  console.log(`Found ${results.body.hits.total.value} products`);
  
  results.body.hits.hits.forEach(hit => {
    console.log(`  - ${hit._source.name} ($${hit._source.price})`);
  });
} catch (error) {
  console.error('Search failed:', error);
}

The Node.js approach is slightly more involved because you need to explicitly handle request signing, but the AWS SDK v3 makes it manageable. The key is that SignatureV4 handles all the cryptographic heavy lifting—you just provide your credentials, region, and service name.

Connection Pooling and Performance Optimization#

When you’re making frequent requests to OpenSearch, connection pooling becomes critical. Both the Python opensearch-py client and the JavaScript OpenSearch client handle pooling automatically, but understanding what’s happening under the hood helps you configure them appropriately.

Connection pooling means keeping a set of reusable HTTP connections open rather than creating a new connection for each request. Opening a new connection involves a TCP handshake and TLS negotiation, which takes time. By pooling connections, you avoid that overhead for most of your requests.

In the Python example earlier, the client manages this automatically. However, if you’re building a high-volume application, you might want to configure pool size:

client = OpenSearch(
    hosts=[{'host': 'my-opensearch-domain.us-east-1.es.amazonaws.com', 'port': 443}],
    http_auth=auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
    pool_maxsize=20,  # Maximum number of connections
    timeout=10  # Connection timeout in seconds
)

For Lambda functions specifically, since Lambda reuses execution environments between invocations, you should create the OpenSearch client outside your handler function:

# Create the client once, reuse across invocations
client = OpenSearch(
    hosts=[{'host': 'my-opensearch-domain.us-east-1.es.amazonaws.com', 'port': 443}],
    http_auth=auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

def lambda_handler(event, context):
    # Reuse the client for multiple requests
    results = client.search(index="logs", body={"query": {"match_all": {}}})
    return {'statusCode': 200, 'body': results}

This pattern eliminates the overhead of creating a new client for every Lambda invocation.

Handling HTTP Error Responses#

Working with OpenSearch means dealing with various HTTP status codes. Understanding what they mean and how to handle them properly is essential for building resilient applications.

429 Too Many Requests indicates that your requests are exceeding the OpenSearch domain’s throughput capacity. This typically means you’re either sending requests too fast, or your domain doesn’t have enough resources. The appropriate response is to implement exponential backoff and retry logic.

Here’s a Python example with retry logic:

import time
from opensearchpy.exceptions import TransportError

def search_with_retry(client, index, body, max_retries=3):
    backoff = 1
    
    for attempt in range(max_retries):
        try:
            return client.search(index=index, body=body)
        except TransportError as e:
            if e.status_code == 429:
                wait_time = backoff * (2 ** attempt)
                print(f"Rate limited. Waiting {wait_time}s before retry...")
                time.sleep(wait_time)
                continue
            else:
                raise
    
    raise Exception(f"Failed after {max_retries} retries")

5xx errors (500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable) indicate problems on the OpenSearch side. These warrant retries as well, but with caution—if you’re getting persistent 5xx errors, your domain may be having issues.

400 Bad Request means your request was malformed. This might be invalid JSON, a syntax error in your Query DSL, or querying an index that doesn’t exist. These require code changes to fix; retrying won’t help.

403 Forbidden indicates that your AWS credentials lack permission to perform the requested action. This is an IAM configuration issue. Check that your credentials have the appropriate OpenSearch permissions (typically es:* for development, or more restrictive permissions for production).

Practical Considerations for Production Applications#

When you move beyond toy examples, a few considerations come into play:

Domain Sizing: Your application’s search patterns directly influence the resources you need. A domain that’s undersized will throttle your requests (429 errors), while an oversized domain wastes money. Start conservative and monitor CloudWatch metrics—particularly the IndexingRate and SearchRate metrics—to understand your actual usage.

Query Timeouts: By default, OpenSearch queries timeout after 30 seconds. If you’re running expensive aggregations or queries against massive datasets, you might exceed this. You can increase the timeout per request, but better yet, think about whether you can optimize your query or data structure.

search_body = {
    "query": {"match": {"content": "search term"}},
    "timeout": "60s"  # 60 second timeout
}
response = client.search(index="logs", body=search_body)

Indexing Strategy: How you structure your data and indices affects both search performance and storage costs. For time-series data like logs, many applications create indices per day or per week, then delete old indices. This is far more efficient than one massive index.

Monitoring and Logging: Enable OpenSearch application logs in CloudWatch to see slow queries and errors. Use CloudWatch metrics to monitor domain health. These insights help you understand where to optimize.

Direct HTTP Requests with Manual SigV4 Signing#

Sometimes you might not want to use an official OpenSearch client library—perhaps you’re working in an environment where no suitable library exists. In that case, you can sign requests manually using AWS SigV4. Here’s a minimal Python example:

import requests
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
import boto3

session = boto3.Session()
credentials = session.get_credentials()
region = 'us-east-1'
service = 'es'

url = 'https://my-opensearch-domain.us-east-1.es.amazonaws.com/products/_search'
body = {
    "query": {
        "match": {"name": "widget"}
    }
}

request = AWSRequest(
    method='POST',
    url=url,
    json=body,
    headers={'Host': 'my-opensearch-domain.us-east-1.es.amazonaws.com'}
)

SigV4Auth(credentials, service, region).add_auth(request)

response = requests.post(
    url,
    json=body,
    headers=dict(request.headers)
)

print(response.json())

The botocore library (which powers the AWS SDK for Python) handles all the SigV4 signing complexity. The SigV4Auth class modifies your request object in place, adding the required Authorization and X-Amz-Date headers. You then use those signed headers with your HTTP library of choice.

This approach is more transparent about what’s happening, but it’s also more error-prone. The official OpenSearch clients handle edge cases and evolving AWS requirements automatically, so prefer them when available.

Wrapping Up#

Querying OpenSearch from your application comes down to three key areas: understanding the Query DSL to express what you’re searching for, properly authenticating using SigV4 when IAM is enabled, and handling the practical details like connection pooling and error responses.

Start with the official OpenSearch client libraries—they’re battle-tested and handle much of the complexity for you. Use opensearch-py in Python and @opensearch-project/opensearch in Node.js, combined with appropriate AWS credential providers. Keep your credentials secure by relying on IAM roles rather than static keys. Monitor your domain’s metrics and slow queries to understand where optimization might help.

The Query DSL itself is expressive enough for most use cases. Master the basic query types—match for full-text search, term for exact matching, range for boundaries, and bool for combining multiple conditions. These four building blocks will handle the vast majority of your real-world search needs.

As you build and deploy your applications, remember that OpenSearch is a resource you share with the rest of your system. Thoughtful query design, appropriate domain sizing, and monitoring will keep your search experiences fast and your costs reasonable.

© 2026 CloudNugget