How To Use OpenSearch Serverless As A Datasource In AWS Managed Grafana

Introduction

At KINTO Technologies' Platform Engineering team, we were not fully satisfied with our current logging solution. With new AWS services available, we saw an opportunity to enhance our logging platform, making it both easier to use and more cost-effective - a win-win situation!

Of course we could not just tear down everything already in place to replace it with the new shiny services - that would be like replacing the engine of a car while it's still running! We needed to investigate what new services we could use and how to configure them to meet our needs.
As part of our exploration of using OpenSearch Serverless for our new log platform we needed to find a solution for our alert system. Currently, we are using the Alerting feature of our OpenSearch cluster, but this feature is unavailable in the serverless instances.

Thankfully, as of AWS Managed Grafana version 9.4, the Grafana OpenSearch plugin could use an OpenSearch Serverless instance as a data source (see the Grafana Opensearch plugin page), so we could use Grafana for our alerting needs! We still needed to figure out how to configure both services so that they could work nicely together.

At the current state of our investigation we had already created an OpenSearch Serverless instance and tested log ingestion from all of the source we wanted to use. The remaining task was to set up a test Grafana instance in our Sandbox to use our serverless instance as a data source.

At the time of writing this article, the AWS documentation is not explicit on how to do exactly that. As engineers, we often don't have a step-by-step guide for every task. This is when we need to explore and experiment with whatever we are building to see what works. We also asked for help from the AWS Support to narrow down all the necessary permissions, where they had to escalate our request for help to both the Amazon Managed Grafana internal team, and to the OpenSearch team as the documentation does not exist yet. This motivated us to write this article to share the knowledge.

A quick self-introduction before continuing: I'm Martin, a Platform Engineer at KINTO Technologies. I joined the team last year and started working with AWS sporadically since then. Working on this project has been a great learning experience for me and I'm excited to share it with you! The biggest takeaway I got from this project is that the AWS Support is a great resource and you should not hesitate to ask for help when you need it.

Setting up our environment

In this article, we'll set up everything using the AWS Console. You can, of course, use your favorite Infrastructure as Code tools with AWS to build the same configuration. This article assumes you are familiar with the AWS Console and already have an Opensearch Serverless instance running.

Please note, the configurations used in this article prioritize simplicity. I strongly recommend reviewing and adjusting these settings to align with your organization's security requirements.

Setting up the IAM role

Before anything else, we will need to create an IAM role for our Grafana instance to use. If you plan to use other AWS services with your Grafana workspace, it might be better to select the Service managed option when creating the Grafana workspace. You can then update that role created by AWS or provide the ARN of your custom role when setting up the data source in Grafana.

Here is the trust policy needed when creating the IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "grafana.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

You can get the same trust policy by selecting the AWS service Trusted entity type and select AmazonGrafana in the Use case section.

Here is the permission policy required for accessing OpenSearch Serverless from Grafana, with special thanks to the AWS Support team for escalating our request to the Grafana and OpenSearch teams to provide us with the minimum necessary permissions.:

{
    "Statement": [
        {
            "Action": [
                "es:ESHttpGet",
                "es:DescribeElasticsearchDomains",
                "es:ListDomainNames"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": "es:ESHttpPost",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:es:*:*:domain/*/_msearch*",
                "arn:aws:es:*:*:domain/*/_opendistro/_ppl"
            ]
        },
        {
            "Action": [
                "aoss:ListCollections",
                "aoss:BatchGetCollection",
                "aoss:APIAccessAll"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:aoss:<YOUR_REGION>:<YOUR_ACCOUNT>:collection/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

OpenSearch Access Policy

On the OpenSearch side, we need to add a Data access policy for our newly created IAM role. Even if we gave our IAM role the necessary permissions to access OpenSearch, we still need to create a Data access policy to allow the IAM role to access the data in the collections. See the AWS documentation for more information.

In the serverless section of the OpenSearch Service page menu, select Data access policies, then click on the Create access policy button. Add a name and a description to your access policy, then select JSON as the policy definition method. Use the following policy, courtesy of the Grafana Opensearch Plugin documentation:

 [
  {
        Rules = [
          {
            ResourceType = "index",
            Resource = [
              "index/<NAME_OF_YOUR_OPENSEARCH_INSTANCE>/*"
            ],
            Permission = [
              "aoss:DescribeIndex",
              "aoss:ReadDocument"
            ]
          },
          {
            ResourceType = "collection",
            Resource = [
              "collection/<NAME_OF_YOUR_OPENSEARCH_INSTANCE>"
            ],
            Permission = [
              "aoss:DescribeCollectionItems"
            ]
          }
        ],
        Principal = [
          <GRAFANA_IAM_ARN>
        ]
        Description = "Read permissions for Grafana"
      }
    ]

Update it with the name of your OpenSearch Serverless deployment and the ARN of the IAM role we created earlier.

A little bit of Networking

Before continuing with the creation of our Grafana instance, we are going to create a few networking resources.

First let's create two Subnet in the same VPC as your OpenSearch Serverless deployment. Each subnet should be in a different Availability Zones.

Once created, we need to update the Route Table of each subnet to add a new route from 0.0.0.0/0 to an Internet Gateway.

Next, create a Security Group accepting Inbound HTTPS traffic from your VPC, and accepting all Outbound traffic on 0.0.0.0/0.

With all of this in place, we can now create our Grafana instance!

Creating your Grafana Instance

Search for the Amazon Managed Grafana service in the Console search bar.

On the service's homepage, use the button that the AWS engineer conveniently placed there to create the Grafana workspace.

For the first step of the creation page, set the name and description of your Grafana workspace. Set the version to at least 9.4. Version 10.4 is the latest version available so I will be using that.

On the next page, for Authentication access, select your preferred authentication method. I'll select AWS IAM Identity Center.

In the Permission type section, select Customer managed and select the ARN of the IAM role you created earlier. I had this weird issue where after creating the Grafana workspace, it was using another IAM role than the role I selected so I had to update the workspace to use the correct role. It could be a bug or a misconfiguration on my side. For the sake of this article, we will agree that I definitely selected the correct role and that this was a bug. Ok? Great!

In the Outbound VPC connection section, select the same VPC as the one in which your OpenSearch Serverless instance is deployed. For the Mapping and Security Groups, select the subnets and the security group we created earlier.

In the Workspace configuration options section, make sure to select Turn plugin management on.

For this tutorial, we will section Open Access in the Network access control section.

Click on the next button and review your settings.

Once the workspace is created, set up your authentification method. I selected AWS IAM Identity Center so I'll simply add my user and make myself admin.

You should now be able to connect!

Grafana Meets OpenSearch Serverless

Before adding our OpenSearch Serverless data source, we need to install the OpenSearch plugin in our Grafana workspace. To do this, follow these steps:

1. In the menu on the left, select Administration, then Plugins and Data, and finally Plugins.

2. On the Plugins page, select All instead of Installed in the field at the top of the page.

3. Search for the OpenSearch plugin and install it. Once installed, you should see an Add new data source button at the top right of the OpenSearch plugin page. Click on it.

Next, configure the data source information to connect to your OpenSearch Serverless instance:

1. HTTP Section: Add the URL of your OpenSearch Serverless instance in the URL field.

2. Auth Section: Toggle on SigV4 auth and select the region where your OpenSearch Serverless instance is located.

3. OpenSearch Details Section: Toggle on Serverless and set the index you want to use.

4. Logs Section: Set the name of your message field and level field.

Finally, click on Save & test. You should receive a message confirming that you have successfully connected to OpenSearch. You can now use this data source to create alerts and dashboards!

Conclusion

I hope this article has been helpful and that you can now set up your own Grafana instance with OpenSearch Serverless as a data source.

For us at KINTO Technologies, using Grafana for alerting looks like a great choice for our new logging solution. With this setup, we'd have a robust, efficient, and cost-effective logging and alerting solution that meets our specifications. Personally, I find creating alert queries in Grafana to be more straightforward and flexible compared to OpenSearch.

By the way, the Platform Group at KINTO Technologies is hiring! We are always looking for talented engineers to join our team. If you're interested in joining our team or want to learn more about what we do and what it's like to work here, please feel free to reach out to us! We have a nice web page with all our job listings here.