Often, we need to create a private API on AWS that is only callable from within the AWS network. We do not want to put our internal servers on the public Internet. But if we don’t put the EC2 instance which implements the API on the public Internet, the API will not be accessible from other AWS services. The answer is to create Lambda functions also deployed in the same VPC as the EC2 instance is and setup security groups in such a way that traffic is allowed on the private AWS network from the Lambda function to the EC2 instance. Then other AWS resources can call the private lambda function through VPC endpoints.
To create private APIs on AWS EC2 instances that are only accessible privately from a VPC or publicly through API Gateway:
- Create an Amazon Virtual Private Cloud (VPC): A VPC is a logically isolated section of the AWS cloud where you can launch AWS resources in a virtual network. You can define the IP address range of the VPC, create subnets, and configure route tables, network gateways, and security settings.
- Launch an Amazon EC2 instance: Launch an Amazon EC2 instance in your VPC to host your private API. Ensure that the security group assigned to the instance only allows traffic from within the VPC.
- Install and configure your API: Install and configure your private API on the EC2 instance. You can use any web server or API framework, such as Apache, Nginx, or Node.js, to serve your API. Ensure that your API is configured to listen only on the private IP address of the EC2 instance.
- Create an API Gateway: Create an API Gateway in AWS and configure it to proxy requests to your EC2 instance. Ensure that you configure the API Gateway to require authentication and limit access only to authorized clients.
- Set up VPC endpoints: Set up VPC endpoints to allow private access to your API from within your VPC. VPC endpoints are an AWS feature that enables you to create a private connection between your VPC and another AWS service. You can use an endpoint to connect to your private API from within your VPC without traversing the public internet.
Test your API: Test your private API by making requests from within your VPC and through the API Gateway. Ensure that you can access the API from authorized clients and that it is not accessible from unauthorized clients
Using a private lambda function to access the API
Another often used approach is to create a private lambda function to access a private API, A private Lambda function is one that has been deployed to a VPC rather than the default space Lambda functions are normally deployed to on AWS,. Then all other AWS resources which need access to the private API would access it via calling the lambda function.
When you you configured a lambda function to run in a VPC, a you can use a VPC endpoint to connect to the private API. To do this, you can create a VPC endpoint for the AWS service that your API is running on, such as EC2 or Elastic Load Balancing, and configure the endpoint to route traffic to your private API. Then, in your Lambda function, you can use the VPC endpoint’s DNS name or IP address to access the API.
Why can’t I access my private API directly?
Even if both the lambda function and the instance the private API runs on live in the same VPC, you will not be able to access the private API, because the EC2 API is normally only accessible through the public internet.
Use a VPC endpoint: If your Lambda function is configured to run within a VPC, you can use a VPC endpoint to connect to the private API. To do this, you can create a VPC endpoint for the AWS service that your API is running on, such as EC2 or Elastic Load Balancing, and configure the endpoint to route traffic to your private API. Then, in your Lambda function, you can use the VPC endpoint’s DNS name or IP address to access the API.
The other way is to configure an NAT Gateway which runs an instance is very expensive. A VPC endpoint is the cost effective solution.
What is a VPC endpoint and how is it different from an NAT Gateway?
A VPC endpoint is a virtual device that allows you to privately connect your VPC to supported AWS services and VPC endpoint services. A VPC endpoint enables you to connect to AWS services or VPC endpoint services without requiring an internet gateway, a NAT device, VPN connection, or a Direct Connect connection.
When you create a VPC endpoint, you can choose to create an interface endpoint or a gateway endpoint. Interface endpoints are endpoint network interfaces that are powered by PrivateLink technology, and they allow you to connect to a specific service within AWS. Gateway endpoints are VPC endpoints that are associated with a specific service and are used to create a route from your VPC to the service over the AWS network.
On the other hand, an NAT Gateway is a highly available, managed Network Address Translation (NAT) service provided by AWS. NAT Gateway is used to allow resources in a private subnet to initiate outbound connections to the internet or other AWS services, while blocking inbound connections initiated from the internet.
The main difference between a VPC endpoint and an NAT Gateway is that VPC endpoints are used to connect your VPC to AWS services or VPC endpoint services without requiring a public internet connection or NAT device, whereas NAT Gateway is used to provide outbound internet connectivity for resources in a private subnet.
VPC endpoints allow you to connect to AWS services or VPC endpoint services without requiring an internet gateway or NAT device, whereas NAT Gateway is used to provide outbound internet connectivity for resources in a private subnet.
Since out goal is to not go through the public internet when creating a private API, VPC endpoint is the preferred solution. When using an ENI, an interface, to create a VPC endpoint, the cost of this connection is also much more favorable than using an NAT server.
How to configure the security groups
Important: For traffic to flow from the private the lambda function through the ENI implementing the network interface through to the instance which runs the private API, we must configure the security groups along the way such that they are chained. The lambda function, the VPC endpoints ENI, and the EC2 instance all need to list the source security group and the target security group to allow traffic and often open up the IP ports as well. One would think that just chaining the security groups is enough but we often needed to open up the TCP ports as well for traffic to flow properly.
In addition to this the Lambda function needs to be configured with a role that can access the EC2 instance.
Do you need help setting up private APIs?
We have 30 year plus veterans of the IT industry that can help. Contact us at info@synkre.com