General Information

This section provides some basic information about AWS networking that is likely to affect Charon-SSP when running in the cloud.

NetworkManager is disabled on Charon-SSP AWS. Therefore, the interface configuration relies on ifcfg-files in /etc/sysconfig/network-scripts.

The information is not comprehensive. Please refer to the Amazon AWS documentation for up-to-date and comprehensive information.

Concepts

VPC: VPC stands for virtual private cloud. It is a virtual network associated with your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. You can launch your AWS resources, such as Amazon EC2 instances, into your VPC. You can specify an IP address range for the VPC, add subnets, associate security groups, and configure route tables. In addition to the default VPC, an account can have non-default VPCs. The default VPC includes an Internet gateway.

Subnet: A subnet is a range of IP addresses in your VPC. You can launch AWS resources into a specified subnet. There are public subnets that can be connected to the Internet and private subnets that have no direct Internet connection. Instances in a private subnet can reach the Internet via a NAT gateway.

Instance: An instance is a virtual machine that is launched into a VPC. It is associated with an image (e.g., Charon-SSW AMI) and a certain instance type representing the virtual hardware.

Address Assignment

Each VPC is assigned a block of private IP addresses. This block can be split by the user to form several IP subnets. Routing between such subnets is automatically enabled.

When an E2C instance is launched into the default VPC and a public subnet, the default behavior is as follows:

If the instance has only one network interface, it is automatically assigned a private IP address and a public IP address from the address range assigned to the public subnet. This network interface is the primary network interface. It is called eth0.
If the instance has more than one network interface, it is automatically assigned a private IP address for each of the network interfaces - but no public IP address.

The default behavior can be modified, for example:

Manually assigning a private IP address from the subnet range.
Enabling or disabling the automatic assignment of a private IP address to deviate from the subnet setting.
Manually assigning a public IP address from the AWS range or the customer range.

Public IP addresses are not directly visible to the instance. The instance operating system always works with the private address. For external connections, the private address is mapped to the public IP address via NAT.

Reserved addresses (important, if manual address assignment is used):

The following address range is reserved to allow AWS to query meta-data about instance configuration: 169.254.0.0/16. This range is automatically configured on every network interface.

The following addresses are reserved in each subnet and cannot be used for E2C instances (shown in the example below for network 10.1.1.0/24):

10.1.1.0: the network address
10.1.1.1: reserved by AWS for the VPC router
10.1.1.2: reserved by AWS in any subnet; the second host address in the base VPC network range is the DNS server for the VPC.
10.1.1.3: reserved by AWS for future use
10.1.1.255: network broadcast address; AWS networks do not use broadcasts.

An automatically assigned public IP address is released (and not re-assigned) by AWS for example if

a second interface is added to the instance and the instance is then stopped and restarted,
an Elastic IP is associated with the the instance,
an Elastic IP address is associated with the primary interface of the instance.

See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html for details.

An automatically assigned public IP address is not persistent. Every time a instance starts, it is assigned a new public IP address. If persistent public addresses are needed, use Elastic IP addresses.

Interface Names

When looking at the instance from the AWS management console, the interface names are eth0, eth1, etc.

On instances without support for enhanced networking the Linux interface names are also eth0, eth1, etc.

However, on instances with support for enhanced networking, the names on the AWS level differ from those on the Linux level:

The first (primary) interface is called ensX (where X is an integer denoting the interface number; example: ens5).
When a second interface is added to a running instance, it maybe initially called eth0.
However, the command ethtool -i eth0 shows that the enhanced network driver (ena) will be used for this interface. This interface will change its name to ensY (where Y is X+1) after restarting the instance. This means that any configuration file created for this interface must use the final name of the interface instead of eth0. Otherwise, the instance may become unreachable after a restart because there is no valid interface configuration (NetworkManager is not enabled on Charon-SSP AWS, so a configuration file must exist to configure the interface properly).
This numbering scheme may change in the future. Currently, it is based on the PCI slot on which the Ethernet controller is presented and which is incremented by one for each additional Ethernet interface added. On the Charon host system, the slot can be verified with the following command:
# lspci -vv | grep -A20 Ethernet

To avoid confusion before the instance can be restarted, the new interface can be renamed to its final name using the command
ip link set eth0 name ensY && ip link set ensY up

Host to Guest Communication Considerations

There are several ways a communication between the host operating system and the guest Solaris system can be implemented. For example:

Internal virtual bridge on the host system:
Such a bridge has several TAP interfaces. The host and the guest systems are connected to this bridge and can communicate directly to one another using L3 and L2 protocols. The bridge uses its own IP subnet that can be defined by the user. Setting up such a configuration is supported by the Charon Manager.
Communication via the AWS subnet LAN:
In this case, a second interface is added to the Charon host system. The second interface is then assigned to the emulated guest system. After the correct configuration, the host and guest can communicate across the AWS LAN using IP. L2 protocols or any protocols that require changing the MAC address to something different than the MAC address assigned to the second interface by AWS will not work.
To connect the guest system to the LAN, the following basic configuration steps must be performed:
- Add the additional interface to the Charon host system.
- Create a configuration file for the additional interface.
- Remove the private IP address assigned to the second interface by AWS from the Linux configuration (if it has been configured).
- Use Charon Manager to assign the interface to the emulated SPARC system.
- Use Charon Manager to set the MAC address of the emulated SPARC system to the same value as the one used on the host system Ethernet interface.
- On the Solaris system, configure the private IP address that was previously assigned to the second interface on Linux and configure the appropriate default route for the LAN.
- Additional steps may be required:
  - If the primary interface has an automatically assigned public IP address, this will be released when the instance is stopped and restarted. Hence the configuration must be changed to use a persistent Elastic IP address first to maintain reachability of the host system.

The section Example of a More Complex Network Configuration provides some hints on how to configure the second interface in the different situations. Please refer to the AWS documentation for up-to-date comprehensive information.

If Layer 2 communication between guests on different Charon hosts is required, a bridged tunnel solution must be set up between the two Charon host systems.

External Communication Considerations

In addition to allowing SSH access to the host system for management purposes, it may be necessary to enable Internet communication to the host and guest system or connect host and guest to the customer's network.

Recommended way to connect the Charon host and Solaris guest systems to the customer network:

To ensure data traffic between the Charon host and guest systems and the customer network is encrypted, it is strongly recommended to use a VPN connection. An example of a simple VPN connection based on an SSH tunnel is described in SSH VPN - Connecting Charon Host and Guest to Customer Network. This connection is based on a bridge between Charon host and guest system and (via an encrypted SSH tunnel) the remote end-point in the customer network. The connection supports L3 and L2 protocols.

AWS also provides a VPN gateway instance that can be added to the customer VPC to connect the VPC to the customer network (for a charge).

Recommended way to connect the Solaris guest system to the Internet:

The Internet connection can be implemented across the VPN to the customer network. In this case, the customer can allow the guest Solaris system to access the Internet exactly following the security policies defined by the customer.

Access to the Internet from private VPC subnets or a Solaris guest system with only private IP addresses:

Access to the Internet for private VPC subnets is possible across a gateway instance providing VPN access to the customer network and allowing (NATted) Internet access via this path. Alternatively, a NAT gateway in the cloud can be used to map the private addresses to public addresses. The NAT gateway can be implemented on a Charon host system or it can be provided by AWS for a charge.

Direct Solaris guest access to the Internet:

This not a recommended standard solution for security reasons. However, should it be required, two interfaces with public IP addresses can be assigned to the Charon host.
One of these interfaces is then dedicated to the guest system which uses the private interface address and the MAC address assigned to the Charon host by AWS (similar to point 2 in section Host to Guest Communication Considerations above).

Using a Charon host system as a Router

If a Charon host system is to be used as a router (for example as shown in Example of a More Complex Network Configuration or to provide Internet connectivity to other Charon host and guest systems), it is not sufficient to configure Linux for IP forwarding.

The following settings have to be made on the Charon host instance via the AWS management console:

For each interface, the source/destination check has to be disabled. Unless this is configured correctly, traffic from and to an AWS instance will only be allowed if either source or destination address belongs to the instance. Transit traffic destined to be forwarded by the router, would be discarded.

Guest to Guest Layer 2 Communication Considerations

Should L2 protocols be required between two guest systems on different host systems, a bridge/tunnel solution similar to the one described in SSH VPN - Connecting Charon Host and Guest to Customer Network must be set up between the two host systems to allow the L2 traffic to pass.

Asymmetric Routing Considerations

This section applies to the case where several interfaces are configured on an instance and they all have IP addresses configured on the Linux level.

From the AWS documentation (https://aws.amazon.com/premiumsupport/knowledge-center/ec2-ubuntu-secondary-network-interface/):

"Adding a secondary network interface to a non-Amazon Linux EC2 instance causes traffic flow issues. These issues occur because both the primary and the secondary network interfaces are in the same subnet, and there is only one routing table with one gateway. Traffic that comes into the secondary network interface leaves the instance using the primary network interface. But this isn't allowed, because the secondary IP address doesn't belong to the MAC address of the primary network interface.

To make the secondary interface work, create a secondary network configuration file, configure the routing table, and then set up rules in the custom routing table policy database so that traffic for the secondary interface uses the new routing table."

The above documentation only describes the required steps for Ubuntu. An earlier article for CentOS and Red Hat was removed from the AWS site. So the information presented here may change in the near future.

Review the section about interface names if using an instance with enhanced networking enabled.

When adding a second IP interface (for example eth1) to the same subnet as the first on the Charon-SSP host, the routing problems described above can occur. To solve them, perform the following basic steps.

Create a configuration file (/etc/sysconfig/network-scripts/ifcfg-<interface-name>) for the second interface (if there is no configuration file for the primary interface, create it as well).
Set the correct interface for default route in /etc/sysconfig/network (example: GATEWAYDEV=eth0).
To prevent the cloud-init from resetting your custom network configurations, add the following lines to the /etc/cloud/cloud.cfg file:
network:
; config: disabled
Restart the network.
Create an additional routing table (use the command: ip route add <path> dev <interface-name> table <table-id>). There must be an entry for every IP address assigned to the second interface and any other route to be used.
Set rules in the Routing Policy Database (use the command: ip rule add from <ip-address-of-second-interface> lookup <table-id>)
Create a static route file (/etc/sysconfig/network-scripts/route-<interface-name>)
Create a static rule file (/etc/sysconfig/network-scripts/rule-<interface-name>)

Please refer to the Linux man pages for ip rule and ip route for more information. The AWS example for Ubuntu may also provide helpful hints.

Further Information

The following sections show sample network configurations:

Charon-SSP 3.1.21 AWS

AWS Networking and Charon-SSP

Contents