...

This section provides some basic information about AWS networking that is likely to affect Charon-SSP when running in the cloud.

 NetworkManager is disabled on Charon-SSP AWS. Therefore, the interface configuration relies on ifcfg-files in /etc/sysconfig/network-scripts.

The information is not comprehensive. Please refer to the Amazon AWS documentation for up-to-date and comprehensive information.

...

Each VPC is assigned a block of private IP addresses. This block can be split by the user to form several IP subnets. Routing between such subnets is automatically enabled.

When an E2C instance is launched into the default VPC and a public subnet, the default behavior is as follows:

...

Public IP addresses are not directly visible to the instance. The instance operating system always works with the private address. For external connections, the private address is mapped to the public IP address via NAT.

Reserved addresses (important, if manual address assignment is used):

The following address range is reserved to allow AWS to query meta-data about instance configuration: 169.254.0.0/16. This range is automatically configured on every network interface.

The following addresses are reserved in each subnet and cannot be used for E2C instances (shown in the example below for network 10.1.1.0/24):

• 10.1.1.100: the network address
• 10.1.1.1: reserved by AWS for the VPC router
• 10.1.1.2: reserved by AWS in any subnet; the second host address in the base VPC network range is the DNS server for the VPC.
• 10.1.1.3: reserved by AWS for future use
• 10.1.1.255: network broadcast address; AWS networks do not use broadcasts.

An automatically assigned public IP address is released (and not re-assigned) by AWS for example if

• a second interface with a private address is added to the instance and the instance or the network is then stopped and restarted,a second interface with a public IP (
• an Elastic IP ) is added to associated with the instance and the instance or the network is restarted,
• an Elastic IP address is added to associated with the primary interface of the instance.

See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html for details.

An automatically assigned public IP address is not persistent. Every time a instance starts, it is assigned a new public IP address. If persistent public addresses are needed, use Elastic IP addresses.

Interface Names

When looking at the instance from the AWS management console, the interface names are eth0, eth1, etc.

On instances without support for enhanced networking the Linux interface names are also eth0, eth1, etc.

However, on instances with support for enhanced networking, the names on the AWS level differ from those on the Linux level:

• The first (primary) interface is called ensX (where X is an integer denoting the interface number; example: ens5).
• When a second interface is added to a running instance, it maybe initially called eth0.
  However, the command ethtool -i eth0 shows that the enhanced network driver (ena) will be used for this interface. This interface will change its name to ensY (where Y is X+1) after restarting the instance. This means that any configuration file created for this interface must use the final name of the interface instead of eth0. Otherwise, the instance may become unreachable after a restart because there is no valid interface configuration (NetworkManager is not enabled on Charon-SSP AWS, so a configuration file must exist to configure the interface properly).
   This numbering scheme may change in the future. Currently, it is based on the PCI slot on which the Ethernet controller is presented and which is incremented by one for each additional Ethernet interface added. On the Charon host system, the slot can be verified with the following command:
  # lspci -vv | grep -A20 Ethernet

 To avoid confusion before the instance can be restarted, the new interface can be renamed to its final name using the command 
     ip link set eth0 name ensY && ip link set ensY up

Host to Guest Communication Considerations

...

1. Internal virtual bridge on the host system:
   Such a bridge has several TAP interfaces. The host and the guest systems are connected to this bridge and can communicate directly to one another . Such a communication also allows L2 traffic and modifications to the MAC addressusing L3 and L2 protocols. The bridge uses its own IP subnet that can be defined by the user. Setting up such a configuration is supported by the Charon Manager.
2. Communication via the AWS subnet LAN:
   In this case, a second interface is added to the Charon host system. The second interface is then assigned to the emulated guest system. After the correct configuration, the host and guest can communicate across the AWS LAN using IP. L2 protocols or any protocols that require changing the MAC address to something different than the MAC address assigned to the second interface by AWS will not work.
   To connect the guest system to the LAN, the following basic configuration steps must be performed:
   • Add the additional interface to the Charon host system.
   • Create a configuration file for the additional interface.
   • Remove the private IP address assigned to the second interface by AWS from the Linux configuration (if it has been configured).
   • Use Charon Manager to assign the interface to the emulated SPARC system.
   • Use Charon Manager to set the MAC address of the emulated SPARC system to the same value as the one used on the host system Ethernet interface.
   • On the Solaris system, configure the private IP address that was previously assigned to the second interface on Linux and configure the appropriate default route for the LAN.
   • Additional steps may be required:
     • If the primary interface has an automatically assigned public IP address, this will be released after the first network restart with two interfaceswhen the instance is stopped and restarted. Hence the configuration must be changed to use a persistent Elastic IP address first to maintain reachability of the host system.
     • If both the primary and the second interface only have private IP addresses or both interfaces have public addresses (Elastic IP) assigned, separate routing tables must be created for both interfaces to enable proper routing.

The section Example of a More Complex Network Configuration provides some hints on how to configure the second interface in the different situations. Please refer to the AWS documentation for up-to-date comprehensive information.

If Layer 2 communication between guests on different Charon hosts is required, a bridged tunnel solution must be set up between the two Charon host systems.

External Communication Considerations

...

Recommended way to connect the Charon host and Solaris guest systems to the customer network:

To ensure data traffic between the Charon host and guest systems and the customer network is encrypted, it is strongly recommended to use a VPN connection. An example of a simple VPN connection based on an SSH tunnel is described in SSH VPN - Connecting Charon Host and Guest to Customer Network. This connection is based on a bridge between Charon host and guest system and (via an encrypted SSH tunnel) the remote end-point in the customer network. The connection supports L3 and L2 protocols.

AWS also provides a VPN gateway instance that can be added to the customer VPC to connect the VPC to the customer network (for a charge).

Recommended way to connect the Solaris guest system to the Internet:

...

Access to the Internet from private VPC subnets or a Solaris guest system with only private IP addresses:

Access to the Internet for private VPC subnets is possible across a gateway instance providing VPN access to the customer network and allowing (NATted) Internet access via this path. Alternatively, a NAT gateway in the cloud can be used to map the private addresses to public addresses. The NAT gateway can be implemented on a Charon host system or it can be provided by AWS for a charge.

Direct Solaris guest access to the Internet:

This is not a recommended as a standard solution for security reasons. Should However, should it be required, two interfaces with public IP addresses can be assigned to the Charon host.
One of these interfaces is then dedicated to to the guest system which uses the private address interface address assigned by AWS and the MAC address of the interface assigned address assigned to the Charon host by AWS (similar to point 2 in section Host to Guest Communication Considerations above).

Using a Charon host system as a Router

If a Charon host system is to be used as a router (for example as shown in Example of a More Complex Network Configuration or  or to provide Internet connectivity to other Charon host and guest systems), it is not sufficient to configure Linux for IP forwarding.

...

For each interface, the source/destination check has to be disabled. Unless this is configured correctly, traffic from and to and an AWS instance will only be allowed if either source or destination address belongs to the instance. Transit traffic destined to be forwarded by the router, would be discarded.

...

Should L2 protocols be required between two guest systems on different host systems, a bridge/tunnel solution similar to the one described in SSH VPN - Connecting Charon Host and Guest to Customer Network must be set up between the two host systems to allow the L2 traffic to pass.

Asymmetric Routing Considerations

This section applies to the case where several interfaces are configured on an instance and they all have IP addresses configured on the Linux level.

From the AWS documentation (https://aws.amazon.com/premiumsupport/knowledge-center/ec2-ubuntu-secondary-network-interface/):

"Adding a secondary network interface to a non-Amazon Linux EC2 instance causes traffic flow issues. These issues occur because both the primary and the secondary network interfaces are in the same subnet, and there is only one routing table with one gateway. Traffic that comes into the secondary network interface leaves the instance using the primary network interface. But this isn't allowed, because the secondary IP address doesn't belong to the MAC address of the primary network interface.

To make the secondary interface work, create a secondary network configuration file, configure the routing table, and then set up rules in the custom routing table policy database so that traffic for the secondary interface uses the new routing table."

 The above documentation only describes the required steps for Ubuntu. An earlier article for CentOS and Red Hat was removed from the AWS site. So the information presented here may change in the near future.

 Review the section about interface names if using an instance with enhanced networking enabled.

When adding a second IP interface (for example eth1) to the same subnet as the first on the Charon-SSP host, the routing problems described above can occur. To solve them, perform the following basic steps.

1. Create a configuration file (/etc/sysconfig/network-scripts/ifcfg-<interface-name>) for the second interface (if there is no configuration file for the primary interface, create it as well).
2. Set the correct interface for default route in /etc/sysconfig/network (example: GATEWAYDEV=eth0).
3. To prevent the cloud-init from resetting your custom network configurations, add the following lines to the /etc/cloud/cloud.cfg file:
   network:
  ; config: disabled
4. Restart the network.
5. Create an additional routing table (use the command: ip route add <path> dev <interface-name> table <table-id>). There must be an entry for every IP address assigned to the second interface and any other route to be used.
6. Set rules in the Routing Policy Database (use the command: ip rule add from <ip-address-of-second-interface> lookup <table-id>)
7. Create a static route file (/etc/sysconfig/network-scripts/route-<interface-name>)
8. Create a static rule file (/etc/sysconfig/network-scripts/rule-<interface-name>)

 Please refer to the Linux man pages for ip rule and ip route for more information. The AWS example for Ubuntu may also provide helpful hints.

Further Information

The following sections show sample network configurations: