OVERLAY(7) Standards, Environments, and Macros OVERLAY(7)
NAME
overlay - Overlay Devices
DESCRIPTION
Overlay devices are a GLDv3 device that allows users to create overlay
networks that can be used to form the basis of network virtualization
and software defined networking. Overlay networks allow a single
physical network, often called an
underlay network, to provide the
means for creating multiple logical, isolated, and discrete layer two
and layer three networks on top of it.
Overlay devices are administered through
dladm(8). Overlay devices
themselves cannot be plumbed up with
IP,
vnd, or any other protocol.
Instead, like an
etherstub, they allow for VNICs to be created on top
of them. Like an
etherstub, an overlay device acts as a local switch;
however, when it encounters a non-local destination address, it instead
looks up where it should send the packet, encapsulates it, and sends it
out another interface in the system.
A single overlay device encapsulates the logic to answer two different,
but related, questions:
1. How should a packet be transformed and put on the wire?
2. Where should a transformed packet be sent?
Each of these questions is answered by a plugin. The first question is
answered by what's called an
encapsulation plugin. The second question
is answered by what's called a
search plugin. Packets are encapsulated
and decapsulated using the encapsulation plugin by the kernel. The
search plugins are all user land plugins that are consumed by the varpd
service whose FMRI is
svc:/network/varpd:default. This separation
allows for the kernel to be responsible for the data path, while having
the search plugins in userland allows the system to provide a much more
expressive interface.
Overlay Types
Overlay devices come in two different flavors, one where all packets
are always sent to a single address, the other, where the destination
of a packet varies based on the target MAC address of the packet. This
information is maintained in a
target table, which is independent and
unique to each overlay device. We call the plugins that send traffic
to a single location, for example a single unicast or multicast IP
address, a
point to point overlay and the overlay devices that can send
traffic to different locations based on the MAC address of that packet
a
dynamic overlay. The plugin type is determined based on the type of
the
search plugin. These are all fully listed in the section
Plugins and their Properties.
Overlay Destination
Both encapsulation and search plugins define the kinds of destinations
that they know how to support. An encapsulation plugin always has a
single destination type that's determined based on how the
encapsulation is defined. A search plugin, on the other hand, can
support multiple combinations of destinations. A search plugin must
support the destination type of the encapsulation device. The
destination may require any of the following three pieces of
information, depending on the encapsulation plugin:
MAC Address An Ethernet MAC address is required to determine the
destination.
IP Address An IP address is required. Both IPv4 and IPv6 addresses are
supported.
Port An IP protocol level (TCP, UDP, SCTP, etc.) port is required.
The list of destination types that are supported by both the search and
encapsulation plugins is listed in the section
Plugins and their Properties.
varpd The varpd service, mentioned above, is responsible for providing the
virtual ARP daemon. Its responsibility is conceptually similar to ARP.
It runs all instances of search plugins in the system and is
responsible for answering the kernel's ARP-like questions for where
packets should be sent.
The varpd service, svc:/network/varpd:default, must be enabled for
overlay devices to function. If it is disabled while there are active
devices, then most overlay devices will not function correctly and
likely will end up dropping traffic.
PLUGINS AND PROPERTIES
Properties fall into three categories in the system:
1. Generic properties all overlay devices have
2. Properties specific to the encapsulation plugin
3. Properties specific to the search plugin
Each property in the system has the following attributes, which mirror
the traditional
dladm(8) link properties:
Name The name of a property is namespaced by its module and always
structured and referred to as as module/property. This allows
for both an encapsulation and search plugin to have a property
with the same name. Properties that are valid for all overlay
devices and not specific to a module do not generally use a
module prefix.
For example, the property
vxlan/listen_ip is associated with
the
vxlan encapsulation module.
Type Each property in the system has a type.
dladm(8) takes care of
converting between the internal representation and a value, but
the type influences the acceptable input range. The types are:
INT A signed integer that is up to eight bytes long
(
int64_t).
UINT An unsigned integer that is up to eight bytes long
(
uint64_t).
IP Either an IPv4 or IPv6 address in traditional string
form. For example, 192.168.128.23 or
2001:470:8af4::1:1. IPv4 addresses may also be encoded
as IPv4-mapped IPv6 addresses.
STRING A string of ASCII or UTF-8 encoded characters
terminated with a
NUL byte. The maximum string length,
including the terminator, is currently 256 bytes.
Permissions Each property has permissions associated with it, which
indicate whether the system considers them read-only properties
or read-write properties. A read-only property can never be
updated once the device is created. This generally includes
things like the overlay's encapsulation module.
Required This property indicates whether the property is required for
the given plugin. If it is not specified during a call to
dladm create-overlay, then the overlay cannot be successfully
created. Properties which have a
default will use that value
if one is not specified rather than cause the overlay creation
to fail.
Current Value The current value of a property, if the property has a value
set. Required properties always have a value set.
Default Value The default value is an optional part of a given property. If
a property does define a default value, then it will be used
when an overlay is created and no other value is given.
Value ranges Value ranges are an optional part of a given property. They
indicate a range or set of values that are valid and may be set
for a property. A property may not declare such a range as it
may be impractical or unknown. For example, most properties
based on IP addresses will not declare a range.
The following sections describe both the modules and the properties
that exist for each module, noting their name, type, permissions,
whether or not they are required, and if there is a default value. In
addition, the effects of each property will be described.
Encapsulation Plugins
vxlan The
vxlan module is a UDP based encapsulation method. It takes
a frame that would be put on the wire, wraps it up in a VXLAN
header and places it in a UDP packet that gets sent out on the
underlying network. For more details about the specific format
of the VXLAN header, see
vxlan(4P).
The
vxlan module requires both an
IP address and
port to
address it. It has a 24-bit virtual network ID space, allowing
for virtual network identifiers that range from
0 -
16777215.
The
vxlan module has the following properties:
vxlan/listen_ip Type:
IP | Permissions:
Read/Write |
Required The
vxlan/listen_ip property determines the IP address
that the system will accept VXLAN encapsulated packets
on for this overlay.
vxlan/listen_port Type:
UINT | Permissions:
Read/Write |
Required Default Value:
4789 | Range:
0 - 65535 The
vxlan/listen_port property determines the UDP port
that the system will listen on for VXLAN traffic for
this overlay. The default value is
4789, the IANA
assigned port for VXLAN.
The
vxlan/listen_ip and
vxlan/listen_port properties determine
how the system will accept VXLAN encapsulated packets for this
interface. It does not determine the interface that packets
will be sent out over. Multiple overlays that all use VXLAN
can share the same IP and port combination, as the virtual
network identifier can be used to tell the different overlays
apart.
Search Plugins
Because search plugins may support multiple destinations, they may have
more properties listed than necessarily show up for a given overlay.
For example, the
direct plugin supports destinations that are
identified by both an IP address and a port, or just an IP address. In
cases where the device is created over an overlay that only uses an IP
address for its destination, then it will not have the
direct/dest_port property.
direct The
direct plugin is a point to point module that can be used
to create an overlay that forwards all non-local traffic to a
single destination. It supports destinations that are a
combination of an
IP Address and a
port.
The
direct plugin has the following properties:
direct/dest_ip Type:
IP | Permissions:
Read/Write |
Required The
direct/dest_ip property indicates the IP address
that all traffic will be sent out. Traffic will be
sent out the corresponding interface based on
traditional IP routing rules and the configuration of
the networking stack of the global zone.
direct/dest_port Type:
UINT | Permissions:
Read/Write |
Required Default Value:
- | Range:
0 - 65535 The
direct/dest_port property indicates the TCP or UDP
port that all traffic will be directed to.
files The
files plugin implements a
dynamic plugin that specifies
where traffic should be sent based on a file. It is a
glorified version of /etc/ethers. The
dynamic plugin does not
support broadcast or multicast traffic, but it has support for
proxy ARP, NDP, and DHCPv4. For the full details of the file
format, see
overlay_files(5).
The
files plugin has the following property:
files/config Type:
String | Permissions:
Read/Write |
Required The
files/config property specifies an absolute path to
a file to read. The file is a JSON file that is
formatted according to
overlay_files(5).
General Properties
Each overlay has the following properties which are used to give
additional information about the system. None of these properties may
be specified as part of a
dladm create-overlay, instead they come from
other arguments or from internal parts of the system.
encap String | Permissions:
Read Only The
encap property contains the name of the encapsulation
module that's in use.
mtu UINT | Permissions:
Read/Write Default Value:
1400 | Range:
576 - 9000 The
mtu property describes the maximum transmission unit of the
overlay. The default value is
1400 bytes, which ensures that
in a traditional deployment with an MTU of 1500 bytes, the
overhead that is added from encapsulation is all accounted for.
It is the administrator's responsibility to ensure that the
device's MTU and the encapsulation overhead does not exceed
that of the interfaces that the encapsulated traffic will be
sent out of.
To modify the
mtu property, use
dladm set-linkprop.
search String | Permissions:
Read Only The
search property contains the name of the search plugin
that's in use.
varpd/id String | Permissions:
Read Only The
varpd/id property indicates the identifier which the
varpd service uses for this overlay.
vnetid UINT | Permissions:
Read/Write The
vnetid property has the virtual network identifier that
belongs to this overlay. The valid range for the virtual
network identifier depends on the encapsulation engine.
FMA INTEGRATION
Overlay devices are wired into FMA, the illumos fault management
architecture, and generates error reports depending on the
search plugin in use. Due to limitations in FMA today, when a single overlay
enters a degraded state, meaning that it cannot properly perform look
ups or another error occurred, then it degrades the overall
overlay pseudo-device driver.
For more fine-grained information about which overlay is actually in a
degraded state, one should run
dladm show-overlay -f. In addition, for
each overlay in a degraded state a more useful diagnostic message is
provided which describes the reason that caused this overlay to enter
into a degraded state.
The overlay driver is self-healing. If the problem corrects itself on
its own, it will clear the fault on the corresponding device.
SEE ALSO
vxlan(4P),
overlay_files(5),
dladm(8)illumos March 30, 2022 illumos