A new flag -address is introduced to support using a DNS record as the control plane endpoint. Kube-vip will do a dns lookup to retrieve the IP for the DNS record, and use that IP as the VIP. An dnsUpdater periodically checks and updates the system if IP changes for the DNS record. Dynamic DNS Support (added in 0.2.1). This guide walks you through the installation of an external control plane. The external control plane deployment model enables mesh operators to install and manage mesh control planes on separate external clusters. This deployment model allows a clear.
-->- Contract: ControlPlane resource Most Kubernetes clusters require a cloud-controller-manager or CSI drivers in order to work properly. Before introducing the ControlPlane extension resource Gardener was having several different Helm charts for the cloud-controller-manager deployments for the various providers.
- One or More API Servers: Entry point for REST / kubectl. Etcd: Distributed key/value store. Controller-manager: Always evaluating current vs desired state.
APPLIES TO: SQL API Cassandra API Gremlin API Table API Azure Cosmos DB API for MongoDB
Control Plane in Azure Cosmos DB is a RESTful service that enables you to perform a diverse set of operations on the Azure Cosmos account. It exposes a public resource model (for example: database, account) and various operations to the end users to perform actions on the resource model. The control plane operations include changes to the Azure Cosmos account or container. For example, operations such as create an Azure Cosmos account, add a region, update throughput, region failover, add a VNet etc. are some of the control plane operations. This article explains how to audit the control plane operations in Azure Cosmos DB. You can run the control plane operations on Azure Cosmos accounts by using Azure CLI, PowerShell or Azure portal, whereas for containers, use Azure CLI or PowerShell.
The following are some example scenarios where auditing control plane operations is helpful:
You want to get an alert when the firewall rules for your Azure Cosmos account are modified. The alert is required to find unauthorized modifications to rules that govern the network security of your Azure Cosmos account and take quick action.
You want to get an alert if a new region is added or removed from your Azure Cosmos account. Adding or removing regions has implications on billing and data sovereignty requirements. This alert will help you detect an accidental addition or removal of region on your account.
You want to get more details from the diagnostic logs on what has changed. For example, a VNet was changed.
Disable key based metadata write access
Before you audit the control plane operations in Azure Cosmos DB, disable the key-based metadata write access on your account. When key based metadata write access is disabled, clients connecting to the Azure Cosmos account through account keys are prevented from accessing the account. You can disable write access by setting the disableKeyBasedMetadataWriteAccess property to true. After you set this property, changes to any resource can happen from a user with the proper Azure role and credentials. To learn more on how to set this property, see the Preventing changes from SDKs article.
After the disableKeyBasedMetadataWriteAccess is turned on, if the SDK based clients run create or update operations, an error 'Operation 'POST' on resource 'ContainerNameorDatabaseName' is not allowed through Azure Cosmos DB endpoint is returned. You have to turn on access to such operations for your account, or perform the create/update operations through Azure Resource Manager, Azure CLI or Azure PowerShell. To switch back, set the disableKeyBasedMetadataWriteAccess to false by using Azure CLI as described in the Preventing changes from Cosmos SDK article. Make sure to change the value of disableKeyBasedMetadataWriteAccess to false instead of true.
Consider the following points when turning off the metadata write access:
Evaluate and ensure that your applications do not make metadata calls that change the above resources (For example, create collection, update throughput, …) by using the SDK or account keys.
Currently, the Azure portal uses account keys for metadata operations and hence these operations will be blocked. Alternatively, use the Azure CLI, SDKs, or Resource Manager template deployments to perform such operations.
Enable diagnostic logs for control plane operations
You can enable diagnostic logs for control plane operations by using the Azure portal. After enabling, the diagnostic logs will record the operation as a pair of start and complete events with relevant details. For example, the RegionFailoverStart and RegionFailoverComplete will complete the region failover event.
Use the following steps to enable logging on control plane operations:
Sign into Azure portal and navigate to your Azure Cosmos account.
Open the Diagnostic settings pane, provide a Name for the logs to create.
Select ControlPlaneRequests for log type and select the Send to Log Analytics option.
You can also store the logs in a storage account or stream to an event hub. This article shows how to send logs to log analytics and then query them. After you enable, it takes a few minutes for the diagnostic logs to take effect. All the control plane operations performed after that point can be tracked. The following screenshot shows how to enable control plane logs:
View the control plane operations
After you turn on logging, use the following steps to track down operations for a specific account:
Sign into Azure portal.
Open the Monitor tab from the left-hand navigation and then select the Logs pane. It opens a UI where you can easily run queries with that specific account in scope. Run the following query to view control plane logs:
The following screenshots capture logs when a consistency level is changed for an Azure Cosmos account:
The following screenshots capture logs when the keyspace or a table of a Cassandra account are created and when the throughput is updated. The control plane logs for create and update operations on the database and the container are logged separately as shown in the following screenshot:
Identify the identity associated to a specific operation
If you want to debug further, you can identify a specific operation in the Activity log by using the Activity ID or by the timestamp of the operation. Timestamp is used for some Resource Manager clients where the activity ID is not explicitly passed. The Activity log gives details about the identity with which the operation was initiated. The following screenshot shows how to use the activity ID and find the operations associated with it in the Activity log:
Control plane operations for Azure Cosmos account
The following are the control plane operations available at the account level. Most of the operations are tracked at account level. These operations are available as metrics in Azure monitor:
- Region added
- Region removed
- Account deleted
- Region failed over
- Account created
- Virtual network deleted
- Account network settings updated
- Account replication settings updated
- Account keys updated
- Account backup settings updated
- Account diagnostic settings updated
Control plane operations for database or containers
The following are the control plane operations available at the database and container level. These operations are available as metrics in Azure monitor:
- SQL Database Created
- SQL Database Updated
- SQL Database Throughput Updated
- SQL Database Deleted
- SQL Container Created
- SQL Container Updated
- SQL Container Throughput Updated
- SQL Container Deleted
- Cassandra Keyspace Created
- Cassandra Keyspace Updated
- Cassandra Keyspace Throughput Updated
- Cassandra Keyspace Deleted
- Cassandra Table Created
- Cassandra Table Updated
- Cassandra Table Throughput Updated
- Cassandra Table Deleted
- Gremlin Database Created
- Gremlin Database Updated
- Gremlin Database Throughput Updated
- Gremlin Database Deleted
- Gremlin Graph Created
- Gremlin Graph Updated
- Gremlin Graph Throughput Updated
- Gremlin Graph Deleted
- Mongo Database Created
- Mongo Database Updated
- Mongo Database Throughput Updated
- Mongo Database Deleted
- Mongo Collection Created
- Mongo Collection Updated
- Mongo Collection Throughput Updated
- Mongo Collection Deleted
- AzureTable Table Created
- AzureTable Table Updated
- AzureTable Table Throughput Updated
- AzureTable Table Deleted
Diagnostic log operations
The following are the operation names in diagnostic logs for different operations:
- RegionAddStart, RegionAddComplete
- RegionRemoveStart, RegionRemoveComplete
- AccountDeleteStart, AccountDeleteComplete
- RegionFailoverStart, RegionFailoverComplete
- AccountCreateStart, AccountCreateComplete
- AccountUpdateStart, AccountUpdateComplete
- VirtualNetworkDeleteStart, VirtualNetworkDeleteComplete
- DiagnosticLogUpdateStart, DiagnosticLogUpdateComplete
For API-specific operations, the operation is named with the following format:
- ApiKind + ApiKindResourceType + OperationType
- ApiKind + ApiKindResourceType + 'Throughput' + operationType
Example
Control Plane
- CassandraKeyspacesCreate
- CassandraKeyspacesUpdate
- CassandraKeyspacesThroughputUpdate
- SqlContainersUpdate
The ResourceDetails property contains the entire resource body as a request payload and it contains all the properties requested to update
Diagnostic log queries for control plane operations
The following are some examples to get diagnostic logs for control plane operations:
Query to get the activityId and the caller who initiated the container delete operation:
Query to get index or ttl updates. You can then compare the output of this query with an earlier update to see the change in index or ttl.
output:
Next steps
In network routing, the control plane is the part of the router architecture that is concerned with drawing the network topology, or the information in a routing table that defines what to do with incoming packets. Control plane functions, such as participating in routing protocols, run in the architectural control element.[1] In most cases, the routing table contains a list of destination addresses and the outgoing interface(s) associated with each. Control plane logic also can identify certain packets to be discarded, as well as preferential treatment of certain packets for which a high quality of service is defined by such mechanisms as differentiated services.
Depending on the specific router implementation, there may be a separate forwarding information base that is populated by the control plane, but used by the high-speed forwarding plane to look up packets and decide how to handle them.
In computing, the control plane is the part of the software that configures and shuts down the data plane.[2] By contrast, the data plane is the part of the software that processes the data requests.[3] The data plane is also sometimes referred to as the forwarding plane.
The distinction has proven useful in the networking field where it originated, as it separates the concerns: the data plane is optimized for speed of processing, and for simplicity and regularity. The control plane is optimized for customizability, handling policies, handling exceptional situations, and in general facilitating and simplifying the data plane processing.[4][5]
Controlplanesecurityenabled Istio
The conceptual separation of the data plane from the control plane has been done for years.[6] An early example is Unix, where the basic file operations are open, close for the control plane and read write for the data plane.[7]
Building the unicast routing table[edit]
A major function of the control plane is deciding which routes go into the main routing table. 'Main' refers to the table that holds the unicast routes that are active. Multicast routing may require an additional routing table for multicast routes. Several routing protocols e.g. IS-IS, OSPF and BGP maintain internal databases of candidate routes which are promoted when a route fails or when a routing policy is changed.
Several different information sources may provide information about a route to a given destination, but the router must select the 'best' route to install into the routing table. In some cases, there may be multiple routes of equal 'quality', and the router may install all of them and load-share across them.
Sources of routing information[edit]
There are three general sources of routing information:
- Information on the status of directly connected hardware and software-defined interfaces
- Manually configured static routes
- Information from (dynamic) routing protocols
Local interface information[edit]
Routers forward traffic that enters on an input interface and leaves on an output interface, subject to filtering and other local rules. While routers usually forward from one physical (e.g., Ethernet, serial) to another physical interface, it is also possible to define multiple logical interfaces on a physical interface. A physical Ethernet interface, for example, can have logical interfaces in several virtual LANs defined by IEEE 802.1Q VLAN headers.
When an interface has an address configured in a subnet, such as 192.0.2.1 in the 192.0.2.0/24 (i.e., subnet mask 255.255.255.0) subnet, and that interface is considered 'up' by the router, the router thus has a directly connected route to 192.0.2.0/24. If a routing protocol offered another router's route to that same subnet, the routing table installation software will normally ignore the dynamic route and prefer the directly connected route.
There also may be software-only interfaces on the router, which it treats as if they were locally connected. For example, most implementations have a 'null' software-defined interface. Packets having this interface as a next hop will be discarded, which can be a very efficient way to filter traffic. Routers usually can route traffic faster than they can examine it and compare it to filters, so, if the criterion for discarding is the packet's destination address, 'blackholing' the traffic will be more efficient than explicit filters.
Other software defined interfaces that are treated as directly connected, as long as they are active, are interfaces associated with tunneling protocols such as Generic Routing Encapsulation (GRE) or Multi-Protocol Label Switching (MPLS). Loopback interfaces are virtual interfaces that are considered directly connected interfaces.
Static routes[edit]
Router configuration rules may contain static routes. A static route minimally has a destination address, a prefix length or subnet mask, and a definition where to send packets for the route. That definition can refer to a local interface on the router, or a next-hop address that could be on the far end of a subnet to which the router is connected. The next-hop address could also be on a subnet that is directly connected, and, before the router can determine if the static route is usable, it must do a recursive lookup of the next hop address in the local routing table. If the next-hop address is reachable, the static route is usable, but if the next-hop is unreachable, the route is ignored.
Static routes also may have preference factors used to select the best static route to the same destination. One application is called a floating static route, where the static route is less preferred than a route from any routing protocol. The static route, which might use a dialup link or other slow medium, activates only when the dynamic routing protocol(s) cannot provide a route to the destination.
Static routes that are more preferred than any dynamic route also can be very useful, especially when using traffic engineering principles to make certain traffic go over a specific path with an engineered quality of service.
Dynamic routing protocols[edit]
See routing protocols. The routing table manager, according to implementation and configuration rules, may select a particular route or routes from those advertised by various routing protocols.
Installing unicast routes[edit]
Different implementations have different sets of preferences for routing information, and these are not standardized among IP routers. It is fair to say that subnets on directly connected active interfaces are always preferred. Beyond that, however, there will be differences.
Implementers generally have a numerical preference, which Cisco calls an 'administrative distance', for route selection. The lower the preference, the more desirable the route. Cisco's IOS[8] implementation makes exterior BGP the most preferred source of dynamic routing information, while Nortel RS[9] makes intra-area OSPF most preferred.
The general order of selecting routes to install is:
- If the route is not in the routing table, install it.
- If the route is 'more specific' than an existing route, install it in addition to the existing routes. 'More specific' means that it has a longer prefix. A /28 route, with a subnet mask of 255.255.255.240, is more specific than a /24 route, with a subnet mask of 255.255.255.0.
- If the route is of equal specificity to a route already in the routing table, but comes from a more preferred source of routing information, replace the route in the table.
- If the route is of equal specificity to a route in the routing table, yet comes from a source of the same preference,
- Discard it if the route has a higher metric than the existing route
- Replace the existing route if the new route has a lower metric
- If the routes are of equal metric and the router supports load-sharing, add the new route and designate it as part of a load-sharing group. Typically, implementations will support a maximum number of routes that load-share to the same destination. If that maximum is already in the table, the new route is usually dropped.
Routing table vs. forwarding information base[edit]
See forwarding plane for more detail, but each implementation has its own means of updating the forwarding information base (FIB) with new routes installed in the routing table. If the FIB is in one-to-one correspondence with the RIB, the new route is installed in the FIB after it is in the RIB. If the FIB is smaller than the RIB, and the FIB uses a hash table or other data structure that does not easily update, the existing FIB might be invalidated and replaced with a new one computed from the updated RIB.
Multicast routing tables[edit]
Multicast routing builds on unicast routing. Each multicast group to which the local router can route has a multicast routing table entry with a next hop for the group, rather than for a specific destination as in unicast routing.
There can be multicast static routes as well as learning dynamic multicast routes from a protocol such as Protocol Independent Multicast (PIM).
See also[edit]
References[edit]
- ^Forwarding and Control Element Separation (ForCES) Framework, RFC 3746, Network Working Group, April 2004
- ^Do, Truong-Xuan; Kim, Younghan (2017-06-01). 'Control and data plane separation architecture for supporting multicast listeners over distributed mobility management'. ICT Express. Special Issue on Patents, Standardization and Open Problems in ICT Practices. 3 (2): 90–95. doi:10.1016/j.icte.2017.06.001. ISSN2405-9595.
- ^Conran, Matt (2019-02-25). 'Named data networking: Stateful forwarding plane for datagram delivery'. Network World. Retrieved 2019-10-14.
- ^Xia, Wenfeng; Wen, Yoggang; Heng Foh, Chuan; Niyato, Dusit; Xie, Haiyong (2015). 'A Survey on Software-Defined Networking'(PDF). Institute of Electrical and Electronics Engineers. 17 (1): 27–46.
- ^Ahmad, Ijaz; Namal, Suneth; Ylianttila, Mika; Gurtov, Andrei (2015). 'Security in Software-Defined Networks: A Survey'(PDF). Institute of Electrical and Electronics Engineers. 17 (4): 2317–2342.
- ^Do, Truong-Xuan; Kim, Younghan (2017-06-01). 'Control and data plane separation architecture for supporting multicast listeners over distributed mobility management'. ICT Express. Special Issue on Patents, Standardization and Open Problems in ICT Practices. 3 (2): 90–95. doi:10.1016/j.icte.2017.06.001. ISSN2405-9595.
- ^Bach, Maurice J. (1986). The Design of the Unix Operating System. Prentice-Hall.
- ^Configuring IP Routing Protocol-Independent Features, Cisco Systems,July 2006
- ^Nortel Ethernet Routing Switch 8600 Configuring IP Routing Operations, Nortel Networks, January 2007