Private Cluster

Private Clusters #

Features #

  • VPC Network Peering

Private clusters require VPC Network Peering. The two VPC networks (Masters and Nodes) are connected using VPC Network Peering.

  • Private Google Access

Private nodes do not have outbound Internet access because they don’t have external IP addresses. For example, Private Google Access makes it possible for private nodes to pull container images from Google Container Registry, and to send logs to Stackdriver.

Requirements #

  • VPC-native

A private cluster must be a VPC-native cluster, which has Alias IP Ranges enabled. VPC-native clusters are not compatible with legacy networks.

  • Kubernetes version 1.8.14-gke.0 or later

The nodes in a private cluster must run Kubernetes version 1.8.14-gke.0 or later.

Limitations #

  • The size of the RFC 1918 block for the cluster master must be /28.

Access to the master #

  • Private endpoints

Any VM in your VPC network, and in the same region as your private cluster, can use the private endpoint.

  • Public endpoints

This is the external IP address of the master. You can configure access to the public endpoint.

Determining cluster endpoints # 

gcloud container clusters describe [CLUSTER-NAME] \
    --zone=[ZONE] | --region=[REGION] \
    --format="get(privateClusterConfig.privateEndpoint)"

gcloud container clusters describe [CLUSTER-NAME] \
    --zone=[ZONE] | --region=[REGION] \
    --format="get(privateClusterConfig.publicEndpoint)"

Access to the cluster endpoints #

Public endpoint access disabled #

Client access to the master’s public endpoint is blocked. Access to the master must be from internal IP addresses.

--enable-ip-alias
--enable-private-nodes
--enable-private-endpoint
--enable-master-authorized-networks

Public endpoint access enabled, master authorized networks enabled #

Restricted access to the master from both internal and external IP addresses that you define

--enable-ip-alias
--enable-private-nodes
--enable-master-authorized-networks

Public endpoint access enabled, master authorized networks disabled #

Access to the master from any IP address.

--enable-ip-alias
--enable-private-nodes
--no-enable-master-authorized-networks

Using an automatically-generated subnet # 

gcloud container clusters create private-cluster-0 \
    --create-subnetwork name=my-subnet-0 \
    --enable-master-authorized-networks \
    --enable-ip-alias \
    --enable-private-nodes \
    --master-ipv4-cidr 172.16.0.0/28 \
    --no-enable-basic-auth \
    --no-issue-client-certificate

At this point, these are the only IP addresses that have access to the cluster master:

The primary range of my-subnet-0.

The secondary range used for Pods.

Using a custom subnet #

Create a network

gcloud compute networks create my-net-1 \
    --subnet-mode custom

Create a subnet and secondary ranges

gcloud compute networks subnets create my-subnet-1 \
    --network my-net-1\
    --region us-central1 \
    --range 192.168.0.0/20 \
    --secondary-range my-pods-1=10.4.0.0/14,my-services-1=10.0.32.0/20 \
    --enable-private-ip-google-access

Create a private cluster

gcloud container clusters create private-cluster-1 \
    --zone us-central1-c \
    --enable-master-authorized-networks \
    --enable-ip-alias \
    --network my-net-1 \
    --subnetwork my-subnet-1 \
    --cluster-secondary-range-name my-pods-1 \
    --services-secondary-range-name my-services-1 \
    --enable-private-nodes \
    --master-ipv4-cidr 172.16.0.16/28 \
    --no-enable-basic-auth \
    --no-issue-client-certificate

At this point, these are the only IP addresses that have access to the cluster master:

The primary range of my-subnet-1

The secondary range my-pods-1.

Creating a private cluster with no client access to the public endpoint #

Your cluster master is accessible only from within your VPC network. You cannot access the master at all from outside your VPC network.

gcloud container clusters create private-cluster-2 \
    --create-subnetwork name=my-subnet-2 \
    --enable-master-authorized-networks \
    --enable-ip-alias \
    --enable-private-nodes \
    --enable-private-endpoint \
    --master-ipv4-cidr 172.16.0.32/28 \
    --no-enable-basic-auth \
    --no-issue-client-certificate

Viewing the cluster’s subnet and secondary address ranges # 

gcloud compute networks subnets list --network [NETWORK]
gcloud compute networks subnets describe [SUBNET_NAME]

Notes about secondary ranges #

You can create five secondary ranges in a given subnet. For each cluster, you need two secondary ranges: one for pods and one for services. This means that you can create only two clusters that use a given subnet.

Other private cluster configurations #

Granting private nodes outbound Internet access #

If you want to provide outbound Internet access for your private nodes, you can use Cloud NAT or you can manage your own NAT gateway.

Troubleshooting #

Cluster overlaps with active peer #

  • Error

Google Compute Engine: An IP range in the peer network overlaps with an IP range in an active peer of the local network.

  • Potential causes

You chose an overlapping master CIDR.

  • Resolution

Delete and recreate the cluster using a different master CIDR.

Can’t reach master #

  • Potential causes

kubectl is unable to talk to the cluster master.

  • Resolution

You need to add authorized networks for your cluster to whitelist your network’s IP addresses.

Can’t create cluster due to overlapping master IPv4 CIDR block #

  • Symptoms

gcloud container clusters create returns an error such as The given master_ipv4_cidr 10.128.0.0/28 overlaps with an existing network 10.128.0.0/20.

  • Potential causes

You specified a master CIDR block that overlaps with an existing subnet in your VPC.

  • Resolution

Specify a CIDR block for –master-ipv4-cidr that does not overlap with an existing subnet.

Can’t create subnet #

  • Symptoms

When you attempt to create a private cluster with an automatic subnet, or to create a custom subnet, you might encounter the following error: An IP range in the peer network overlaps with an IP range in one of the active peers of the local network.

  • Potential causes

The master CIDR range you specified overlaps with another IP range in the cluster. This can also occur if you’ve recently deleted a private cluster and you’re attempting to create a new private cluster using the same master CIDR.

  • Resolution

Try using a different CIDR range