KubeCon + CloudNativeCon North America 2017

Exposing internal HTTP-based services to the Internet is a well supported and documented feature of Kubernetes. What's less well understood is how to do it for thousands of services running on behalf of hundreds of possibly competing customers, in particular how to do it securely, protect the privacy of each customer, and support binary protocols other than HTTP. This is the problem that our company solved for our SaaS business which requires hosting and operating the control plane of popular infrastructure management software (e.g. Openstack, Big Data, and Kubernetes itself) as a service for our customers. Those control planes contain services exposing protocols as varied as MySQL and AMQP. This talk describes the challenges we faced and how we solved them using multiple technologies from the Kubernetes ecosystem. The solution includes a system that automatically creates namespaces, provisions certificate hierarchies, and manages ingress controllers for new customers, then wraps services with a set of side-car containers to handle tasks such as TLS termination. We describe how we employed Kubernetes native constructs such as Custom Resource Definitions to automate those tasks. For network communications, we discuss how to securely handle ingress, outgress, pod-to-pod, and cross-namespace traffic. To support both HTTP and TCP-based protocols, we describe a two-level network routing system consisting of both a "k8sniff" and an nginx ingress controller. For ensuring customer data privacy we compare these approaches: (1) Network Policy + Layer 2 virtualization; (2) TLS encryption of all pod-to-pod traffic; (3) a combination of the two. Finally, we debate whether the process isolation model of Linux containers is sufficient, and discuss our experience with stronger virtualization-based mechanisms such as Frakti / HyperContainer.

Kubernetes provides the ability to extend the platform with your own custom types and controllers. We will walk through a tutorial to write a custom controller, also known as an operator. Patterns will be reviewed that will make your application a natural extension of the platform through CRDs and desired state management, all with the same security, lifecycle management, and API surface that native Kubernetes applications expect.

Kris Nova (Heptio) and Robert Bailey (Google) join forces and begin the difficult task of looking into the future of the infrastructure layer of Kubernetes. We start the talk with a brief summary of the state of infrastructure today and explain the differences between “infrastructure as code” and “infrastructure as software”. We look at how the lack of definition in the most fundamental layer of the stack has fragmented our community and caused problems with adoption of Kubernetes.

We propose a new way of representing infrastructure (the cluster API) for the Kubernetes community and take a deep dive into its implementation in kubicorn. We look at the structure of the cluster API and share valuable insight on how we took lessons from other areas of Kubernetes to form what it is today. Furthermore we look at the power of having a declarative approach to infrastructure as we start to treat the infrastructure layer the same as the application layer.

The audience will walk away with a clear understanding of the infrastructure layer, as well as a new way of thinking about the infrastructure in the future via the cluster API.

The serverless paradigm is bringing a new type of applications to the forefront of application architecture. Distributed, containerized, scalable, event-driven and ephemeral with fine grained billing. In this talk we will go through several application use-cases that are driving the serverless movement (e.g data processing, IoT, mobile-backends,machine learning) and demonstrate how these applications can be developed and deployed on top of Kubernetes using an open source serverless solution called kubeless. Through live demos and examples, we will show that Kubernetes with its rich and stable core API is the perfect platform to build FaaS solutions.

Database as a Service is one of the most interesting and challenging domains on the cloud industry. In eBay, we implemented a cloud-native geo-distributed document service based on the kubernetes. eBay extended the kubernetes to support local disk volume on bare metal machine, which enables the high performance DB can be deployed on the kubernetes as a Pod. On top of the kubernetes platform, we develop a control layer to orchestrate the databased pods and enable it can be distributed on multiple cluster, and expand the WISB model to use a workflow to auto manage the database cluster.

Solutions to enable customers to apply consistent traffic management and security policies regardless of whether their applications are running on-premises, in a Public Cloud or in a managed Kubernetes environment.


The demos (please below) use Equinix as the cloud interconnection; however, the solution would work very similarly with other colo/interconnect providers.
The demos are using Google Container Engine but this would work very similarly in AWS, Azure and Oracle Baremetal Cloud.

Demo
------------
How it works
---------------------
The F5 Container Connector configures the F5 BIG-IPs to expose applications in a Kubernetes cluster as virtual servers, serving North-South traffic.

Components
---------------------
F5 Container Connector (http://clouddocs.f5.com/containers/v1/kubernetes/)
running in Google Container Engine (GKE).

F5 BIG-IPs running in Equinix and doing IPSec VPN to Google Cloud Platform (using Google Compute Engine VPN gateway).
The F5 BIG-IP routes traffic to the container networks via the IPSec tunnel.

The reality of multiple Kubernetes deployments typically leaves you with varied cluster profiles, deployed on a mix of on-prem and public cloud environments. Production ops for large distributed systems is hard enough in a single environment, but becomes even more complex with hybrid conditions.

In this talk, we’ll dissect how to leverage federation for Kubernetes governance across capacity management, micro service dependencies, infrastructure upgrades, versioning, and security, as well as, global high availability, continuity, and resiliency, in a hybrid environment.

kubeadm is the Kubernetes tool that helps you set up a Kubernetes cluster quickly and easily. kubeadm is different from other Kubernetes setup tools in that it doesn’t assume or depend on any special infrastructure. It assumes that you have one or more machine available and those machines can connect to each other via the network.

The master plan is to make kubeadm work both as the “fast path” to getting a best-practice Kubernetes cluster with a couple of easy-to-remember commands and as a toolbox for higher-level solutions like GKE, kops and Tectonic.

But how does kubeadm actually set up a cluster? How is it so easy to add a node with the Bootstrap Token? How does it self-host the control plane? How does it upgrade clusters smoothly with only one command? What is the plan for achieving HA without relying on any external infrastructure?

After this talk, you will be able to describe how:

• kubeadm runs the different tasks in different stages
• the network traffic between the cluster components flow
• self-hosting of the control plane works
• the Bootstrap Token works
• the `kubeadm upgrade` command works
• kubeadm will support multiple masters that are dynamically rotated
• you can extend kubeadm to build your higher-level Kubernetes deployment tool

Kubernetes has a growing array of security controls available, but knowing where they all fit in, what the highest priorities are, and how it all helps against real attacks is still far from obvious. In this talk we’ll take a vulnerable application, exploit it, install tools, escalate privileges, propagate between containers and gain control of the cluster. At each stage of the attack we’ll demonstrate how proactive steps could have prevented these actions (or at least made them more difficult), from the container build process to writing RBAC/PodSecurity/AppArmor/Network policies, and more. Since configuration of each defence could be the subject of it’s own deep-dive talk, we’ll mainly focus on the big picture of “what” technologies you’d use to configure your cluster securely and “why”.

In recent releases, we've enabled node admission and configuration APIs that eliminate configuration requirements for Kubernetes workers. This allows cluster operators to add and remove nodes from clusters without a configuration management tool driving the process. This fully automate node management behavior allows physical data centers to be much more cloud-like and lights-out.

In this session, we'll run this process as a demo and decompose the various parts that must work together for success. We'll discuss the specific APIs and how to implement them in a coordinated way that ensures node security and minimizes workload disruption. We'll also discuss how to improve node security by using trusted platform modules (TPM). By the end of the session, operators will be able to duplicate the steps on their own to learn the process.

While we have a focus on bare metal infrastructure for this session, the lessons learned are equally usable on cloud infrastructure.

In this talk we will demonstrate an approach to management of kafka clusters in kubernetes deployments. We will show how we can provision kafka clusters and configure it using kubernetes concepts and an operator process. The kafka and zookeeper cluster elements will be provisioned using StatefulSet. As these applications benefit from high performance storage, we will also show how we can use node selectors or persistent volume claims to schedule instances on correct hardware. In order for clients to use it, the necessary message topics have to be configured in kafka cluster. We will show how using an operator process, based on kubernetes custom resources or ConfigMaps we can manage this configuration in descriptive manner and ensure consistent configuration across different development and operations stages as well as cluster restarts. Finally we will discuss how all this ties in with service catalog.

With our adoption of Kubernetes at Capital One, we have simultaneously reduced our application delivery time-to-market while providing a common platform for streaming pipelines. We leverage Kubernetes to manage stateful decisioning applications for multiple tenants and provide a host of analytical tools as platform services to help data scientists iteratively improve decision models. We will discuss the challenges in operating these pipelines which consist of Apache Nifi canvases/flows for data ingress/egress, Kafka as persistent stream backbone, Flink for decisioning, and a number of other popular open source data analytics packages such as Apache Drill and Zeppelin forming our “Analytical Environment.”

This talk will focus on the recent changes & challenges in Kubernetes to address the need for consistent & secure access to local persistent storage and raw block storage.