spec: define pod lifecycle

[jonboulle]

We need to address several aspects of pod lifecycles in the spec:

• Garbage collection lifecycle of the pod filesystem including:
  • Format of app exit codes and signal
  • The refcounting plan for resources consumed by the ACE such as volumes
• Lifecycle of apps within the pod including:
  • All exit or first to exit
  • Start and stop order
  • Can a pod's apps be updated while the pod is running?

[jonboulle]

Another aspect to this, which has come up in various discussion channels, is the relationship between the pod's lifecycle and identity. What is a pod's identity, and to what extent should it be immutable?

Should a pod be considered a resource envelope (similar to the alloc in Google's Borg), which individual applications can be added to or removed from? Or is a pod's identity more tightly linked to its constituent applications and their respective lifetimes; i.e. if a single application dies within a pod, or it is desired to stop one of the applications within a pod, then it is necessary to tear the entire pod down and create it again? (Coming back to the old pets vs cattle metaphor - the pod/resource-envelope that can be mutated at-will is the pet, the pod which must be destroyed + recreated is the cattle.)

Opening the door to things like "updating an application within a pod" fundamentally throws into question the whole idea of identity (by updating or stopping/starting every application in the pod in turn, we can easily end up with an entirely new set of applications in the pod, so what is really left to link it back to its original identity?).

I am personally in strong favour of leaning towards the cattle side of the equation and retaining the pod as the fundamental scheduling unit; it is inherently easier to reason about, implement, and we don't need to define a complicated "inter-pod" API/ABI (i.e. what operations are permissible for individual apps vs on the pod as a whole). But there are practical arguments for allowing at least some operations to performed at a greater level of granularity.

/cc @thockin who has plenty of thoughts on this, /cc @vbatts

[kelseyhightower]

@jonboulle Will there be support for just containers without pods?

[jonboulle]

@kelseyhightower please elaborate
Am 27.04.2015 11:04 schrieb "Kelsey Hightower" notifications@github.com:

@jonboulle https://github.com/jonboulle Will there be support for just
containers without pods?

—
Reply to this email directly or view it on GitHub
#276 (comment).

[vbatts]

per chat, I think as pods become like templates, there ought to be a bullet that covers pod discovery and signature. At this point, they are seeming like their own *.acp or similar. Which would have a minimum a /pod manifest.

[jonboulle]

@vbatts not to derail this thread too much (since I'd consider that a separate issue), but the nice part about that direction is that the spec then addresses the use case of related efforts like nulecule - it clearly fulfils those listed goals:

Provide a simple, flexible way to describe a multi-container application, including all dependencies.
Provide a way for an application designer to describe an application while allowing a sysadmin a clear way to parameterize the deployment at runtime.
Provide a versioned specification for developer tools and runtime implementations to agree on.

[vbatts]

As for the cattle versus pet, one question is the content addressed image requirement in the pod manifest, and largely duplicating the image manifest components. There may need to be a floating aspect to it. For instance, I may want to enforce that I'm using image foo.com/bar from identity foo.com, maybe not a precise build.

[xiang90]

Should a pod be considered a resource envelope (similar to the alloc in Google's Borg), which individual applications can be added to or removed from?

From our discussion at #google-container around 4:22 PDT, a pod is a fixed resource envelope and a spec. alloc (a flexible resource envelope) is too flexible and all most everyone just wants to have a one-to-one mapping from a task(pod) to a alloc.

if a single application dies within a pod, or it is desired to stop one of the applications within a pod, then it is necessary to tear the entire pod down and create it again

In pod, there is a restart policy. The lifecycle of a pod should not be the same as a container inside it. So I guess if there is a container dies and needs a restart, we should only restart the container without tearing down the pod. (so in our spec, we should also support this use case?)

If the pod spec is changed, we should tear down the entire pod and restart all.

[thockin]

A whole bunch of topics. It's hard to say what are right answers and what are simply answers that we chose before.

Identity. Kubernetes pods have a mortal identity - once a pod is bound to a node, that pod's UID (GUID) is burned and never ever reused. If that pod dies and is restarted on the same node it is the same pod (same UID). If that pod is no longer viable on that node, for whatever reason, the pod is destroyed and a new one is created with a new UID. Now, this is somewhat different than what some people expect, and the point has been debated in recent weeks, but I believe fundamentally that this is the correct model.

Add/remove apps from a container. This is something we naturally support in Borg because of the Alloc abstraction, but we do not support in Kubernetes (but only by punt, not by intent). I could see an argument for not supporting add/remove of containers, but it has to consider...

Updates. If you can't update a pod's (or apps within that pod) resource requests without a restart, you've blown it. That is so fundamental to how we operate at scale that we would grind to a halt without it. Once you accept that, it's an easy walk to seeing how much you can support without a restart. We get a lot of value from in-place updates, and we go to great lengths to make all updates as minimally invasive as possible. One of the things that Kubernetes does differently from Borg is that replicas of a pod are not tightly coupled in a grouping abstraction. This means that, once birthed, a Pod is free to live its own life. One replica can get more memory while another gets less. They can be manipulated independently, but they are considered fungible from an administrative point of view (replication controller).

So now - can you update the version of a container that an app-in-pod is running (e.g. v1 -> v2)? Obviously that requires restarting the one app. Does it require killing the pod? Hopefully not. If not, can you update the actual image name that an app-in-pod is running (e.g. nginx -> haproxy). If so, that not the same as adding and removing? There's a line to draw somewhere, but it's really not an obvious line. Sometimes you need to allow things you think are a bad idea and let policy be set by higher levels of the stack.

so what is really left to link it back to its original identity?

Is this really an important question? Just because something is treated like cattle does not mean you can't individually care for the cattle. What harm comes of this evolution over time?

If the pod spec is changed, we should tear down the entire pod and restart all.

You might get away with that for now, but it should not be a resting state. There are lots of things about a pod that should not require app restarts to change.

[kelseyhightower]

@thockin Would I be correct in thinking the fact Docker does not implement pods contributed to the success of pods in Kubernetes? I could image a world where every container runtime implemented pods in a different way, which would make it really hard for Kubernetes to manage containers at the granularity you've outlined above.

[thockin]

Kelsey, an interesting take on it. Docker gave us primitives to build
higher abstractions. If Docker gave us similar but different pods, we'd
probably have to abstract or adapt.

On Mon, Apr 27, 2015 at 7:07 PM, Kelsey Hightower notifications@github.com
wrote:

@thockin https://github.com/thockin Would I be correct in thinking the
fact Docker does not implement pods contributed to the success of pods in
Kubernetes? I could image a world where every container runtime implemented
pods in a different way, which would make it really hard for Kubernetes to
manage containers at the granularity you've outlined above.

—
Reply to this email directly or view it on GitHub
#276 (comment).

[xiang90]

@thockin

Updates. If you can't update a pod's (or apps within that pod) resource requests without a restart, you've blown it. That is so fundamental to how we operate at scale that we would grind to a halt without it.

Does not update simply mean a restricted remove and add? The image name is not the identification of the container statically or dynamically, right? So this restriction only enforces the total number of containers in the original pod in reality.

They can be manipulated independently

This means the same pod (in the view of when it get scheduled and replicated) might diverge and it is by design?

There's a line to draw somewhere, but it's really not an obvious line.

Once we allow the update, then the line is super unclear. Right?

[jonboulle]

well, let's not lose track of the fact that the whole point of the spec is
to define this stuff and try encourage consistency in implementations :-)

On Mon, Apr 27, 2015 at 10:37 PM, Tim Hockin notifications@github.com
wrote:

Kelsey, an interesting take on it. Docker gave us primitives to build
higher abstractions. If Docker gave us similar but different pods, we'd
probably have to abstract or adapt.

On Mon, Apr 27, 2015 at 7:07 PM, Kelsey Hightower <
notifications@github.com>
wrote:

@thockin https://github.com/thockin Would I be correct in thinking the

fact Docker does not implement pods contributed to the success of pods in
Kubernetes? I could image a world where every container runtime
implemented
pods in a different way, which would make it really hard for Kubernetes
to
manage containers at the granularity you've outlined above.

—
Reply to this email directly or view it on GitHub
#276 (comment).

—
Reply to this email directly or view it on GitHub
#276 (comment).

[thockin]

Updates. If you can't update a pod's (or apps within that pod) resource requests without a restart, you've blown it. That is so fundamental to how we operate at scale that we would grind to a halt without it.

Does not update simply mean a restricted remove and add? The image name is not the identification of the container statically or dynamically, right? So this restriction only enforces the total number of containers in the original pod in reality.

No, update here means "adjust the cgroup settings, don't touch the
running processes". Why would you kill my process just to increase
the memory limit? Even decreases can be done safely, though slowly.

They can be manipulated independently

This means the same pod (in the view of when it get scheduled and replicated) might diverge and it is by design?

Yes. Particularly in things like resources. You could maybe draw the
line at version updates. It seems like a reasonable starting place.
Don't underestimate the value of uptime. Once people have it, they
don't want to give it up.