> Packing everything together is a "selling point" until you find that a service can fill up ephemeral storage and take down other services, or consume bandwidth without limit.
Ephemeral storage has resource requests/limits in pods.
Traffic shaping/limiting can be accomplished using kubernetes.io/{ingress,egress}-bandwidth annotations. It's not as nice as resources (because there's not quotas and capacity planning, and it's generally very simplistic) but you can still easily build on this.
Pods can also have priorities and higher priority workloads can and will preempt lower priority workloads.
> Let's not forget the potential security implications of not keeping things properly isolated.
For hardware isolation isolation, you can use gVisor or even Kata containers.
> People who were around when provisioning on bare-metal was still a thing already learned all these lessons. Somehow it seems they have been forgotten by all the people driving hype around Kubernetes.
Kubernetes explicitly aims to solve resource isolation. It was built by people who have decades of experience solving this exact problem in production, on bare metal, at scale. Effectively, Kubernetes resource isolation is one of the best solutions out there to easily, predictably and strongly isolate resource between workloads _and_ maximize utilization at the same time.
Ephemeral storage has resource requests/limits in pods.
Traffic shaping/limiting can be accomplished using kubernetes.io/{ingress,egress}-bandwidth annotations. It's not as nice as resources (because there's not quotas and capacity planning, and it's generally very simplistic) but you can still easily build on this.
Pods can also have priorities and higher priority workloads can and will preempt lower priority workloads.
> Let's not forget the potential security implications of not keeping things properly isolated.
For hardware isolation isolation, you can use gVisor or even Kata containers.
> People who were around when provisioning on bare-metal was still a thing already learned all these lessons. Somehow it seems they have been forgotten by all the people driving hype around Kubernetes.
Kubernetes explicitly aims to solve resource isolation. It was built by people who have decades of experience solving this exact problem in production, on bare metal, at scale. Effectively, Kubernetes resource isolation is one of the best solutions out there to easily, predictably and strongly isolate resource between workloads _and_ maximize utilization at the same time.