This is actually something I've been playing with. Containers/VMs managed by a daemon with lifecycles that an agent can invoke sessions on and execute commands in, using OPA/Rego over gRPC. The cherry on top is envoy for egress with whitelists and credential injection.

One cool thing is that you can run a vscode service on these containers and open the port up to the outside world, then code in and watch a project come to life.