I need to get serious about doing all development inside a virtual machine. One project per VM. There are just too many insidious ways in which I can ignorantly slip up such that I compromise my security. My only solace is that I am a nobody without secrets or a fortune to steal.
IDEs, plugins, development utilities, language libraries, OS packages, etc. So much code that I take on blind faith.
Vagrant’s popularity seems to have died down with Docker containers but it’s by far my favorite way to make dev environments.
Several years ago I worked somewhere that prohibited web browsers and development tools on laptops. If you needed to use a browser, you’d have to use one over Citrix. If you needed to code, you’d use a VDI or run the tools in a VM.
At the time I thought their approach was clinically insane, but I’m slowly starting to appreciate it.
I still like Vagrant. But I believe it's yet another victim of the Hashicorp license change debacle from a year or two ago.
Unlike with Terraform/OpenBao, I know of no community effort effort to keep the open-source version of this project alive. The latest open source version is still available on the Ubuntu repo, but who knows who long it will work until somefor of bit rot occurs.
> I still like Vagrant. But I believe it's yet another victim of the Hashicorp license change debacle from a year or two ago.
The license change is irrelevant - from the licensing page:
> All non-production uses are permitted.
Devs who use Vagrant in a development environment can do it as they used to do it before.
> The latest open source version is still available on the Ubuntu repo, but who knows who long it will work until somefor of bit rot occurs.
Hashicorp products have always been intended to be downloaded from the website, since they're statically linked binaries (I don't like that they're huge, but matter of factually, they make distribution trivial).
A customer of mine still uses vagrant on a project, for local development. That project started in 2016. We are developing on a mix of Linux, Mac, Windows and it's not as straightforward as it could be. Linux is easier, Windows is messier.
A newer project fires up VMs from a Python script that calls an adapter for EC2 (with the boto library) when run on AWs and for VirtualBox (by calling VBoxManage) when running locally. That allows us to simulate EC2 locally: it's a project that has to deal with many long jobs so we start VMs for them and terminate the VMs when the jobs are done. That also runs better on our mix of development systems. WSL2 helped to ease the pains of developing on Windows. We call the native Windows VirtualBox, not the one we could have installed inside WSL2, but we keep most of the code that runs on Linux.
Devcontainers[1] are the new incarnation of this pattern. We use them at work and they are a dream for onboarding new developers. The only downside is the VSCode lock-in but if that's a concern there's always DevPod[2].
It looks like the team behind it have been moving it towards more of an open standard over the last year. There's now a CLI reference implementation, and the Jetbrains IDE's have an implementation for it.
There's also a thread for Zed about a path to implementing it there [0]. Hopefully it'll become a bit more common over 2025.
At my first job almost 10 years ago we had the concept of "X-in-a-box" using Vagrant + VMs and I miss that pattern so much ever since (multiple job skips later).
None of my jobs since have had any semblance of a better way to set up a local dev environment easily.
It was just way easier to encapsulate services or other things in a quickly reproducible state.
I started using Ansible a few years back to set up VMs (or Raspberry Pis) with a consistent environment. Once I wrapped my head around it, I've found it very nice for any situation where I need to treat systems as livestock rather than pets.
Vagrant is still kicking! But yeah not as popular as back in 2014-2016?
A hybrid(?) alternative is enroot, which is pretty neat IMO, it converts a docker container into a squashfs file that can be mounted rw or used in an ephemeral way. https://github.com/NVIDIA/enroot
The real problem is video performance in VMs. It still just...kind of sucks. Running Cinnamon in a VM is just about impossible to get GL acceleration working properly.
nvidia gates it's virtualized GPU offerings behind their enterprise cards, so we're left with ineffective command translation.
IMO: I can tolerate just about every other type of VM overhead, but choppy/unresponsive GUIs have a surprisingly bad ergonomic effect (and somehow leak into the performance of everything else).
If we could get that fixed, at least amongst Linux-on-Linux virtualization, I think virtualizing everything would be a much more tenable option.
There are ways around it. There is a community of people who use Nvidia enterprise cards with vGPU for gaming, performance is excellent, or PCI pass through an entire GPU.
If you can't do that because it's for company/corporate purposes then I can sympathise with not wanting to pay Nvidia's prices.
You can get good security without virtualization, for example SeLinux and namespaces in Linux. Jails in BSD and zones in Solaris. We would have many viable and competing solutions if it wasn't for Microsoft monopoly.
But would it matter much for development? Either SSH into the VM and use vi/emacs or use an IDE/editor with remote support. VS Code even lets you use a container as a development environment (I know, not a VM by default):
I don't know about VS Code's dev containers extension but the SSH extension's README says:
> Using Remote-SSH opens a connection between your local machine and the remote. Only use Remote-SSH to connect to secure remote machines that you trust and that are owned by a party whom you trust. A compromised remote could use the VS Code Remote connection to execute code on your local machine.
> When a user installs an extension, VS Code automatically installs it to the correct location based on its kind. If an extension can run as either kind, VS Code will attempt to choose the optimal one for the situation;
It's horrible that trust is being eroded so much, and seeing monthly GB updates to my OS doesnt reassure me at all. I like the idea of having a stable isolated VM for each project. Are there standard open-source tools to do this?
Specifically I'm transitioning my Go and Zig development environments from an old mac to an M1 with Asahi Linux and getting a bit lost even finding replacements for Truecrypt and Little Snitch. Do these VM tools support encrypted VM's with firewall rules? I saw Vagrant mentioned here and that sounds like it might cover the network isolation, but what else would you suggest?
I run all my dev environments under LXD. Even the IDE: full graphical Emacs (or Vim) over X11 forwarding over SSH. Host is Wayland, so security concerns with X are handled. WayPipe also works, but is jankier than X, probably because X, unlike Wayland, was designed for network transparency.
LXD, unlike Docker, doesn't play fast-and-loose with security. It runs rootless by default, and I don't allow non-root access to the LXD socket on host. Each container is a full userspace, so it's much more convenient to configure and use than Dockerfiles.
SSH from a container to a remote works transparently because I forward my SSH Agent. This is secure because my agent confirms each signing request with a GUI popup (on host).
Can you point to a write-up somewhere that details this setup?
Part of the appeals of VMs is that they were built with security as a primary objective. I probably have to do something stupid to break that isolation. A custom ad hoc configuration makes me a bit nervous that I will unknowingly punch a Docker sized hole through my firewall and have less security than if I ran a stock workflow.
For me, I don't use LXD, but use Proxmox containers. These are non-root Linux containers by default. Super lightweight compared to a VM. Proxmox makes managing LXC containers a little easier with a UI, compared to managing containers strictly using command line.
If you go this route, create a container template that has everything you want in every instance. And then spin out new containers whenever you need one.
I always used to do that, using Vagrant. Mostly because it was the only practical way to maintain independent environments for the tools I was using.
These days I work in JavaScript and rarely have issues with project environments interfering with each other. I've gotten lazy and don't use VMs anymore.
In theory docker type setups could work but they just seem so much effort to learn and setup.
Seconding vagrant - especially because it's the only reasonable way I found so far to test linux release on my windows rig (would prefer to dev on linux, but windows-only company is windows-only company).
Basically I put a Vagrantfile in src folder, then run docker compose with db, caddy, app server and other services inside it - then I forward ports 80 and 443 from vm and use localhost.whateverdomain.igot with self-signed cert on caddy (since https is just enough different than http that I otherwise get bitten by bugs every so often).
When I start a new project I can usually just copy the Vagrantfile with minimal changes.
I know where you are coming from and I considered this myself again and again. For me and for now it is not something I want to do and not primarily because of the effort.
The VM might protect me, but it will not protect the users of the software I am producing. How can I ship a product to the customer and expect them to safely use it without protection when I myself only touch it when in a hazmat suit?
No, that is not the environment I want.
My current solution is to be super picky with my dependencies. More specifically I hold the opinion that we should neither trust projects nor companies but only people. This is not easy to do, but I do not see a better alternative as for now.
i develop on linux, on various projects. i'm mostly concerned with all the tools, build scripts and tests that may read sensitive data, or accidentally destroy data. so i'm limiting access to files when working on a project with linux namespaces, using bubblewrap.
i've got a simple per-project dot file that describes the file system binds. while i'm working on a project, new terminals i open are automatically isolated to that project based on that dot file. it has very low (cognitive) overhead and integrates pretty much seamlessly. i suspect many developers have similar scripts. i looked for projects that did this some time ago, but couldn't find it. either because it's too simple to make a project about, or because i don't know how others would describe it. if anyone has pointers...
i don't limit network access (though i did experiment with logging all traffic, and automatically setting up a mitm proxy for all traffic; it wasn't convenient enough to use as regular user).
there is still a whole kernel attack surface of course. though i'm mostly concerned about files being read/destroyed.
I actually did try to install Qubes over the holiday, but I repeatedly encountered installation failures and could not ever login to the system. Someone had posted an identical issue, but they were similarly stymied. I should revisit, but my initial foray tells me I am going to have to withstand quite a few papercuts in order to get the isolation I want.
never had issues with qubes like that but i did pick something tested (hw). u can check hardware compat list. it has also some good links to forums for specific hw related tweaks u might need.
that being said, runing qubes fully and workin with it is something else... i decided i am uninteresting enough just to use ubuntu these days :p... maybe sometime ill have the patience again.
I think a lot of the issues in this particular example is the ease with which api keys, once leaked, are single factor passwords.
If you ran a key logger on my machine you would never get into any major site with mfa. You couldn't watch me log on to the azure console with passkey and do much with it. But if you scrape a saved key with publish abilities bad things happen.
What's to stop me from installing custom certs and MITM your login session proxying the info. Or an extension to harvest the data after you login. I'm pretty sure if I have root it's game over one way or another. The surface is massive.
At that point you've done something much more invasive and detectable than exporting a .env file and you've walked away with a very short lived token. There's always "something more an attacker can do", I'll stand by the view that requiring further authentication to perform interactive actions and pushes is worthwhile.
I wonder how this is mitigated by my current workflow of running jupyter and vscode from a docker container.
I did not start doing this because of security, but just to get something more or less self managed without any possibility to break different projects. I am tired of my team spending too much time on extensions, versions, packages, ...
Docker compose files have saved our team many hours, even if it's extremely wasteful to have multiple vscode instances running alongside each other
I started doing development under a separate non-admin user on my MacBook. I switch to another user for personal stuff, or the admin user to install stuff with Homebrew. Doesn't protect from zero days but it's better than nothing.
I toyed around with this a bit, and it feels like it has significant merit. User separation is about the only security boundary built into Linux from the beginning. I was not totally happy with the workflow I adopted, but it is probably going to be less burdensome than the VM approach.
With Fast User Switching on macOS it's pretty convenient too. The difficulty is remembering to switch user when changing contexts. I tried to set a different wallpaper/icon for each user to make it more obvious which user I'm on, but macOS just resets them all to be the same.
I guess so… there seems to be absolutely no consequences to getting hacked, so from a business perspective it makes a lot of sense.
It's not up to me to decide what policy to use, and if it was I couldn't just do whatever I wanted, I'd have to justify its cost. And every company does the same…
The security, and overall application stability attack vector, is why I now vouch for processes with OS IPC instead of shared libraries, even if it requires more resources.
It doesn't fully sort out the trust issue though, even if everything is sandboxed in some fashion.
I know of IPC, but it has never occurred to me to view as an alternative to shared libraries. It's an intriguing viewpoint I'm having trouble wrapping my mind around. Are there battle-tested real-life examples of IPC being used where shared libs could have been used instead?
All the stuff using Android intents, out-proc COM extensions in Windows, XPS in macOS, are other relevant set of examples.
I assume you are kind of new to the computing world, OS IPC is how we extended applications almost 40 years before shared libraries became common feature across all major operating systems.
Naturally with them being around, shared memory in-process was much easier, and less resource intensive. IPC calls require processes, which take more kernel resources, and context switch.
Microservices isn't a new concept, rather re-branding.
Sun had as marketing quote, "The network is the computer", exactly because of how it used to be.
I wonder how long until this is standard, and PFYs coming into the industry look at our current practices much like people now look at non-encrypted credentials being sent over the network.
It is good to wind down every now and then during work time.
Also, many people here work on multiple projects for different customers. Having a security breach for one affecting the other is not something you'd be happy with.
I never said anything about personal stuff on a work machine? I want my own hardware to have isolation between my email/banking/etc and side project programming.
IDEs, plugins, development utilities, language libraries, OS packages, etc. So much code that I take on blind faith.