I have found that managing dotfiles is not enough. The dotfiles serve no purpose without the software that uses them. My method is to write small scripts, I call them setuplets, that install the software and then symlink the dotfile to its master that I manage in git. In the simplest case, it is just a two line script in a directory, but I have one for each program, and a tool to select which I run when setting up a new linux machine. The more complex ones may install a number of tools and set up environment variables etc. For environment variables, the script drops a script file in a directory, where my bashrc sources it.
+1 for nix and home-manager. Most other solutions make it hard to manage installing packages on different flavors of Linux. You can get even more fancy by using the direnv integration to have specific versions of binaries available when entering a directory. See https://www.mathiaspolligkeit.de/dev/exploring-nix-on-macos/ . By checking in the nix files into your projects directory, you get really reproducible projects, because all you have to do is to checkout the project then run direnv allow and nix/home-manager will install all tools needed to run/build that specific projects. Different projects can use different versions of tools.
I used to do this for macOS and Linux but it was a fragile approach, since using Nix and NixOS I've had great success using home-manager[0] which takes care of what software is installed and any overrides or patches I desire.
I also use GNU stow in my dotfiles[1] especially so that my setup still works on systems without Nix installed.
I don't want to sound condescendant, but are you not basically doing packages ? Instead of standard packages your distro gives you, you have "custom" packages that also contain more specific installation instructions (write file with specific content, set environment variables, ...). I know that PKGBUILDs in Arch allow you to do such things if needed.
Bonus: now even configuration files can be linked to the software that needs it; when you remove a software you also remove all the files related to it and there are no leftovers.
Because package manager are different across different distro? Also, custom setup allows you to have those tools even if you don't have root permission to install them with package manager.
I've been logging each install as a markdown document with directions, and manually symlinking my dotfiles, but tiny scripts that do both are a great idea!
Yes, as soon as I have installed a program which I like and think is a "keeper", I write my setuplet script, usually by copying and pasting from the bash history.
I am not an expert in Ansible, or any other software of the kind, but I think this is one of the easier use-cases of such a program. I wrote an ansible script to compile emacs from source, after some initial configuration, and it serves me pretty well.
I started with Ansible for automation, and still use it, but have abandoned it for setting up my personal environment. I still use it to set up services, such as sshd, webserver, printer, samba, smartd etc. I found it was more cumbersome for personal environment setup than my script method. My method of re-initializing my environment rests on three pillars:
Ansible, for server/OS setup. Things run by init/systemd, involving anything outside of $HOME basically
Backup, using borgmatic. I use this to restore most things in $HOME, except random dotfiles. My documents, my checked out git repos, etc.
Script "setuplets". These I run on demand, to set up my environment piece by piece. Perhaps I do not want to restore my programming environment just because I want to have my custom prompt on a host, for example.
Finding the balance between these have been difficult. What bootstraps what, and especially, how to handle credentials. My backup is encrypted, but how would I make sure I had the keys to restore it? I could restore it from my pass password store, but how would I get the gnupg keys in place first? I have not solved this completely satisfactorily yet
May I recommend Makefiles? Nothing fancy, just a bunch of .PHONY targets. It's a converient way to bundle a bunch of scripts (or single arcane commands) into one file, with autocompletion.
I have a similar setup, but I don't have to write that script myself. Instead, I uses Zinit (https://github.com/zdharma/zinit) to let it setup the tools for me.
> My method is to write small scripts, I call them setuplets,
Finally I have a proper term for what I, too, have been doing all these years! :-) It's indeed the best-possible approach I've found, though there are a number of things that I haven't yet solved for myself in a satisfactory manner:
- With shell scripts there are no idempotency guarantees and there is no easy undoing / uninstalling / clean-up, especially after updating a setuplet.
- With shell scripts everything is defined imperatively as opposed to declaratively. In particular, setuplets usually operate on the filesystem directly and testing and dry runs are almost impossible.
- No status report as to what a setuplet wants to set up (software, configs, cronjobs…) and what is already set up on the current machine. That is, no diffs. This makes sharing setuplets and configs between multiple machines (say, personal and work laptop) rather cumbersome. For instance, I might forget to re-execute a setuplet on the second machine which could then lead to a missing software dependency or a mismatch between config and software.
- No simple, out-of-the-box way to have different configs/dotfiles for different machines, in particular: no config templating.
- It's hard to share common settings across applications without duplicating them everywhere. For instance, I would like to define a common set of colors / a common theme for my window manager, my terminal, my editor and so on. Similarly, (some) keybindings should be the same across applications. Moreover, I have a set of common directories in my home dir (for binaries, logs, cache etc.) that all my setuplets & dotfiles should use.
- Dependencies and interactions between setuplets are often implicit. They interact with and depend on one another through a myriad of ways, like software dependencies (of course) but also PATH modifications, cronjobs, bash aliases, file system modifications … These are very hard to recognize and, even worse, to refactor.
- Bash scripts are error-prone and cumbersome to write and debug (and refactor).
- My setuplets don't have a common command line interface and their relation to one another is unclear. (In which order should they get executed?) I tend to write scripts that invoke all the setuplets in the right order but it still seems messy and error-prone.
I've tried solutions like Ansible but I've found that its purely declarative DSL is not flexible enough to cover all my use cases in an elegant manner.
…which is why I'm currently working on a small Python library that will hopefully solve or at least ease the above pain points for me. Once the library is finished, I will rewrite my setuplets in Python (using the library to do the hard and tedious work), so that I end up with one single Python project of dotfiles and setuplets (exposed through one single command line tool) that, once executed, will automatically set up an entire machine for me within a few minutes. One nice thing would be that all inter-setuplet dependencies would be expressed through Python code (with proper typing, encapsulation in modules and everything) which could then easily be explored (and also refactored) with an IDE. Sure, this sounds like a lot of work but given that I intend to use my dotfiles for a couple more decades, it seems well worth it.
Of course, whether my approach will ultimately be able to solve all the challenges above remains to be seen but I'm at a point right now where I'm convinced that proper software engineering methods (especially dependency injection, type checks, tests etc.) would be a real boon for managing my (hundreds of) dotfiles.
Have you played with NixOS or Guix at all? They attempt to solve this problem from the ground up for the entire OS. It obviously has trade offs, but IMO it is the best solution around today (other than Kubernetes, but that is an abstraction level higher).
I really love nix and nixOS and I‘m also using homemanager on my macOS machine and an WSL instance. But I decided to turn away from it. Using just the nix packagemanager on a distribution feels off and doesn’t bring you all the benefits. On macOS it can be very frustrating to find a programm that is in the mix packages but not available for macOS. Plus some tools tend to be very outdated and use very general compile settings (light theme etc). I also have issues how libraries etc are linked. I have a rust project that for the love of god I could not compile on a arch installation with nix package manager managing all tools (rustup, git, etc). It was failing with some error around a standard c lib. It was working fine on a popOS installation with nix so I have no clue what went wrong where. The rabbit hole is very deep with nix. It is very very awesome but you also leave behind the conventional way of Linux when going all in. I played around with nixOS and really loved the idea and how easy it is to reconfigure the system from a single config. But if you don‘t want to write custom package overrides and dable with nix-expressions I would not use it as a dotfilemanager. If anyone knows a good way to integrate nix with arch that does not feel like a strapped on solution I‘m all open.
