There was a major industry change in technical interviewing practices after Google hit it big. They very publicly optimized their process to minimize false positives even at the expense of a high rate of false negatives. This included live coding and "brain teaser" type questions. Google was then wildly successful and so people in the industry assumed that their interviewing process was one of the reasons why. So a lot of other tech companies superficially copied the Google process in a "cargo cult" approach.

And I'm not claiming that the old Google approach was necessarily wrong or bad (I understand their current process has significantly changed but don't know the details). As an industry we still haven't figured out what the best practice should be. Everyone is still guessing. Every company seems to think they have an excellent hiring process but there are no real controlled experiments or hard data in that area. Who knows?

Let's not forget it became a business. Gayle Laakmann wrote a book, became a consultant, and I'm sure earned a whole lot of money convincing companies that she'd found the perfect path to hiring great engineers.

I think she had a willing audience because a lot of companies weren't sure they were interviewing the 'right' way. It's always easier to tell your bosses you are following the advice of a top consultant than to try to tell them why you have a better strategy than the FAANGs.

AFAIK Gayle sold her book and services as a way to train for interviews, not conduct them. The problem is the companies, not her.

Exactly! She was the one instrumental in publicizing this shitty way of hiring. Or course she could care less about the damage caused because she cashed in, actually still is.

> Everyone is still guessing.

This isn't actually true, as the article points out; there is actually tons of empirical research.

No one is capable of measuring programmer productivity objectively.

You can, if you have clear objectives.

Nah. It's impossible, unless you define "productivity" in such a narrow way that it ceases to be relevant to actual business value creation.

You just defined it, "business value creation", from there, you narrow it down, short term vs long term, strategic vs revenue generating, and so on.

The idea that you can't measure productivity is align with people who think "economics is not science."

Nah. It's impossible to link business value creation to individual contributors. No one has ever figured out how to do this in a accurate, reliable, and repeatable way. When it comes to knowledge work, productivity can only be measured in aggregate for larger groups.

Most of economics is junk science. Very little of it is quantitatively rigorous or falsifiable in the real world.

I think it’s possible some of the time for some projects to measure how much business value an IC created.

But I also think there is no company with more than 50 software engineers employees that is able to accurately attribute business value creation to the majority of their software engineers.

Software engineers rarely pick what they work on, so in most cases you can only measure how well they meet the objectives they were assigned to work on.

And even then you can only get any kind of accuracy at the team level.

And those numbers aren’t comparable between teams because teams aren’t assigned to work on the same things.

People who think they can accurately measure things like velocity will bring out arguments based on the law of large numbers. But that only works when the number of samples is much larger than the number of variables, which is not the case for software engineering.

You can believe that economics is science. But still believe that it’s impossible to figure out how much GDP growth is attributable to each individual member of congress.

> Software engineers rarely pick what they work on, so in most cases you can only measure how well they meet the objectives they were assigned to work on.

There it goes, your other metric.

> People who think they can accurately measure things like velocity will bring out arguments based on the law of large numbers. But that only works when the number of samples is much larger than the number of variables, which is not the case for software engineering.

You can't accurately measure any coastline, but we don't stop at that.

>There it goes, your other metric.

1. It’s impossible to measure that except in aggregate

2. The only thing you can actually measure is how soon did it launch because the quality of the thing is too wrapped up in what it’s supposed to do.

How many bugs were there? That’s heavily dependent on how complicated the problem it’s solving.

The absolute best thing anyone has come up with is velocity, which just measures how good a team is at estimating. That also only works at the team level and it’s only accurate for teams repeatedly doing similar work that haven’t changed in composition for years.

> You can't accurately measure any coastline, but we don't stop at that.

We actually can accurately measure the relative lengths of coastlines such that we can compare them by picking a measurement resolution and using that for all of them.

You can’t do that here. It doesn’t work.

If we could accurately map business value creating back to ICs, tech companies would look very very different.

Even if you could objectively measure how good an IC is at meeting objectives. The developers I’ve met who were best at that metric tended to be terrible at their jobs in general because they’d just take anything product wanted and build it exactly as described.

So you want some amount of pushback. But not too much or they just end up not doing anything.

Try to come up with a metric for that. To do that though it needs to be normalized by how good the PM is at designing features. If the PM was terrific, you wouldn’t want pushback.

So now you have to rate all your PMs before you can rate your ICs. Then you need to rate the PMs bosses and so on. Until you have an uncomputable mess.

The best you can do is have enough EMs so they can get a feel for how ICs are doing. You can also probably expose the absolute worst performing engineers with metrics because they’re just literally doing nothing. But even then, you’ll get some false positives.

Virtually none of that research is actually high-quality, reproducible, and correlated with organizational outcomes. I don't see it as really actionable one way or the other.

Exactly this for me the last couple of years. Interviewing at most companies these days results in me getting through the final rounds only to be presented with “feedback” about all the secret things they really wanted that I didn’t meet the bar for. The feedback is often extremely vague but is always a list of shortcomings and rarely a list of strengths. The companies that do provide strengths are the ones I feel bad about missing.

Interviews aren't always confrontational. I have been using the same pair programming exercise for roughly the last hundred interviews. There are no tricky algorithms, just walking through the implementation of a basic rest endpoint. It's cooperative and I want the candidate to ask questions. If you can code at fizzbuzz level, are comfortable writing tests, and know a little about databases, it's easy.

There are probably great programmers out there that think pairing is horrible and stressful and this is the interview from hell. I accept that, but I also think that being able to pair and communicate is an important job skill. I want a team, not brilliant lone wolves.

The pair programming thing works great as an actual work sample test. I think it works best when neither person knows the answer though.

If one of you already knows what they want to see, it’s not really pair programming.

Either way though your process is already better than 90% of companies.

For the most part I think this is right though I'm not opposed to a quick screener. It reminds me more of the traditional engineering interview, which is more about how you think. Knowledge is good, but it usually isn't hard to obtain. But the way you think? That's much harder to change. I'd love to optimize for both, but if I have to pick I'd rather have someone who has a good engineering mind. Seems even more important these days, because if I just wanted to optimize for knowledge I'd just use an LLM and RAG.

My last three jobs all did pair programming so we structured our interviews to be similar, and they’re still my favorite interviews I’ve conducted. Many candidates also praised the format, whether they were hired or not.

This is how I've conducted a few interviews at a startup. I take pains to emphasize that:

1. I'm just looking for pseudocode, nobody cares about whether you do length(items) or items.size(), etc. The code won't even be run.

2. Invent functions without necessarily defining them, I'll object if doAllTheWork() needs to be fleshed out.

3. The problem/docs presented are the whole thing for the interview. There might be bugs to uncover, but there's no secret second phase to the boss battle.

Ultimately, I'm looking for them to assemble the basic building blocks, and see what suggestions they have when I point out issues like error, handling, stale data, security, etc.

This really resonates with me. It's hard to describe just how enjoyable and more importantly interesting tech interviews used to be. I can't think of one in the past decade that's left an impression on me. Now interviews feel like I'm playing Trivial Pursuit with a very non-technical George Costanza, trying to convey years of experience verbally when the card says Moops.

> This resonates. I'm now in my early 60s. Oddly, in my 20s and 30s I could interview reasonably well. But as time went on, I think the interviewing styles became more confrontational.Whereas earlier on they were trying to find reasons to hire you, later on it was more that they were trying to find reasons not to hire you. Possibly this is somewhat attributable to ageism, but I think it was a change in the industry as well.

At one point in my past, I was planning to do a career change over to Investment Banking (I know, I know... don't laugh), so I interviewed at a bunch of banks. These guys are notorious about how annoying and uncomfortable they make their interviews. They'll deliberately grief you and try to throw you off your game to see how you react, soft-skill-wise, under stress. Like they'll ask you a question about calculating a discounted cash flow, and then while you're answering they'll make a phone call to someone, just to see how you deal with disrespect.

While tech interviewing hasn't gone to those extremes, I definitely agree we seem to be moving that direction: interviewers seem to be deliberately looking for ways to stress/haze the candidate, for whatever reason. Something I never experienced interviewing in tech < 2005 or so.

> interviewers seem to be deliberately looking for ways to stress/haze the candidate, for whatever reason.

Probably caused by a tight market where there's greater pressure to filter a larger pool of candidates, and company cultures have worsened due to economic pressures.

That sounds something like the infamous interviews that Admiral Hyman Rickover conducted for the US Navy Nuclear Reactors program. He deliberately wanted to weed out officers who couldn't perform under intense pressure (although it's impossible to know whether that was really an effective approach).

https://www.usni.org/magazines/naval-history-magazine/1992/m...

> earlier on they were trying to find reasons to hire you, later on it was more that they were trying to find reasons not to hire you

Have you by chance been pursuing roles at increasingly larger and more lucrative orgs?

I've worked at several startups, and those were clearly more looking for reasons to hire. Now at a FAANG, the interview process was clearly more in the looking for reasons not to hire direction...

Pretty sure he's talking about roles that pay close to seven figures TC, not a paper money startup.

> I think it was a change in the industry as well.

It's very true, I did not interview from about 2006 to 2014, and had no problems before, but had a noticeable feeling of "wtf is going on?" after this (and still do feel that way tbh).

It's the money. It's all about the money. Things are pretty low stress and low stakes when you're hiring for a 50k pure salary role. When things blew up into the mid six figures with stock incentives, it became a whole thing. This pervades the work environment too. Everyone is scared to death of saying the wrong thing or messing something up or sounding stupid because it means your 800k mortgage won't be getting paid next month. I hate it.

Your 800k mortgage not getting paid is one thing, but when there's a partner and kids who need money to eat, it's a whole other level.

Throw in that your health insurance tends to be tied to your job in the US.

Employers hate just as much giving 500k to an employee who does not deliver.

Nowadays it's the same poor interview when they are hiring for 20k pure salary overseas.