Why PhDs whiff the onsite, and how to find a diamond in the rough

Lessons from hiring 100+ data scientists

Throughout my (Duncan’s) career, I’ve interviewed or advised hundreds of new PhDs trying to break into data science in tech. One year at Uber, I even hired five new Harvard economics PhDs — about 20% of the graduating class.

New PhDs can be total amateurs when it comes to the job market. They’re going from the academic world — where their papers and CV build an encyclopedic, multi-year, portable track record of their accomplishments — to a new reality where a single 45-minute interview can decide their fate. They’re often wildly unprepared. No research. No practice. With a straight face, they’ll say things like, “Why do you want to hire someone with my skills?”, “I didn’t expect a coding interview!”, or even“I’d prefer to do research; tech is my backup”.

When I gave some tips to job seekers on LinkedIn recently, my network knew exactly what I meant. Data science leaders chimed in with comments like:

• “I once had a rookie PhD candidate who, when [asked] what sector they were targeting for their first job after school, said proudly ‘Academia, tech, litigation consulting, nonprofits, and maybe government.’... Needless to say, the job market didn't treat the unfocused candidate well. “

• “One of my favorites is ‘I am interviewing here because the academic job market is tight right now’. I still hired that person, and they were great — they just shouldn't have said that in an interview!”

Today I’m going to extend that post, but from my point of view as hiring manager: Knowing these candidates are going to say some silly things — sometimes unintentionally — how can you separate the wheat from the chaff?

What to look for during data science interviews

It’s important to remember that the point of the interview isn’t to find someone good at interviewing — rather to find someone who will be good at the job. Most PhDs won’t be good at interviewing! So over the years, I’ve identified a few red flags — and green ones — to look for.

1. Excitement to learn all parts of the job

Ideally, your candidate would have proven abilities in every bullet point on your job description. But PhDs tend to be very spiky in a single dimension, as many spend six years focusing on learning basically one thing. So it's not realistic that they'll be good at everything. The key thing is that they show high interest in learning.

Take writing quality code. It’s a necessary part of the job but often a new pursuit when coming from academia. During the interview, look for green flags: they’ve taken a computer science course, are familiar with git, show awareness of concepts like scaling and latency. Maybe they even know big O notation.

Conversely, if the candidate shows no familiarity with coding, asks no follow-up questions, or flat-out tells you “I don’t code”, that’s a big red flag. After all, tech companies are in the business of building technology.

Pro tip for any candidates reading this: read the job description before your interview. It might look like boilerplate, but it really is what the company is looking for. Then show excitement about all those dimensions!

2. Evidence of multidisciplinary thinking

Data science requires the ability to think about problems from many angles. Your candidate may not have mastery of all of these disciplines, but they should demonstrate the ability to reason about what they don’t know, and discuss where they will go to work through a problem.

• Statistics — how am I going to analyze this distribution?

• Operations research — how am I going to optimize this model?

• Computer Science — how am I going to do this in a performant way?

• Economics — will this model have complex effects on behavior?

A mistake I often see is that someone with a math background wants to optimize something with no awareness of the statistical complexity of the data (e.g. which could be super noisy). When I point this out, do they get it and take a step back? Or do they have a hammer for which they desperately need to find a nail?

Pro tip for candidates: multidisciplinary thinking is hard to build overnight. My advice is to take classes in an uncomfortably wide array of disciplines, so that you build this muscle before you need it.

3. Navigating multiple levels of altitude

The best data scientists operate fluidly at different altitudes. They zoom out and summarize the big-picture business problem, then zoom in and talk about the architecture of a specific component — and then make sure that component solves the big picture problem!

Take surge pricing at Uber. I might look for evidence that candidates understood why the problem mattered in the first place — how having the wrong number of riders relative to drivers would break the product, and why surge pricing was a huge innovation over how taxis worked. Then we’d go deeper — discussing the architecture we’d need to deliver surge pricing models. And finally, we’d  get into the weeds, talking through the requirements and subtleties of building a specific component.

For many PhDs, their natural state is in the nitty-gritty — obsessing over the finer points of a model without demonstrating an understanding of or interest in the macro view. If you sense a candidate will have a hard time stepping back to engage with the big picture, that's likely to be a big problem.

Pro tip for candidates: this is a muscle that can be trained fairly quickly. I’ve found the best approach is to practice case-study thinking, which forces you to rapidly work through the basics of a wide variety of business problems. You’ll find this muscle will be highly valuable in both  interviews and on the job. The go-to book to crank through is Case in Point.

4. Burning to get things done

It’s hard to overstate how different the pace of the tech world is from grad school. In academia, students are often used to years-long research and publication cycles; in tech they’ll need to accommodate much faster cycles of iteration — months or even weeks. So you need to find people who tolerate long research cycles simply because that was the only way — but whose natural state is vastly faster. These are people who have a need for speed.

I like to dig into this by asking them about how they manage their day to day in grad school. Are they putting in the hours, thinking about the next thing to do? How do they balance quality vs. quantity?

You can learn a bit of this from their CV: a lengthy CV can demonstrate their ability to put out substantial amounts of quality work. That’s someone who will get it done for your business — so don’t let them slip past just because they didn’t crush the interview.

Pro tip for candidates: this one is easy. Make it clear that you like to go FAST!

5. Bonus: public speaking superpowers

Some PhDs have a powerful skill you may not be accustomed to looking for in non-DS functions: teaching and presentation. These folks are used to teaching classes and presenting their work in front of audiences.

This is (surprisingly) not typically part of interview loops, but translates to tech in super useful ways. How often are data scientists asked to explain models to stakeholders, or give a presentation on the latest metrics? (All the time.)

Look for candidates’ experience in and passion for teaching, as well as teaching awards, which are a strong signal of excellence. You might have a candidate give a short presentation on their research — this has the side benefit of giving the candidate warm fuzzies, since it looks like you are interested in their research.

Pro tip for candidates: If this is you, make sure to highlight it on your resume and speak to it in interviews — presenting work to stakeholders is a key part of the DS job.

What to do upfront

There’s some upfront work you can do to set your candidates — and yourself — up for success.

1. Do your homework

Different disciplines (statistics vs. computer science vs. economics, etc.) tend to function very differently, with different job markets, and special failure modes — e.g. economists are notorious for failing catastrophically on live coding interviews. Whenever possible, find somebody from that field to help educate you so you can better understand your candidates’ backgrounds and context.

2. Prep your candidates

Especially if your candidates are brand new to the job market, they’ll benefit from extra hand-holding prior to the interview. Send over the job description, set specific expectations around technical interviews, and share any blogs or internal documentation that will give them a clearer picture of what this kind of work actually entails. Emphasize that the live interview is make-or-break, and that prep really makes a difference.

3. Connect with schools and universities

PhD students can be shockingly disconnected from the corporate world — and may have no idea your company is an option for them. At Uber, there was surprisingly high value in going on roadshows to local schools like Stanford and Berkeley and across the country to Harvard and MIT to actively recruit grad students. We found many had no idea what a statistics PhD at a company like Uber does. Forge those connections yourself through active recruiting and marketing with universities. Reach out to career centers, conduct virtual or live talks, and plan in-person visits when possible to meet with students.

Especially in a competitive hiring environment, this can be a highly effective way to find some diamonds in the rough.