Rushabh Doshi

Ship week at Hiro: MCP, Portfolio, and Import 2.0

Rushabh Doshi — Fri, 13 Mar 2026 18:08:03 -0700

Hiro is not a financial advisor. It is an AI powered financial assistant. Any responses should not be considered financial, tax, or legal advice.

The Hiro team has had an epic ship week. We launched two new features and completely redid a critical data import flow. One feature for agents, another for humans, and the third to keep all your financial data clean and usable. Without much further ado, let’s get into what these are and why they are worth your time playing with.

I don’t charge for my newsletter. If you like my writing, I’d appreciate it if you 1) try Hiro 2) send us feedback 3) spread the word.

MCP Server#

We launched an MCP server! Super easy to set up if you’re into MCPs, works with Claude, Codex, Openclaw, whatever your client choice is.

This launch will really matter to readers who are already on the forefront of experimenting with OpenClaw and Codex and Claude. What the Hiro MCP server provides is an up-to-date snapshot of your financial data, secured away in Hiro. Why not just give Claude a bunch of your CSVs? You absolutely can, but then you’ll have to repeat the process next month and the month after, and remember how you categorized each transaction, and that the random credit union account has something strange with polarities. We handle all this for you and continue to handle it.

So what can you do? Let me illustrate with a few examples: Our COO Samar Shah is one of the most AI-pilled execs I know. He vibed up a bunch of apps with his MCP connection. You can see his post here. Pictures speak a thousand words, so here are some ideas from his post:

Customized cashflow dashboard#

Everyone has a different view of their ideal cashflow dashboard because we think about life differently. Samar made his own that fits his own mind.

Position solar system#

This is for pure fun - but who hasn’t wondered what their portfolio looks like as a solar system? (We are huge 🤓s at Hiro) So here you go

Iran impact#

My partner in crime Ethan took things in a different (more serious) direction. He wanted to know the impact of the Iran conflict on his portfolio. He technically did this without MCP by manually creating spreadsheets of his data (all of this pain is now gone), but the analysis that Claude generated is quite fantastic.

Iran Conflict by Ethan Bloch

Investment Impact Analysis

Read on Substack

What will you build?#

In all cases, the key is that the MCP server provides cleaned, normalized access to your data, that continues to update itself in the background and lets your agents / claws / Claudes go to town and do whatever analysis they want. I’d love to see what you build - if its exciting and you want us to feature it, please let us know.

Portfolio tab#

What if you follow the MCP stuff but aren’t that into it, and would prefer a more normal human interface to things? Well, we have a fresh new update for you. Announcing our Portfolio tab.

Why this matters:

Checking in on your net worth is often the number one thing that people want to do on a somewhat frequent basis. As humans, we just want to know what that number is and whether it is moving up or down. We want to know why. The portfolio tab is designed to help you understand that. We are still in early days, so we don’t have advanced tools like backtesting and sensitivity analysis, but those are coming. If you’re excited about this, give it a go, tell us what we’re missing and we’ll build it.

Import flow part 2#

Financial data is extremely messy. That’s just an unfortunate reality. If you’re a lay customer, you probably have some financial statements or CSV dumps of data. If you use Hiro to connect your financial accounts via Plaid or other providers, you start seeing data going forward, but historical data is limited. Moreover, for transactional data, you may have spent a lot of time categorizing transactions, and cleaning up your data and that is effectively gone when you establish a new connection.

To solve this problem, we built an import flow that supported arbitrary PDFs, and imports from Monarch and Mint. It was good, but not great. We were doing a bunch of client side processing and the flow was fragile. If you ended up in a bad state, your account was toast. There was no undo.

We have now solved these problems (and more). Our import flow is completely undoable - go through the import, it doesn’t work, no problem, just hit a button and it never happened. As far as we know, almost no other provider supports this, though it is absolutely critical in case of a mistake. Moreover, we’ve replaced our old fragile CSV parser approach with an AI agent that can deal with any sort of CSV. So for friends using Personal Capital - go nuts - we support you now.

Bonus#

Three features is not enough. We’ve historically had a lot of trouble with some banks and financial institutions connecting via Plaid. We now support Finicity and MX as alternative bank connectors. Our institution picker does its best to direct you to the ideal option, but you can always override it if you want.

I’m very excited about what we are shipping and building at Hiro and the direction we’re headed in. Please give Hiro a shot and send us any feedback you may have.

Cognitive Debt

Rushabh Doshi — Tue, 17 Feb 2026 22:08:45 -0800

After using many claudes non-stop for a month, I feel stupider. Not in a good way - I am not awed by the sheer brilliance of super intelligent coding agents - but in a way where I find myself becoming increasingly lazy, deciding not to put in the cognitive work required to build a computer system of any meaningful complexity, and looking for quick “productivity hits” rather than outcomes that result from slow, meaningful labor. I heard about the term Cognitive Debt from Simon Willison’s blog which led to Margaret-Anne Storey’s article which led to a research paper by Kosmya et al., 2025.

Your brain on ChatGPT (arXiv)#

To explore the cognitive effects of using ChatGPT, the authors chose a narrow task - SAT style essay writing - and created 3 groups of 18 students: those that could use ChatGPT, those that could only use a search engine, and those that had nothing. They then measured the quality of the essays and brain activity using EEGs. Quality is difficult to measure, so they used a few different approaches to triangulate things.

The conclusions of the paper match my lived experience using Claude Code. Quoting directly from the discussion section:

Taken together, the behavioral data revealed that higher levels of neural connectivity and internal content generation in the Brain-only group correlated with stronger memory, greater semantic accuracy, and firmer ownership of written work. Brain-only group, though under greater cognitive load, demonstrated deeper learning outcomes and stronger identity with their output. The Search Engine group displayed moderate internalization, likely balancing effort with outcome. The LLM group, while benefiting from tool efficiency, showed weaker memory traces, reduced self-monitoring, and fragmented authorship.

Computer systems vs. computer programs#

When we were early in our careers, studying computer science, most of our work involved computer programs, usually in the context of homework assignments. We had no burden of maintenance or bug fixing or evolving our programs once the next feature came along.

Going from a student to professional, or even early- to mid- career, required closing the gap from programs to systems. Computer programs, taken together, formed computer systems and gave rise to an immense amount of complexity, that cannot be captured by any single program. In many ways, this reminds me of Stephen Wolfram’s work on cellular automata, and complexity emerging from a simple series of rules (ref: longish article).

Dealing with this complexity requires us to understand the systems that we (or others) have built. I swear by Feynman’s quote: What I have not built, I do not understand. I often do not understand the code that Claude is spitting out, especially in areas where I do not have domain expertise. I suspect that most people moving at the speed of Claude don’t either.

The thing with computer systems is that you don’t see the complexity for a while. The start of any project is easy, and initial velocity when you don’t have users, or bugs, is very fast. Complexity develops rapidly - usually the first 10 users do something wonky with your system that you didn’t anticipate, and now you need to tweak it, add a feature here, move some databases there. The first instinct here might be to just have Claude hammer away at it and fix your system until you get what you need. But you don’t understand this beast, so you cannot critique what Claude did and whether it makes any sense. This is the road to software that is going to be extremely hard to maintain and modify, even with Claude.

I think the term Cognitive debt perfectly captures this feeling. Claude has demolished Technical debt and replaced it with Cognitive debt. I can code with impunity and crush bugs and build stuff that I don’t really understand. To make matters worse, we are gorging on this debt like its candy. We do not understand when and how this debt comes due. Over the past week, I have seen some glimpses.

The clearest indicator is when I have to go back over my own code - either as part of dealing with code review comments or changing something - and realizing that I don’t really understand what’s going on. I didn’t write the code and reading code, no matter how carefully, is just not the same. I end up with no good choices:

Debt payoff: Pseudo re-implement the code to really understand the system. This is time consuming, often more so than implementing the thing the first time around.
More debt: Get claude to fix the problem, without understanding the code. May or may not work, might give you some wonky results.

The clearest indicator that you’re in some sort of debt cycle is when a reasonably simple PR is taking 10 commits to pull together, when from experience you know that if you had implemented it by hand, you’d have done it in half the number of commits. When you start asking claude to fix all comments without really understanding things - you’ve essentially declared bankruptcy.

The way out#

First, we need more research. This whole cognitive debt cycle is an emergent phenomenon. The paper I referenced above is from 2025. I could not find related work in the programming domain that would replicate similar results. If someone enterprising grad student reads my work and does the study, the world would be extremely grateful.

We need to understand what happens to software engineers with over reliance on coding tools, over a span of time, working on a difficult systems. Are they net faster? More productive? Happier? Unlike essays, we have objective measures of code quality - bug rates, security issues, cyclomatic complexity - as well as subjective measures - give the code to a few smart people and let them review it. We need to find out what happens when we over rely on these things.

Second, we don’t need to wait for research to tell us that we get dumber through over reliance on Claude. Learning (output) seems intuitively proportional to the amount of cognitive labor (input). Ownership, satisfaction, and overall velocity seem intuitively proportional as well. We don’t need to mentally flog ourselves for every small thing. However, using claude code is a choice, and we should use it as a tool, in the same way that debt (financial, technical, cognitive) is a tool.

The easy stuff:

When exploring and trying to build quick prototypes, delegating to agents is the way to go. They are fast, you don’t really worry about long term maintenance.
When dealing with bugs that you understand, but are just too lazy to create a test, and write the fix, the agents seem ideal. You know exactly what needs to be done, and if it doesn’t do it, you can fix quickly.
When writing documentation for work that you wrote, it seems ideal. You have to be careful to keep things succinct so you don’t just check in oodles of slop, but with that caveat, there is almost zero harm in generating docs and diagrams that others can consume.

Be careful:

Non-trivial changes in an adjacent domain. I am most comfortable in python / backend / systems layer. I understand the frontend/web layer and can write some code, I absolutely do not understand the frontend / mobile layer. When making a mobile change, I have a few choices: rely on claude and just jam the fix through, or go through the long process of learning mobile. For some fixes, that are reasonable to eyeball and then test, it makes sense to use Claude. However, anything more than that and I’m just creating a lot of work for the code reviewer and I would have better off either punting the work to an expert or going through the painful process of learning mobile.
Complex changes in core domain. I have a reasonable level of expertise in our core domain. Even here, when I completely delegate to Claude, I can feel my brain just turning off. I do not recognize the code that comes out. It takes a lot longer to think through the edge cases that Claude might have missed. Ultimately, I often wish that I’d just done the whole thing from scratch myself because it’s way harder to implement and review.

An unsatisfying conclusion#

I don’t have deep concluding thoughts here. I’m documenting my journey here, my experiences, and my research into trying to understand them. Amusingly, this almost certainly means that I’m going to use Claude code less. I give full license to my colleagues to say “I told you so” or otherwise give me a hard time. I’m going to go back to using tab models more. My PR velocity will almost certainly drop, but my satisfaction will go up, I will have much more faith in the code that I wrote and I will understand the system better.

Stop counting tokens

Rushabh Doshi — Wed, 11 Feb 2026 18:57:03 -0800

I attended the Pragmatic Summit this week.

One striking thing from the summit was the amount of airtime dedicated to justifying the financial cost of continuously running Opus 4.6 / Codex 5.3. Brags like “We gave all our engineers an unlimited budget for tokens” immediately begged for the counterfactual - you mean that was a real constraint at your company before you stepped in? Why?

In the early days of video on the web (circa 2006), YouTube’s video format was 320x240px - video so small and pixelated that even the worst compression artifacts today would be 10x better. We literally didn’t have the CPU capacity to process more video, or the bandwidth to stream all this video at reasonable cost.

We spent an inordinate amount of time optimizing the heck out of video so it could look good at reasonably low bitrates. We had formats solely for India and other countries where the internet infrastructure hadn’t caught up and people didn’t have fancy 3G on their phones.

Today, these stories feel quaint - in an “okay grandpa” sense.

This is exactly how I feel about these discussions of “token budgets” right now.

Fast forward a few years, and we’re all going to look back at this time and wonder what we were all thinking. Why were you being so stingy with electricity? Couldn’t you just throw a million Claudes at a problem?

Infrastructure feels expensive and something to optimize around, until the costs fall to zero and we focus on building what’s important.

Simon Willison, Django co-creator and tinkerer supreme, was recounting a story of how he vibe coded a custom app using his phone, to help him cook two Christmas meals simultaneously. Laura Tacho vibed an app to match nail colors while waiting to get her nails done. I think this is exactly what I’m talking about.

I think we should all switch into the abundance mindset as soon as possible. The economics of Opus or Codex costs are shaping your ambition. Spend the money, move fast, run agents 24/7 and build everything you want.

What would you build if tokens cost the same as electricity?

Multi-Clauding Like a Boss

Rushabh Doshi — Sun, 11 Jan 2026 10:13:12 -0800

In December 2025, with the release of Claude Opus 4.5, the software engineering world realized that things will never be the same again. Opus 4.5 is the first model that is able to run complex reasoning chains and create code that is correct, succinct, and feels like code you would have written yourself - including following your code style and conventions across multiple files and refactors.

When ChatGPT originally came out, in November of 2022 (which feels like an eternity ago), there was already a hint that LLMs and coding were a great match. One of my first learning projects was llm-code in May 2023, exploring how GPT3.5 could be coerced into writing code on the terminal.

Thanks to LLM assist, I’ve written more code in 2025 than I have in any other year of my career, including a decade where I was a full-time engineer.

Generated using git-wrapped

I fully expect that by the end of 2026, the entire world is effectively using LLMs to write software and hand-written software will soon get delegated into the educational / learning category or for niche software without adequate training data. This claim would have seemed absurd a few years ago, but doesn’t feel as crazy now. To understand the basis of this claim, one has to use Claude Code on a production codebase for a few weeks and the conclusion feels obvious.

Like most futures, this one is here but unevenly distributed. I’ll go into some detail here on how our setup at Hiro works, what works for us and what doesn’t (yet), and provide some references. Prognostications on the future of software development and engineering will come in a follow-up post ~week from now.

Contents

Setup and Install — Claude subscription, Claude Code, Terminal, Code editor
Commands — Repository init, monorepo setup, code checkouts
Workflow — Plan/Write/Review, Think/Test/Write/Review, permissions, subagents, MCP
Multi-Claude like a boss — Running multiple Claudes in parallel
Important note — Code review expectations
References — Links, tmux.conf, commit command

Setup and Install#

Claude subscription#

First thing you need. I strongly recommend the $200 max subscription if you’re doing development professionally, or at least the $100 subscription. You can upgrade pretty easily as you go along, but the thing you don’t want to do is try and use Sonnet or some of the weaker models. You’ll have poorer results and get frustrated quickly because weaker models will deliver suboptimal results that need substantial manual review and editing to get the code over the line.

Claude code#

From official Claude docs you can install it using curl, npm, or homebrew. In our case, we use nvm pretty religiously and pin all our npm deps, so using npm to install it makes sense. Claude does auto-upgrade and managing claude upgrade cycles has not proven to be challenging.

Terminal#

Ghostty is my terminal of choice, iTerm and Warp get honorable mentions.

tmux is a MUST. It doesn’t take long to learn it. Configure tmux using Claude Code - this way you don’t really need to learn the intricacies of the file. I’ll put my tmux.conf in the references section below.

Code editor#

This almost doesn’t matter. I use Cursor because I like its tab completion model, but regularly switch back to VSCode and nvim for quick terminal based edits.

Commands#

From here on out, I’ll assume you have Claude Code up and running and can fire commands.

Repository initialization#

/init - Quickly generate your CLAUDE.md file that is at the repo root. Optional for now: Read through the file and make manual edits (or getting Claude to edit) to tune it to your codebase.

Monorep setup#

Hiro codebase is a small-medium sized monorepo with various directories for backend (python), frontend/web (typescript), frontend/mobile (typescript), terraform, data etc. We have multiple Claude.md files across the monorepo. We fire claude from root - it seems to reference the correct CLAUDE.md file, in addition to the root one, when working on a particular part of the codebase.

See the “Customize your setup” section in Claude best practices from Anthropic.

Commands, plugins, subagents#

These are important and easy to setup, but we will explore these in the context of workflows 👇🏽

Code checkouts#

I tend to have about 5 checkouts of my codebase, in different directories, each tied to a different tmux window. I do not use git-worktrees - I tried it, couldn’t really get it to work well because our repo has other dependencies like a sqlitedb that must be maintained with migrations. It’s just much easier to have a complete checkout living in a directory.

Each checkout is its own reality - I can run dev servers and mess with the local db without affecting anything else. Each checkout often has a bunch of branches; I use git-town to do stacked changes (sapling is another alternative) but that’s a whole different post by itself. One branch may be in a few checkouts; it mostly doesn’t matter as long as we maintain that each tmux window and subsequent checkout is its own sealed world.

Note: 5 is a bit of trial-and-error. My brain maxes out around 3-4 claudes running in parallel, the 5th is for random one-offs.

Workflow#

Use Ctrl-Tab to quickly cycle through modes.

Plan, Write, Review#

When building a new feature, I almost always start in Plan mode. You tell Claude what you’d like it to do, or point to a Linear ticket or Slack thread and let it go to town exploring. Once claude comes up with a plan, I will go through and read the plan, making edits inline. If the edits are substantial, I’ll just prompt claude to keep planning in the direction that I want to go.

I consider most plans and explorations to be throw-away. If I don’t like the direction things are going, I hit /clear and just start again.

Once the plan is done, I let Claude rip using Accept Edits mode (Ctrl-Tab your way into this, Claude will also prompt you).

Here you’ll hit your first roadblock, with Claude asking you for a bunch of permissions.

I do not like running Claude in Dangerously accept all permissions mode. Turns out, for most operations on a reasonably tuned codebase, you don’t need a ton of crazy commands. Most of our commands tend to fall in the Bash(ls:*) or Bash(git diff:*) type buckets. For all of these, when we see us needing to give Claude these permissions, we add it to the project permission set, so everyone benefits from it. It should take a few iterations, but you’ll rapidly find yourself rarely needing to give Claude permissions.

Sidebar: Shared `.claude` directories#

I strongly recommend having project level claude settings, at the root of your codebase. This allows everyone working on that codebase to contribute while having everything version controlled and merged in. This requires everyone in the company to agree on coding practices and conventions - if you don’t have that, then this serves as a good forcing function.

Let’s continue.

Once claude is done writing all your code and your tests, you’ll want it to execute tests and review the code. Here you have two options:

Just tell claude how to execute tests. In our case we make liberal use of justfiles to capture common commands and have Claude execute those. However, this does pollute context quickly if your tests spew a lot or you have one tests that blows up at the beginning of the run etc.
Use a sub-agent to do the test linting, formatting, testing. The benefit of this approach is that your main agent context stays clean.

Now that we have a clear use for sub-agents, let’s set one up. Simply hit /agents from claude code and follow the prompts to setup your first sub-agent. We set ours to be project wide, so they can be shared across the team.

Similar to CLAUDE.md, it is worth reviewing these and tuning them. One important adjustment my colleague made to our agents file was to tell Claude to run just the tests that it thinks exercise the code that was written, thus reducing the amount of time the agent takes to run (instead of running the world).

Moving on.

Once we have our code in a good place, it’s time to commit and push to GitHub. You can just tell claude to do so, and it’ll do the right thing. However, you’ll find yourself doing this a lot, so let’s be lazy and write a shortcut. Introducing commands.

Commands are literally aliases for prompts. You can add commands by dropping a file into the .claude/commands directory. We prefer to kebab-case our commands like pr-commit-and-push.md. The entire commit-and-push command is in references below.

At this point, claude is done. My followups here are to manually test and check if we actually got things right, and to carefully review the code in question.

When making small adjustments (like a docstring here or there), it’s much easier to pop into an editor and tune quickly. For large adjustments, I will either have claude do it, or potentially throw everything away and start again.

Think, Test, Write, Review#

The second workflow is typically used when fixing bugs.

I start with describing the bug or pointing Claude at Linear, and then having it think and write a test to explore the bug.

Setting up MCPs in claude is straightforward. /mcp will open up a list, most tools out there have instructions on how to setup their MCP integrations. Linear’s is quite good and supports both token and OAuth based auth. If using token auth, be careful not to merge your token into your codebase.

Moving on: when the test is written, Claude usually prompts me to let it continue on and fix the code, which is also straightforward.

The format, lint, test and review steps are not substantially different from the one above.

Multi-Claude like a boss#

Claudes can take some time to run, it’s nice to be able to do a few things in parallel.

I have multiple checkouts of our code (about 5, as mentioned above). I have one tmux session running in a terminal, with good naming so that window-1 points to hiro-1 points to the same named directory. This mostly helps me stay sane.

The central limiting factor with multi-claude is my ability to keep 5 different bugs or features in my head, test them thoroughly, review the code, etc. My brain gets completely fried after doing Claude x3 for ~2 hours.

Important note#

Since code generation is no longer the limiting factor, it can be really easy to write a thousands of lines of bad code and send it over to your colleagues to review. We require all engineers to stand by the code that they are putting up for review. Typically, most PRs will have comments about which parts were completely vibed (increasingly all of it) and specific comments to guide the reviewer through potentially tricky areas. We give each other feedback around reducing the slop (especially around tests where Claude really wants to test the implementation and be thorough) and continue to tune our Claudes.

References#

Claude best practices from Anthropic. Excellent, readable tips.
Boris Cherney’s X thread on how the creator of Claude code uses it. Maps to the best practices article that he wrote.

My tmux.conf file#

# split panes using | and -
bind | split-window -h
bind - split-window -v
unbind '"'
unbind %

# switch panes using Alt-arrow without prefix
bind h select-pane -L
bind l select-pane -R
bind k select-pane -U
bind j select-pane -D

# Enable mouse control (clickable windows, panes, resizable panes)
set -g mouse on

# Make active pane visually distinct
set -g pane-active-border-style 'fg=green,bold'
set -g pane-border-style 'fg=colour238'

# Reset window background to default (no tint)
set -g window-style 'default'
set -g window-active-style 'default'

Our commit-and-push command#

Goal: Commit outstanding changes to a branch and push upstream.

Tasks:

- If the current branch is main, create a new branch and PR using the `pr-new` command.
- Commit all the outstanding changes to the current branch.
- Push the current branch to origin.

Coding for Managers

Rushabh Doshi — Thu, 25 Dec 2025 09:29:14 -0800

Many people managers I know are facing some variant of the following crisis:

wants me to consider coming onboard as an IC. But I haven’t coded professionally in 10 years. I’m not sure I’m a good engineer anymore.

If you’re a professional manager who isn’t coding regularly, this probably feels familiar.

After talking to a host of smart friends, I’m convinced this isn’t a crisis of ability. It’s a crisis of confidence.

Good news: there has never been a better time to dip back into being technical. Picking up where you left off is easier than you think.

A few opinionated notes from going through this myself:

Setup#

Don’t spend time setting up your machine, dev env, and tool chain. This used to be really challenging and is still an incredible nerdsnipe. You could lose weeks tweaking your settings files and not make any real progress. Here is my stack. I’d recommend just copying it.

Use VSCode+Copilot or Cursor for your main editor. The specific choice doesn’t matter (Cursor is a fork so they both feel identical). I go back and forth between the two - keybindings are pretty easy to translate over.
Get Claude Code and Codex CLIs. If you find yourself running out quota on Claude, upgrade to Max ($200/mo) if you can afford it. I have not experimented with Gemini much.
Ghostty for your terminal. Straightforward, fast, easy to use. What’s not to like.
tmux. You’ll need this to run a few different things together, especially if you’re running frontends and backends. Use Claude to fix your configuration.

What you don’t need:

Expensive machine with a $5000 NVidia GPU. Even if you’re training your own models, you could rent GPUs when you need them. That being said, I admit that the NVidia DGX Spark is calling to me.

Note:

I am quite liberal with my $20 subscriptions, which add up quickly. If you can afford it, I think it’s worth spending freely on these tools for a few months without worrying about optimizing your spend. You can easily discard tools that don’t work for you.

Languages and frameworks#

A common mistake is to layer on too many new things at once. For example, learning a new language (say Mojo or Rust), at the same time as figuring out hosting and deployment, and learning machine learning, is overwhelming. Moreover, if you choose something non-standard, you will likely find yourself in configuration hell.

My strong suggestion is to pick one thing that sits at the intersection of personal curiosity, interest, and a real project that you can ship.

If your goal is to get back into programming, pick Python or Typescript. Both these languages and associated ecosystems are excellent, fully featured, and easy to go from toys to production.

At Hiro, we use Python and Django on the backend, with Typescript, React, and React-Native on the frontend. When I started on my journey ~2 years ago, I used vanilla Python and over time, layered on a number of packages that I’ve found to be very useful, for example:

Rich and Click - build great CLI utilities using these.
Pydantic - easy validation for objects, super useful when talking to other services that primarily talk REST. Most AI frameworks (OpenAI, Anthropic) use Pydantic under the hood.
FastAPI - easiest way to get rolling building your own servers. Even though we chose Django and like its built in ORM and Admin tooling, it’s quite likely that if we were starting today, we’d choose FastAPI instead.

If your goal is to go deeper on LLMs, I strongly suggest not using any AI framework to start with. OpenAI and Anthropic APIs are incredibly straightforward, and programming against them directly will give you a much better understanding of the available knobs e.g. reasoning level, temperature, bias. When you decide to choose a framework, Langchain and PydanticAI are the two leading ones. I strongly recommend PydanticAI - it’s tastefully designed and very powerful, while not abstracting bits that it doesn’t need to. We use this extensively in production.

Taste still matters#

As you start coding, you will quickly discover that what used to matter still does - a lot.

In fact, AI tooling has amplified some of the important experiential learnings because when you can generate an infinite amount of code really fast, editing becomes paramount. AI coding tools won’t give you software that is anti-fragile, structured beautifully and layered to make extension easier.

It’s also very tempting to delegate test generation to AI coding tools. Tests themselves are code debt. Without paying attention, it’s easy to accumulate this debt with tests that are effectively testing each line of the implementation. Go back to basics and re-remember your testing rules.

Thankfully, refreshing all your old knowledge is very straightforward. Dust off your GoF book, reread CAP theorem papers, talk to ChatGPT or stick things in NotebookLM.

Be public and consistent#

Writing a Jupyter notebook is easy. Writing scripts is easy. Don’t fool yourself in ways that make your imposter syndrome worse. Put stuff into a public github repo and push it out. This will help on a few dimensions:

It forces you to think of deployment. Despite what people say online, deployment is still hard and requires configuring a few things. There is some initial pain, but the result is very satisfying.
It raises the bar on what you build. Software is often about the last mile - this is where all the annoying issues and bugs pop up. By posting something publicly, you have to cover a lot more than the easier first 80%.
It’s still interesting to someone. I am constantly surprised by the people who ping me and say “Hey that thing you did was super interesting and helpful.”

Do not worry about the project not being good enough. Let it be embarrassing. I can assure you - https://demos.rushabhdoshi.com/ is extremely cringe - but it’s worth it.

Finally, be consistent. Write a little code here and there every day. You don’t need to go into a remote hut for a month - it’s perfectly okay to chip away at things.

It’s time to build

There has never been a better time to build. Software is easier than ever to pick up. Yet, the number of things that need to be built continues to grow.

Happy holidays to you and yours! I look forward to seeing what you create.

A note on hiring#

If you care deeply about the craft of software engineering, about building incredible financial tools that help real people, and want to work at a small startup where you can have tons of impact, we’re hiring software engineers and a product designer.

Please apply directly or reach out to me.

On Learning Faster

Rushabh Doshi — Sun, 30 Nov 2025 00:00:00 +0000

We live in a world where human knowledge is more accessible than it has ever been, yet people continue to use old, linear methods of learning (books, papers, videos) as if nothing has changed. The modern LLM stack compresses months of learning into days.

This post is about how I learn today. Not theoretically, but the literal tools and workflows that enable me to make progress in solidifying my fundamentals (what is a tensor, how is it different from a matrix, why PyTorch vs NumPy), or picking up entirely new concepts (what does attention really mean, why does the KV cache matter).

When ChatGPT came out in November of 2022, I was serving as the interim CTO at Oportun, a public financial company that had acquired our startup Digit. I believed then that LLMs were an inflection point similar to the internet - if anything, this may have been an underestimate. After quitting in February of the next year, I immersed myself in learning ML from the ground up (my formal background in CS is in systems). I used books, papers, YouTube (thank you Jeremy Howard and Andrej Karpathy) but struggled. The problem wasn’t the material, but the limits of unidirectional learning.

Fast forward to today, as we are deep in building out Hiro, that pace of learning has only accelerated. To pick up all this material, some of which has half a century of research behind it, and put it to productive use, would have been impossible had we been stuck with the traditional modes of learning. My learning stack today is anchored by three workflows that I could not live without:

1. ChatGPT with branching chats#

Sure, everyone uses ChatGPT (or Gemini or Claude). I’d like to draw attention to two extremely important things:

Use branching chats to explore a vast body of work in different threads. This is such an incredible feature, yet, as of this writing, only ChatGPT supports this 🤯. Most of my learning chats result in lots of little side branches - I use it to explore related topics or background that I don’t fully understand, while keeping the main bits in context.
Use the best model possible, even if you have to pay for it. The $20 subscription is well worth the money - the newer models really are better. Besides, you get a host of features (including privacy). As of this writing, this means you should use GPT5.1, Opus 4.5, or Gemini 3, amongst the typical set of widely available consumer LLMs.

2. Use NotebookLM#

Google’s NotebookLM tool is incredible when trying to understand dense research papers. If you’re unfamiliar with it: the tool allows you to add links and documents in a “notebook” and then ask questions of everything in context, or generate a “podcast” narrating the content. If you like, it can even pull in more context using search.

Most papers have a central point or two, but have a huge body of reference material that one must be nominally familiar with, to contextualize and understand the novel bits. I tend to put the initial paper, along with some background papers into NotebookLM and ask it to generate a podcast. Stick it on my phone, go on a walk, and get a quick understanding (along with some fresh air and exercise).

3. ChatGPT voice mode#

This is another ChatGPT feature that people don’t talk about enough. I will regularly have long 15-20 minute conversations with ChatGPT about something that is not very clear to me. I love the fact that I have an infinitely patient, superhuman teacher, who can explain things to me in ways that I understand.

Note: the voice model can be weaker than GPT5.1 - use it for more well understood, fundamental learning, which is well-trodden material. Nuanced, cutting-edge stuff is likely to be imperfect.

Learning is hard#

If your brain doesn’t hurt after an hour or two of trying to learn something new, you’re not trying hard enough. I have many days of mental exhaustion, some quite frustrating as progress seems stuck. It is important to remind yourself that learning new things is supposed to be hard, it’s supposed to suck, and that the reward on the other side justifies the pain.

Additionally, just reading (or listening) is never going to be enough. I live by Feynman’s quote: “What I cannot create, I do not understand”. Without actually implementing the concepts (or the equivalent of implementation, such as solving math problems), it is very likely that I’m deluding myself into believing that I understand something, only to come up short when I need to explain it or implement it.

New tools help make the process faster and better, but learning is still very difficult work.

About Hiro#

Building Hiro is a multi-domain learning exercise - we’re building something truly exceptional at the intersection of personal finance, LLMs, and great product engineering. If learning and mastering hard things resonates with you, if you want to dive in headfirst into new domains that most people haven’t explored, and you want to work with a small, incredible team, drop me a line. We’re hiring great software engineers, remote/hybrid in the US.

Your stack#

Given how new things are, and how quickly they are evolving, I’m certain my stack is incomplete and won’t stand the test of time. If you have figured out things that work for you, I’d love to learn about them. Please comment below or drop me an email.

Rushabh Doshi

Rushabh Doshi — Fri, 28 Nov 2025 00:00:00 +0000

Hi, I’m Rushabh.

I co-founded Hiro Finance to build the first AI-powered financial advisor. We are a small, passionate team working on challenging problems, and we care just as much about the people we build with as the products we build.

Before Hiro, I was the CPO at Digit where we helped millions save over a billion dollars and eventually joined forces with Oportun. Prior to Digit, I spent years at Facebook and YouTube (Google) building products used by billions of people. Very early in my career, I was an engineer at Microsoft, working on clustering Windows machines together.

I’m an engineer at heart, with a BS and MS in computer science from Illinois and Stanford. I love building things - products, teams, systems, companies - and I care deeeply about elegance, craft, scale, and reliability.

When I’m not working, I’m with my family, or playing squash, or hitting the slopes on my board. In a previous life, I drank way too much good beer.

If you’d like to get these posts in your mail, you can subscribe to my newsletter.

Vibe Coding Safely

Rushabh Doshi — Wed, 20 Aug 2025 10:02:25 -0700

Everyone at Hiro cares deeply about building a great product - one that solves real problems while maintaining an extremely high quality bar. We take inspiration from other software products like Linear (the first issue tracking product that has actually given me joy) where every interaction feels intentional and polished. This obsession with quality isn’t just about aesthetics - we want Hiro to be something that people can rely upon, software that doesn’t break in unexpected ways. In an age of AI driven development, this has become more achievable and paradoxically more challenging.

A constant tension against the high quality bar is a desire to move fast by cutting corners. This is often a false choice - moving quickly this way feels fast, but is often followed by weeks of “fixes” to make the thing really work. This has not stopped teams, including ours, from falling into this trap - the temptation to push something out before it is properly engineered or ready.

Enter LLMs: Tools like Claude, ChatGPT, Cursor and VSCode/Copilot have transformed the software development workflow as we know. 2021 (and before) feel like the dark ages of coding. As a company that is on the cutting edge of LLMs and software development, we want to use the latest, greatest tools to their fullest extent. But this represents a fundamental tension: LLM coding tools today can create a lot of slop, which isn’t the only problem.

Amongst some of the issues we have encountered first hand:

LLMs tend to be verbose and want to write oodles of code that looks right but can hide subtle bugs. This is especially true in “agent-mode” where the LLM is doing multi-file edits.
Misdiagnoses of bugs can result in a large volume of code and entire rewrites when a surgical fix, or a completely different approach, is appropriate.
LLMs prefer to follow commonly accepted styles (based on their training set) rather than the styles and frameworks of your codebase. For example, we use tailwind, but have specific components like from scratch.
Even when the code is reasonable, the overall architecture can be inelegant. LLM code can be repetitive and miss similar code in other files.

The dichotomy of obvious LLM coding superpowers combined with the aforementioned issues can create problems that must be addressed through a combination of tooling and culture. Through discussions with other engineers and leaders at AI startups, we know that we are not alone here. Everyone faces this question of “how much AI coding is okay”. Tab completions are obviously fine, but what about letting agents run amok? What about vibe coding entire features?

Our goals are to move quickly while maintaining excellent software quality. To achieve these, we want to lean into agents and llm coding tools as much as possible. Everyone at Hiro can, and is encouraged to write code. Ethan (my co-founder and co-CEO) regularly writes PRs for minor changes and will even ship larger features. Contrary to traditional thinking, we want everyone to write code. That’s the whole point - with LLMs the barrier is significantly lower, we need to lean into it.

To enable this degree of code democracy while also allowing us to sleep at night, we are very rigorous about how we structure, write and maintain our code. Here is a list of some of things that have worked for us:

You own your vibed code. Vibe coding makes writing code feel a lot easier but reviewing vibed code is painful. Throwing some vibed stuff over the fence to another engineer to review is lazy and rude. We treat vibed code as our own and carefully review it before sending it over for a formal review. There are cases where we’re not sure - for example, I’m no terraform expert and often have to rely on the LLM to generate the right things. In these cases, we make it really clear in our PR which bits were vibed, and need extra eyes and help.
Monorepo forever. This can be a religious lightning rod, but we have taken a very firm stance on putting all our code in one repo. This makes it really easy for the LLM to grep and find files across frontend and backend and make changes that span different layers in the stack. Some of our frontend code is shared between web and mobile (react native) and having things split into different repos would make life difficult.
Explicit STYLEGUIDE.md. Our styleguide is a file that is checked into our repo and referenced via CLAUDE.md (see below). Any updates to the styleguide necessarily warrant a PR and a discussion.
Leaning into type systems. Type systems are back in vogue (thanks to Typescript and Rust) and we couldn’t be happier. Our codebase is primarily python/django and Typescript and we lean heavily into enforcing types in both. As an example of how maniacal we’ve gotten, we turned on the no-explicity-any eslint rule, effectively disallowing people to get around the type checker by casting to any. On the python side, we treat all mypy issues as errors that must be fixed before merging.
Tons of (fast) unit tests. Unit testing used to be a huge chore until LLMs essentially took that pain away. In 2025, there is pretty much no excuse for not having tons of unit tests to check all aspects of your code. We don’t mandate that all new lines of code need to be unit tested but culturally, almost every new PR is. Frontend components are one place where we don’t unit test things; relying on other tools like Storybook instead.

The net result of leaning extremely heavily into type checkers and unit tests, is that the LLM can quickly loop on itself to fix these problem, resulting in eliminating a whole slew of issues with vibe coded output. 6. Create abstractions and tell the LLM about them. When using LLMs to vibe frontend code, we’d run into problems where it would create correct react/ts code, but would be building from Figma to code resulting in lots of component duplication and subtle differences. After all, it’s really easy to change a px-4 to a px-5 in that one instance without caring about everything else. We changed our codebase to create abstractions, like our component library, based on our design system. Then, we add something like look in @/components/ for the right components to our prompt to get the LLM to do the right thing. 7. Share your CLAUDE.md. We use claude-code pretty heavily and have shared CLAUDE.md files that we check-in and update periodically. How Anthropic teams use Claude code is a pretty great whitepaper with tips on using claude code together as a team.

I’d love to hear your thoughts on effective ways you’re using LLMs to develop code. I’m particularly interested in cultural and tooling changes you’ve made to accommodate and accelerate LLM adoption.

If you’re an engineer and like nerding out about this stuff, and care about personal finance, we’d love to work with you. We are a small, fully remote team (another post on that coming up), obsessed with building the future of personal finance. We are looking for senior full stack software engineers proficient in at least Python or Typescript (ideally both), who can learn quickly, are very curious and interested in AI and LLMs, and have a passion for personal finance. If that sounds like you, please reach out. Note: we cannot sponsor visas at this time, and are only looking for engineers that live in the Western United States.

A Return to Writing

Rushabh Doshi — Tue, 22 Jul 2025 23:49:25 -0400

I’m back to writing after nearly two years. A lot has happened since then.

First, my explorations around AI and LLMs kept increasing my excitement about the area. This excitement has proven well founded, as we are collectively in the midst of one of the greatest revolutions in computing that we have ever seen.

Second, I started a company! Ethan and I are back into personal finance, but this time with LLMs. We believe there is no world in which everyone doesn’t have a personal financial expert in their pocket in the next ~years. We want to build this future we believe in. To see what we’re cooking, head over to Hiro and sign up. If you sign up for the beta and are waiting to get in, just drop me a line and I’ll bump you up.

Finally, I spent some time updating this site to the latest greatest Hugo and Tailwind. It’s a lot simpler, a lot less code. There is probably a bunch of other code that can go, but this is good for now. I used Claude Code extensively to get me here, there is probably a large post on effective Claude Code usage coming up in the future.

LLM Leverage: Where to Build and Where to Bet

Rushabh Doshi — Mon, 18 Sep 2023 12:28:34 -0700

Introduction#

What do electricity, steam engines, printing presses, and large language models (LLMs) have in common?

They are (likely) General Purpose Technologies: technologies that have a broad impact on all sectors of the economy.

Given this broad applicability, how do we reason about the impact of LLMs? If you’re an investor or builder, where do you start?

The GPTs are GPTs (1) paper by OpenAI provides one way to reason about the impact of LLMs. I replicate this paper using GPT-3.5 as a classifier and look at the complete impact, according to this methodology and the most exposed industries.

The following analysis is targeted at two sets of people:

Application builders who are looking for areas to build in.
Investors who are trying to understand likely areas to invest in.

TL;DR#

We use “exposure” as our primary measure. At the task level, an “exposed” task is one that can be done more productively given some LLM technology. This definition can be extended to occupations (which comprise of tasks) and industries (which comprise of people doing occupations).

Greater exposure means more tasks can be aided by LLMs or tools built on top of LLMs. There is no surprise that industries that deal primarily with information are the most exposed. Back-offices of every industry are exposed.

I like exposure as it gives us a way to look at different industries. It also allows us (in the future) to change the definition subtly (if we want to look at job replacement vs. productivity increases) and rerun the whole analysis in the same way.

We define 3 types of exposure:

Alpha: What is the impact of a general chatbot like ChatGPT on productivity increase.

This measure is a base level measure. This represents the key opportunity for Foundation Model companies and anyone providing generalized tools on top of foundational models - aka Google and OpenAI.
Zeta: What is the impact of ChatGPT + Tooling on productivity increase.

This measure looks at the impact of tooling. It’s a superset of alpha, so no major surprise that the top industries are mostly similar.
Delta: What is the marginal impact of Tooling: delta = (zeta - alpha). This is a narrower version of zeta exposure. This is the key market for application builders.

Top-5 industries by Delta#

Finance and insurance
~~Management of companies and enterprises~~ (Ignore)
Professional, scientific, and technical services
Education services
Information

Note: Management is ignored because while it comes up as an occupation code, it doesn’t show up in industry output (where it gets layered in).

Top-5 industries by Output-Delta#

Different industries have different output. If we look out output-weighted delta, we get a slightly different picture. If you believe that one should be playing in a big pond, this is the list to look at:

Finance and insurance
Manufacturing
Professional, scientific, and technical services
Information
Health care and social assistance

This list intuitively feels like the right list to be building tooling in. Huge output, large exposure and large delta values (tooling has a major impact). If you’re investing or building in AI tooling (outside of hardware, foundational models, and orchestration software) this is a good lens to start with.

Methods#

The method is based on the GPTs are GPTs paper (1). The high level approach is as follows:

We have data from the US Bureau of Labor Statistics on employment by industry and occupation.
For each occupation, we have a set of Tasks and for each Task, we have a set of Detailed Work Activities (DWA).
We can classify these DWAs into 3 categories:
1. Direct exposure: The DWA can be done by a LLM alone.
2. Exposure by LLM-powered applications: The DWA can be done by a LLM + Tooling.
3. Exposure given image capabilities: The DWA can be done by a LLM + Tooling + Image capabilities.
We can then back out the exposure of each occupation and industry to LLMs.

For precise classification prompts, see the appendix or the code.

Important note: We are estimating exposure to LLMs. An exposed job simply means that the job could be done with far greater efficiency using LLMs. It does not imply job loss or redundancies.

For example, we’re seeing a lot of LLM use for software development (eg. Github Co-pilot, ChatGPT, Cursor.so). These applications seem to be making software developers more efficient, not redundant. Similary, we would expect a legal assistant (like Harvey) to make paralegals and lawyers more efficient, not redundant.

It’s simply too early to tell whether LLMs replace jobs (like switchboard operators) or just them more efficient, causing the industry to expand (like accountants after spreadsheets).

Data#

To understand and contextualize the results, we need to understand the data used.

O*Net Database (2) maintains data on all occupations (eg. CEO)
Each occupation is broken into a set of Tasks (eg. “Direct or coordinate an organization’s financial or budget activities to fund operations, maximize investments, or increase efficiency.”)
Each task is broken into a set of Activities (eg. “Analyze impact of legal or regulatory changes.”)
US Bureau of Labor Statistics (3) maintains data on employment data by industry and occupation. This matrix has the employment mix of every industry (eg. the “Rental and leasing services” industry employs “Marketing managers”, “Sales managers”, etc.)

Example#

Let’s work through an example to understand how the whole flow works.

industry_output[(industry_output["Supergroup"] == "Information") & (~industry_output["is_supergroup"])]

This gives us a look at everything that is part of the “Information” supergroup. Truncated list:

Software publishers
Computing infrastructure providers, data processing, web hosting, and related service
Web search portals, libraries, archives, and other information services(3)
Publishing industries(3)

Let’s look at Software Publishers which is NAICS Code 5132, by employment:

bls_matrix[bls_matrix["industry_code"].str.startswith("5132")].nlargest(10, "employment_2022")

The “Software Publishers” industry employs people who do the following jobs (truncated):

Software developers
Computer and information systems managers
Computer user support specialists
Sales representatives, wholesale and manufacturing, technical and scientific products

A Software developer does the following tasks (truncated):

task_statements[task_statements["onetsoc_code"] == "15-1252.00"]

Analyze information to determine, recommend, and plan installation of a new system or modification of an existing system.
Analyze user needs and software requirements to determine feasibility of design within time and cost constraints.
Confer with data processing or project managers to obtain information on limitations or capabilities for data processing projects.
Confer with systems analysts, engineers, programmers and others to design systems and to obtain information on project limitations and capabilities, performance requirements and interfaces.
Consult with customers or other departments on project status, proposals, or technical issues, such as software system design or maintenance.

An example task has the following detailed activities:

tasks_to_dwas[tasks_to_dwas["task_id"] == 21669]

dwa_reference[dwa_reference["dwa_id"].isin(["4.A.2.b.4.I03.D15", "4.A.4.b.4.I06.D07"])]["dwa_title"].to_list()

Develop testing routines or procedures.
Manage information technology projects or system activities.

We can take each of these and classify them. For example, we get the following classifications:

1 - Direct exposure
2 - Exposure by LLM-powered applications

We can now back this classification out all the way, through the employment matrix and get to an overally alpha, zeta etc. number for the entire “Information” industry.

Analysis#

The analysis section is graph heavy. Each graph follows a similar format: in a 2X3 grid, we see a list of industries or occupations. The first row is alpha, zeta, and delta scores, the second row is the same, but multiplied by the output of the industry.

Top Supergroups#

A Supergroup is a set of industries, bunched together.

Top Groups#

Groups look at one more level of granularity compared to supergroups.

Looking at these, a few industries stand out as being worthy of deeper dives:

Finance and insurance
Professional, scientific, and technical services
Information
Educational services
Health care and social assistance

Deep Dives#

The following industries deserve their own deep dives:

Finance and insurance#

Two interesting aspects of finance and insurance:

Base level alpha is very high. As a result, the delta value is lower. Building tooling in this space could be lucrative but one has to be careful about building too close to a general purpose LLM-enabled task.
The output is massive. Finance is a huge, huge industry. The scaled impact of even small tooling improvements could be massive.

Professional, scientific, and technical services#

This is the motherload of the tooling bonanza.

Legal
Advertising
Accounting
Design
Architecture

So much whitespace to build amazing, bespoke tooling.

Information#

Some whitespace for software. Like finance, high base levels of alpha. I admittedly do not understand why telecom is such a huge output industry or why the scaled output is so high.

Healthcare#

I am not a healthcare expert. If someone wants to collaborate and add a commentary, I would welcome it.

Conclusion#

Thanks for making it all the way here!

Using exposure classification to figure out what industries are highly exposed seems like a reasonable first pass to understand the impact of LLMs on the broader economy. I like it because it widens my aperture to think of all the areas where people work, where better tooling will lead to productivity increases. It’s not perfect, but it’s certainly a start.

What about Foundational Models or Orchestration?#

Martin Casado at a16z had a great post about the Generative AI Platform.

The Model provider and Orchestration layers lend themselves to reason - we understand model copabilities and dimensions of price, speed, data privacy etc. Similarly, on the tooling layer, we know some of the problems associated with LLMs and how to build tooling to overcome them.

However, the application layer remains both the great unknown and the great promise. I hope to understand this layer, in a systematic way, to provide guideposts for those of us looking for gold. Hence this analysis solely looks at the Application Layer.

Reference#

GPTs are GPTs paper: https://arxiv.org/abs/2303.10130
O*Net Database: https://www.onetcenter.org/database.html#all-files
Bureau of Labor Statistics, Occupation Data: https://www.bls.gov/emp/data/occupational-data.htm
How will Language Modelers like ChatGPT Affect Occupations and Industries: https://arxiv.org/abs/2303.01157
a16z: Who owns the Generative AI Platform: https://a16z.com/who-owns-the-generative-ai-platform/

Appendix#

Software#

All the data and software for this is availble and open source. You can find the datasets above, the software is the following repo:

https://github.com/radoshi/gptsaregpts

Detailed classification prompt#

## 1 – Direct exposure
Label tasks 1 if direct access to the LLM through an interface like ChatGPT or the OpenAI playground alone can reduce the time it takes to complete the task with equivalent quality by at least half. This includes tasks that can be reduced to:
- Writing and transforming text and code according to complex instructions,
- Providing edits to existing text or code following specifications,
- Writing code that can help perform a task that used to be done by hand,
- Translating text between languages,
- Summarizing medium-length documents,
- Providing feedback on documents,
- Answering questions about a document, or
- Generating questions a user might want to ask about a document.

## 2 – Exposure by LLM-powered applications
Label tasks 2 if having access to the LLM alone may not reduce the time it takes to complete the task by at least half, but it is easy to imagine additional software that could be developed on top of the LLM that would reduce the time it takes to complete the task by half. This software may include capabilities such as:
- Summarizing documents longer than 2000 words and answering questions about those documents
- Retrieving up-to-date facts from the Internet and using those facts in combination with the LLM capabilities
- Searching over an organization’s existing knowledge, data, or documents and retreiving information

Examples of software built on top of the LLM that may help complete worker activities include:
- Software built for a home goods company that quickly processes and summarizes their up-to-date internal data in customized ways to inform product or marketing decisions
- Software that is able to suggest live responses for customer service agents speaking to customers in their company’s customer service interface
- Software built for legal purposes that can quickly aggregate and summarize all previous cases in a particular legal area and write legal research memos tailored to the law firm’s needs
- Software specifically designed for teachers that allows them to input a grading rubric and upload the text files of all student essays and have the software output a letter grade for each essay
- Software that retrieves up-to-date facts from the internet and uses the capabilities of the LLM to output news summaries in different languages

## 3 – Exposure given image capabilities
Suppose you had access to both the LLM and a system that could view, caption, and create images. This system cannot take video media as inputs. This system cannot accurately retrieve very detailed information from image inputs, such as measurements of dimensions within an image. Label tasks as 3 if there is significant reduction in the time it takes to complete the task given access to a LLM and these image capabilities:
- Reading text from PDFs,
- Scanning images, or
- Creating or editing digital images according to instructions.

## N - Not confident
If you are not confident in your classification or if you believe it could go either way, label a task as En.

Two books

Rushabh Doshi — Thu, 20 Jul 2023 11:44:30 -0700

Instead of asking GPT to write my material, I used it as a critic to help strengthen my writing. I like the result: it feels much more authentic and less like generated text.

Ministry for the Future#

I enjoyed Ministry for the Future by Kim Stanley Robinson. This book is difficult to pigeonhole into a particular genre - somewhere in between “climate fiction” and “hard sci-fi”. It paints an optimistic future of saving the Earth through geo-engineering and crypto currency utopianism. It’s set in Zurich, which made me nostalgic for all the time spent hanging out there. The closest comparable recent book is Neal Stephenson’s Termination Shock which explores similar themes of geoengineering to save the planet. Where Termination Shock is full of action and adventure, set against a climate change background, MftF is more mellow, focusing on the workings of a small international governmental agency trying to save the planet. Besides climate change, the book also explores themes of global inequality and the power of individuals to make a lasting change. The opening scene will be seared into my memory forever: a harrowing heatwave in North India which results in millions of deaths. Given the heat waves ripping through America and Europe today, this possibility feels all too real. Ultimately, the book left me feel optimistic and hopeful. It’s unlikely we can get through this without geoengineering, but the fact that there is a possible way out of this mess is heartening. Recommended.

When the Heavens Went on Sale#

I listened to Vance’s latest book When the Heavens Went on Sale via Audible. I find the audio format more compelling for dramatized non-fiction - the details are less important than the story and the narrative arc. I’m very interested in space - it presents a completely new, unexplored frontier, vast and unimaginably huge. The book is about engineers, investors, and various characters building companies to conquer near earth space: Planet Labs, Rocket Lab, Astra, and Firefly. Planet Labs launched small satellites to constantly image the world, RocketLab and Astra are launching rockets to deliver payloads to space, and Firefly is building engines to power these rockets. The stories lose some steam as we go through the companies - Planet and RocketLab are the strongest, Astra is weird, and things sort of fall apart with Firefly. Given that at least three of the companies are public, it is interesting to see how things unfolded after Vance’s coverage of these companies ended. Regardless of some story weakness toward the end, I came up with a great sense of excitement about everything going on a few hundred miles above our heads. Recommended for a non-fiction summer read / listen and for anyone at all interested in space and its potential to impact life on earth.

Trivia

Rushabh Doshi — Mon, 12 Jun 2023 16:54:16 -0800

A fun little trivia game, co-written with my friend Vivek Patel.

Enjoy: https://demos.rushabhdoshi.com/trivia/

Bartender

Rushabh Doshi — Wed, 31 May 2023 16:54:16 -0800

I used GPT to generate cocktail recipes. A bunch of recipes are pre-generated and cached, but you can generate new ones on the fly.

🥂: https://demos.rushabhdoshi.com/bartender/

llm-code

Rushabh Doshi — Mon, 22 May 2023 14:19:46 -0700

`llm-code`#

tl;dr: llm-code is a CLI tool that helps you write code. It’s powered by OpenAI’s models. It’s open-source and free to use. Check it out on github or install it with pipx install llm-code.

I’m really excited about this project. Excuse the enthusiasm.

I’ve been messing around with LLMs since leaving my job in February. In part, I’ve tried to get back to my technical roots by coding up a storm, and reading paper after paper. However, the honest truth is that I’m just having a lot of fun.

As I explore what LLMs can do, I started writing little utilities to help me program. For example, one of the most boring things an engineer has to do is write a ton of unit tests. The net result: we get lazy and unit test some bits and then hope the rest works out. What if we all had a little assistant that did the boring bits for us - like writing unit tests?

I was messing around with bits and pieces of what would become llm-code for a few weeks. Then I saw Simon Willison’s post about his llm tool and was inspired to write and release llm-code.

What is `llm-code`?#

llm-code is an open-source CLI tool that helps you write code. It’s really straightforward to use, with a few quality of life features thrown in. I won’t go into all the things you can do here - there are a lot more examples on the github page - github.com/radoshi/llm-code, but here’s a sample.

llm-code 'write me a hello world program in rust'

fn main() {
 println!("Hello World!");
}

Simple enough.

Let’s get more complicated.

llm-code """Write a python program that uses SQLAlchemy to model the following database class:
id - int autoincrement primary key
name - string
phone - string

Add a function to create the database and a function to add a new entry to the database.
"""

from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker

Base = declarative_base()
engine = create_engine('sqlite:///phonebook.db')
Session = sessionmaker(bind=engine)

class Contact(Base):
 __tablename__ = 'contacts'
 id = Column(Integer, primary_key=True, autoincrement=True)
 name = Column(String)
 phone = Column(String)

def create_database():
 Base.metadata.create_all(engine)

def add_contact(name, phone):
 session = Session()
 new_contact = Contact(name=name, phone=phone)
 session.add(new_contact)
 session.commit()

create_database()
add_contact('John Doe', '123-456-7890')

That’s pretty darn good!

Quality of life features#

A common pattern that I’ve been using is to get the LLM to output something, examine it, and then do something with it. Copy-pasting console output becomes tedious, and in theory, one could use tee to store it away, but what if the tool just cached the previous response and repeated it if you asked it exactly the same question?

llm-code """Write a python ...""" > contact.py

No round-trips to OpenAI necessary.

Didn’t like the output? Want to reroll or use the more expensive GPT-4 model? No problem. Changing any parameters or using the -nc option gets rid of the caching.

llm-code --gpt-4 """Write a python ..."""

Remembering responses#

llm-code by default stores all interaction with the llm in ~/.llm_code/db.sqlite (just like llm). You can examine all old ouputs, even look at the input and output tokens to estimate costs (if that’s your thing). Recommend using Datasette to view this database easily, but any tool will suffice.

Installation#

Install quickly using pipx

pipx install llm-code

Configuration#

You need an API key from OpenAI. Store it in an environment variable called OPENAI_API_KEY or in an env file in ~/.llm_code/env.

Enjoy coding again#

The whole goal of this tool is to help take some of the pain away from doing great software engineering. It’s very evident, after using Copilot and ChatGPT and now llm-code that we’re in a whole new age of programming productivity. These tools don’t make software engineers obsolete - they give us super powers and help us go 10x faster.

Happy coding!

A Shift in Focus

Rushabh Doshi — Sat, 20 May 2023 08:22:20 -0700

A Shift in Focus#

More AI, more random, less focused#

I wrote this post as a catchup for my substack newsletter. Reposting it here.

Hello dear reader! It’s been a while - nearly a year to be precise, since the last substack post. I’ve been writing more regularly on my blog but didn’t crosspost here because the content was simply not worthy of the lofty ambitions of this blog. This bar was a mistake - instead of worrying about writing content that meets some arbitrary expectations, I’m going to do away with those expectations completely. If you still find my content interesting - I’m glad to have you here.

This substack is becoming a lot less focused on leadership and managerial things. I’m widening the aperture to include some life updates, but more important, my thoughts and musings on AI, with a bent toward generative AI.

Quick catchup#

A lot has happened in my life in the last year:

I joined Digit in March 2020, right as the pandemic started. I will remain forever grateful to Ethan Bloch for bringing me on this ride. It was incredible. Startups are amazing, highs and lows like nothing else.

Digit was acquired by Oportun in December 2021. We spent some time integrating Digit into Oportun, building out Oportun’s app offering that ties Savings, Banking, and Lending in one place. I served as the interim CTO for 6 months, and eventually left in February 2023.

Since February, after taking some time to do personal stuff, I’ve been diving deep into the world of Generative AI.

Generative AI#

OpenAI’s release of GPT-3 in 2022, followed by GPT-3.5 and GPT-4 has taken the world by storm. I firmly believe these large language models (LLMs) constitute a new platform shift and have materially moved humans closer to Artificial General Intelligence (AGI).

I’m following a three-pronged approach to understanding GenAI:

Bottoms up education by reading the foundational papers, and following courses such as fast.ai, cs231n, and Andrej Karpathy’s excellent neural network zero-to-hero series.
Building demos using APIs and publishing them for others. I’ve primarily been using OpenAI’s API, and have dabbled some with other models hosted on replicate.com and 🤗 model hub. Projects are all published on my projects page.
Writing memos about the business impact and opportunities that are being created with this shift.

I am fully in the AI hype bubble. Between reading lots of interesting writers like Ethan Mollick and Simon Willison, writing a bunch of code, and talking to interesting people, my work life is quite full.

Future expectations#

This blog and newsletter are going to expand to include AI things. To keep a reasonable signal-to-noise ratio, I won’t repost summaries of things you read on Twitter.

I hope you enjoy the content - if there are things that you’re particularly interested about that you’d like me to write, please drop me a line.

My Development Stack

Rushabh Doshi — Wed, 17 May 2023 22:01:05 -0700

My Development Stack#

I’ve been programming since for 30+ years now. There have been a few times in my life when I’ve felt like a kid in a candy shop who couldn’t wait to get to their computer and start coding.

First - the very early days with Basic and a junky ZX Spectrum clone.#

Everything was new and amazing and 🤯s were common. This was pre-internet, so we had to get programs on cassettes, load them up with, and then start to mess with them. The internet changed everything of course - programming took a bit of a hit as surfing (the web kind, non-athletic nerd here, thanks) took over.

Second - joining Google in 2008.#

It’s hard to overstate how ahead of the rest of the industry Google was back then. Engineering practices that are considered obvious were anything but, except at Google. Code reviews were mandated, you could only choose amongst a few blessed languages, documentation was excellent, new libraries, tools, and infrastructure were being written every month that gave everyone super powers (protobufs, gfs, colossus, borg).

Third - now.#

LLMs are crazy and every week, I’m breaking a new frontier in coding. The development world has evolved greatly over the last 30 years and gotten significantly more complex. This complexity is needed - as a species, we compute more than we’ve ever computed, and optimization matter. In the past, we’ve tried to wrap and hide this complexity by trying to create libraries that simplify things, only to inevitably run into the one time when we need to break the abstraction, and then everything goes to hell. However, a new fighter has entered the areana; two new fighters actually. One: LLMs in the co-pilot flavor, as an in-context assistant, and two: LLMs in the ChatGPT flavor, as an open-ended chatbot. Between these two, dealing with esoteric libraries, or arcane syntax, is easy again. I can write code in mostly-python and get correct python, or write code in english, and get mostly-python. Either way, coding is FUN again.

The Stack#

If you’re doing any AI LLM stuff, your stack almost certainly involves either python or Typescript or both. I welcome this - I love python’s clean and efficient syntax and the batteries included philosophy of the Python community. With the development of newer languages like Mojo🔥, I expect Python becomes even more powerful (by potentially morphing into Mojo.)

As a result, I’m all in on Python and believe that developing one’s Python familiarity and using it as the primary stack language is a good choice. Here is a list of libraries I’m using regularly. I intend to keep this list current as I continue to build things:

Python basics#

Pydantic - provides nice class helpers, similar to dataclass. By providing types, we get free validation while constructing objects, and serialization to dicts and JSON objects, both of which matter when dealing with OpenAi APIs (and other LLMs). pydantic[dotenv] helps in dealing with setings and .env files.
FastAPI - My servers are pure JSON API servers, written in FastAPI. It makes writing servers a breeze, has async request handling (needed for dealing with LLMs because everything takes so long), generates OpenAPI specs automatically, along with a neat little frontend where you can play with the API directly.
jupyter - I find it easy to tinker with things using a notebook, and then convert stuff to a library or app once I have a handle on things.
pytest and pytest-watch - Pytest has made unit testing a breeze. pytest-watch just monitors files for change and runs your test in a loop. Open a shell, start ptw and code away. Plus coverage is easy with pytest-cov.
click - Makes dealing with command line arguments a breeze eg. when writing a batch script.
poetry - Great way to handle python package management. python -m venv venv or virtualenv or other options work great as well. Poetry is just easier.

Not critical but very useful:

rich - Provide prettier output for most objects. from rich import print as pprint and then use pprint(obj) to get beautiful output.

AI stuff#

openai - OG AI lib for LLM interactions. Great API out of the box and easy to use.

Interesting but not absolutely required:

langchain - Langchain is almost its own little world and you have to understand it to use it. Since its often a wrapper around a bunch of careful prompts, one doesn’t absolutely need langchain and for a lot of applications, it’s overkill.
chromadb or pinecone - Similarly, most people don’t need vectorstores. But if you do, they’re great options.

Early but interesting:

marvin - Build a chatbot quickly. The idea of an AI Function - essentially an ai_fn decorator that you can stick on any function, give it a docstring, and let the LLM do the rest, is incredible.

Frontend stuff#

My frontends are written in simple React. I compile them into a set of static files and serve them up via the same FastAPI server above. This gets rid of all the CORS crap (its coming from the same server, so its all fine) but adds an additional build step. Oh, and its not optimized in the way some of the crazier Vercel/Next stuff is.

To be honest, I find frontend work quite intimidating and challenging. There are thousands of JS packages and frameworks, and all seem to have slightly different ways of expressing the same underlying complexity - how to manage application state and get the UI to work with said state. I eventually broke through with an understanding of React, thanks to Josh Comeau’s excellent Joy of React course. Still, I consider myself a 101 level frontend developer, so take everything with a grain of salt.

One breakthrough tool for frontend coding has been Vite. It takes all the mess of packaging and bundling and minifying and whatnot, and just puts it all together in one thing. Just use Vite and simplify life.

Test coverage#

Unit testing is (was) painful. Not only would you have to think of how to test your code, you’d have to then construct all the mock objects and inject dependencies and what not. Net result - I almost never unit tested non-production code.

This has changed completely. I simply plug in my python code into ChatGPT and ask it to generate unit tests. It does a reasonably good job, I fix it up, have pytest running continuously in the background, et voila! 100% code coverage for everything.

It’s amazing to be coding toys with 100% coverage. Making changes and doing drastic surgery is just fine - unit tests are catching things left and right and if coverage drops, GPT fixes things up again.

The Joy of Hacking#

If you’re a recovering engineer like me, there has never been a better time to dive into coding again. The tools are amazing, there are new developments every day, and you can build things that were impossibly hard just a few years ago. I’d love to see your hacks and projects - please reach out on Twitter and drop me a line.

Are Foundational Model Companies Overvalued?

Rushabh Doshi — Wed, 29 Mar 2023 10:40:03 -0700

Random thoughts on the valuation of foundational model companies

Companies like OpenAI, Anthropic, and Google are currently leading the development of foundational models, which are highly valued for their potential to unlock fast value across a range of industries. OpenAI’s ChatGPT, Github co-pilot, Bing’s search assistant, and Jasper’s writing assistant are examples of the value of foundational models in action. However, this moment may be short-lived, as new technologies and developments are likely to drastically reduce the cost of building and deploying foundational models.

Firstly, the cost of training, deploying, and running distilled foundational models is expected to be significantly lower than that of traditional models. Distillation involves using a large, complex base model to train a smaller, more efficient model that performs just as well. This process is much cheaper than training a new model from scratch and is becoming increasingly common. As distillation becomes more widespread, the cost of training models will continue to decrease, making it possible for companies of all sizes to build and deploy sophisticated deep learning models.

Secondly, while research costs will remain a significant expense for foundational model companies, it is expected that training and inference costs will eventually go to zero. Building a foundational model requires extensive research to develop a deep understanding of the data and how it can be effectively modeled, which will continue to require significant investment in research. However, as the field of deep learning matures, more and more people will train their models on the same publicly available datasets, such as Wikipedia. This will reduce the cost of acquiring and preparing training data, making it more affordable for companies to train their models. Additionally, advancements in hardware and software engineering will continue to drive down the cost of inference, and code optimizations and specialized hardware will enable more efficient training. As a result, the cost of training and inference will eventually go to zero, making it possible for companies of all sizes to build and deploy sophisticated deep learning models at a lower cost.

Finally, foundational models are expected to become increasingly vertically integrated, without accruing value to foundational model companies like OpenAI. As advancements in hardware and software engineering continue to drive down the cost of inference and training, the value of foundational models will increasingly be captured by companies that integrate them into their products and services. For example, Replit trained Ghostwriter (their version of copilot) while Neeva trained their own search assistant, without relying on or integrating with OpenAI. This means that foundational model companies will need to adapt and innovate in new ways to continue to drive progress in the field of deep learning.

In conclusion, while foundational models are here to stay, cost structures will evolve, making it harder for Foundational Model companies to exist without providing value added services on top, as others rush to clone these models and vertically integrate them into their own products.

Written with ChatGPT, obviously.

Persona

Rushabh Doshi — Wed, 29 Mar 2023 09:54:08 -0700

Live Demo #

A small chat bot to talk to a historical figure. I try to explore whether chat is a fundamentally better didactic medium (I believe so) and play around with python and react.

Github is private. Let me know if you want the code.

YouTube screen cap: https://youtu.be/guUCG5yh-Ws

Audio Summarizer

Rushabh Doshi — Thu, 09 Mar 2023 07:04:42 +0530

https://github.com/radoshi/audio-summarizer

A toy summarizer for english language podcasts. Testing with audio lengths of 15-30 minutes. The summarizer uses OpenAI APIs to transcribe speech and generate summaries from the transcriptions.

That's a Wrap

Rushabh Doshi — Mon, 13 Feb 2023 18:27:05 -0800

That’s a Wrap#

Last week, I wrapped up my three year journey with Digit and Oportun.

Grateful to so many - few I’d like to call out in particular. First, Ethan Bloch for bringing me along - from 🍕 at Delfina, to joining three days before the pandemic, to building products that helped people become financially healthier, to building a team that I’d go into the trenches with any day, to selling the company and getting acquired - it has been an EPIC journey. Thank you.

Second, Hemant Taneja for connecting me and Ethan in the first place so many years ago, and always being willing to take calls and give advice as we navigated our ship.

Third, Raul Vazquez and the Oportun team for acquiring us and making us feel welcome.

Finally, to all my Digits, who are still at Oportun or have moved on to other adventures: I’d go into the trenches with you again anytime. 🫡

What’s next?#

I’m convinced that we are in the early innings of a sea change brought about by generative AI. I am going back into student and builder mode - figuring out the underlying technology and engineering, and imagining the products that could be built with it.

DALL-E: a realistic painting of a path forward, climbing snow capped mountains, at sunrise, 16:9 aspect ratio

Cloudflare 🤯

Rushabh Doshi — Tue, 15 Nov 2022 00:05:02 -0800

I was trying to build a really simple redirector to get back into the mode of building and deploying things for real. It’s been a while - management and children don’t leave that much time for hacking on the side.

I started by building a simple thing with Go. I love Go. It’s simple and straightforward. I cheated. I used copilot to get me through rusty syntax. Copilot is amazing - easily worth the $10 per month. Got the go app built in about 15 mins (all it does is redirect rushabhdoshi.com/linkname to https://linktarget with a 301 or 302.)

Okay cool, lets deploy. Started with Google Cloud. I guess I still have some part of me that is loyal to Google and wants to start there first. Oh boy, what a mess. Cloud Workers are terrible, lots of versioning issues with the new version being mostly complete and the old version being deprecated (familiar to any Googler). DNS redirects not working. Yikes.

I dialed in the cavalry (aka my friend Brian.) He suggested looking at Cloudflare workers. I’d heard of them before without playing with them. Took another look.

Workers are super interesting. They deploy everywhere, in the same process. Only JS workers are supported because of this execution model. Anything that compiles to webasm neatly is fine (Rust, C++). Here is a great post by Kenton detailing why Go isn’t supported (and unlikely to be supported anytime soon.)

Okay, fine. It took about 15m to write the damn thing, how long can it take to rewrite it in JS? Turns out … about 15m. TS really, but samesies.

Deployment otoh - SO MUCH SIMPLER. Get the app up and running in one command. Fabulous. Now to fix DNS.

This opened up another rabbit hole - getting Cloudflare to be my default nameserver instead of Google. Fairly straightforward instructions. I had another toy domain that I wasn’t using, so tried that out first. Smooth as butter. 👏🏽👏🏽👏🏽 Cloudflare - I’m starting to get really impressed. Created a small website there - rad-labs.io - and deployed it. Whoa.

And then it turns out that Cloudflare has redirects built in. 🤯

Okay, so now we need to move this site off Github pages and on to Cloudflare. And add redirects. And Cloudflare autodeploys on commmit, so get rid of deploy.sh. ✅ ✅ ✅

Holy smokes. So over the space of an evening - 2 hours really - I’ve gone from zero to:

Moving DNS for both my domains to Cloudflare
Deploying a brand new website.
Moving rushabhdoshi.com to Cloudflare.
Adding a redirector so rushabhdoshi.com/go/coda will go to Coda.

And it’s all free.

🫡 Cloudflare. You won me tonight.

Digit + Oportun

Rushabh Doshi — Thu, 20 Jan 2022 00:00:00 +0000

Happy 2022 everyone!

We at Digit joined forces with Oportun just before Christmas!

2021 has been a wild year. We started the year by refocusing our efforts on building the world’s smartest bank account from the ground up. We strongly believe that people want to lead financially healthy lives but need help getting to their goals. We set out to extend our automatic savings functionality to the bank account. Your bank account should know how much money you need for upcoming liabilities, like your bills. It should set aside money for emergencies and other life goals. It should put money away for the long term in retirement and investment vehicles—all without you needing to do anything.

After extensive beta testing, we rolled out Digit Direct toward the end of the year and started accepting members off our waiting list to sign up. We’ve been thrilled to see people using Direct, paying bills on time, and putting money away for a rainy day.

At the same time, we found a fantastic opportunity to join forces with another company and a leader that shared the same values as Digit – focusing on helping hardworking people achieve their financial goals. The more we learned about Oportun and Raul Vazquez, the more excited we got. We had a complementary fit of mission, values, and products. By joining forces, we can build a complete set of financial solutions to meet the needs of every person in this country - from savings, banking, investing to credit and lending. In December of 2021, we completed the deal and closed the acquisition. The Financial Health Network wrote an excellent piece that explains why we believe this merger can be a real force for good.

This brings us to 2022. We are at the beginning of a journey to build a modern consumer financial service that is focused on helping hardworking people be financially healthy and achieve their goals. And even though we have a lot of the pieces, there is a lot to build. Digit is a distinct business unit within the Oportun family. If you’re excited to work in fintech, want the flexibility and culture of working at a startup with the stability and backing of a larger organization, join us at Digit. We are hiring for various positions, from IC to management, across product, design, engineering, and security.

Accelerate

Rushabh Doshi — Sat, 27 Nov 2021 12:24:16 -0800

Accelerate was written in 2018 following four years of survey based research data compiled by its authors. In 2021, the book feels a bit dated: the practices mentioned in the book are treated as fairly commonplace in high performing software development organizations. Most of the ideas presented in the book make intuitive sense are accepted without much argument in today’s software companies. The link between presented ideas and team performance is based on survey measures. The authors do not make conclusive claims about the overall performance of a company (using their stock price for example) and good devops practices.

The book is in three parts. The first is of interest to an engineering audience, the second goes into their survey methods, and the third is a case study. If actionable takeaways are the sole things of interest, most of the book can be skipped for Appendix A Capabilities to Drive Improvement.

Key Ideas#

Software development team performance can be measured and improved.
Teams that develop devops capabilities and follow good practices ship software faster, and at a higher quality. Teams are also happier and more retained.

Performance Measurement#

The authors assert that team performance can be measured. Prior attempts to measure team performance have been unsuccessful because they have focused on outputs rather than outcomes. For example, lines of code, or story points, are outputs. They are highly subjective. Instead, measures should focus on global outcomes. Company level goals, for example, are global outcomes.

I saw this idea implemented at Facebook where all functions – engineering, product, design – were graded on whether they achieved a common, measurable goal. This prevented the ownership disparity where one function owns the product outcome (“did likes go up by 1%?”) and another function owned stability (“the app had 99% uptime”).

The authors recommend four performance indicators to measure software delivery:

Delivery lead time: Time from spec to full rollout. Does not include design lead time (time for design and specification development, nor iteration to get the product right.)
Deployment frequency: Faster is better.
Time to restore service: All systems fail. How quickly can a complex system be restored?
Change fail rate: What percentages of submitted PRs result in a patch, hotfix, or need to be rolled back.

[These results demonstrate that] there is no tradeoff between improving performance and achieving higher levels of stability and quality

Capabilities#

Team performance can be correlated to a base of capabilities that the teams have developed, broken down into five categories:

1. Continuous Delivery#

Defn: Set of capabilities that enable us to get changes of all kinds - features, configuration changes, bug fixes, experiments - into production or into the hands of users safely, quickly, sustainably.

Capabilities

Version control.
Fully automated deployments. No manual scripts needed.
Continuous integration. Each check-in triggers a set of tests to discover serious regressions.
Trunk based development. Somewhat challenging with GitHub flow, but a straightforward idea nonetheless. Use extremely short lived branches.
Test automation.
Test data management. Maintaining clean test data is just as important.
Shifting Left on security. Pull security reviews up in the early part of the development cycle. Infosec contributes to all parts of the software development. Use pre-approved libraries, packages, toolchains, and processes.
Continuous delivery. Code check-ins result in changes getting pushed out all the way to customers.

2. Software Architecture#

Authors found no significant correlation between system type and delivery performance, with the exception of engineers working on custom software developed by another company (outsourcing partners).

Capabilities

Loosely coupled architecture
Let teams choose their tools.

3. Product and Process#

The capabilities mentioned by the authors are implemented in most modern software shops and taken for granted. They do not discuss the difficulties of implementing some of these capabilities (eg. experimentation is great, assuming you have sufficient sample size to experiment with.)

Capabilities

Gather and implement customer feedback
Everyone should understand how their work flows into the whole.
Work in small batches.
Enable team experimentation.

4. Lean Management#

Capabilities

Limit work in progress
Create and maintain visual displays showing key quality and productivity metrics and current status of work (including bugs)
Use data from application performance and infrastructure monitoring tools to make decisions.
Lightweight change monitoring: no approvals or peer-reviewed approvals to check-in code.
Leaders should try and follow the principles of Transformational Leadership

5. Culture#

The authors define Culture using Edgar Schein’s model in Organizational Culture and Leadership. The model posits that culture built on three concepts

Basic assumptions. These are not stated but implicitly agreed upon.
Values. Stated missions and way we approach problems.
Artifacts. Public and conspicious declarations of values and assumptions.

Authors use R. Westrum’s typology of organizational cultures:

Pathological (power-oriented)
Bureaucratic (rule-oriented)
Generative (performance-oriented)

The goal for all organizations is to be a generative culture. Westrum posited that the culture predicts how information flows within an organization.

The authors’ theories on changing culture reminded me of Switch, especially some of the analogies about the Elephant and the Rider, based on Jonathan Haidt’s work.

Working Backwards

Rushabh Doshi — Wed, 04 Aug 2021 16:27:18 -0700

Key Takeaways#

Establish and spell out leadership principles (see Amazon’s fourteen leadership principles). Enforce these principles through mechanisms. Three mechanisms extensively used at Amazon: annual planning process, S-Team goals process, and compensation plan.
Create Single Threaded Leadership to drive autonomy and accountability. See Single Threaded Leadership
Communicate using longform narratives instead of powerpoint. Two extensively used narrative styles are the Six-Pager and the PR/FAQ. Meetings start with a silent read of the document, followed by discussion and debate. See Communication.
Depict your flywheel. Understand how inputs translate into outputs, which in turn affect inputs. Focus on defining, measuring, and analyzing controllable input metrics. Weekly metrics review is the most important operational meeting. See metrics

In addition, the book provides a good historical narrative on the creation of the Kindle, Prime, and AWS.

Amazon’s Leadership Principles#

Customer obsession. Start with the customer and work backwards. Work vigorously to earn and keep customer trust. Keep an eye on the competition but obsess over customers.
Ownership. Think long term and don’t sacrifice long-term value for short-term results. Act on behalf of the entire company. Nothing is “not my job.”
Invent and Simplify. Expect and require innovation and invention from teams and find ways to simplify. Look for new ideas everywhere and don’t limit by “not invented here.”
Be Right, A Lot. Leaders are right a lot. They have strong judgement and good instincts. They seek diverse perspectives and work to disconfirm their beliefs.
Learn and Be Curious. Leaders are never done learning and seek to improve themselves. They are curious about new possibilities.
Hire and Develop the Best. Raise the performance bar with every hire. Recognize exceptional talent and be willing to move them anywhere in the organization. Leaders develop leaders and take the coaching role seriously.
Insist on High Standards. Have relentlessly high standards. Continuously raise the bar and drive the teams to deliver high-quality products, services, and processes. Ensure that defects do not get sent down the line and fixed things stay fixed.
Think Big. Create and communicate a bold direction that inspires results. Think differently and look around corners to fulfill customers.
Bias for Action. Speed matters. Take calculated risks. Reversible decisions do not need extensive study.
Frugality. Accomplish more with less. Constraints breed resourcefulness, self-sufficiency, and innovation.
Earn Trust. Listen attentively, speak candidly, and treat others respectfully. Be vocally self-critical, even when doing so is awkward or embarrasing.
Dive Deep. Operate at all levels, stay connected to the details, audit frequently, and be skeptical when metrics and anecdotes differ. No task is beneath them.
Disagree and Commit. Respectfully challenge decisions when you disagree. Have conviction and be tenacious. Do not compromise for the sake of social cohesion. Once a decision is made, commit fully.
Deliver results. Focus on the key inputs and deliver results with the right quality and in a timely fashion. Never settle.

Mechanisms#

Mechanisms bring leadership principles to life. Three key mechanisms are the annual plan, S-Team goals, and compensation philosophy.

Annual plan (Operating Plan 1, OP1) is a detailed plan that includes an assessment of past performance, key initiatives for the following year, a detailed income statement, and requests for resources (headcount, marketing budget, equipment budget, and other fixed assets.)

S-Team goals (there are lots) are detailed list of input-metric focused goals. They are aggressive and leaders are expected to hit 75% of them.

Compensation is heavily biased toward the long term, on purpose. Rewarding short-term goals at the expense of long term value creation, or rewarding the achievement of localized departmental milestones at the expense of company impact, are considered anti-patterns for compensation.

Single Threaded Leadership#

The best way to fail at inventing something is by making it somebody’s part-time job.

Started with 2-pizza teams, evolved into single threaded leadership. 2-pizza teams were a good idea – small, autonomous teams, that were evaluated by a “fitness function” (weighted average of a number of metrics). However, in practice, they turned out to not be the best solution for all problems (some products needed more investment, lots of arguments over the fitness function, hard to find great 2-pizza team leaders.)

2-pizza teams evolved into Single Threaded Leadership, where a particular person was given autonomy and ownership to build and grow chunks of the Amazon business.

Communication#

No slides. Amazon wants narrative documents. Two key docs:

Six pager. Used to describe and review any idea, process, or business. No particular template; intended to go into detail about the idea being discussed.
PR/FAQ. Key to “Working Backwards” Press Release shows customer centricity and brings out potential problems up front. Internal FAQ goes into detail around expected questions e.g. What is the TAM? How much will it cost?

PR/FAQ structure#

PR structure

Heading
Subheading
Summary paragraph
Problem paragraph
Solution paragraph
Quotes and Getting Started

FAQ is less structured

TAM
Economics and P&L

Metrics#

Metric focus is on managing inputs because the CEO or other leaders rarely have complete control over their output metric (e.g. share price).

Weekly Business Review is a weekly meeting to go over all the metrics in detail. Presented as a detailed deck, comprising mostly of charts, graphs, and data tables. Review emerging patterns. Plots of graphs against comparable previous times (week-over-week or month-over-month or year-over-year).

During the review, the focus is on variances and not on business-as-usual. Business owners are responsible for understanding and explaining variances.

Metrics Lifecycle#

Select and define the metrics to measure.
Understand the flywheel. How do input metrics lead to output metrics and back again.
Identify the correct, controllable input metrics. This step sounds simple but can be very tricky. A mistake here will result in the wrong outcome.
Measure.
Analyze: develop a comprehensive understanding of what drives the metrics. Use the 5-Whys method to understand what is going on.
Improve. Once the previous steps are fulfilled, you can improve upon your metrics.

Taming Slack

Rushabh Doshi — Sun, 01 Aug 2021 00:00:00 +0000

Photo by Scott Webb on Unsplash

Digit is a Slack-only company. While we use email for automated alerts and GitHub emails, nearly all of our communication happens on Slack. We’re north of 100 employees, and without proper usage, Slack can get incredibly noisy and distracting.

While one may be tempted to blame Slack, the root of the problem is human communication – communication volume grows quadratically with the growth of a community. Organizational hierarchies are one solution to this problem: they impose a tree-like communication structure that, in theory, dampens the volume of communication. However, deeply hierarchical organizations have other issues, resulting in modern companies trying to be “flat.” In a flat, trusting, information-should-be-free company, there are lots of messages floating around.

Using Slack in such an environment gives rise to some common anti-patterns:

Missed messages. Important messages that need a response get buried in a sea of business-as-usual stuff. To ensure that everyone reads important messages, people start using @here and @channel
Channel disorganization. Lots of channels, lots of messages. Nobody knows the “right” channel to post or follow a particular message. So people post to the most convenient channel, often breaking a team’s flow or causing problem #1.
Interwoven threads. We commonly have more than one conversation going on at any time in one channel. Because each message is a top-level object, multiple conversations get woven together, making one stream challenging to follow.
No inbox zero. For people (like me) who like to clear their queues, Slack can be challenging. Where do you store messages that you want to get to later?

Not all of these problems are specific to Slack. We’ve certainly seen variants of all of these with email. Email has had a more extended timeframe for people to build tooling around it and specialize to suit their needs. Slack is relatively new – and so we have a greater need to develop conventions from scratch.

Here are some of the things that have worked to help tame Slack, divided into team-related conventions and personal workflow.

Team Practices#

1. Have a channel naming convention#

Channels in Slack are free and can explode. Channel names should signify a channel’s purpose. Names help people understand where to post things without having to read through all the channel’s content. For example, #engineering-fyi, #product-fyi, #release-fyi, and other #-fyi channels are meant for important information about a particular topic. They are generally low volume, but when you see something pop up, it’s probably critical. #savings-team, #onboarding-team, and other #-team channels, on the other hand, are for everyday team conversations. High volume, a mix of business and banter, generally not intended for people who are not on the team (but we keep them public, more on that later.) Some channels (e.g. #feedback) have specific conventions associated with them and automated bots to help run these channels. “How to run a triage channel” is an excellent example of a triage channel, which we implemented using a custom slack bot.

2. Encourage threaded replies#

For high-volume channels, a conversation can become difficult to follow if messages are interspersed. Essential conversations can get lost or be left incomplete while being hard to follow for interested parties. This problem is more acute in broad channels than DMs or small groups. Using threaded replies solves this problem – however, getting everyone to use threaded replies is challenging. We use a cute :shame: emoji to gently remind people to use threaded replies – it works well, and people laugh at the inevitable slip-ups.

3. Ban `@here` and `@channel`#

@mentions can be incredibly distracting because they notify people in most circumstances. @channel and @here take this distraction to a new level by notifying everyone in a channel; when used in a broad channel, like #fyi can essentially notify the entire company. There are certain rare occasions when breaking through peoples’ filters is necessary: @channel we're shutting down the office because of covid-19. In most cases, however, using these notification mechanisms is rude and should be avoided. It assumes that people don’t read their Slack messages or that whatever you’re posting is important enough that they need to look at it now. @here makes very little sense in a distributed, remote world – but teams tend to use it as a sort of lightweight @channel, as if notifying the people who happen to be working right now is okay, but notifying others is not. There are circumstances where these make sense: @here I'm locked out, can someone please open the door but other than that, these should be banned.

4. Discourage group DMs#

How many times have you started a “quick message” between a few colleagues as a group DM, which has subsequently escalated into an entire meaningful conversation, which needs some more people … and you’re stuck. Because Slack doesn’t allow expanding group DMs (come on, Slack folks!) There is a solution - it’s free and easy – it’s called a channel. Just convert group DMs into a private channel as soon as possible. We encourage people to start public channels for brief discussions with a naming convention - #tmp-something-something - and archive it once complete. Group DMs aren’t indexed well either and generally suck. Channels are free; use them.

5. Post in the right channel, cross-post for visibility#

Often, we don’t know where we should post a particular message. Not knowing is okay – we post in a broad channel and ask or take our best guess. Have a little emoji to indicate that you should probably ask the question in a different channel: we use :polite-racoon:. Sometimes, a message is important enough for a few related channels. We post the message to the most appropriate channel in such cases and share it (Slack s) to other channels.

6. Setup emoji conventions#

Slack doesn’t let people know you’ve read their message or taken care of something: a vexing problem when, for example, delegating something to someone. We use the 👀 :eyes: emoji to indicate that we’ve read something, ✅ :white_check_mark: to suggest that something has been taken care of. Some channels have even more specific emoji conventions in use, but people broadly understand these two across the company. When we have a call to action, we post a message similar to Please do this task and ✅ this when you've done it. – this helps us keep track of who’s done something, and we can follow up accordingly.

Personal Tips#

Team tips are great to help the whole team use Slack in the same way. To achieve productivity zen, you need to combine these with ways to manage your unread queues and reach inbox zero.

1. Use `/mute` liberally#

Lots of channels mean lots of messages. Everyone wants to be in the loop on everything, but the reality is, much like world news, most of it doesn’t affect you daily. To get some control over information overload (so you’re not just reading slack messages all day), you can do two things: leave irrelevant channels or mute them. I prefer to mute them to read them at my leisure, without affecting my unread count or showing up in my Unread section.

2. Use `/remind` or `Save` to keep your GTD flow#

There are two GTD flows that I’ve seen people use (or some mix of both): Remind and Save. Remind essentially takes a message and asks Slack to bring it back as unread in some time: similar to Gmail’s snooze feature (itself of Inbox fame.) The Slack command is /remind.

The alternative is to use a queue and clear it every morning and evening (or whatever cadence suits you.) Clearing queues is my preferred approach – I use Save or hotkey `a’ when reading my messages – to keep track of things I need to get back to.

3. Use scheduled posts for after-hours posting#

While we operate on mostly west coast hours, we’re split all over the world. Moreover, we all have different working schedules – with my family, I sign out starting around 5 pm but am incredibly productive between 9 pm - 1 am (a combination of my night owl-ness and my family sleeping peacefully.) However, sending people tons of slack messages at midnight is … not good. It can come across as a poor work flex (look at how hard I’m working), and it doesn’t work: you’re not going to get replies in the middle of the night anyway. So use scheduled posting (🙏🏽🙏🏽🙏🏽 Slack for finally shipping this) to post at more reasonable hours. Alternately, I queue up some drafts and fire them off first thing in the morning.

4. Use `Notify me` for threaded conversations#

Sometimes you’ll see a fascinating topic, but you’ll lose out on the subsequent conversation because your team uses threads. Instead of chiming in with a Following or similar message, you can hit the ⠇ button on the thread and ask Slack to notify you of future updates. Now everything shows up in your threaded list and is easy to clear, like any other queue.

5. Quickly read through `All Unreads` and `Threads`#

Both of these help me go through all incoming messages rapidly and clear my queues. When I find myself consistently thumbing through a particular channel without responding to it, I mute it.

6. Organize channels by topic#

Slack launched custom sections to allow creating folders for channel organization. Folders let me manage my channels by topic, such as “team,” or “recruiting,” or “engineering.”

I hope these tips are helpful. If there are other things you or your organization do to make Slack usage more effective, I’d love to know - please drop me an email or DM.

Thanks to David Kossnick for thoughtful suggestions and edits.

Are Your Lights On

Rushabh Doshi — Sun, 23 May 2021 00:00:00 +0000

“Are Your Lights On” is a humorous exploration into the heart of Problem Solving. The book is fun, light-hearted, and profound, all at the same time.

Allegory#

The book’s namesake allegory goes thus: There is a tunnel through a mountain, leading to beautiful Switzerland. As the engineer building the tunnel, you put a sign at the start that says WARNING: TUNNEL AHEAD PLEASE TURN YOUR HEADLIGHTS ON

After going through the tunnel, tourists regularly stop by the roadside to take in the beautiful views and have a picnic. However, they also forget to turn on their headlights, leading to an unwelcome surprise at the end of their lovely picnic: their car no longer starts because the battery has been drained.

As the engineer in charge of the tunnel, you must solve this problem. How should you do it?

The book explores a variety of questions that help one explore the heart of the problem: Whose problem is it? What is the actual problem? Should this problem be solved? What is a good solution?

In the case of our intrepid engineer, the problem is relatively straightforward, but the solution ranges from the over-complicated (put a sign that says turn your lights off, except if it’s night, in which case leave them on) to a gentle nudge. We learn that putting up a simple sign at the end of the tunnel does the trick. ARE YOUR LIGHTS ON?

Key Ideas#

Problem solvers rush in with solutions before taking the time to define the problem being solved. Even experienced people fall into this trap, exacerbated by social pressure and a demand for haste. When in this mode, people find many solutions and compete for acceptance of a favored solution; they accuse each other of not being open to a particular solution or an alternate point of view.

The authors suggest a deeper exploration of the problem itself before jumping into coming up with solutions. In particular, they suggest asking some questions:

What is the problem?#

A problem is defined as The difference between things as desired and things as perceived.

People often pass on a solution as the problem definition; it isn’t. Especially if it’s your solution method.

Defining the problem is very important: small word changes can substantially alter the problem definition. Play with it until it’s right.

Whose problem is it?#

Close the gap between the person experiencing the problem and the person solving the problem.

Where does the problem come from?#

Was it created by someone, e.g., an examination problem? These problems are often Puzzles, not Problems.

Do we really want to solve the problem?#

Solutions beget other problems. Thus one problem is displaced with another problem. Our goal is to simply replace the current issue with one that is more tolerable (or one that doesn’t affect us.)

Not every problem should be solved. People may not know what they are asking for until you give it to them (and then they realize it isn’t.)

We never have enough time to consider whether we want it, but we have plenty of time to regret it.

Building Cathedrals

Rushabh Doshi — Tue, 27 Apr 2021 00:00:00 +0000

The York Minster Cathedral took 242 years to build Photo by MatzeTrier CC BY-SA 3.0

Building great products takes enormous hard work over a long time. When we’re deep in the execution tunnel, blocking and tackling the problems that come our way, we can lose sight of the greater goal. Whenever that happens, motivation sinks, and both the volume out of output and the quality of work suffer.

We have to remind ourselves and our teams of the greater purpose constantly. What is the big problem we’re solving? Why does it matter? Where does it go over the next decade?

One of my favorite stories:

Two bricklayers are busy at work.

You walk up to them and ask: “What is your job?” “I’m laying bricks,” they say.

Ask the second: “What is your job?” and they reply: “I’m building a cathedral.”

We are all building cathedrals. We need the occasional reminder because we have to lay down a million bricks.

Complex Problem Solving: The Ever Given Saga

Rushabh Doshi — Sun, 18 Apr 2021 00:00:00 +0000

The Problem#

The Ever Given ran aground March 23rd, 2021, at 05:40 GMT and blocked the Suez canal.

via NYTimes

The ship was stuck like a giant windsail in the middle of a sandstorm with 46 mph winds.

Ships operating in constricted waterways like the Suez are subject to the Bank effect: the ship’s stern tends to stick closer to the bank.

Between being pushed by the wind and due to the bank effect, the ship’s stern ran into the left bank, followed by the bow going into the right bank, completely blocking the canal. Here’s a great video showing the ship’s maneuvering difficulties.

Modern ships have a bulbous bow: a protrusion at the bow (or front) of a ship just below the waterline. The bulb modifies the way the water flows around the hull, reducing drag and thus increasing speed, range, fuel efficiency, and stability. Here’s a great video explaining what a bulbous bow is for.

What’s not evident in this widely shared photo was just how deep the bow was wedged into the ground.

Suez Authority Facebook Page, via gCaptain

I won’t go into why this was a huge deal - the press focused enough on the economic implications of the Suez shutdown: delays in the global supply chain, lost revenue for the Egyptian government, millions of dollars of losses for shipping and container companies; the list goes on.

What is the Problem?#

Imagine you’re in charge of sorting out this problem. What do you do?

When presented with a problem of this nature, it’s tempting to jump in and immediately solve the readily available problem.

The ship is wedged, so let’s get some tugboats and pull it out.

However, for problem-solving in general, but especially for high-pressure situations, we must first fully understand the problem. And the problem may not be apparent at first glance.

So what is the problem?#

The Suez canal is completely blocked. The southern part of the Suez has no alternative routes (in contrast to the northern section.) Global shipping is halted and will stay stuck for days or weeks while pulling the Ever Given out.

What else is a problem?#

What happens if we stick a 100 ft sail in the middle of a giant windstorm? We may have lots of containers in the Suez, potentially blocking it off for months.

What else?#

When the tide goes out, we will have a very long, heavily loaded container ship, with its bow and stern stuck on the ground like poles. Long container ships are subject to hogging and sagging - the ship could bend at its middle and potentially break into two.

That sounds very bad - could it happen? Yes, in fact, it has - with the MOL comfort snapping in high seas due to severe weather.

via Maritime Executive

If the Ever Given were to snap and sink in the middle of the Suez, it would block the canal for years, potentially requiring building a new channel to get around the blockage.

These are some pretty distressing and alarming facts. But then again, you’re not called into high-pressure situations when things are going swimmingly.

Enter Mind Maps#

Mind Maps are one of the best tools to work through problem understanding and solving. Mind maps allow you to organize hierarchical information visually. They allow you to see the entire picture and quickly zoom in and out of the problem.

For each sub-problem, we can develop mutually exclusive but completely exhaustive (MECE) options to solve it.

To solve for the ship breaking, we need to prevent sagging. We can move the ballast to the sides of the ship; we could remove fuel to lighten the load (and reduce the possibility of a catastrophic oil spill in the Suez in the event of the boat cracking); we could move containers off the ship.

To prevent tipping, we need to move containers out (also incidentally helping the sagging problem). However, moving containers off a 100ft mega-ship without using land-based cranes is virtually impossible.

To unstick the vessel:

We could use tugs. However, we don’t have enough tug capacity in the world to pull the vessel out. Some simple friction math says we need about 33kTons of force to pull out the ship. The world’s best tugboat - the Far Samson - can do 423 tons. And there is only one Far Samson, and it’s nowhere near the Suez.
We can excavate the bow and the stern and free them. However, excavating sand vertically underwater is not possible due to the way sand moves. There are no amphibious excavators with sufficient reach.
We can use a dredging ship instead.
We can wait for the high tide. As the tide lifts the boat, with sufficient dredging and some tugging, we can hopefully pull the boat out.
We can use high-capacity cranes to lift the containers off the boat. This is very challenging given the boat’s height and the lack of existing cranes for the operation. Given the sheer number of containers, this would also be a very long operation.

We can see all this in one place in a solution mind map.

The Technique#

A more generalized approach looks like this:

1. Understand the problem#

This is the first step. The real problem may not be the obvious, evident problem staring you in the face. To get to the real problem, start by creating a problem map. Answer the prompt: What is the problem? and dig in - with each child ask Why? and to generate more children ask What else?.

2. Prioritize#

Once you understand all the associated problems, not just the first one you saw, figure out the order in which you should solve them. The function used to prioritize depends on the context and not germane to this section.

3. Create independent solutions for each problem#

For each identified problem, create a set of solutions that would effectively solve the problem. Dig into the ideas to understand the feasibility of implementation, the cost, the time, and other such attributes. An important note: we’re solving each sub-problem in isolation. A solution for one may aid or hamper another – we must pay attention to these dependencies (solutions may not be mutually exclusive).

4. Execute#

You may be able to execute in serial or parallel, and you may find some jugaad way of doing multiple things at once. Knowing your prioritized problem map helps ensure the work is directed toward the highest priority first.

The Ever Given Conclusion#

The Ever Given was towed to the Great Bitter Lake, where it remains as of April 18th, 2021. The Egyptian authorities are demanding nearly a Billion Dollars in damager. Given the boat’s interesting international situation (owned by a Japanese company, operated by a Taiwanese company, registered in Panama, technically managed by a German company, crewed by Indians, stuck in Egypt.), who knows how long it will stay there.

Codanames

Rushabh Doshi — Thu, 25 Feb 2021 00:00:00 +0000

I love codenames and coda. Decided to implement codenames using coda:

https://coda.io/@rushabhdoshi/codanames

Switch

Rushabh Doshi — Mon, 15 Feb 2021 00:00:00 +0000

Key Ideas#

Change is hard. To execute change, leaders have to act differently and motivate their teams to act differently. This book details a framework to effect change.

Elephants and Riders#

Based on Jonathan Haidt’s work, the authors describe how each person has an emotional Elephant side and a rational Rider side. What motivates one does not help with the other. To effect change, you must appeal to both the Elephant and the Rider.

Directing the Rider#

Find what’s working. Instead of focusing on problem analysis, find the thing that is working, and clone it. Follow the bright spots.
Script the critical moves. Find a series of small, but pivotal, steps to change, and script them. The critical part is important: else you are unsustainably scripting everything.
Point to the destination. Create a Big, Hairy, Audacious Goal [[BHAG]] or a Destination Postcard. These do double duty by showing the Rider where you’re headed and show the Elephant why the journey is worthwhile.

Motivate the Elephant#

Find the feeling - rational logic isn’t enough; make people feel something viscerally. People think the sequence is ANALYZE-THINK-CHANGE but in reality, for most big changes, the sequence is SEE-FEEL-CHANGE.
Shrink the change - break down the change until it not scary or demoralizing. Every large change starts with a small change, make that change a small win.
Grow your people - cultivate a sense of identity and instill a growth mindset.

Shape the Path#

Tweak the environment - change the situation instead of changing the behavior.
Build habits - look for ways to encourage habits rather than banking on motivation.
Rally the herd - behavior is contagious. Help it spread.

Other interesting ideas#

Prefer to use positive emotion to motivate, vs. scaring people or using negative emotion such as the Burning Platform Memo (choice between a fiery death or a cold sea and shark food.)

The Haddon Matrix is a simple framework that provides a way to think systematically about accidents by highlighting three key periods of time:

Pre-event interventions include things that would prevent. wrecks from happening - installing bright lighting on highways, painting clear lane markers on the roads.
Event interventions will accept that crashes will happen and ask how we can reduce the changes of injury. Seat belts and air bags.
Post-event interventions acknowledge that both crashes will happen and people will get injured. Minimize the severity of injuries and optimize the health outcomes - speedy emergency crews, jaws of life.

The Checklist is a great tool to accomplish two things at once: 1) tweaking the environment and 2) building habits.

Burnout

Rushabh Doshi — Sun, 14 Feb 2021 00:00:00 +0000

Key Ideas#

The authors present a systems thinking approach to dealing with Stress.

Stressors create stress which accumulates until the Stress Cycle is completed. If left to continuously grow over time, stress results in Burnout - defined by emotional exhaustion, depersonalization, and a decreased sense of accomplishment.

Approaches to completing the stress cycle: Physical activity of 20-60m per day (most effective), deep breathing, positive social interaction, and creative expression.

Approaches to reducing stressors: planful problem solving, positive reappraisal, redefining winning and failing, and finding meaning.

Details#

The definition of Burnout according to Herbert Freudenberger in 1975, is a state of:

Emotional exhaustion charachterized by fatigue.
Depersonalization resulting in a depletion of empathy, caring, and compassion.
Decreased sense of accomplishment leading to a deep sense of futility.

Stress vs. Stressors#

Stressors activate the stress response.
Stress is the neurological and physiological shift that happens in your body when you encounter a stressor. For example: Being chased by a hippo (stressor) generates neurochemicals that initiate a physiological response to aid survival (stress response).

According to the authors, we focus too much on controlling our stressors, and not enough on completing the stress cycle. The lack of completion of the stress cycle results in stress accumulation, eventually resulting in burnout.

Completing the Stress Cycle#

Physical activity - 20 to 60 minutes per days is the most efficient strategy for completing the stress response cycle.
Breathing - slow breaths with long exhalation. Most effective when the stress isn’t that high.
Positive social interaction - external sign that the world is a safe place.
Laughter
Affection
Crying
Creative Expression - engaging in creative activities that lead to more energy, excitement, and enthusiasm.

Dealing with Stressors#

Planful problem solving - analyze the problem, make a plan based on your analysis, then execute the plan. Works for people who naturally make lists and follow calendars.
Positive reappraisal - post-hoc rationalization that things are actually worth it. Look at the positive side and realize the opportunity hidden in the stress.
Redefining winning and failing - winning can often be defined in binary terms eg. Climb Mt Everest, which is incredibly hard and subject to failure. Instead, it could be defined as a shorter goal - making it to base camp, making it to the next stage, and so on. Redefined goals should be:
- Soon
- Certain
- Positive
- Concrete
- Specific
- Personal
Finding Meaning or Something Larger
- Meaning offers a positive, final value, than an individual’s life can exhibit. It isn’t always fun, and not all activities are meaningful. But meaning is invariably good for you.
- Three kinds of sources
  - Pursuit and achievement of ambitious goals that leave a legacy.
  - Service to the divine or other spiritual calling.
  - Loving, emotionally intimate connection with others.

2021

Rushabh Doshi — Sat, 06 Feb 2021 00:00:00 +0000

I cannot believe we are in February already. 2020 was a clusterf*** of a year – COVID, forced isolation, millions sick, hundreds of thousands dead, a nerve-wracking election, and the aftermath that would not end. January felt like a continuation of December'20 to some extent. But thankfully, things are looking a little better. Vaccines are out, we have a government made of adults, and things are looking up.

I’m starting the year with an upgrade to the website. You’ll notice a new theme and some splashes of color – they’re nice, but the key upgrade is below the hood. I’ve moved off Jekyll to Hugo. I don’t love Ruby; I don’t love how Jekyll has tons of ugly _s everywhere; I’m not fond of managing dependencies. Built using Go, Hugo is clean and fast. Structure and templates make sense. The community has ported most of the Jekyll templates over. I’m still using GitHub Pages to host (free and straightforward) but moved the site generation over to Hugo.

Enough about SSGs. Though the world of SSGs is fascinating and worth a peek. Why have fancy database-driven websites (I’m looking at you, WordPress) when a simple statically generated site could suffice? And the interwebs are very good at pushing bits of text around – no fancy database lookups, and everything works blazingly fast.

I care about writing, and this upgrade will let me write more. My goal is to write shorter posts more often. I hope you, dear reader, enjoy them.

Here’s to a great 2021 and a giant f-you to 2020.

Hooked

Rushabh Doshi — Fri, 25 Dec 2020 00:00:00 +0000

Key Ideas#

Habit forming products can be created using the Hooked model: Trigger, Action, Variable Rewards, and Investment.

Details#

Habits#

Habits are defined as automatic behaviors triggered by situational cues.

Habits give us the ability to focus our attention on other things by storing automatic responses. Our brain takes a shortcut and stops actively deliberating over what to do next. The brain then codifies behaviors that provide a solution to whatever situation it encounters.

New habits are hard to form and easy to lose. To convert a behavior into a habit, it needs to happen frequently and deliver high utility.

Habit forming products are both Painkillers (solve an obvious need, relieving a specific pain, have quantifiable markets) and Vitamins (appeal to users emotional, rather than functional needs.)

The Hooked Model#

The hooked model is a 4 step process to building habit forming products.

1. Trigger#

A trigger actuates the behavior by providing the spark. There are two types of triggers:

External triggers
- Paid triggers: Ads
- Earned triggers: Favorable press mentions, app store feature placements, viral videos.
- Relationship triggers: One person telling another about the product, Social media shares, word of mouth.
- Owned triggers. Email newsletter, app icon, notifications.
Internal triggers
- When a product becomes associated with an emotion, thought, pre-existing routine, these become internal triggers to use the product. For example, people take a photo with the intent of posting it to Instagram.

2. Action#

The author’s rely on BJ Fogg’s Fogg Behavioral Model to illustrate the MAP method for when people will take an action.

Some hearistics to increase motivation:

Scarcity effect: Appearance of scarcity increases the value of a product. Eg. a limited time offer.
Framing effect: Surroundings and context creates motivation. Eg. a concert pianist playing in a subway vs Carnegie Hall gets different audience attention.
Anchoring effect: People anchor on one piece of information when making a decision. Eg. “Everything 30% off” fails to consider the markup or whether something was needed.
Endowed Progress effect: Eg. For a loyalty card, starting with a blank card, with 9 spots to reward is worse than starting with a card with 10 spots and marking one off.

3. Variable Reward#

The variable part is important. Without that, we have a vanilla feedback loop.

Types of rewards#

Rewards of the Tribe: make us feel accepted, attractive, important, and included.
Rewards of the Hunt: based on the need to acquire physical objects.
Rewards of the Self: intrinsic motivation to conquer obstacles, even for simply the satisfaction of doing so.

Finite vs Infinite Variability#

Finite variability is the same script, different characters, different show. This is commonly found in Hollywood (eg. Special Victims Unit, the best show ever created). The same model is exploited by Zynga and other FTP games. Companies making products with finite variability often operate under the studio model.
Infinite variability. Multiplayer games, UGC sites like Facebook, YouTube, Pinterest. The variability never follows the same script.
Inherent variability. When the act of using your product itself is variable, there is no need to add variability on top.
- Eg. Google searches and Uber rides.

4. Investment#

Getting users to do a bit of work increases the odds that they will make another pass through the entire cycle. Strategies include:

Escalation of commitment: the more users invest time and effort into a product or service, the more they value it. Heuristics to understand why this happens:

Ikea Effect - irrationally valueing our own efforts. Ikea furniture looks great because we made it ourselves.
Consistency with past behavior - foot in the door effect. Once we’ve done something small, doing the next thing is easy.
Avoiding cognitive dissonance - we value consistency with past behaviors, thoughts, and attitudes.

Storing value: getting users to store value in your product increases the chances they will use it again and reduces their propensity to churn. Ways to create stored value:

Content Eg. Spotify listening history.
Data Eg. Entering all financial information into Mint
Followers Eg. Twitter follower graph.
Reputation Eg. Online marketplaces like Yelp, AirBnB, Upwork
Skill Eg. Photoshop or Figma or any prosumer tool that users have learned to use.

How to Take Smart Notes

Rushabh Doshi — Sun, 29 Nov 2020 00:00:00 +0000

Key Ideas#

Generating insights is key to accelerate research and learning. To generate insights, Ahrens recommends using the Zettelkasten method to create a Second Brain that helps see relationships between concepts, discover open holes, and validate or invalidate understanding.

Read by taking notes and noting concepts in your own words. This process helps deepen our understanding.

Details#

How do you plan for insight, which, by definition, cannot be anticipated?

Ahrens’s book, while directed at students and researchers, is applicable to everyone that cares about learning new things.

To generate more insights, Ahrens recommends changing the learning workflow. Reading alone is insufficient and having read more does not mean more ideas.

Zettelkasten system to store and connect ideas#

The Zettelkasten system relies on a series of interconnected notes. The original system was built using pen and paper by Niklas Luhmann. He relied on writing notes that were organized by topic, putting physical pieces of paper next to each other in a slip-box. Each piece of paper also had linkages to other notes. He relied upon these linkages to generate insight and connect one thing to another.

Replicating the system today is easier, thanks to technology.

Fleeting Notes#

The first step is to capture all your ideas - whether they’re coming from things you read, or during a shower, or on a walk, or when you’re browsing the internet. Use whatever means you’d like to capture fleeting notes - pen and paper, voice memos, apps on your devices.

When reading books and articles, I find it particularly easy to highlight things – Kindle makes it very easy to highlight things, as well as revisit all highlighted content. Importantly, the act of highlighting is insufficient. Treat highlights as another fleeting note.

I also use Notion to maintain a web reading queue. Alternatives include browser bookmarks, Pocket and Instapaper.

Permanent Notes#

All forms of fleeting notes can be converted to permanent notes, though not all should. There are plenty of fleeting notes that weren’t great ideas or didn’t generate any particular value, and can be discarded.

Permanent notes require more thought to put together and are short summaries of fleeting notes. I use Roam Research, though Obsidian is a very reasonable alternative.

Links#

Linking permanent notes together is an extremely important step. Tools like Roam Research allow you to very easily link permanent notes together. Moreover, they automatically generate backlinks and maintain your link graph.

You can use a tag scheme to link notes together, however, be cautious. An Archivist tags notes to figure out where to store them. A Researcher tags notes to figure out how to retrieve them. As yourself the question: In what circumstances will I want to stumble upon this note, even if I forget about it?

GTD vs. Insightful Writing#

David Allen’s Getting Things Done system is an excellent way to be productive. The central insight in GTD is to collect everything in one place and process it in a standard way. This system forces clear prioritization and continuously evaluates how tasks fit into the bigger picture.

GTD is different from Insightful Writing (Ahrens’s method). They are hard to combine – for GTD to work, you must know where you’re trying to go (clear objective) and the path can be decomposed into smaller steps that guarantee progress. Research, writing, and insight generation are not linear and we have to constantly jump back and forth between tasks.

Learn by Writing#

If you want to learn something for the long run, you have to write it down.

Write with a slip-box. Develop topics, questions, and research projects from the bottom up within the system. Turn the permanent notes into a rough draft. You should be able to see the holes in the argument and do more research.

Read by Writing#

Take Notes

I would advise you to read with a pen in your hand and enter in a little book short hints of what you feel that is common or that may be useful; for this will be the best method of imprinting such portcullis in your memory. – Benjamin Franklin
Be careful about Confirmation Bias. We are often seeking supporting evidence instead of reading with an open mind to truly understand. Darwin’s technique for avoiding confirmation bias was to force himself to write down and elaborate upon the argument that were the most critical of his theories.
Get the gist. Summarize and pull out only the most important bits.

Anti-Brainstorming#

Brainstorming is an old technique, introduced to the world in 1958. It tests for memorized knowledge, without much indication of insight. Someone can come up with a lot of ideas during brainstorming sessions but give no indication about their quality. Brain prioritizes ideas that are easily available, not the best or the most relevant. The brain also remembers things that have happened more recently. Things that are abstract or vague get pushed down the list. Additionally we love our own ideas and get wedded to them pretty quickly. Prefer the bottom-up, deliberate approach to generating ideas and insights to a “brainstorm”.

Hacking Growth

Rushabh Doshi — Mon, 23 Nov 2020 00:00:00 +0000

Key Insights#

The Pirate Stack, or where a Growth team works (with lots of concrete ideas about what to do in each phase.)

Acquisition
Activation
Retention
Revenue

Avoid Growth stalls

Don’t get complacent due to overconfidence in market positioning.
Don’t lose focus on core products or services.
Innovate.

Grow or Die

Double down on ideas that are working.
Run a disciplined team and process.
Experiment, experiment, experiment.

Details#

How Growth teams are commonly structured#

Growth teams are a multidisciplinary team with a focus on speed and experimentation. They usually comprise of:

Product Managers with deep understanding of strategy and business goals.
Data Analysts focusing on Experimental design and analysis. They generate insights into customer behavior and segmentation.
Marketing
Leadership. Growth teams should be led by the equivalent of a Battalion commander:boots on the ground, managing the day to day, and actively participating in the idea generation and experimentation process. they should have basic fluency with data analysis, product management, designing and running experiments.

There are two flavors of Growth team leadership and org structure:

Product Led Growth Team. They report into product leadership. Advantage is that they are more tightly integrated with the rest of product.
Independent Led Growth Team. They report into the CEO. Advantage is that they have more autonomy to work across the company.

Growth Hacking Process#

Analyze data and gather insights.
Prioritize experiments.
Run the experiments.
Generate ideas.
Back to step 1.

Grow a successful product#

Make sure that a product has Product-Market Fit (PMF) before growth. Determining PMF using a survey:

How disappointed would you be if this product no longer existed tomorrow?
- a) Very disappointed
- b) Somewhat disappointed
- c) Not disappointed (it really isn’t that useful)
- d) N/A—I no longer use it
What would you likely use as an alternative to [name of product] if it were no longer available?
- I probably wouldn’t use an alternative
- I would use:
What is the primary benefit that you have received from [name of product]?
Have you recommended [name of product] to anyone?
- No
- Yes (Please explain how you described it)
What type of person do you think would benefit most from [name of product]?
How can we improve [name of product] to better meet your needs?
Would it be okay if we followed up by email to request a clarification to one or more of your responses?

Getting to Must-Have#

Talk to users (or risk building features that nobody wants.) Listen and observe. Be careful to not sell.

Analyze your data:

Create a data lake.
Set up event tracking
Look at feature correlations for

Fundamental Growth Equation#

A growth equation is a simple formula that represents all of the key factors that will combine to drive your growth. It is different for every product or business. Common drivers:

New user acquisition
Higher activation
Better retention

North Star Metric#

A number that we can obsess over. It must be simple eg. for Uber Rides, the metric was Rides Completed. For Facebook sharing, our number was Original Posts. This metric isn’t set in stone and can change over time. Don’t obsess about getting it perfect.

A good plan violently executed now is better than a perfect plan tomorrow. – General George Patton

Growth Experiments#

Growth teams have a high experiment cadence (20-30 / week at scale). There are two phases, corresponding to two meetings:

Idea generation. Everyone is welcome to add ideas; focus area owners must add ideas. Ideas should be crisp and clearly articulate what we expect to move.
Weekly prioritization. Prioritize the list of ideas using some scoring system. Examples of scoring systems:
- ICE
  - Impact
  - Confidence
  - Ease
- TIR
  - Time
  - Impact
  - Resources
- PIE
  - Potential
  - Importance
  - Ease
Experiment review. Look at all existing experiment for success or failure. Use a 99% confidence level instead of 95%.

[[Acquisition]]#

Start by crafting a compelling message. The message is not just about your brand but improves the product as well.

Pick a channel. Be selective and put resources on fewer channels rather than spreading bets around. See the book for examples of channel categories, and Brian Balfour’s strategies around picking the right channel.

Activation#

Get new users to the aha moment faster:

Map the route to the aha moment.
Create a funnel report of conversions and dropoffs.
Eradicate friction.

Optimize the New User Experience (NUX). The first time someone experiences your product is a unique, one-time encounter. The NUX should be treated as a product by itself. Craft a special experiencee.

Retention#

Retention is fundamentally about providing customers with a product or service of high quality that continuously addresses their needs, or delights them, and which they come to regard as must have.

Phases of retention:

Initial. This is an extension of activation work. What gets users from merely activated to commited?
Medium. The Hooked model by Nir Eyal provides a model to turn products into habits. Here’s my summary of Hooked.
Long term. Optimize existing features. Introduce a steady stream of new features. Resurrect zombie customers.

Revenue#

Authors give a survey based method for pricing SaaS products.

Pricing Relativity: people’s perceptions of prices are influenced by the prices of other options they are offered.

Pricing requires a study of Consumer Psyechology. References:

Predictably Irrational by Dan Arielly
The Art of Choosing by economist Sheena Iyengar.
Robert Cialdini’s principles from Influence.

Building a Virtuous Growth Cycle#

Avoid Growth stalls

Complacency due to overconfidence in market positioning.
Lose focus on core products or services.
Failure to innovate.

Grow or die

Double down on ideas that are working.
Dig deeper for data gold.
Dive into new channels as they are built.
Open up the ideation process.

The Humble Note-Taker

Rushabh Doshi — Thu, 03 Sep 2020 00:00:00 +0000

Welcome to your first team meeting as the new PM. You’re busy trying to learn what the team does and who everyone is. The team is discussing an important decision they need to make. What should you do?

Do nothing. You don’t have all the context, how could you contribute?
Jump in with your ideas. You’re a person of action and this is why they hired you.
Offer to take notes. Ask a clarifying question. Post the notes after the meeting.

Doing nothing, while gentle, is ineffective. You don’t make progress in building trust with your team or extend the understanding of the subject matter.

Jumping in with your ideas is plain obnoxious. You have no context. In your rush to “add value” you’re showing everyone that you know barely anything.

Taking notes, asking the occasional clarifying question, perhaps suggesting a minor reframing, and posting the notes publicly — these are activities that simultaneously add value and improve your own understanding and knowledge.

Note-taking is historic record-keeping. Nobody will remember what they discussed or agreed upon, a few days after the meeting. What everyone will look at are the notes. If you have any doubts about this, try and think of your precise position on a meeting you were in, one week ago.

Note-taking is an incredibly powerful way to learn. People learn through writing. By taking notes, you force yourself to understand things and ask questions. In some cases, you find references to read through, deepening your knowledge. In other cases, the questions you have are the same questions everyone else has. Either way, your learning accelerates rapidly through taking notes.

Note-taking progresses rapidly to decision-making. You start by taking notes for your team. You post them publicly. Everyone is so grateful that you took on a menial and tedious task. Slowly, you develop the most in-depth knowledge of everything the team is doing. People start asking you for your opinion. At some point, they delegate the decision over to you, because you have the most knowledge and most up-to-date view.

Thanks to Shishir Mehrotra, for the original idea, years ago. Thanks to Jules Walter and DeVaris Brown for proof-reading and feedback.

10 Tips For Using OKRs Effectively

Rushabh Doshi — Thu, 18 Jun 2020 00:00:00 +0000

We are in planning season at Digit. Like a number of other technology companies, we use OKRs. I was puzzled by the wide range of OKR effectiveness at different companies. My study of the nuances of different approaches led to the conclusions outlined in this note. OKRs are incredibly useful but it takes a long time for OKR usage to become effective. Shortcut your company’s learning process by understanding and implementing the following ten ideas. If you find them helpful, drop me a note.

What are OKRs?#

OKRs stands for Objectives and Key Results. They capture two things:

Objective: The goal a company wants to achieve (“What”)
Key Results: The path to achieving that goal (“How”)

OKRs are an invaluable management tool for organizations of all sizes. Used properly, they allow an organization to unlock four superpowers:

Focus and commit to priorities.
Align and connect for teamwork.
Accountability through tracking.
Stretch to achieve the impossible.

[Measure What Matters, John Doerr]

OKRs are a tool. The act of using OKRs does not guarantee Google-like performance, innovation, or market dominance. OKRs are not a strategy, nor do they make up for lack of strategy. OKRs force a company or organization to think rigorously, to hold themselves accountable, to stretch, and to learn and grow. Persistent usage over time guarantees increased focus, alignment, and execution. Here are 10 ideas to use them more effectively 👇🏽

10 Tips for using OKRs effectively#

1. Objectives must be Big and Motivating#

A great, challenging goal is inspiring. It brings teams together. It ignites that thing in us that led to human spaceflight and exploration of unknown frontiers. Inspirational objectives are critical at the company level, since company OKRs cascade down to teams and individuals. Everyone should feel proud to accomplish their objectives and together achieve a higher goal.

Objective: Win the Super Bowl
KR1: Passing attack amasses 300+ yards per game.
KR2: Defense allows fewer than 17 points per game.
KR3: Special-teams unit ranks in top 3 in punts return coverage.

[whatmatters.org/get-examples]

2. KRs must be measurable#

There are three important components of a key result:

The metric by which you will measure progress.
Where you are starting and what the goal is.
When you’ll be done. The time is assumed to be end of the execution period (quarter or half) if not mentioned explicitly.

For example, let’s say you want to reduce your AWS costs by improving server efficiency.

Objective: Reduce AWS costs.
KR1: Increase server utilization.

How will we know whether we have achieved our OKR or not? When should we stop working on this or work harder? The OKR could be rephrased to make it clearer:

Objective: Reduce AWS costs from $100 to $80.
KR1: Increase server utilization from 65% to 80%
by the end of the quarter.

3. Use binary KRs sparingly#

A binary KR is one that is either done or not done. For example:

Objective: Reduce AWS costs from $100 to $80.
KR1: Turn on auto-scaling for all services
by the end of the quarter.

Binary KRs are measurable — they were either accomplished or not — but usually need to be combined with quality counter-metrics to be effective. More on metrics and counter-metrics below.

Another example of binary KRs are “Ship KRs”. They are often used when rolling out new features. Without baseline data, there is no way to measure or predict how a feature will perform. The team can put in work to predict outcomes, but sometimes, it is easier to ship something and see the impact. Ship KRs have their place, use them with caution.

4. All Key Results must have dashboards#

The natural consequence of measurability is the scoreboard. Every measurable KR should have an associated dashboard. These need not be fancy — an excel sheet is sufficient to get started.

Some results are only measured fully in retrospect or take a while to get an accurate read. Retention is an example of such a measure — we don’t know upfront which user will churn. In such cases, teams should use alternate approaches, such as:

Create a proxy “operational” metric that correlates well with the goal metric. 1-day retention can be used for 30-day retention if we find that the bulk of churned users churn within a day.
Use a forecasted or predicted number as the operational metric. Monitor the predicted metric to keep the error within bounds.

5. Key Results must be exhaustive#

Another common trap that leads to a feeling of inadequacy at the end of the quarter is missing an objective despite accomplishing its KRs. Imagine we take on a feature activation goal that ladders to some critical company objective.

Objective: Increase usage of feature X from 100 DAU to 200 DAU by the end of the quarter.
KR1: Improve activation funnel efficiency from 85% to 90% by August 1.
KR2: Improve feature X retention from 90% to 95% by September 1.

We get to the end of the quarter, both KRs are green, and feature usage hasn’t gone up at all.

What if the lack of feature X had more to do with how you marketed it? If the problem is at the top of the funnel, increasing funnel efficiency is unlikely to achieve the goal.

Missed KRs are abundantly evident in hindsight, but much harder to think of ahead of time. One approach is to ensure that the goal (or the first KR) captures the intended outcome (100 DAU to 200 DAU) and let the team add KRs along the way if enough progress is not being made.

6. Pair Metrics with Counter-Metrics#

There is natural pressure and temptation to achieve progress at all costs. There have been innumerable examples in the corporate world, with Enron and Wells Fargo being the most recent, where the drive to achieve a particular outcome, such as “Open more accounts,” compromised the company’s core values.

To guard against such adverse outcomes, you can pair metrics with counter-metrics. For example, Wells Fargo could have paired “number of active accounts” with defensive metrics such as “each account must be active,” “each account must have a certain balance,” “no more than X accounts per person in a week.”

In product-feature land, there is a natural pairing of growth-based KRs and quality-based KRs.

KR1: Feature X will have 100 users by the end of the quarter.
KR2: 7-day retention for Feature X will be 90%.
KR3: P0 and P1 bugs will be closed within a day.

7. Distinguish between Committed and Aspirational OKRs#

OKRs are scored on a scale of 0.0 to 1.0. What’s a “good” score? Many tech organizations believe the answer is 0.7, based directly on Google’s scoring approach [whatmatters.org — how to grade OKRs]

However, the Google scoring system applies to Aspirational OKRs — typically Big, Hairy, Audacious Goals that you don’t know how to hit. You know it’s going to be hard, and you’re likely to fall short. But if you don’t reach for the moon, you’re never going to get there. The purpose of aspirational OKRs is to stretch you, to motivate you to solve more significant problems and to think differently.

In contrast, some OKRs are expected to be delivered in full. Doerr calls these Committed OKRs. For the business to succeed, for cross-functional teams to depend on each other, these goals must be achieved in full. You are pushing for complete success, not setting aspirational visions. The expected score for these OKRs is 1.0, with low variance.

Distinguishing between committed and aspirational OKRs is essential. If a team biases too much toward commitment, they may not stretch enough. If you go all aspirational, you may not hit any goal. Leadership must make deliberate choices about how their company should think about their OKR bias.

My current personal leaning is to bias toward committed OKRs for nearly everything, with one or two aspirational OKRs for things that are truly game-changing.

8. Cascade OKRs up, down, and laterally#

High functioning organizations speak the language of OKRs. When you depend on another team to hit your goal, you should expect to see that dependency in the other team’s OKRs. If you don’t, your likelihood of failure has increased significantly.

Similarly, company Key Results often become Objectives for organizations within the company. Organization Key Results become Objectives for individual teams.

OKRs can also cascade upward. For example, a team could discover that they have no way of hitting their goals, given their level of staffing. They refuse to take on the OKR, which in turn changes the OKRs above them, and so on.

This cascade of OKRs through the organization is a critical step of aligning the entire company. While it may sound chaotic in theory, the company or organization should quiesce in a week or two.

9. Personal OKRs are powerful. Use them to accelerate your career#

Company and Team OKRs drive organizational alignment and focus. You can create your personal OKRs too. While some will inevitably be congruent with your team’s OKRs, there could be others that are about making progress in your career or growing as an individual. Personal OKRs clarify your priorities with your manager and your peers. They hold you accountable for accomplishing what you set out to do.

I think of Personal OKRs as writing my self-review, six months ahead of time. I have been creating personal OKRs, putting them into a document, and sharing them with my manager for about six years. They have helped me be clear with my manager about what constitutes success and failure, what is paramount in their minds, and keeps me accountable for making progress. I strongly recommend doing Personal OKRs and taking charge of your career.

When writing personal OKRs, you must focus on outcomes, not activities. This trap is avoided at the team and company level by making all KRs measurable. However, when looking at personal OKRs, we start using language like “Participate in interviewing” or “Be a part of X team.” Consider rephrasing into outcomes instead, such as “Hire 3 engineers by the end of quarter” or “Conduct an average of 1 interview per week.”

10. Prefer a small number of tightly focused OKRs to a long list#

OKRs should add focus. A large number of OKRs, whether individual or team, imply a lack of focus. When push comes to shove, which OKR is going to drop? Don’t bother adding OKR priorities — that is simply a bandaid over the root cause: you have too many OKRs.

OKRs should also represent commitment. Picking one OKR means that we cannot do something else. In your OKR discussions, you could include a list of things you chose not to do to illustrate an effort to add focus.

Bonus: Effective OKR usage takes years#

If you’re getting started with OKRs, know that your initial implementation will suck. You may have too many, or they may not be measurable, chaos may reign as OKRs cascade across the organization. Know that there is no pinnacle of OKR usage — Google struggles with them as much as a series-A startup. Their struggles are different, and they’ve had more than two decades to get it right. But keep at it and make your version of OKRs better over time. Be patient.

Reference#

This note is a combination of practical experience from a decade of practicing OKRs and Goals at Google and Facebook, and reading through three books:

Thanks#

Many thanks to Leo Shklovskii, Lily Zhang, Jules Walter, DeVaris Brown, and Ethan Bloch for reviewing early drafts of this work and providing invaluable feedback.

Joining Digit

Rushabh Doshi — Tue, 07 Apr 2020 00:00:00 +0000

Non-COVID news: I’ve joined Digit as Chief Product Officer.

We are living through unprecedented times. Humanity is collectively battling an enemy that we cannot see, one that doesn’t recognize national boundaries, and kills without remorse. The economic impact is horrendous. Nearly 6 million people have filed unemployment claims in the US in the last week alone [source] — a number never before seen in our country.

When people lose their livelihoods, they need cash to survive. Emergency funds are built for hard times, such as the ones we are experiencing right now. However, saving money, when you’re living paycheck to paycheck, is incredibly hard. Human psychology works against you. We are wired to think in the present and to value immediate rewards. We need tremendous will power to look at every paycheck, and put some money away for a rainy day.

Digit started as an app to help people save. You connect your checking account to Digit, it analyzes your cash flow, and it intelligently moves your money to a Rainy Day Fund. You don’t think about saving money, but you’re saving. It is magical. Nobody wants to lose their job, but if it happens, at least you have some money saved up.

Digit’s mission is to make financial health effortless for everyone. Our app supports saving for any goal, not just a Rainy Day. We help you automatically pay off your student loans. We help you automatically pay credit cards in advance to reduce your debt. The list of features is long and growing. We want people to feel less anxious about money, knowing that Digit is there to help them become and remain financially healthy. If the worst happens, and you lose your job because of a global pandemic, Digit is there to help.

Joining a company is a big decision. Being genuinely in love with the mission is a requirement, but it is not sufficient. Work is a large part of life. I believe in working with great people — who have done amazing things, are smart and driven, and above all, are exceptional human beings.

I have known Ethan Bloch, Digit’s founder and CEO, since 2017. I have always admired Ethan’s leadership — he stays true to his values and the mission, and does not hesitate to make hard calls. He has been a friend and someone I seek out for advice and counsel. Over time, he has built an exceptional leadership team with Michael Murray, Vishwas Prabhakara, Carolyn Satenberg, Karthik Hariharan, and Samar Shah. I’m thrilled to join this crew and call them colleagues and friends, along with Ryan Nier, our head of Legal and Compliance.

Digit is a fantastic company with great people who are dedicated to the company’s mission. We’re growing, and we’re hiring, virus be damned. We are looking for product designers and product engineers who love building products that help people. If you’re excited about our mission to make financial health effortless for everyone, come join us!

Product Designer: https://boards.greenhouse.io/digit/jobs/2004653
Software Engineer, Product: https://boards.greenhouse.io/digit/jobs/922676

Welcome To Remote Work

Rushabh Doshi — Mon, 02 Mar 2020 00:00:00 +0000

The Trigger#

As of March 2nd, 2020, the coronavirus outbreak has officially hit the tech sector. The virus, SARS-CoV-19, causes the disease, COVID-19. There are several different strains of the virus at this point, as it mutates at an average of about two mutations per month, resulting in several different strains in different places. [source]. There is a rash of cases in the Seattle area, with a smaller number reported in a host of other places, including California and the Bay Area.

COVID-19 Global Cases, ex-China. Data by Johns Hopkins CSSE

Elad Gil published an excellent post about the impact of the virus on tech companies, including detailed but readable background information on the virus and it’s spread. Some useful Twitter accounts to follow are in the resources section below. Most important things to remember (summarized from Elad’s post):

We will likely see a widespread outbreak of COVID-19. However, most cases will be mild, and most people will recover.
Symptoms manifest between 3-14 days from exposure, including fever, cough, and shortness of breath. [Mayo clinic]
Case Fatality Rate (CFR: deaths / people infected) is estimated at 2-3% but could be much lower, with better healthcare. People over 60 have a higher risk, as do patients with existing cardiovascular disease, diabetes, chronic respiratory disease, hypertension, and cancer. [source] There have been no reported fatalities in infants and children. [source]
R0 value estimate is between 2 and 3. Every infected person will likely infect 2-3 more. In contrast, flu R0 is about 1.5. [source]

[Edit March 2021] In hindsight, using the below mnemonic was insensitive. When it was written, it tied the site of the biggest early infection to what people should do. In retrospect, after the term “China” was co-opted as a vehicle for racial hatred, it is wrong.

~~Quarantine and focused personal hygiene appear to be our best defenses against the virus at this point. Use the WUHAN mnemonic as a reminder:~~

~~W - Wash hands with soap frequently. [YouTube]~~

~~U - Use a mask if you are sick and infected.~~

~~H - Have your temperature taken daily if you have been exposed.~~

~~A - Avoid large crowds and gatherings.~~

~~N - Never touch your face with your hands.~~

The Response#

Tech companies are starting to take action to protect their employees. I expect this trend to continue and accelerate over the next week. The responses generally follow the same principles:

Minimize or cancel all work-related travel.
Cancel large conferences and gatherings.
Stay home if you exhibit any symptoms or believe you have been exposed to COVID-19.
Working from home is encouraged for everyone and mandatory for critical employees.
Interviews are over video instead of in-person.

Sources: Stripe’s public post, Twitter’s post, Coronavirus company response tracker

The Shift To Remote#

COVID-19 represents a watershed moment for how we work. Tech companies have been trying to figure out how to move more of their workforce to remote, to take advantage of talent pools outside of major hubs like San Francisco, New York, and Seattle. Most companies that start in one location advance step-by-step, moving toward a hub model, and some like Stripe, move to full remote in addition to having a central hub.

After the outbreak of the virus, no tech company has a real choice. Every company will become a remote-work company out of necessity. Leaders need to face the reality of not having everyone in the office — they should be proactive and not get caught flat-footed. Here are a few things to consider:

Async Remote vs. Sync Remote. If your current culture is set up to work in one office, within one timezone, you should maintain the concept of “working hours” where everyone is present on Slack or Teams. Jumping to fully async remote, where everyone works any hours they like, might be too disruptive for your company’s flow.
Check your tools. Do you have enough zoom licenses? How are you going to do 100 people conferences remotely? Do you have a way to do collaborative whiteboarding? Are you capable of visually providing design feedback?
Dry run. If your company isn’t going full remote right away, you can still mandate staying home for a few days at a time, to work out the kinks. Do you have a process to deal with emergencies and outages? Do you have phone numbers in an easily accessible directory?
Preserve culture. Every company has its customs and traditions, from Monday donuts to Friday beers. People celebrate achievements together, have ways of communicating and collaborating. Art — ranging from murals to laptop stickers — gives us a constant reminder of belonging and shared values. How will you preserve this when you don’t have a common place to meet? For example, Fil Fortes uses a green screen to change up the background and keep things fresh.

Good books on remote work include Jason Fried and David Heinemeier Hansson’s Remote: Office Not Required. The book not only makes a strong case for remote work (not required at the moment), but has several good tips, learned through experience.

Self and Family Care#

There is a lot of information about COVID-19, and it’s evolving fast. The consensus at this time, for yourself and your family, seems to be the following:

Be prepared for a self-quarantine of 14 days or more. Stock up on food. ThePrepared has an excellent guide to getting food. Note that a lot of vendors are running out due to the increased demand. There is a distinction between food that will last for a decade (of which there seems to be short supply) and food that will last for a year (which is available in most grocery stores and Costco). The sooner you buy, the better.
Be prepared for medicine shortages. Supply chains will get affected, and there could be runs on common drugs. Get a 90-day supply of your medication.
Wash hands regularly. Regular hand washing (say ABC twice) is one of the best defenses against the virus spreading. The primary entry points are eyes, mouth, and nose, but we touch them with our hands roughly 23 times per hour. Hand sanitizers are only useful if they are 60% alcohol, and you rub your hands for 20 seconds or more. They’re also out of stock (in the Bay Area.)
Avoid large gatherings. Regularly wiping down commonly used surfaces, like door handles, will help prevent the spread of the disease in schools and other shared places.

I hope this has been helpful, and everyone stays healthy and safe.

Resources#

Elad’s advice for startups: https://docs.google.com/document/d/1hYPrTMKfcEFux2rETlK07TAmW1wAW3003a00YRDn9qw/edit
GitHub data repo updated regularly from Johns Hopkins: https://github.com/CSSEGISandData/COVID-19
WHO-China Joint Mission on COVID-19: https://www.who.int/docs/default-source/coronaviruse/who-china-joint-mission-on-covid-19-final-report.pdf
ThePrepared guide on preparing: https://theprepared.com/wuhan-coronavirus/

Company plans#

Public list of all plans: https://docs.google.com/spreadsheets/d/1awyG207-dR8HfpZR2T4GrPqixdT5Rht-Ex04pZ1KLNk/edit#gid=0
Stripe: https://stripe.com/newsroom/news/covid-19
Twitter: https://blog.twitter.com/en_us/topics/company/2020/keeping-our-employees-and-partners-safe-during-coronavirus.html

Twitters to follow

4 Interesting Reads - Feb '20

Rushabh Doshi — Mon, 24 Feb 2020 00:00:00 +0000

– Generated using repper.app

I received positive feedback and encouragement for my previous 4 Interesting Reads post. Here is the next one, in the same vein. Feedback is always welcome, please reach out.

Making Uncommon Knowledge Common by Kevin Kwok#

https://kwokchain.com/2019/04/09/making-uncommon-knowledge-common/

Rich Barton has founded three companies valued at over a billion dollars: Expedia, Glassdoor, and Zillow. In the linked article, Kwok explains Barton’s central strategy: take public knowledge that is hard to obtain for historical or structural reasons, make it easily accessible, and as a result, own customer demand. When looking for flights or hotels, you start with Expedia. When you’re looking to sell your house, your journey begins at Zillow with its Zestimate. This is data that has been converted to content that people want. All this “content” substantially lowers customer acquisition costs (CAC), the bane of a startup’s unit economics. The takeaway for product people: cheaply create high quality, personalized content, and use it to drive down CAC, and provide value.

The Limits Of High Speed Rail by Ivan Rivera#

https://mappingignorance.org/2020/01/22/the-limits-of-high-speed-rail/

I am fascinated and genuinely interested in the engineering problems faced in building things that are world record holders - the fastest car or train, the tallest building, the largest dam. For this reason, I could not put down Adrian Newey’s autobiography How To Build A Car, about his journey in the world of Formula 1 racing, and the nonstop engineering ingenuity to build some of the world’s fastest cars. In a similar vein, Rivera’s article goes into detail about the engineering required to break the world record for the fastest train — SNCF’s TGV V150 (named because 150m/s == 540km/h, the target speed) clocked 574.8 km/h on April 30, 2007. He goes into a lot of fascinating detail around wheel-rail contact points, why diesel-powered trains can’t really go beyond 240 km/h, the role of aerodynamics, and the evolution of the pantograph-catenary contact (for electrical power transmission). I firmly believe that we advance the state of all human inventions and knowledge when we push the limits of what we can achieve, and for this reason, studying record-breaking inventions is worthwhile.

The New Business Of AI by Martin Casado and Matt Bornstein#

https://a16z.com/2020/02/16/the-new-business-of-ai-and-how-its-different-from-traditional-software/

The stellar people at Andreesen-Horowitz write incredibly insightful commentaries on entire industries: find the thing you’re interested in (SaaS? Marketplaces? Crypto?) and follow the relevant a16z partners. Casado and Bornstein’s post on the fundamentals of AI startups is packed with insights and hard to compress further. They posit that AI startups have lower gross margins (50-60%) compared to traditional SaaS businesses (60-80%) because of two reasons. 1) High cloud costs due to training, retraining, and inference over large data sets. 2) Human-in-the-loop costs, both for high-quality data (cleaning, labeling) and for checking the AI’s results. AI startups have a harder time scaling because edge cases are much harder to deal with - there is a very long tail of edge cases due to the sheer volume of high dimensional space. Finally, they have weaker defensive moats because a lot of model development is being done in academia, or is open-sourced, and data scale effects are weaker than people think. I believe we are still in AI frontier land: new discoveries, new tools, new rules. Hence, understanding the business of AI is as vital as grokking the technology, and makes this article worth the time.

Group Chat: Group Stress by Team Basecamp#

https://basecamp.com/guides/group-chat-problems

One of the most significant challenges of today’s Slack based tech work environment is that there is no time to think. I don’t believe Slack is the root cause; it is somewhere in between an enabler and an accelerant. But, there are too many channels to follow, lots of FOMO about stuff that does not matter, shallow thinking, and fast responses, no preservation of context, and forced ephemerality. The linked article (obviously biased, Basecamp is a competing product), lists out all these problems in greater detail. The central point, which I wholeheartedly agree with, is that “chat attacks attention, and severely hinders deep work.” If you’re in a Slack based environment and have figured out a way to do deep work without constant interruption, kindly write a post about it and help out the world (and let me know!)

Bonus: The Man In The Arena by Theodore Roosevelt#

Doris Kearn Goodwin’s book Leadership is a fantastic read about how 4 of America’s great leaders (Johnson is controversial) suffered and dealt with significant personal hardship, and how it shaped their presidency and their leadership. Theodore Roosevelt is my favorite leader because of his bias toward action, his values and morality centered around always doing the right thing, his boldness, and his courage. I come back to this quote often, whenever I’m feeling low about my work, or overly criticized, or when an idea that I thought was going to change the world turns out to be a dud, or when I am scared of taking a leap. I hope it helps you as much as it has helped me.

It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.

— Theodore Roosevelt, Citizenship in a Republic

3 Frameworks For Making Complex Decisions

Rushabh Doshi — Fri, 14 Feb 2020 00:00:00 +0000

Life is full of complex decisions: capital purchases, such as a car or a house, planning a vacation, choosing a new job, picking a product strategy, prioritizing roadmaps, hiring someone. Complex decisions have several shared traits: the list of options is often extensive, evaluation criteria are ill-defined, the outcomes are hard to predict, input data is unavailable or incomplete. Humans understand large systems by building mental models, which are more straightforward than the reality they represent. Mental models are a great thing: they allow us to make progress without getting bogged down in every little detail. But they also have their flaws. Most notably, human cognitive biases, our failures to think and communicate clearly, lead us to sub-optimal decisions.

Psychologists and behavioral economists have spent a considerable amount of time and mental energy dedicated to understanding the human decision-making process. By systematically understanding our cognitive biases and flaws, smart people have come up with frameworks to counteract their ill effects. Using these frameworks can lead to decisions with better eventual outcomes., how we fail, and how to make decisions that lead to better results. Amos Tversky, Daniel Kahneman, Richard Thaler, Dan Ariely, and Chip Heath have done seminal research in this field — and distilled their ideas into highly readable books. Thinking Fast and Slow, Nudge, Predictably Irrational, and Decisive are amongst my favorites. I strongly encourage you to read these to gain a deeper understanding of the field.

In this post, I present three practical frameworks to improve decision- making in different contexts. Frameworks are hard to understand in the abstract. Just reading theory leads to a shallow understanding of how to apply them in practice. To make things more concrete, I use two practical problems that I have solved using some combination of these three frameworks.

How to buy a car: Large capital purchases, such as buying a car or a house, can make for challenging decisions. Some input data for our car-buying determination: I have a growing family with small kids. I have a short commute to work. I care about style and comfort. I don’t intend to race my car on a track. I care about the environment and would like something efficient. I have a nominal budget of $50k in mind. A cursory examination of the car market should quickly reveal a broad spectrum of options. For the sake of this post, let’s narrow that down to 1) a minivan (Honda Odyssey), 2) a hybrid sedan (Prius), and 3) a pure electric (Tesla Model 3).
How to pick the next feature: We make many complicated decisions at work. Product Managers and organizational leaders often need to decide what part of their product they should focus on given their goals. This strategic choice is perhaps the most impactful, on par with perfect execution. Input data: our app is in the market. It is growing slowly. Churn is higher than we’d like. Research shows that the current set of customers like the app, but don’t love it. Should we focus on acquiring new users, increasing lifetime value, or churning fewer users?

How Not To Decide#

1. Gut Feeling#

Listening to your gut is probably the most common approach to decision making. It’s the way we make most decisions - if we did an exhaustive process to decide what to eat for lunch, we’d never get anything done or be able to make any progress. Instinct is your subconscious brain pattern matching inputs with what it has seen in the past and making a quick, shortcut decision. Our brains are fantastic at taking in vast amounts of data and making gestalt decisions; don’t fight your instincts.

However, when it comes to highly complex decisions, the very brain that helps us make rapid decisions and move forward with life, deludes us into making bad decisions. Our fast decision-making process is often known as the “reptilian brain” or “System 1 thinking”. It is the reason we survived on the savannah: when we thought we saw a lion, our brain didn’t take its time working through whether it was a bird, or a blade of grass, or a zebra. It told us to climb the tree first. Our deep-thinking, thoughtful cousins were pruned out of the family tree by the lion. These instant reactions, fight-or-flight instincts, all the shortcuts our brains use, can show up as cognitive biases in decision-making.

Let’s use the car buying example. Imagine we walked into the local Toyota dealership. It’s a boiling hot day in the middle of summer. Salespeople are extremely busy, overworked, and slightly rude. They give us the keys to a car that’s been baking in the sun. We test drive it, hate it, and pass summary judgment: it’s a pile of rubbish. The car is way too hot, takes forever to cool down, drives like a sloth. Additionally, we’re unhappy about not being treated like royalty and don’t want to buy from that dealer anyway — hard pass.

We have attributed the rudeness of a particular salesperson to not just the entire dealership, but all the dealers of this specific car manufacturer. This mistake is called a fundamental attribution error. We have attributed the car’s inability to cool down instantly to a manufacturing flaw. We have ignored the base rate: all vehicles sitting in the sun on that day were hot and would take time to cool down. As a result of these biases, we may have discarded a perfectly reasonable option thanks to our instant decision making brain.

2. The Giant Spreadsheet#

I love spreadsheets. They allow me to organize my life (and I love organization) and view things at various levels of detail. It is very tempting to distill every decision to some formula and take the flawed human out of the loop. The formula can be straightforward: weighted sums seem to do the trick. Every decision now becomes so precise, so mathematically elegant. Don’t like the outcome? You must have gotten the inputs or weights wrong.

Let’s visit our product feature prioritization decision. We could build features that target acquisition, LTV, or churn. Each row has a cost and an impact estimate. Any Product Manager worth their salt will come up with a table of priorities, each of these features as rows, and drop columns to show potential impact. Some complex mathematical jiu-jitsu comes next, and the potential impact column has numbers and color coding from red to green. We must pick the greenest feature because our matrix just told us so!

This approach is problematic because it reduces humans to automatons and throws out all intuition. Moreover, it overly simplifies the model by diminishing highly complex information into a number. In the made-up example above, working on notifications seems to win over everything else, using the scheme I’ve put in. But that the model itself is biased - cost and impact estimates might be completely bogus, our gut might tell us that focusing on acquiring new users is more important, or that the cost of doing notifications is probably higher.

Intuition is essential — our brains are pattern-matching against past experiences and predicting the future. Numerical models create a false sense of precision and delude us into trusting the models. Our minds are excellent at translating vast amounts of information into decisions, and we should trust them while finding ways to correct their shortcomings.

Decision Frameworks#

The next section outlines the three decision frameworks that I have used in some shape or form. None of these frameworks are mine - I have merely adapted them for my purposes and found them to be applicable and relevant.

Framework 1: Reducing Dimensionality#

The credit for this idea goes to my friend and colleague Josh Williams. The principles are easy to understand and apply on the fly, require little formal work, and help break through a decision making logjam.

Complex decisions are often challenging because they contain an overwhelming number of dimensions. Decomposing the problem results in a large number of smaller choices along each dimension. However, dimensions are not orthogonal — changes in one affect another. Trying to optimize all dimensions at the same time quickly gets overwhelming.

Take the car-buying example: we need to make individual decisions about passenger capacity, gas mileage, styling, manufacturer, safety features, cargo hauling, maintenance, buy vs. lease, and so on. In the example above, we need a car that can carry five humans, is efficient, stylish, safe, easy to maintain, and costs less than 50k. A Porsche 911 is stylish and safe, but doesn’t cost less than 50k or carry five humans. A minivan fits most of the requirements but is on the lowest end of the style spectrum. A Prius is in the “meh” range on most things but does excellent on efficiency. The perfect car simply doesn’t exist. What do we do?

A good approach in such circumstances is to reduce dimensionality. If you magically cared only about your budget and passenger capacity, the answer would become much more apparent. We can reduce dimensionality in 3 ways:

Aggressively ignore dimensions that you don’t care about. In the car example, we could stop caring about maintenance. Maintenance plans are straight forward. Almost every major manufacturer has a good policy. Let’s get rid of that completely.
Create “threshold” dimensions that you care about up to a certain point, but not beyond. For example, safety matters to my family, with our small children. But beyond a specific safety rating, any car is sufficiently safe, and we don’t need to optimize any further.
Establish trade budgets. This is not dimension reduction per se, but helpful in understanding the relationships between different things. For example, if we care more about efficiency than style, and getting a high gas mileage is worth twice as much as having a sexier car. This approach gives us a rough calculator to prioritize the dimensions we genuinely care about.

The beauty of this framework is that you can quickly sort through the dimensions that matter and devalue or completely discard the ones that don’t. Moreover, when we end up with a few real choices at the end of the process, we are assured that all of them satisfy our constraints and would make us happy. Beyond this point, all decisions are good decisions.

Framework 2: Mediating Assessments Protocol (MAP)#

This approach is from an excellent article by Daniel Kahneman, Dan Lovallo, and Olivier Sibony. If this summary piques your interest, I encourage you to read the article in full. It is clear, easily understandable, and practical. If you’re at a tech company, such as Google or Facebook, and are using a structured interviewing process, you’re already using MAP without knowing about it.

Remember the “giant spreadsheet” approach to making decisions? The problem with that approach was that it threw out all human intuition. What if we kept an element of intuition in the mix, but had a way to neutralize a variety of cognitive biases? This is the central idea behind the MAP framework proposed by Kahneman and team.

Let’s revisit the feature prioritization problem. We need to make a decision on which feature to build next. There is a trap here — we can easily mislead ourselves into believing that we are following a structured process by sitting through presentations about each option, evaluated in its entirety, with pros and cons, followed by a decision making or voting meeting. This method is subject to precisely the same biases - confirmation bias for things you like, and recency bias for the last option presented. It is essentially the equivalent of a holistic gut call.

Here is the MAP alternative:

Agree upfront on what the goals are. To continue with our example, let’s say the objectives are 1) increase the number of daily users, 2) improve the performance of the app, and 3) reduce our operational costs.
One presentation per goal. This method allows us to compare all proposals, on a particular dimension, instead of looking at all the aspects of one proposal. If we have a scoring rubric, we can score proposals per goal at this stage. These assessments are called mediating assessments.
A final evaluation of all proposals, while looking at the mediating assessments. Note: we are not merely taking a weighted average of the intermediate scores. Instead, we are using our judgment at this juncture while keeping all the data in front of our eyes.

The changes seem subtle, but the impact can be profound. The best proof of this approach is in the use of a structured interviewing process to evaluate candidates. If you have interviewed at modern tech companies, like Facebook, Google, or most modern startups, you have experienced this. Instead of having each interviewer simply provide an overall score, the interview process involves a series of mediating assessments.

In structured interviewing processes, each person interviews and makes a judgment about one area of competency - coding, system design, communication, people management, etc. Interviewers score candidates per dimension. The hiring committee looks at all the intermediate scores and then determines an overall rating. This approach is different from each interviewer judging the candidate in all of the different areas and giving one overall score. Structured interviewing is the norm in almost all tech companies. Studies on personnel selection have conclusively shown that using such approaches to interviewing leads to more accurate long term outcomes.

Framework 3: WRAP#

This framework is a summary of the WRAP process outlined in the Heath Brothers’ fabulous book Decisive. I strongly encourage you to read the book, as well as use the summary resources on their website (free registration required.)

The WRAP framework focuses on avoiding or overcoming cognitive biases that creep into all human thinking. It is easy to understand and practical to apply. Each maxim can be used independently toward decision making; apply a few or all.

1. Widen the frame#

Let’s go back to the car-buying example. We are trying to choose between a minivan or a Prius. This problem statement implies a particular frame: we have to decide between A and B.

However, the car is a means to an end - commuting to work, transporting children to school, picking up groceries, or traveling for leisure. In our choice, did we consider solving the more significant problem using some other means? Do we need a car at all? Could we use an electric bike to commute? Or Instacart for all groceries? How would that change our set of options?

A narrow frame is a common decision-making trap. It focuses our thinking on available options, instead of opening our minds to all possibilities, some of which may solve the problem in unique or non-traditional ways.

A classic sign of this trap is the “whether or not” question. When you hear your friend ask you “whether or not they should quit their job” or “whether or not they should build a feature” or “whether or not they should buy an iPad,” you should smell a trap. One way out of this trap is removing the option you are leaning toward and making that a non-option. What if you absolutely could not quit your job or buy the iPad? What would you do then?

2. Reality Test Your Assumptions#

When we survey the set of available options, we build models in our head of how those options are going to work out. These models get tested when they meet reality, and usually don’t survive. We try to improve our models by finding evidence that supports or disproves the model. However, because of confirmation bias, we are much more likely to seek validating proof, rather than the contrary, or disconfirming evidence.

One way to get around this pernicious problem is to look for opposing or disconfirming evidence. Imagine we are in love with a particular feature. Instead of looking for reasons to support our instinct, look for the holes in our reasoning. Why could this feature fail or underperform?

How do other similar features perform? This line of questioning helps us determine the base rate. If most similar features underperform (low base rate), it is unlikely that this particular one is going to be the breakout.

Looking for disconfirming evidence can be difficult, especially when we’re already heavily biased toward pursuing a particular path. One trick is to do a joint “premortem” exercise. Get together in a room, and imagine that you’re six months into the future. The feature has been built and launched and isn’t doing well. What went wrong?

Another approach to reality testing assumptions is to dip a toe in without diving in all the way. In the car buying example, we could rent a minivan for a week, followed by renting another car for a week, to test out what it would feel like living with that car. The cost of a mistake (perhaps you hate the way the minivan drives or turns out that the sedan is entirely too small for your family) is tiny compared to buying the car and discovering you made a mistake.

3. Attain Distance#

One of the most striking passages in Andy Grove’s book “Only The Paranoid Survive” is about Intel’s decision to pivot from making computer memory to making microprocessors. Intel started as a memory company - and they were the world leader in manufacturing memory chips in the late 70s through the early 80s. The microprocessor business was niche, dwarfed by the massive memory business. However, the memory business was seeing enormous pressure from Japanese manufacturers and steadily losing margin. Pivoting the company from their roots, to go all-in on microprocessors was an incredibly difficult decision. Grove described how they finally did it:

— via Google Book Search

For complicated psychological reasons, we seem to make clearer decisions when we are deciding for others instead of ourselves. One of the most effective techniques for attaining distance is to ask:

“What would you tell your best friend to do in the same situation?” — Personal Context

“If you were let go and we hired someone else, what would they do in the same situation?” — Professional Context

4. Prepare To Be Wrong#

We typically overestimate the impact of any particular decision. In reality, most decisions are reversible, or at least have escape hatches that are less catastrophic than we initially believe them to be. Jeff Bezos summarizes the concept of reversibility:

Some decisions are consequential and irreversible or nearly irreversible – one-way doors – and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation. If you walk through and don’t like what you see on the other side, you can’t get back to where you were before. We can call these Type 1 decisions. But most decisions aren’t like that – they are changeable, reversible – they’re two-way doors. If you’ve made a suboptimal Type 2 decision, you don’t have to live with the consequences for that long. You can reopen the door and go back through. Type 2 decisions can and should be made quickly by high judgment individuals or small groups.

— Jeff Bezos, Amazon Annual Shareholder Letter

What if we made a wrong car-buying decision? We own a car that we don’t like, which we need to sell subsequently, and buy another car. There is a quantifiable dollar cost and some hassle in selling and buying cars and filing paperwork. But that’s it. With that understanding, we no longer fear making a decision, knowing that the cost of reversing that decision is not life-altering.

In Conclusion#

Life is full of decisions. In the majority of cases, our instinct is a great decision-maker. However, when faced with highly complex decisions, the evolutionary processes that helped us survive the lions on the savannah can mislead us into making poor, often irrational choices. Using frameworks to make such complex decisions allows us to counter some of those cognitive biases and make good long term choices.

4 Interesting Reads - Jan '20

Rushabh Doshi — Mon, 27 Jan 2020 00:00:00 +0000

A different post this week. Instead of the usual dose of problems and frameworks, here are four articles I came across in the past few weeks, along with summaries of why they’re worth reading.

How Will You Measure Your Life by Clayton Christensen#

https://hbr.org/2010/07/how-will-you-measure-your-life

Prof. Christensen passed away last week. He was a giant in the business world, with his theory of how low-end disruption can topple the biggest, most well-funded companies, explained in The Innovator’s Dilemma. He also gave us the Jobs To Be Done framework, in his more recent book, Competing Against Luck.

The above article is a break from business and explains his ideas on how to measure life. It is a beautiful and sobering read.

Have a purpose and follow a strategy to achieve that purpose
Allocate time wisely, knowing that intimate and loving relationships with family and friends are the most potent and enduring sources of happiness.
Create a culture, at work, and home, of doing things that are hard and learning what works.
Never compromise your principles. “It’s easier to hold to your principles 100% of the time than it is to hold to them 98% of the time.”
Remembering to stay humble, even if you’re the smartest person in the room.

Functional vs. Unit Organizations by Steven Sinofsky#

https://medium.learningbyshipping.com/functional-versus-unit-organizations-6b82bfbaa57

Sinofsky’s post is an essential read for any leader planning an organization’s structure. This long post includes the history of Sinofsky’s organization of Office, followed by Windows. He goes into a lot of detail on the pros and cons of functional and unit organizations. Functional organizations are organized by discipline (eg. engineering, product, design, research), with functional leaders all reporting to the CEO. Unit organizations are led by general managers, with smaller functional teams reporting to them. Lots of unit organizations constitute the company. Neither are correct per se, and the pendulum swings between the two. Sinofsky also points out his three golden rules:

Don’t ship the org chart (or accept that you will always ship the org chart, and design the org to match what you’d like to ship)
Know the problem the re-org is solving.
Ship schedule is everything. (bonus: ship dates are dates. “January” or “Q3” are not ship dates. January 15th is.)

Pioneers, Settlers, and Town Planners by Simon Wardley#

https://blog.gardeviance.org/2015/03/on-pioneers-settlers-town-planners-and.html

Wardley argues that different phases of innovation need different types of people and thinking.

Pioneers. They can explore never-before discovered concepts, the uncharted land. They show you wonder, but they fail a lot. Half the time, the thing doesn’t work correctly. You wouldn’t trust what they build. They create ‘crazy’ ideas.
Settlers. They can turn the half baked thing into something useful for a broader audience. They build trust. They build understanding. They make the possible future happen. They turn the prototype into a product, make it manufacturable, listen to customers, and turn it profitable.
Town Planners. They can take something and industrialize it taking advantage of economies of scale. You trust what they build. They find ways to make things faster, better, smaller, more efficient, more economical, and good enough.

The article was an interesting way to think about the different mindsets, people, and teams, that work well on products in various phases of development.

How To Hire Smarter Than The Market by Erik Bernhardsson#

https://erikbern.com/2020/01/13/how-to-hire-smarter-than-the-market-a-toy-model.html

With the help of a toy statistical model, the author argues that there is an arbitrage opportunity in hiring engineers. When companies collectively place a hefty premium on some trait (e.g., graduate of a top tier engineering school), they ignore a large pool of candidates that another company can go after. He lists several such traditionally undervalued traits: candidates who didn’t get a CS degree, or who have gaps on their resume, or who are not confident in interviews, who don’t have experience with your exact tech stack, the list goes on.

This article made me think, rather than being immediately actionable. I don’t know how to efficiently screen and interview candidates without the help of some traditional markers. I do agree with the hypothesis that we are collectively ignoring a lot of great talented people out there.

Selfish Note#

If you liked this sort of post, or if you hated it, I’d love to hear from you. I spend a lot of time reading articles and books about leadership, engineering, product management, and technology. I would love to share more summaries and pointers if they are of interest. Thank you for being a reader.

On Execution

Rushabh Doshi — Mon, 13 Jan 2020 00:00:00 +0000

Thanks to Ben Liebald, Lily Zhang, and Maher Saba for ideas and feedback.

“Our customers love our product and are hungry for more. We have a solid strategy. But we seem to be moving very slowly. Our team is great, but we can’t seem to execute.”

If this struck a chord, I have good news. First, you’re not alone. Most leaders I talk to ask for “execution” help. Second, execution can be improved through hard work and persistent, directed effort. However, execution is a discipline, and there are no silver bullets, despite what the internet will tell you. Improvement will take time and effort, but if you’re committed, read on.

Building A Race Car#

— Photo by Tim Carey on Unsplash

Racing cars is a fascinating sport: it requires skilled drivers, but the world’s best driver cannot win with a poorly designed car. Take Formula One racing as an example. Formula One, also called F1, is a series of races around the world, with the world’s best drivers in custom-built cars. F1 cars are built for extremely high cornering speeds: hitting peak forces of 6.5g, while maxing out speeds of 350km/h (215 mph). F1 is highly regulated - accidents can be hazardous, sometimes claiming lives, such as the tragic death of Ayrton Senna at the Imola Grand Prix in Italy in 1994. The regulatory authority, F1A, changes rules often, sometimes minor changes, but the occasional major change that requires a substantial redesign of the car. Car design and development is such an essential part of F1 that there are actually two prizes: the Driver’s prize and the Constructor’s prize. Building F1 cars is as much an artistic endeavor, as it is a feat of ultimate engineering. It requires concerted hard work, year over year, improving cars, fixing existing problems, adjusting to rules changes, adapting to different drivers, stretching for varying road and weather conditions.

I think of high performing teams as a custom-built race car. The car comes together as a whole — engine design matters, but there is more to it. Similarly, any process a team employs matters, but there is much more to execution than process. The internet discussions on execution often point to a change in process, from waterfall to scrum, or having daily standup meetings, or weekly retrospectives. These things matter, but like the car’s engine, do not capture the complete picture. I believe that you exact processes must be tuned for the team; there are broad rules of what everyone must do (all cars have 4 wheels, are under a certain weight, and have an engine), but within the general framework, there are a lot of choices to be made.

This post is divided into three broad sections:

Defining execution and making the case to treat it as a discipline, rather than a process.
The Five pillars of execution.
A toolbox of things that have worked: practical ideas for you to copy-paste and edit.

What Is Execution?#

The words “strategy” and “execution” are insufficiently understood by product development teams. My post, “Strategy and Tactics,” attempts to explain the difference. Here’s the short version:

Vision is what the future looks like if you’re successful.
Strategies are possible paths that allow you to attain your vision. A strategic choice or decision is the act of choosing which particular way to follow.
Execution is walking that path.

Success depends on all three; which one matters more is highly dependent on the nature of the work, the company, the team, and a variety of other factors.

At a company level, the vision is typically to change the world in some meaningful way by making some problems go away. Google seeks to understand the world’s information and make it available. Facebook aims to connect the world.

Strategy is how you get there. Exceptional strategic thinking is hard to come by because it involves so many leaps of faith. Strategies are forward-looking and often hard to test on any meaningfully short timescale. Amazon’s strategy is three-pronged: decrease prices, increase product selection and increase customer convenience. Amazon could have chosen a variety of other approaches — build bricks and mortar stores, or purely match buyers to sellers, or designing and selling only their own products, or making a white-label e-commerce site. Notably, each of these strategies has a company built around it: Barnes and Noble, eBay, Everlane, and Shopify.

Execution is about the nitty-gritty. Designing experiences. Shipping hardware or code. Nimbly iterating to get through or around new roadblocks. Similar to other disciplines, excellent execution takes sustained effort over a long time. It is not a Thing to be achieved and checked off. The principles of Deliberate Practice apply here at the team level. Having a clear goal, focus and hard work, feedback, accountability, and deadlines — these are the pillars of a high achieving team.

Pillars of Execution (OSPAD)#

One Clear Goal
Sustained Focus
Progress Indicators
Accountability
Deadlines

1. One Clear Goal 🏁#

To execute well, teams must know what they are executing toward. It all begins with a clear and concrete goal, one that everyone can understand and rally behind.

One of the biggest challenges that organizations face is not the lack of goals, but having too many. When you have multiple goals, priorities get muddled. Given a choice, teams will follow the path of least resistance and focus on goals that are achievable over goals that are important. It is paramount, for a leader, at any level, to identify and set a small number of concrete goals for their organization, ideally one.

The second challenge is communication. Ask 5 people at different levels in the organization what their goals are. If you hear different goals depending on who you ask, or worse, “I’m not sure what our goals are,” you have a problem. After picking a goal, you must communicate it as clearly and as often as you can. The test comes at goal evaluation time, at the end of the quarter. If the team hits a secondary, unimportant goal, but misses the primary one, and they believe they did well and deserve commendation, then you have failed at clear goal-setting and communication.

A note on measurability: we in tech are obsessed with measurement. “What we cannot measure, we cannot improve.” Sometimes things are just not measurable, or worse, the only thing you can measure is a misleading proxy of the actual thing. Don’t let the lack of easy measures stop you from setting goals. For example, shipping is a perfectly good goal — especially when a team is building a new product with no priors.

Your job as a leader is to do the following:

Pick a clear goal. This is the goal that you consider critical to the success of the team or organization. It is the goal that everyone should rally behind. If you have a dozen goals, prioritized into big and small goals, whittle the list down to one or two. This exercise is challenging, but worthwhile.
Talk about your goal as often as possible. If your priorities are clear to you, and perhaps your executive team, but not to every engineer, PM, designer, and analyst - you haven’t talked about them loudly or often enough.

2. Sustained Focus 🧘🏽‍♀️#

YouTube in 2011 was primarily known as the site for cat videos that people forwarded and shared on Facebook. There was a blossoming creator ecosystem, but creators were incentivized to make small, bite-sized content. The reason was simple: YouTube algorithms valued views as the primary metric and overwhelmingly recommended short videos. Sometime in 2012, leadership made the decision to switch to watch-time as the primary metric. The central idea was that as long as people were watching content on YouTube, we didn’t care whether it was short or long, just that they were entertained (or informed) by the things that they watched. The analytics team came up with a chart that showed a growth curve to 1 billion hours of watch-time. We all thought that a billion hours was impossible, but the chart served as strong motivation to hit a clear goal. When did YouTube hit the billion-hour goal? 2017.

Large bets take time to build. We, in Silicon Valley, are obsessed with instant feedback, numbers moving up and to the right instantly, trying to get to product-market fit as quickly as possible. If it doesn’t work in 3 months, it must be a terrible idea and should be abandoned because of opportunity cost.

I believe many organizations are afflicted with organizational attention deficit disorder. We do lots of annual and quarterly roadmap planning, and the minute we see something shiny, internally or externally, we collectively go “Squirrel!” and chase after it.

Execution requires focus. Goals are the setup. Focused effort is the follow-through. As a leader, you should keep your team focused on the goals, for months to years, based on your conviction about your strategy.

3. Progress Indicators 📈#

A game without a scoreboard does not inspire players to do their best. The mere presence of a score is not enough, the score must be visibly present, so everyone sees it, every day.

Picking a good progress indicator is an art. These are operational metrics, not business metrics. As a result, the number you select to keep tabs on progress can be different from the overall goal chosen in section one. Rules of thumb for operational metrics:

Indicators should be fast. The point of a visible indicator is to motivate the team. A rapidly moving number helps build momentum.
Use leading indicators instead of lagging ones. When I was trying to lose weight (a perpetual goal), I made the mistake of using my weight as the indicator. It is a lagging indicator, and will move down after strenuous effort along other fronts, is subject to a lot of intra-week variation, and is mostly demotivating. Better markers like activity-minutes, or calorie counts, or standing or walking time possess the dual properties of being on the leading edge and moving quickly.
Use indicators connected to the eventual goal. Measuring the wrong thing is akin to going really fast around the wrong race track. Things will feel good, but nobody will make progress.

Examples of good indicators:

Kanban boards. I love these for projects where the primary goal is to ship something. They publicly and visibly capture the set of things that are in flight, and who is responsible for what. They get updated in real-time. Kanban boards are supported by most task management software; putting up a large TV with a display of your team’s kanban board is a great way to check off the Visible Indicators pillar.
Real-time dashboards. When working on projects to improve performance, improve efficiency, reduce bugs, I like dashboards that reflect the current state of the world (latency measurements, bug counts, etc.) I get inspired by seeing steady progress toward the horizontal goal line.
To-do list. Low tech solutions are on occasion, even better than the high tech ones. A whiteboard chart showing deals closed and progress toward the ARR goal can be more inspiring than a fancy screen with a soulless number.

4. Accountability 💪🏽#

In 2018, the Facebook app was slow, and bugs were out of control. Most of Facebook’s mobile users are on Android; device and OS fragmentation make Android development particularly painful and error-prone. We knew we weren’t doing a good job serving our users but were stuck in the constant pull between new feature work and stability. Leadership eventually decided that enough was enough, and something had to be done. We had clear goals: fixing bugs and faster startup. The progress indicators were natural (bug count and startup latency). Leadership all the way through Mark was committed. Things still didn’t move.

The solution was a weekly meeting with all the engineering leaders and the head of the Facebook app. When some team wasn’t doing well (I was a frequent offender), people would ask questions about why and what they could do to help. The social pressure, my word as a leader, made me feel very accountable to deliver on my commitments.

As humans working cooperatively toward a higher goal, our word stands for something. I try my best to deliver on public promises. Holding a mirror up to my face when I fail is enough of a stick to keep me motivated.

The word “accountability” has frequent connotations of negative performance management, or of letting people go. I believe this is far too extreme; accountability is simply about a feeling of responsibility. Every parent knows they can’t control their young children but have a sense of responsibility for their actions.

I have two simple techniques for driving accountability:

Write detailed weekly status updates, publicly visible to the whole company. The leader must write these updates themself, rather than delegating them to an assistant. The process of writing forces the leader to read the constituent material and stay on top of execution issues in their team. Much more on this in the Execution Toolbox section below.
Weekly check-ins for the most critical projects. There is nothing that beats face to face (or video) interaction for driving accountability. These meetings are large and can get expensive really fast, so preparation and a concerted effort toward efficiency pays dividends quickly. Balance is essential — a status update meeting for every project in the organization is an egregious waste of time and unnecessary.

5. Deadlines ⏱#

Confession: I am a serial procrastinator. Why do something today that I could push off until tomorrow? The harder something is, the more likely I am to procrastinate. I know I am not alone. And teams procrastinate just as much as individual humans do.

Deadlines are an effective deterrent for a few reasons:

They create pressure to get things done. Creative work is not steady; it happens in bursts. Fixed periods provide a way to channel that bursty energy in a coordinated way.
Deadlines serve as a focusing function. When people have a deadline to get something done, other priorities get downgraded.
They provide opportunities to change directions. We all make incorrect decisions at times, bite off more than we can chew, or underestimate the complexity of a project. Changing decisions midway kills productivity and focus. However, changing direction at a particular deadline is a reasonable way of re-evaluating decisions and correcting course.

Deadlines go hand in hand with the curse of estimation. No sufficiently complex software project I have been on has been estimated correctly at the start. Are estimates even worth the work? I don’t believe in long term estimates — these are mostly garbage and serve to create artificial pressure by committing teams to death marches that are months long. Short term estimates - in the order of 2-6 weeks - are reasonably accurate and should be done. Call it a sprint, call it a milestone, call it whatever you want. Software engineering teams should be able to predict, with a small margin of error, what they can accomplish in 2-6 weeks and then reliably deliver on this estimate.

Execution Toolbox#

The toolbox is a set of things that have worked. I’ve included three ideas from leaders that I have worked with and respect greatly.

Weekly updates: perfect communication.
Daily office hour blocks: faster decision making.
Who-What-When: world’s most straightforward project tracking.
Recognition: positive reinforcement.
Retrospectives: feedback about how we work.

1. Weekly Updates#

Teams and leaders often write weekly updates, but without care and effort, these can feel like a lot of make-work. Weekly updates are one of the most powerful tools a leader has. They can keep tabs everything, as well as tell the company what’s on their mind.

Tricks to writing regular weekly updates:

Build a process that works for your team. In my case, PMs wrote an update for every major work-stream, with a deadline of Friday evening. I spent 2-3 hours over the weekend, understanding and pulling updates together. I would tune it on Monday and send it out Monday evening.
Avoid unanchored metrics — “Participation moved up by 0.5% last week because of 3D photos.” Is this good or bad? What is the baseline? What is the goal? Vague numbers engender more questions. Instead, try: “We rolled out 3D photos to 100%. As expected, participation rate increased to 3% (incremental 0.5%), bringing us close to our goal of 3.2%. We expect some follow on increases, as the rollouts continue.” Longer, but provides way more context and understanding than the former.
Remember the audience: exec leadership and the rest of the company. Stick to what is essential. It is okay for a week to be quiet and not much to report. If there are many weeks of quiet, leaders should check-in.

Getting to meaningful weekly updates took months. My most effective technique was rewriting individual PM updates as feedback. It helped everyone on the team write in one voice and significantly reduced the burden on me when writing the full update.

2. Daily Office Hour Blocks#

Most teams schedule leadership or exec reviews at a regular cadence of weeks. These reviews fall into two broad categories: decision making and information dissemination. When a team is deep in execution mode, slow decision making can break a team’s flow. Rebuilding that flow could take days.

In mid-2017, my team at Facebook was charged with building an AR-enabled camera feature for the Facebook family of apps. Our teams were running as hard as they could, but would get stuck when decisions needed to be made. Groups disagreed with one another and, on occasion, disagreed with themselves. At times, they just needed advice or someone to say, “go ahead.”

We would typically have reviews once a week. Due to packed schedules, any given team could get review time perhaps once a month. This was too long to wait on a decision, especially in the middle of an intense sprint.

Our solution was to devise a 2 hours block of office hours, every day, broken into 30-minute slots. The entire leadership team committed to being present. We decreased the level of homework needed to get into office hours — pre-reads were still ideal, but a two-paragraph email was a sufficient pre-read, instead of a ten slide deck.

The net result of this was that teams rarely sat on decisions for more than 48 hours. They could come in, talk about their problem, get help, advice, or a decision. Everyone on the leadership team would be aware and aligned. Teams would still have to communicate the decision out to the broader organization. However, thanks to liberal audience management, key team members were instantly in sync.

I found this incredibly useful but extremely taxing on time. There is some risk that you train teams to come to you for every small decision that they should be making or to break every minor disagreement that they had. This did happen, and we did ask teams to go back and sort it out on their own. However, in most cases, the issues were legitimate, and teams were able to move significantly faster.

3. Who-What-When#

Thanks to Maher Saba, Facebook Engineering VP, and leader extraordinaire.

Modern startups are drowning in project management tools. Asana, Jira, Monday, Notion, Coda, Google Sheets. These tools are flexible and sophisticated because they try to model an inherently messy, complex system. They can also be overkill — imposing concepts of owners or timelines or points or whatever ideas were built into the system.

Saba’s technique was to have the most straightforward tool possible: a doc tracking 3 things:

Who is accountable/responsible?
What is the item being delivered or shipped?
When will it be shipped by?

Below the table, there is a log to keep track of everything that is happening every week. This sounds ridiculously simple compared to all the powerful tools out there. Because it is. But it works well to run projects of small to medium size and complexity.

The goal of the process is to have an ongoing, credible plan where the dependencies and workload are well understood and balanced.

Here is a sample coda doc to duplicate if you’d like to implement this system.

4. Recognition#

Thanks to Ben Liebald, Head of Engineering at Branch.co

Execution grinds can take a toll on team health and morale. Good leaders find ways to keep spirits up. Celebrating wins along the way, or even milestones with significant progress, can do wonders. Here are some ideas to create opportunities for people (not just the leader) to recognize good work:

Friday Demos and Beer. At the end of the day on Friday, the team would gather, and anyone could present things they had been working on. The bar for demos was low — this was a safe space for showing off work in progress, not a polished presentation for the CEO.
Recognition rituals. One of my teams at Facebook would award a giant stuffed pusheen cat to the person who went above and beyond their job to help the team. This person would then pass the award on to the next person at the team’s weekly meeting. It was a great way to recognize work that would otherwise go unacknowledged publicly.
Gratitude emails. I love receiving emails that say something like, “I appreciate your hard work on feature X. I know this was challenging, and we were under a lot of pressure. You really stepped up and delivered. I’m so glad to have you on the team.” Sending these emails is effortless for the sender, but carries a lot of meaning for the recipient.

5. Retrospectives#

Thanks to Lily Zhang, Director of Engineering at Instacart.

Teams, like people, need time to reflect on what they have done, and where they could have done better. It is hard to find time for this reflection during regular business since everyone is chugging away at their task and moving as quickly as possible.

To improve, we need feedback loops. Immediate product feedback loops are rare, or the signals are often weak. Leaders can ease people into a habit of seeking feedback and improving, by providing a safe space to collectively reflect, give each other feedback, and figure out how to work better. There are two natural points for retrospectives:

Incident level. When there is a significant outage (“SEVs” in Facebook parlance), the first order of business is to deal with the outage. Facebook has this down to a science: the tooling and coordination needed for crisis management has been tuned to the point of near perfection. Once the team is past the crisis, there is a meeting called the “SEV Review,” where the person causing the SEV presents an analysis of what happened, why, what could have been done to prevent it, and the next steps to minimize chances of a similar future incident. Senior engineering leaders are in attendance — giving feedback by pattern matching against other SEVs. This is not a blame meeting; it is strictly a learning meeting where everyone in the company learns as a result.
Project level. One benefit of having deadlines (pillar #5) is that the end of a particular milestone presents an opportunity for reflection and improvement. I am a fan of using sticky notes to individually write what is working and what isn’t, before opening up the discussion to the entire room. We are human and biased, writing before presenting helps reduce availability bias.

#executionishard#

This post is not meant to be read once, understood, and put on the side. I hope you scanned it once, and return to it, when you have specific problems that these techniques can help with.

Remember, execution is really hard. It takes a long time to build execution muscle. If you’re a leader: be patient and keep improving with your teams. If you’re an individual contributor, and you see some key pillar missing, ask your leadership to fill in the gap.

Onward!

Buddha's Office

Rushabh Doshi — Sat, 28 Dec 2019 00:00:00 +0000

Buddha’s Office is the latest book by my friend, Dan Zigmond, who is a buddhist monk with a day job. I’ve worked with him at two companies (a rare pleasure): he’s a computer scientist turned data scientist turned leader. I have always admired his calm demeanor, his gentle but insightful arguments, and wondered how he manages to stay calm through intense situations. In this book, he gives a lot of practical advice on applying Buddhist principles toward that essential part of life: work.

Awakening#

Buddhist philosophy centers around the concept of leading an Awakened life - one where we are fully in the moment, accepting and appreciating life in all it’s aspects, without allowing ourselves to get sucked in by pain and unpleasantness.

Life is dukkha, which is often translated as “pain” but could be better interpreted as “struggle.”

We can run away from pain and suffering by avoiding it and chasing fun instead. It never lasts. And as soon as we know it’s not going to last, it stops being as much fun.

We can go the other extreme, and surround ourselves with suffering, inflicting it to the maximum possible extent. This doesn’t work either - all it accomplishes is making ourselves more miserable.

There is a middle path, built around acceptance.

Instead of running away from or toward pain, you accept it. By accepting that life involves pain, but pain doesn’t fully define life, you rob it of its power.

In other words, it is all about balance.

In the workplace, we all run into obstacles and setbacks. We don’t seek them out, but we can’t avoid them. Perhaps we didn’t get that job we really wanted, or the promotion we thought we deserved, or someone else got selected to work on the project we were really excited about, or our boss or colleague or customer treated us poorly, or we’re up against an impossible deadline, or just had a shitty day. These things happen.

The key to awakening is not to avoid problems or to ignore them. It is to accept that they happen and move on.

Practices#

The rest of the book is about practical advice to apply the above theory to work.

1. Mindfulness#

Buddha preached the Eightfold Path to Awakening:

Right views
Right intention
Right speech
Right action
Right livelihood
Right effort
Right mindfulness
Right concentration.

Mindfulness is the one that underpins the entire Eightfold Path.

Mindfulness is the state of being attentive to and aware of what is taking place in the present.

Meditation is the way we practice mindfulness.

2. Beginner’s Mind#

A beginner’s mind is an empty and ready mind: it is open to everything and full of possibilities.

Becoming an expert is appealing, especially at work. We like to feel competent and capable, we don’t like making mistakes. However, an expert mind can be narrow, less open to other ideas and possibilities, leading to carelessness and inattention.

We have all experienced a Beginner’s Mind at some point: starting a new job, embarking on a new project. That feeling of incompetence, the certainty of making mistakes. We can use this energy, this acceptance of not knowing, toward becoming great.

Tricks to cultivate a Beginner’s Mind at work:

Take a break. Perhaps the most effective way to snap out of monotony.
Change the task. We all have a number of things to do. Rotate between tasks to keep things fresh.
Change the setting. Desk work getting hard? Move to the library. Need to do some thinking? Go on a walk.
Work with new people. Are you an expert? Team up with someone that is just starting out. Are you the novice? Work with a master.
Get a hobby. Learning something new outside the workplace will give you a fresh perspective and self-awareness that you can bring to your day job.

Finally, resting and taking a break is really important: both during the day, but more importantly after work.

3. Exercise#

Do regular exercise. Physical activity improves literally everything.

4. Sleep#

Buddha worried a lot more about his disciples sleeping too much. In today’s world, the problem is too little sleep. Tips to improve your sleep:

Meditation. This helps everything.
No phones or electronic devices at night. This includes e-readers, such as the kindle.
Darkness. Use a mask if your surroundings don’t allow complete darkness.
Naps. If you can’t get enough sleep at night, take short naps during the day.

5. Diet#

Eat mindfully. Don’t eat at your desk.
Avoid alcohol. Alcohol and work absolutely do not mix.

6. Speech#

Right Speech is an integral part of the Eightfold Path. Buddha suggested asking the following questions before speaking:

Is it true?
Is it helpful?
Is now the right time?
Is it kind?

“One should speak confident, measured words, clear in meaning, delighting the mind, pleasing to the ear, soft and slow, and stemming from compassion.”

— Kate Crosby and Andrew Skilton, The Bodhicaryavatra

This is a good test for any speech. When delivering or receiving criticism, Dan suggests a few more things to keep in mind.

Timeliness is key, whether we’re sharing praise or dispraise.
Show care for the person, save criticism for the work.
Build trust by caring about your colleagues every day.
Accept criticism as a reflection of your work, not your self, even if the criticizing coworker is not skilled at making that distinction.

7. Discussions, Decisions, and Meetings#

Rules of argumentation and disagreement:

Don’t be a bully. If someone is questioning you about something, it is never the right response to “crush him, ridicule him, and seize upon a slight error.”
Focus on ideas, not people. It doesn’t matter who’s right — it matters what’s right. There is no value in scoring rhetorical points, or exploiting some unintended misstatement.
Summarize the opposing point of view. This is a good exercise to make sure you understand your opponent’s perspective and argument. Sometimes it even results in one changing their mind!

Meetings are how a lot of modern work gets done. Meetings are one of two types:

Decision making meetings. Clear goal (make a decision), small audience, required participation, active and lively meeting.
Information sharing meetings. Wider audience, more passive meeting. These meetings are still necessary, in the age of email and Slack, because reading fails to convey emotion and nuance.

Tips for effective meetings:

No interruptions. Interruptions are contagious, but waiting your turn to speak, and indicating that you’d like to - is contagious as well!
Everyone gets to speak (in decision making meetings). No one monopolizes the conversation. If the meeting was correctly constructed, everyone was invited for a reason. There should be no pure spectators.
Follow the rules of disagreement 👆🏽
Make a decision. Don’t let the perfect be the enemy of the good enough.

8. Burnout#

Do. Or do not. There is no try. — Master Yoda, Definitely Not a Buddhist.

Unlike Yoda, Buddhists believe in trying. Buddha worried about laziness, and knew that anything worth doing takes effort, and that includes everything from career advancement to spiritual awakening.

A constant striving can lead to burnout. Techniques to avoid burnout:

Spend at least twenty percent of your time on the aspect of your work you love.
Morning ritual: make a list of things you are going to accomplish that day. Writing it down helps stay motivated and remember everything you set out to do.
Evening ritual: reflect back on your day. How did you spend your time? What did you accomplish? Did you get to the parts of your job you love?

Other techniques mentioned above — eating well, sleeping, meditation, exercise — all contribute to reducing the chances of burnout.

9. Breathe#

Everyone has bad days.

Everyone has days when all the real problems feel like more than you can handle. Things to do when work gets overwhelming:

Trust your colleagues. You don’t have to do everything. Imagine that you got sick or had an accident. Is the company going to grind to a halt? Unlikely.
Don’t fight for work. If someone wants to do a particular task, let them. Don’t get territorial. Accept help when it’s offered. There are very few tasks that you must do yourself.
Admit your limitations. Don’t exude confidence all the way to the deadline and then fail. Let people know and be vulnerable. Tell your boss, let them help you.
Breathe. Breathing isn’t limited to mindfulness meditation. You can use breathing throughout the day as a means of overcoming stress and despair.

10. Work-Life Balance#

Buddha avoided the question of “work-life balance” by entirely renouncing both work and home life.

Work is a part of life. Life has many parts that need to be balanced. In addition to our careers and personal lives, there’s sleep, wakefulness, activity and rest, friends and family, social connection and solitude. Balancing the time we spend in and out of work is only one of many tradeoffs and not necessarily the hardest or the most important. Make conscious choices in seeking out this balance. The likelihood of finding the perfect balance is low, but the chances of stumbling upon it by pure chance are zero.

Keep in mind that asking this very question is a privilege. For much of human history, and for a large part of the world, working is a matter of subsistence and survival. If you’re able to exercise control over when and how much you’re working, you’re already one of the lucky ones.

11. You Are Not Your Job#

This chapter made the whole book worthwhile for me. It spoke to me in a way that nothing else on this subject has before.

You don’t exist.

Imagine your car. It exists. It’s got wheels and an engine and seats and a steering wheel and everything. It is your Car.

Now let’s say we replace a wheel. Is it still your Car? Yes.

What if we replace all 4 wheels and the seats? Maybe.

What if we slowly replaced every single part in the car? Is it still your Car? Or just some other car that happens to occupy the same physical space? Maybe not.

If it’s not your car anymore, when did it stop being your car? If you can’t point to a specific moment when it stopped being your car, then it could not have been your car to begin with! The concept of “your car” is just that - a concept, an idea.

We have the same dilemma with people as we did with the car. You are constantly changing. And if so much about you has changed, in what sense are you the same you? And if you’re not the same you anymore, maybe you were never quite you to begin with.

If there is no unchanging thing that defines our true self, maybe there is no self at all.

Perhaps you don’t accept the above. Maybe there is some essential constant that defines you. However, this essential constant cannot have anything to do with your job: you could lose your job, you could quit, your boss could go crazy, your company could implode. Stuff happens.

If whatever constitutes you can survive years and years of physical and mental change, it cannot possibly depend on who issues your paycheck.

You are not your job. Whatever you are — you’re definitely not that.

When we define ourselves by our jobs, we invite all manner of suffering. We set ourselves up for countless disappointments when things go wrong at work. We start holding on too tightly to our jobs, living in fear of ever letting them go.

This delusion that we are what we do is an especially dangerous one.

You are not your job. Your colleagues are not their jobs either.

12. Flow#

Buddha placed great value on concentration, and much of his teaching on meditation describes how to get into the deeper and deeper states of concentration that he called “absorptions.” In popular psychology, this total absorption is usually called “flow”. It is exhilarating and productive, but very fragile.

Flow is that “subjective state that people report when they are completely involved in something to the point of forgetting time, fatigue, and everything else but the activity itself.”

Distraction is the enemy of flow. How do we reduce distractions? Meditation and quieting electronics help. If working from home, have a specific work place, and some rituals to prevent slippage into other life distractions.

13. Buddhism#

Buddhism is not a faith that asks you to be exclusive or to give up any other beliefs. Buddhism doesn’t really ask you to believe anything.

Buddhism is more about doing things, practicing things, having experiences rather than beliefs, and then paying attention to those experiences and results.

Coaching (Feedback 3)

Rushabh Doshi — Mon, 23 Dec 2019 00:00:00 +0000

The final note in the series brings theory and practice together into two ideas: first, in order to improve at anything, you need fast feedback loops. Second, coaches are a great way to create these feedback loops in complex environments. I believe coaching is an essential part of management and part of a manager’s job. I also suggest two approaches to getting coaching when your manager is unable to coach.

– Photo by Moises Alex, Unsplash

Atul Gawande is a master surgeon. Specializing in endocrine surgery, he has performed thousands of operations over a long career. For years, Gawande had been steadily improving - his rates of complications were dropping steadily, and he consistently beat the averages, when compared to national data. Then he hit a wall. It seemed like he had hit peak performance, and the only place he could go from there was down.

On an out of town trip, Gawande was trying to play a game of tennis, for fun. He had been an aspiring tennis player growing up but had long since given up that dream. He couldn’t find a game, and the ball machine was reserved for members. He paid the local pro to play with him for some time. After running Gawande around, the pro couldn’t help get into coaching mode. He pointed out that Gawande could get more power into his serve by paying attention to his feet. Gawande gave it a go and within a short period, the pro had Gawande serving at least 10 mph faster than before: a new personal record.

Everyone knows that athletes have coaches. So do musicians. Why not doctors?

Gawande decided to try an experiment and got another expert doctor to coach him. The coach sat in Gawande’s surgery, took extensive notes, and gave him lots of feedback about things that he could improve. Bear in mind: this is one of the world’s foremost surgeons, an expert in their field, at their performance peak, getting feedback and improving!

– excerpts from Personal Best, Atul Gawande, New Yorker

Building Fast Feedback Loops#

By this point, I have hopefully convinced you of the importance of feedback. Implementing feedback loops, however, is not easy.

Sometimes, tasks have in-built fast feedback loops. Remember Ericsson’s experiment in Deliberate Practice? If the student was able to accomplish the job, it got harder. Else it got easier.

We see this characteristic in many software problems, as well. Imagine an engineer working on making their service faster. The performance of the service either improves or doesn’t (or worse, regresses): an in-built fast feedback loop.

Some feedback loops can be very long. One can learn management by building a team, leading them into disaster, learning, starting again, creating the next team, etc. The feedback loop exists, but just takes years to run its course, and includes a large number of confounding factors.

To get good at anything, we need fast feedback loops - ideally realtime, but failing that, as close to realtime as possible.

Feedback Loops For Complex Domains#

Coaches for athletics, music, and even medicine, are effective because they are taking in a vast amount of complex information, pattern matching against what they know to be optimal, and efficiently communicating the result.

This process currently requires a human. I imagine that over the next decade, we will build better AI-powered coaches (using computer vision to analyze your tennis serve), and the human coaches will move on to increasingly complex domains.

People management in any organization is a very complex domain. Everyone working in a modern company is seeking to improve. To improve, they need fast feedback.

Managers as Coaches#

Managers are uniquely positioned to provide fast and accurate feedback. They observe most of what their team is doing and the team’s struggles. They may have done the same things before, or they may have done similar things and can pattern match, or they may just have a wider lens. Regardless of the reason, they can provide helpful feedback and dramatically improve their reports’ performance.

I believe that coaching is a crucial part of a manager’s job. If you’re a manager and aren’t providing great coaching to your team, you’re failing at an essential part of your job.

Coaching is not the only thing that managers do - they have to build teams, deliver outcomes, and in general, lead. Managers are human - and neglecting the important in favor of the urgent is a very human fault. To be great coaches, managers must put in time and effort toward this goal.

Other Coaches#

Sometimes your manager cannot provide you all the coaching you need. They may not be great coaches or are working on their coaching skills. They may not have the time - this is a frequent occurrence in executive situations. They may not believe in its value or have the inclination. In any case, if your manager cannot provide you adequate coaching, here are some other ideas to explore.

Peer coaching. Your peers have skills that you may be lacking and vice-versa. Put enough people in a group together, and there can be a mutual sharing of skills that helps the whole group improve. Running coaching circles takes time and effort: there has to be a set of people dedicated to running them, bringing people together, setting the agenda, and helping moderate the discussion. When done well, coaching circles can be effective and help form close bonds among colleagues as a side effect.
Hire a coach. Many large tech companies, such as Facebook and Google, offer employees at a certain level, the ability to get external coaches. Executive coaches are expensive, and the company will likely partner with another organization or agency to pair coaches with people. One can also directly hire a life coach. Many companies will help you find the right one. However, I have no experience with any of these companies or their services, and cannot recommend one platform over another.

Important note when hiring a coach: a great coach can improve your game; a bad coach can destroy it. Be careful from who you seek counsel. Check referrals, talk to them, and do your homework.

Recap#

This is the third of three notes on Feedback. This note made the case for building rapid feedback loops, getting a coach, and why managers should treat coaching as a critical part of their job.

Delivering Constructive Feedback 2. Be timely. The effectiveness of feedback decays rapidly over time. 3. Be firm and kind. Don’t beat around the bush, don’t sugar coat, and don’t be an asshole. 4. Show the way. Point out flaws, but illustrate solutions with examples.
Deliberate Practice 6. Have a specific goal. 7. Focus on achieving that goal. 8. Get rapid feedback. 9. Persevere through the pain.
Coaching 👈🏽this note. 11. Feedback is critical. 12. Build fast feedback loops, automated or human. 13. Coaching is a manager’s job.

{% include about.md %}

Deliberate Practice (Feedback 2)

Rushabh Doshi — Tue, 17 Dec 2019 00:00:00 +0000

This is the second of three notes on Feedback. The first one covered pragmatic tips on delivering constructive feedback; this note goes into the theory behind achieving peak performance.

How do we become experts? How can we learn new skills in depth? What is the process, what are the steps, and who do we need? Deliberate Practice is one way to learn and grow: have clear goals, focus, get feedback, and overcome frustration.

– Photo by Jasper van der Meij on Unsplash

I have been playing squash off and on for many years. Some friends got me into it, and I got hooked. It’s easy to get started, a fantastic cardio workout, and immensely fun. However, my game wasn’t particularly good, and I belonged in the lowest tiers of the club. Starting last year, I started getting coaching. During each session, my coach would point out three things to focus on, work with me on those three things, and then ask me to practice those things in solo practice over the week, before coming back to the next coaching session — the result: more improvement in 3 months than ten years of playing the game.

– Nour El Tayeb vs Raneed El Welily, Netsuite Open Finals, San Francisco, 2019

This story is not surprising for anyone that has seriously pursued sports, or music, or art: I got some coaching, and my game improved. Where’s the story?

Sports stories are interesting because people are engaging in Deliberate Practice without explicitly trying. Deliberate Practice is a theory by Anders Ericsson, described in his book Peak (YouTube summary video).

“Most individuals who start as active professionals or as beginners in a domain change their behavior and increase their performance for a limited time until they reach an acceptable level. Beyond this point, however, further improvements appear to be unpredictable, and the number of years of work and leisure experience in a domain is a poor predictor of attained performance (Ericsson & Lehmann, 1996). Hence, continued improvements (changes) in achievement are not automatic consequences of more experience, and in those domains where performance consistently increases aspiring experts seek out particular kinds of experience, that is deliberate practice (Ericsson, Krampe & Tesch-Römer, 1993)–activities designed, typically by a teacher, for the sole purpose of effectively improving specific aspects of an individual’s performance.

– Anders Ericsson, https://psy.fsu.edu/faculty/ericssonk/ericsson.exp.perf.html

In other words, it is easy to get started. Most people then hit a wall. Getting over that wall takes effort, focus, feedback, and grit. This formula applies to any learning activity and is how people become experts.

10000 Hours#

Remember Outliers, Malcolm Gladwell’s 2008 book that shot up the NY Times’s bestseller list? He examined what factors differentiated the great people from the merely accomplished. He repeatedly brought up the concept of 10000 hours - the amount of time that experts dedicated to their craft, vs. about 2000 hours for amateurs.

Over time, we took this idea and ran with it - to be good at anything, we need to put in 10000 hours of work. In other words, put in this much work, and voila - you’re an expert. It turns out this isn’t true, and there are other factors besides the time that goes into building real expertise.

10000 Hours Of Deliberate Practice#

What is “Deliberate Practice”? Ericsson breaks this down into four things:

🎯 Specific goal.
🕯 Intense focus.
⏱ Immediate feedback.
😢 Discomfort.

The best illustration of these is through the experiment that Ericsson’s research group ran, as described in Peak.

A graduate student is committed to spending an hour a week, working on getting better at a simple task: take a string of 7 random numbers, memorize them, and recite them from memory. They started at seven because that’s widely believed to be the number that humans can remember easily. When the student could quote them accurately, they increased the length of the number string (8, 9, 10 ..), and when they made a mistake, they reduced the series.

The student struggled with this for a while. It got frustrating. Then he made a breakthrough! He got to 11 digits. Then - frustration again. Struggle → Breakthrough! 22 digits! This pattern kept repeating itself till he got to 82 digits! 82 is an insane number of digits to memorize, and the student got there over the course of 200 sessions of an hour each.

Looking at this example through the lens of deliberate practice:

🎯 Specific goal: remember and recite strings of digits of increasing length.
🕯 Intense focus: no interruptions during the hour-long session, with a singular focus on the task at hand.
⏱ Immediate feedback: when he slipped, the string shrank.
😢 Discomfort: it was hard and frustrating.

Then they went one step further: they trained another student in the same task, but the first student was their coach and taught them his techniques in digit memorization. Result: the second student learned even faster and developed their techniques when the coach’s methods no longer worked for them.

Deliberate Practice At Work#

Whether you’re an engineer, a designer, a data scientist, or a manager, you’re trying to be better at your job. There is a degree of intrinsic satisfaction in a job done well; there is a feeling of growth as you conquer and solve previously unsolvable problems. There are the extrinsic benefits of growth: bonuses and promotions, to go along with increased responsibility.

The best way to get better at what you do is to apply the principles of Deliberate Practice to whatever you want to get better at. Imagine you’re an engineer, looking to get better at your craft. What should you do?

🎯 Set a clear, measurable goal. It’s not enough to build a system; the system should have some performance or scale or other criteria that are hard to hit.
🕯 Focus. The modern, open workplace makes this challenging, but you can focus. Buy some noise-canceling headphones, clear your calendar, find a quiet room.
⏱ Get feedback. Engineering systems have intrinsic feedback built-in, via compilers, unit-tests, and end-to-end tests. Get human feedback through code reviews and design reviews, with people who are better than you. Managers can provide non-technical feedback, helping you pick the right problem, and communicate solutions.
😢 Be one with the discomfort. It won’t be easy. There will be days or weeks of no progress. Learning and growth have plateaus, and that is okay.

The Manager’s Role#

What about managers? Managers are both coaches that are helping their reports, and humans who are improving their management skills. The best thing you can do for your team as their manager is to make sure they are adhering to the principles of deliberate practice:

🎯 Make sure everyone has clear goals. The weekly 1-1 is an excellent time to check in. Goals should form the beginning of every performance review conversation.
🕯 Defend their time. Focus is important.
1. Block “maker time” on everyone’s calendars.
2. Move all team meetings to a couple of days in the week, and keep the rest of the days blocked off for work.
3. Encourage reading email twice a day and shutting down slack notifications a few hours at a time.
4. Buy your team good, noise-canceling headphones. Bose QC-35-ii is my favorite. $250 here will be repaid in growth and productivity quickly.
⏱ Give them timely feedback. Giving good constructive feedback is hard, but crucial to growth. I’ve documented my thoughts and tips on how to give good feedback here.
😢 Be patient. Know that learning is not a smooth linear curve. It goes in spurts. Be patient when they are struggling with a plateau, knowing that if they put the effort in, the growth will happen.

I was a Peloton skeptic, and late to the party. It’s an incredibly expensive exercise bike, and I figured it was a craze. I could not have been more wrong.

The magic of Peloton is not about bikes being well built, or the instructors being amazing (they are), or the social pressure of leaderboards (it can be fun at times). Peloton is magical because it tricks you into deliberate practice without realization. Pelotons have:

🎯 Goals. You can set your own - get on the bike every day, compete with others, compete against yourself - all of these are easily trackable through the app.
🕯 Focus. It is impossible to do anything while you’re in a class.
⏱ Instant feedback. Your cadence, your resistance, your output.
😢 Pain. Thirty minutes into a 45-minute class, all you can think of is giving up.

But it works. Here’s my total output over time for 30-minute intense efforts as evidence.

Tricking people into Deliberate Practice is something that a run-of-the-mill exercise bike cannot do. You can be on your exercise bike for 10000 hours vs a Peloton for a fraction of the time, and I guarantee faster progress toward your fitness goals on the latter.

Recap: The Feedback Series#

This is the second of three notes on Feedback. We went into the theory of deliberate practice, which provides some hints on how to become an expert at anything.

Delivering Critical Feedback
1. Be timely. The effectiveness of feedback decays rapidly over time.
2. Be firm and kind. Don’t beat around the bush, don’t sugar coat, and don’t be an asshole.
3. Show the way. Point out flaws, but illustrate solutions with examples.
Deliberate Practice 👈🏽this note.
1. Have a specific goal.
2. Focus on achieving that goal.
3. Get rapid feedback.
4. Persevere through the pain.
Coaching

{% include about.md %}

Delivering Constructive Feedback (Feedback 1)

Rushabh Doshi — Sat, 14 Dec 2019 00:00:00 +0000

Everyone screws up, everyone makes mistakes. When we err, hopefully, someone pulls us aside and tells us about our blunder. Feedback, when delivered on time and in a constructive manner, is critical to learning and growth. Three things that have made my feedback stick: 1 provide it as quickly as possible, 2. be firm and kind, and 3. suggest how you would have acted in that situation.

1. Timing Is Everything#

Bill is a promising PM on the team. He has excellent product insight and can execute projects well. However, he struggles with selling his ideas to the Bigwigs. Good ideas get rejected, and people leave the room in confusion.

During his annual performance review, Bill gets feedback from his manager Amy: “Bill should work on his communication skills. In particular, he can do a better job selling his ideas to executive leadership.”

Feedback? ✅

Helpful? 👎🏽

The effectiveness of feedback decays rapidly over time. If you want the feedback to stick, deliver it while it is still fresh in everyone’s minds. Don’t wait for the annual review or that perfect opportunity. Don’t even wait for the next 1-1. Give feedback immediately while the events are fresh in everyone’s minds.

As an aside, this is the reason that annual reviews are mostly a waste of time from a learning and growth perspective (though they are necessary for other purposes). By the time yearly reviews come around, the illustrative events have become a faded memory, and the opportunity for learning is lost.

2. Be Firm and Honest. Be Kind and Human.#

Joe gets a code review request from Rita. It’s not an outstanding PR. The code is hard to read, and she didn’t include unit tests. Joe doesn’t want to hurt Rita’s feelings. He knows she’s trying hard. She is new to the company and doesn’t know the standards. He accepts it, making a note to himself to go back and add tests and fix things up.

Oops - Joe just lost a valuable opportunity to give feedback because he was trying hard to not hurt Rita’s feelings.

– XKCD 1513, https://xkcd.com/1513/

On the other side of the room, Adam sends some code over to Lisa. Lisa is a veteran and has been with the company forever. Adam is new and less confident, given his background. He doesn’t know the company’s style guide, and frankly, doesn’t see the value of having consistent code style. Lisa blasts the code review and publicly questions Adam’s knowledge and standards, using aggressive language. In short, she is being a jerk.

Another lost opportunity - given Lisa’s demeanor, Adam will get defensive and refuse to listen to feedback, likely to repeat the same mistake again.

Two rules for managers:

Don’t be overly empathetic. People are adults, are here to do a job, and get better. Too much empathy confuses or kills the message.
Don’t be an asshole. You’re telling someone they messed up, and they’re likely going to be hurt. Rubbing it is, belaboring some point, repeating facts in different ways - these things don’t help. Your goal is to build trust, not destroy it.

We can look at communication challenges as a clash of two cultures: Ask Culture and Guess Culture.

In some families, you grow up with the expectation that it’s OK to ask for anything at all, but you gotta realize you might get no for an answer. This is Ask Culture.

In Guess Culture, you avoid putting a request into words unless you’re pretty sure the answer will be yes.

– Metafilter comment, tangerine, 2007

In ask-culture environments, people have no qualms about asking for things, favors, help, things they want. They also expect to get turned down a lot.

In guess-culture environments, people don’t ask for something unless they are reasonably certain the answer is yes. And even then, in these cultures, the appropriate thing to do may be to refuse the help and let the helper insist on helping.

Your boss, asking for a project to be finished early, may be an overdemanding boor – or just an Asker, who’s assuming you might decline. If you’re a Guesser, you’ll hear it as an expectation.

This is a spectrum, not a dichotomy, and it explains cross-cultural awkwardnesses, too: Brits and Americans get discombobulated doing business in Japan, because it’s a Guess culture, yet experience Russians as rude, because they’re diehard Askers."

– Askers vs Guessers, The Atlantic

What happens when an Asker meets a Guesser?

Asker thinks that the Guesser doesn’t speak their mind, is two-faced, is confusing, and hard to comprehend.
Guesser thinks that Asker is very rude, extremely harsh, loud, obnoxious, and uncomfortable.

Keep these differences in mind when delivering feedback and tune your language appropriately!

3. How would you approach the problem?#

Pointing out problems in time gets us halfway there. The rest of the way is teaching someone how to do it better. While nobody is perfect, the person giving feedback typically has some additional information - in the form of more experience, or broader context, or greater depth - that they can use to help the recipient.

Back at Facebook, we were working on Stories, and things weren’t going particularly well. We were in the doldrums; the team was getting unmotivated. Chris Cox, Facebook’s then Product Chief, suggested writing a note explaining why Stories mattered, to get the team excited. Adam Mosseri, my boss, suggested that I take this challenge, and I enthusiastically signed up.

And utterly failed. I could not, for the life of me, write something that came across as remotely engaging or motivating. I’m used to writing dry, technical docs. Even now, my writing tends to be terse and to the point, not exactly motivational speak.

After giving me some time to struggle, Adam checked in. And he asked if I’d prefer he write it this time. And in about 10 minutes he wrote one of the best, heartfelt notes, about our work. People loved it. And I learned how to write better.

When the person giving feedback also illustrates with an example, when your coach tells you what to do, in addition to pointing out errors, it dramatically increases your ability to learn.

I suggest following up with “here’s how I’d do it” - not merely as a thing to copy (though that is perfectly fine), but as a means of illustrating and teaching. Encourage the recipient to take what they like, make it their own, and get better.

Note that this is different from “micro-management.” You’re not telling people exactly what to do and how to do it. You are teaching what you know and encouraging them to find their path.

Reading#

I strongly recommend Kim Scott’s Radical Candor (book, 20m YouTube video) as an easy to understand framework on how to deliver feedback. She espouses a 2x2 (my friends know how much I love 2x2s) that goes something like this:

The central idea behind Radical Candor is to care personally and challenge directly.

Caring personally is to care about your report’s well-being and growth at a deep, emotional level. This care should go across the current project and team. Many people who reported to me are close friends to this day, across organizations, and companies. Some of my best bosses have cared deeply about me, and my well-being - not only are we still friends, but I’d work for any of them again in a heartbeat.

To directly challenge someone, you have to be able to tell them, to their face, when they’re screwing up; when they’ve let you down; when they’ve disappointed you. You expect more, you know they can do better, and you invest in them by telling them to step up.

Scott uses these two axes to create the above 2x2: any of the three quadrants in gray result in little or no change in employee behavior, while the 4th orange quadrant helps people get better.

I’d encourage you to at least watch her video and ideally, buy and read the book.

Recap: The Feedback Series#

This is the first of three notes on Feedback. This note is the practical punchline: how to give effective, constructive feedback, without being an asshole or being overly empathetic.

Delivering Critical Feedback 👈🏽 this note.
1. Be timely. The effectiveness of feedback decays rapidly over time.
2. Be firm and kind. Don’t beat around the bush, don’t sugar coat, and don’t be an asshole.
3. Show the way. Point out flaws, but illustrate solutions with examples.
Deliberate Practice
Coaching

{% include about.md %}

Tips for Better Meetings

Rushabh Doshi — Wed, 04 Dec 2019 00:00:00 +0000

(Photo by Matteo Di Iorio on Unsplash)

Managers work through meetings. In this short post, I posit three dimensions along which we can improve meetings and present tips that have had the most impact on my meetings along these dimensions.

Growing up as an engineer, I equated meetings with wasting time. Naturally, when I become a manager, I tried to reduce or eliminate meetings, which did not work (duh!). I failed to understand that meetings are a manager’s work. I might as well have been trying to become a champion swimmer by eliminating water.

Andy Grove’s High Output Management was the bright light that cut through the darkness of my anti-meeting feelings. Grove devotes an entire chapter to meetings, calling them “the medium of a manager’s work” (good summary). He decomposes meetings into different classes, identifies why they are necessary, and builds a framework around how much time and energy should be devoted to these. For the first time, things made sense.

Since that step-changing moment, like any competent engineer, I have been focused on optimizing meetings. To make something better, we have to agree on what “better” means. I think of improving meetings on three dimensions:

Correctness: did the meeting achieve a good outcome? If it was a decision-making meeting, did we make the right decision?
Efficiency: how quickly did we get to our goal? Did we communicate clearly?
Culture: did people feel included and heard? Did people move forward as a team, even if they disagreed with the outcome?

In the sections below, I present tips that have made material differences along each dimension. This note is not a comprehensive survey; it’s meant to be a quick 1-2-3. For the impatient: skip directly to the tips.

A disclaimer: This material is not original. The business world has studied meetings forever. This 1976 article (HBR) is one of the best on running good meetings. Here’s New York Times and Rands In Repose. My contribution is to distill tips that you can copy without having to read through tons of meeting-theory.

Another disclaimer: Meetings can range from huge (company all hands) to tiny (1-1s). Somewhere in between are the 5-15 people meetings where a company’s crucial work gets done. This article is about those meetings.

Correctness ✅#

I am a big fan of Kahneman’s Thinking Fast and Slow (video summary). Our savannah evolved brains are hardwired to do what he calls “Type-1 Thinking”. When our ancestors saw a lion, they didn’t spend much time thinking deeply about their choices; they climbed a tree. The ones that thought hard aren’t our ancestors. When we sit in meetings where we see material for the first time and have to pass judgment under the gun, we are engaging in Type-1 thinking.

The alternative is to engage the slow brain. This deliberate, rational process is called “Type-2 thinking” in Kahneman’s lore. Engaging the slow brain is challenging: it is genuinely hard work and time-consuming. But it leads to better decisions on complex topics.

Tip: Send all materials out 24 hours ahead to be read as homework. 📖

Engage your slow brain. Read in the peace of your home or wherever you can think deeply. Think about it in the shower, or on your run, or wherever you do your best thinking.

Sending everything out as homework is taxing. Gathering materials and putting things together a day early shifts the schedule. It also forces more work on everyone attending the meeting: they need to spend time reading, digesting, and preparing for the meeting the day before.

My advice is to ease into this: try doing this with your most important meetings first, then work backward until pre-reads become team culture.

Efficiency ⏰#

People have put much energy into making meetings more efficient. Good meeting hygiene goes something like this:

Send the agenda out ahead of time.
Define and focus on the goal.
Keep track of time.
Make a decision.
Write whatever was decided and send it out to everyone.

There is some variation, depending on the type of meeting, but mostly variations on the same theme.

I believe #2 is the most important. It is amazing how many meetings people go to and ask, “why are we here?”. Or we are halfway through the allotted time, and we haven’t even talked about the most important thing we are here for. What’s the problem? We are missing a goal.

Efficiency Tip: Have a clear and stated goal. Focus the conversation on said goal. 🎯

Meetings with unclear goals tended to meander off course, people ratholed on things that were secondary to the task at hand, and these meetings mostly resulted in follow-up meetings, with no actual progress made.

Start your meetings with an articulation of why you’re all in one place and what success is.

“Let’s talk about making things fast.” is not a goal statement. We can talk about the topic for a long time, but to what end? Is it because we think the site is slow? Is someone complaining? Or do we believe in speed as an engineering value and want to enforce it?

“We are here to decide whether or not we should invest in performance right now.” This statement clarifies our purpose, what success would look like (a decision, positive or negative), guides the conversation (is your point germane to deciding whether we should invest now or later?)

To build this discipline, start each meeting by asking two questions:

Why are we here?
What would success look like at the end of the meeting?

You may feel this is rote work, but soon, the team will adapt, and these questions will be answered in the material before anyone has to ask.

Culture ✨#

Humans are social animals. Meetings aren’t just about getting work done. There are a lot of other things going on in meetings - the establishment of social status, signaling on who’s a Big Deal, validation of ideas, people, and their work. We can pretend to be robots, but we aren’t (yet).

The cultural aspects can go very right - people come out of meetings with clarity, purpose, camaraderie, eagerness to make progress - or it can go very wrong - people not feeling heard, hurt feelings, anger, and frustration at adverse outcomes.

Leaders tend to focus a lot of correctness and efficiency, but forget about the cultural element. To build a strong, cohesive team and organization, pay attention to this part.

When I was working on Facebook Stories, we observed a phenomenon in any prominent meeting. As things were getting started, people would walk into the review room — and make a beeline for the chairs on the side while the table was unoccupied. Even worse, the people sitting off-table were often women or minorities or people who were shy or reserved by nature. They might be the primary authors and experts — their work was about to be presented, but they didn’t sit at the table.

Culture Tip #1: Everyone sits at the table. 🪑

(The credit for this tip goes to Adam Mosseri, Head of Instagram, boss extraordinaire. He would publicly call out people and get them to sit at the table, even if he had to stand on the side.)

We may think this sort of physical signaling doesn’t matter. And most meetings nowadays have some people dialing in remotely. Our non-scientific evidence would suggest otherwise.

Designers should present designs. Researchers: research. Data scientists: analyses. This idea seems obvious but is rarely the case in practice. Typically, the Product Manager, the one synthesizing the materials to create one cohesive narrative, presents the work. Not only does she present the work, she also takes the first stab at answering questions, all while the primary author, the expert, is sitting in the room. Net result: people feel voiceless.

Culture Tip #2: Authors present their work. 👩🏽‍🔬

Let primary authors talk about their work, even at the cost of interrupting flow. Changing speakers mid-flight seemed awkward at first. Presenters want to control the room because they feel it will get them to better outcomes. However, we’re not in the middle of a sales pitch; we’re in the middle of trying to come together as a team and work things out.

One way to ease into speaker rotation is to enforce crediting. “Liz and the research team found this through their survey work. I am going to talk through this, but they’re the experts.” Clear attribution gets you halfway there.

In its optimal state, speaker rotation makes everyone on the team feel valued. They feel like the leader knows who they are, and cares about them, and binds them closer as a result.

Every meeting has someone that talks too much (I know; I used to be that guy) and people that don’t speak up, even when they have the best knowledge or the most astute and relevant opinion. It is the leader’s job to make sure they have heard from everyone in the room by creating space for the quietest people to speak.

Culture Tip #3: Everyone speaks at some point, and nobody interrupts. 💬

The way to accomplish this is to quiet the loud person — indirectly: by giving them high cognition tasks like being the scribe, or directly: by telling that you understand their perspective and would like to hear from others instead.

Getting the quiet people to speak can be harder. The most effective technique, in my experience, is to directly ask them. “Jo, what do you think?” Keep a mental track of who has spoken and remember the people dialed in on Zoom.

Also — getting people to speak up is not useful if they get interrupted or passively ignored. Both these behaviors make people feel like nobody cares or values their thoughts. Fixing interruptive or ignoring behavior takes work and courage. One has to call out the interrupter publicly, and often, the person interrupting is the leader. The best people do this, gain the respect of everyone in the room, and build stronger teams as a result.

Parting Thoughts#

I hope you incorporate one or more of these tips into your meeting repertoire. If you had to pick only one, I’d recommend the pre-read / homework tip. I guarantee it will make your decisions and outcomes better.

Do you have tips that have made your meetings infinitely better? I’d love to hear them (so would the world!). Drop me a line.

{% include about.md %}

Durable Decisions

Rushabh Doshi — Tue, 26 Nov 2019 00:00:00 +0000

One of my favorite internal posts at Facebook outlined a framework on making sticky decisions. This post is a rewrite of that original “Durable Decisions” post by Ficus Kirkpatrick, in turn based on ideas and input from Nikhil Chandhok. The credit for the idea and framework go to them; this post is written by me since I have more cycles to write.

Your team needs to decide whether to focus on making your product faster or building a feature that is going to be the next Big Thing. The team follows best practices, does its homework, runs it by you. Decision made. We are focusing on building the feature.

Fast forward two weeks, and you hear rumblings. The star engineer on the team is working on speed. The feature isn’t making as much progress as everyone would like. Three weeks in, the team has started to work on speed and drop the feature altogether. It just happened. Four weeks in, the feature is not built, and the product isn’t significantly faster either. Oops.

So, what happened? We made a decision four weeks ago. Then something changed. And the decision unmade itself over the course of a month. Can this be avoided? How?

Enter durable decisions.

A durable decision is one that doesn’t change if its inputs haven’t changed.

Inputs#

1. Goals#

What are the goals of the project? What are non-goals? Is a project results-bound (done when metric X is at threshold Y?), delivery-bound (done when feature 𝝰 is shipped), or time-bound (done in a month)?

The goal discussion is almost always the right place to start. Disagreements at this phase result in significant downstream thrash. Have precise goals (and non-goals) and write them down.

2. Assumptions#

We are baking in many assumptions when we come up with various options. Time, money, technical capability (CPU, GPU, RAM, disk, network), people (how many people, how much bandwidth), this list goes on.

Assumptions are the second biggest reason (after unclear goals) for decisions to fall apart. People either assumed different things, or those assumptions changed. Exhaustively documenting every assumption is not a good use of time; documenting the major assumptions, those that are likely to break or cause disagreements is a good use of time.

3. Options#

Come up with a good list of options, including tradeoffs between them, or the rationale to choose one over another. Take care to not get too attached to one option over another; we are human, and this is natural. A mentor gave me a pretty good trick to get over this attachment. He said:

You don’t understand your opponent’s argument until you can argue their points, just as strongly, on their behalf, without them being in the room.

Understand your options thoroughly and drumroll please write them down.

4. Decider(s)#

Be explicit about who is making the decision: some person, some committee. When things change in the future, you’ll want to know whom people should go to talk. Gathering Inputs

To make a sticky decision, get everyone’s buy-in. The best way to get them bought in is to get them involved in making the decision early on. You need to consult people to gather input, to build your set of options, to understand all the assumptions that are being baked in.

Consult the stakeholders. Talk to everyone that is likely going to get affected by a significant decision. More buy-in at this stage reduces the chances of follow-on debates.

And finally, surprise!, Write It Down. Somewhere where everyone can see, read, comment, access. Be transparent, disseminate all your materials widely. Remember: there are going to be people who’re not even on the team yet, that will want to know “why did we do X” in the future. This document will explain things clearly, including the conditions, set of choices, and the decision making rationale.

Make The Decision#

This post isn’t about how to make the decision itself. There are scores of books written on this subject; my favorite is Chip and Dan Heath’s Decisive. This book lays out a useful 4 step framework to make sure you’re defending against your own biases, considering all the information, and giving your slow brain a chance to think. This post is a good summary, along with their WRAP cheat sheet.

Regardless of the framework or process you use, this is the time to make the decision. You have all the inputs, the assumptions, the options. Make a call.

Communicate, Communicate, Communicate#

Communicate the decision: quickly, clearly, and as widely as possible. Repete thyself: people are unlikely to understand everything on the first try. For strategies on effective communication, see “Communication is The Job,” by Boz, an excellent communicator and head of AR/VR at Facebook, where he lists out eight strategies he uses.

Did I mention that you should write it down?

From Durable Decisions to Disagree and Commit#

The start of this article defined durable decisions. The side effect of making durable decisions is that if the inputs don’t change, the decision shouldn’t change either.

Often, there will be a set of people that disagree with the decision. It is tempting to grumble about this, let it lie for a bit, and then re-open the debate and revisit the decision at a later point in time, with the hope of overturning it.

If the decider lets a decision by revisited, without substantial new input, the whole system falls apart.

Even worse, people stop believing that decisions are sticky and put less effort into the decision making process because they know those decisions aren’t final.

Having durable decisions means that they cannot get overturned if nothing has changed. This principle forms the basis of “Disagree and Commit”: you may disagree with a decision but have to be fully committed to making the product/team/company successful by going along with the implications of the decision.

The hardest decisions are typically the closest calls.

Knowing that you’ll revisit the decision if the circumstances change will give you the confidence to get behind it. We make the most progress when we’re running together at full speed, not pulling against each other in doubt.

{% include about.md %}

Situational Leadership

Rushabh Doshi — Thu, 14 Nov 2019 00:00:00 +0000

A Most Useful Management Framework#

Management books are full of different leadership and management frameworks and advice. A lot of these use contrived examples and war stories that are hard to replicate. Over the course of a decade of managing engineering teams, I have found Situational Leadership to be the one framework that applies to a broad range of scenarios, is practical enough to be easily understood and applicable, and tremendously valuable once you know it.

Life at FizzBuzz.co#

Amy and Bob are two engineers on the FizzBuzz team, building the next great feature your product needs. Amy thinks deeply about the underlying data structures and problems; Bob is a front-end ninja.

However, managing them is rather challenging. Amy often feels like things are a bit too easy; her output is variable; you think she might be unmotivated. There is no shortage of work, but she hesitates to pick up new work before asking you.

Bob, on the other hand, is really into FizzBuzz. He was so into the product that he applied to join your company out of school and couldn’t wait to get started. Bob works hard but often gets stuck on things. Even worse, the quality of his code can be variable - he makes bad architectural calls, copy-pastes when he shouldn’t, and doesn’t test enough.

What should you do?

Employee Behaviors#

Amy and Bob are two different engineers; they need different approaches to being managed. What are some specific things you can do to make them both individually successful?

The underlying core thesis behind Situational Leadership is the solution to this problem.

We can segment the engineers along two axes - ability/competence and willingness/confidence/security. Depending on where they are in their journey, you can use different techniques to help them. This theory is not new. Ken Blanchard and Spencer Johnson wrote about this in One Minute Manager, first published in 1982.

Note: My 2x2 is slightly different from what you’ll find by Google searching situational leadership. I’ll explain the traditional model below and why I prefer to start with the people view rather than the manager view.

In our case, Bob seems highly motivated and excited about his work. But he may not be able to do the work - he just joined FizzBuzz out of school, and this is his first job.

Amy, on the other hand, is highly skilled. She can knock stuff out at will, and produce high-quality products. She may lack the confidence or the motivation to get her work done and be the independent rockstar you know she can be.

In other words, the picture that emerges is something like this:

Our goal is to get both Amy and Bob to be high performing engineers - who are highly competent, motivated, and self-directed.

How do we get there?

Manager Behaviors#

How should our behavior change when working with Amy or Bob taking into account their different stages? We can think of leadership on two axes (surprise!): Supportive and Directive.

Highly directive leadership is mostly “Do this, in this way, by this time.” There is no autonomy and little opportunity to be creative. The opposite is complete autonomy: the freedom to pick problems, priorities, solutions, and timelines.

Highly supportive leadership is akin to being a great coach. You give feedback. You ask hard questions. They can come to you when they are stuck or need to problem solve. The opposite is being left entirely alone: engineers solve their problems, and the results are the feedback (did the product succeed?).

Bringing It All Together#

Now we a rubric for helping Amy and Bob.

In Amy’s case, we need to Sell (S3 in situational leadership parlance) - give her the confidence to take on more and more challenging tasks, encourage her to seek out problems, to go after things she thinks are essential without having to check back with us. We still need to give her feedback to both help her grow and increase her confidence and security. But in no case should we micromanage her or give her task-specific direction.

For Bob, we need to be more directive (S2 in situational leadership parlance). Some might interpret this as micro-management, and that’s okay. But Bob needs a tight feedback loop and smaller, lower complexity tasks. As Bob grows in his ability, he will graduate to more challenging projects while maintaining a high level of quality and delivery.

Where does it all go?#

The end goal is to get to Delegation holy land.

Doesn’t that put the manager out of a job? Yes! That’s the whole point - build enough leverage and capacity on your team that you don’t actually have to do anything - and then grow the team or seek other challenges.

In delegation mode, the manager is both low directive and low supportive. They are not giving the employees direction or feedback. Does that sound bad? Every CEO, every leader of a sizeable organizational unit, every person who is out on their own lives in this mode. If the employee is so good that they need no management, they are ready to take over your job!

Errata#

One Minute Manager might have been plagiarized from Arthur Carlisle’s work in MacGregor.

{% include about.md %}

Types Of 1-1s

Rushabh Doshi — Mon, 11 Nov 2019 00:00:00 +0000

This is a post about the different types of 1-1 I have encountered in my life. It is almost as if 1-1s were animals in a zoo. Seasoned managers will quickly identify a member of each species and know how to deal with them. Newer managers might mistake a tiger for a house cat, and lose a limb in the process. As a manager with plenty of scars (yours truly made nearly every wtf mistake in the post below), I hope to help newer managers not get mauled when they misidentify a 1-1 animal.

A quick aside: this post is not about the reason for 1-1s, or why you should have them regularly, or why you should have a 2-way agenda, or what you should discuss in them, how to give and receive feedback, etc. There are a plethora of articles on that, including a whole app to help you run them better. Instead, this post is about real-world situations managers encounter, for which an app can’t tell you what to do. If you’re interested in learning more about the mechanics of 1-1s, drop me a line, and I’ll write it out.

If you’ve been managing people for a while, these 1-1 types will appear familiar to you. I’d love to hear about ones that I missed to add to the zoo. Comments and feedback appreciated - @radoshi on Twitter.

🐶 No Agenda Chill#

Engineer: “Hey, boss, what’s up? I thought we could just hang, I got nothing on the agenda.”

I’m a type-A organizer. This message is my kryptonite. A meeting without an agenda? Say what? Why are we even meeting? Should we cancel?

The obvious answer is no. But it often takes me a few deep breaths to get there. Me: “Sure, let’s hang.”

25 minutes of conversation into the “no-agenda” 1-1: Me: “And how is X going?” Engineer: “Oh, it’s a total disaster, and the team is really clowny, and I’ve been super stressed about it and not sleeping, but didn’t know how to bring it up, because I’m so stressed, what should I do?!?”

Well, that escalated quickly. Let’s get into emotional support and problem-solving in the last 3 minutes.

As a manager, the no-agenda-chill 1-1s are the second-worst (Quitting is the worst). You never know what to expect - it might be a waste of time, you might have to prompt the engineer to write an agenda next time, or you come out with some scars because you were in the dark and didn’t ask enough probing questions. 🤷🏽‍♀️

Approaching the No Agenda Chill:#

Go really broad on topics, ranging from how they are doing, personally and professionally, and probe for areas of concern. When you find a problem, follow it up with the immediate help solving the problem. Think of ways to avoid surprise next time.

🧯Everything Is On Fire#

Engineer via email or slack, sometime before 1-1: Hey, I’m really concerned about Thing. Can we spend 5 mins in our 1-1 talking about Thing?

5 minutes. Yeah, right. This Thing is going to take up the rest of your 1-1. If you got anything else, get it done before you talk about the Thing.

These 1-1s are amongst my favorites. As a manager, I don’t have to dig around for where the fire is. Staring at a blazing inferno, being asked to point the firehose. My help! Yay, I feel useful.

The natural response, of course, is to jump into problem-solving mode immediately. I am an engineer (well, I was an engineer, but self-delusion is a powerful thing). This is what I do best. I solve problems. Let’s go.

The fly in the ointment is that, as a manager, you don’t really know what the problem is. One thing it most definitely isn’t: the thing your report tells you it is. Not because they aren’t smart; au contraire, they are better than you at their job. If they knew what the problem was, they would have likely solved it by now. By immediately jumping into solution mode, you’re probably just pointing out the obvious.

Approaching the Everything Is On Fire:#

Do a lot of active listening. (If you are not familiar with active listening, I strongly recommend watching a few YouTube videos on the topic. It will change your 1-1s forever.) Ask clarifying questions. Try and understand the problem in multiple different ways. Do not offer solutions in the 1-1. Think about it offline and write some suggestions within 24 hours. Often, the people talking through the problem, with the aid of your questions, will solve the problem for themselves. This is an excellent outcome. Use the credits to unlock the next level in the Manager RPG.

🦨 That Team (or Person) is Terrible#

This 1-1 is a variant of Everything Is On Fire, but we are dunking on a person or team instead.

Engineer: “This person is an idiot. They did everything wrong, and now I have this whole mess to clean up. Why can’t we just fire them? I don’t want to work with them again because they’re just so hard to deal with.” Or: “That team has no clue what they are doing, but they won’t let me get my work done. They are blocking my commits, won’t answer my questions. Moreover, their code or API or whatever is just a complete mess. Why are they even around?”

Oof.

This is a dangerous 1-1 and must be approached with care. On the one hand, you must walk the engineer off their rage-fueled, emotionally flooded edge. On the other hand, you must help them put things in context and approach problems constructively. Additionally, there might be an element of truth to the engineer’s rage, and you must investigate things further.

There are two large danger signs associated with this 1-1. Engineer says something overtly racist, sexist, against company values, or deeply offensive, in the heat of the moment. Depending on the severity of the infraction, this can result in a stern talking to and a reminder that certain types of speech are not acceptable, or in the extreme, a discussion with HR that can lead to censure or dismissal. It can also be really tempting to join in on the gossip and slam the other team or person. As a manager, you’ve just made your position infinitely worse by doing this - you’ve sent a message that this sort of behavior is okay, you’ve given the engineer the impression that you agree with it (and they’re definitely telling their friends), and put yourself in a vulnerable position.

Approaching That Team Is Terrible:#

De-escalate as quickly as possible. Help Engineer to calm down. Focus on facts. Re-focus the conversation on people having good intentions, but not being capable (everyone is learning/growing), or circumstances conspiring a certain way etc. Dig into the exact complaint. Investigate the engineer’s claims. Follow up with others as appropriate. Follow up with Engineer to close the loop. If appropriate, get HR involved, or escalate to higher leadership.

🐰 It’s Urgent#

Engineer: “Hey, can I chat with you urgently?”

Uh oh. This one is important but seldom good.

If we are lucky, this urgency is about a time-sensitive decision that needs to be made. However, most decision problems get sent over slack or email, and there is usually no need to be coy about the agenda.

This is the 1-1 where the dire stuff comes out. Someone got sexually harassed, someone found something deeply wrong that violates company guidelines or is straight out illegal, anything else that is just Serious.

Approaching It’s Urgent:#

Make time for this asap. If you need to get out of a meeting, do so. If the report is serious, escalate to HR or senior leadership as appropriate. If in doubt, check with HR, they are here for precisely this reason. Follow up with engineer (there are some circumstances where HR or legal would require you not to, but those aside, you should follow up to make sure the issue was addressed.)

🦘Feedback Hour#

There are two flavors of these - the conversation after the formal performance review, and the ad-hoc feedback conversation. The former is more mechanical: prep beforehand, send the recipient a writeup that they can read and digest beforehand, then spend time going over it in person.

The ad-hoc feedback conversation is more interesting.

“Do you have any feedback for me?”

🤔

Approaching Feedback Hour:#

Rookie version: “Sure. Here are 3 things you could be doing better.”

Battle scar version: “How do you think you’re doing with area X, Y, Z? What things are going well, and what aren’t?” You have a list of things they need to be working on (right? RIGHT?), and this is your opportunity to check their self-perception on their growth. “I agree with X and Y, think you could be doing more on Z, here are two specific ways I would have approached it if I were in your shoes.”. ✅

Remember: you don’t have to diagnose, create, and deliver feedback all on the spot. It is perfectly fine to follow up over email or slack or another 1-1 in 24h, as long as you do actually follow up.

🦡 Promote Me#

Engineer: “What do I need to do to get promoted?” or even better: “Reviews are coming up. Are you going to promote me?”

Side note: I love working with people that are bullish on themselves and straight up ask for a promotion. That being said, remember that there are a lot more people who are shy, who have cultural issues against asking for a promotion, who suffer from imposter syndrome, or a bad case of Dunning-Kruger. Care about these people, possibly more than than the ones that ask you to get promoted.

Back to regularly scheduled programming.

The decision tree here is straightforward. You believe your report is ready for a promotion. You are genuinely going to fight for your report in whatever promotion process your company has (unless you’re Google, in which case managers have little say 🤷🏽‍♂️.) You don’t think they are ready for a promotion and are not willing to put them up or fight for them.

In either case: Inviolable Rule: Be Honest. There is no better way to break long term trust than to make things up.

Approaching Promote Me:#

If they are ready for promotion, go into expectations management in case the promotion does not happen: “I think you’re close and are crushing it on many fronts. Here are two areas that are still questionable. I’m going to get feedback from other senior people at the company (or whatever the promotion process is) and make a good call.”.

If they aren’t ready, let them know: “I’m not sure you’re ready yet. We believe in trailing promotions, and you need to be performing at the next level for at least N months. You need to demonstrate more of X and Y for this to be a slam dunk promo. However, I’m going to bring up your case and get feedback to make sure I am calibrated.”

☠️ I Quit#

My worst 1-1. And they almost always start innocuously. Engineer: “How’s it going, how’s the family, isn’t the weather great?”. It’s like some human desire to push away talking about difficult things until the last possible moment, even in the very meeting that you’re supposed to talk about it. Once the engineer feels the manager is warm enough, they drop the bomb: “So, I was thinking about doing something else in the future.”

Me, blinking furiously, half my brain telling the other half to keep calm because they’re not quitting: “Okay, what were you thinking of?”

Engineer: “Well, my friend started this company, and I have an offer to go be their CTO. I’m thinking of taking it.”

Me: “Ffffffffff”

Approaching the I Quit:#

After a moment of panic, take 5 deep breaths, and let’s unpack this.

🤦🏽‍♂️Manager person WTF! How did you fail to detect that your star engineer is bored, tired, or seeking new challenges? In some cases, there is nothing you could have done - a chance meeting leads to something bigger, and suddenly, they have an offer. But in most cases, you could have seen it coming and knew in the back of your mind that you should do something about keeping them motivated.
💰What is the right thing for the engineer? Sometimes, the best opportunity does indeed fall into their lap, and the best thing you can do is encourage them to take it. People is a long game. If you’re consistently doing the right thing for your people, you’ll work with them again.
🙅🏽‍♂️Some times, the deal they are offered is genuinely not good. CTO might sound great, but the company is going nowhere and has been around for a year. Or Founding Engineer looks excellent; they are getting screwed on equity. In cases like this, I’ve helped my reports actually model these things out and understand how ownership, vesting, and dilution work. Incredibly smart engineers sometimes have a hard time building a spreadsheet, so I create one for them (which they then take and make 100x better and add many complicated factors).
❤️More than sensible advice and help (in the form of spreadsheets), they are also looking for emotional support in two dimensions: 1) knowing that you and other leaders at your company care about them and 2) they have a strong connection to their team and the mission here. Reinforcing these emotional points, along with the rational artifacts 👆🏽, leads to a 50% chance of saving the engineer from leaving your team.

{% include about.md %}

Hiring Engineering Leaders

Rushabh Doshi — Mon, 04 Nov 2019 00:00:00 +0000

This post is about the interview process for an engineering leader. Hiring involves a lot more than the interview part - everything from reviewing resumes, sourcing great candidates, getting them to come onsite, the interview itself, evaluation, offer, and close. In this post, I focus primarily on mechanics, propose a concrete interview process, along with sample questions for the non-technical parts, and propose a template that companies can use to build a custom process suited to their needs. This post is based on my long experience managing engineering teams at Google and Facebook and interviewing numerous amazing managers for various roles. This entire post is available as a coda doc that you can duplicate and use.

Your startup is going great. You have a cool product with hundreds of thousands of users, a strong engineering team, and amazing leaders of design, sales, and other functions. You want to scale your engineering team and are looking to hire your first engineering VP. You chatted over coffee with a few candidates and have convinced them to go through a formal loop. What are the types of interviews in your loop? What are the goals of each interview? Are there some questions that are better than others? I propose to answer these questions, focusing on the non-technical parts of the interview. First, we need to align on what the role is about, on diversity, and on having a great candidate experience.

The Basics: What does a VP of Engineering do?#

Martin Casado’s excellent post is the best summary of why you need a VP of Engineering and the value they should provide. A great engineering leader should be able to function across the following axes:

Business / Product impact#

Ensuring engineering execution is one of their most important jobs. “Execution” refers to repeated, on-time, high-quality delivery of things the engineering team is tasked to build. The leader should be able to balance offense (e.g., growing the product, building features, improving performance) with defense (e.g., reducing the bug load, rewriting systems to pay off bug debt, improving efficiency, and reducing cost).
They should provide critical input into the product planning and development process. VPEs are a thought partner for other executive leaders in the company - the CEO, CPO, Head of Sales. They should have an opinion on relative prioritization and technology investments the company makes.
Communication is a significant part of a VPE’s job. They must be excellent at communicating upward, downward, and sideways. They need to manage the CEO, so the engineering team doesn’t get thrashed, they need to communicate to the engineering team, to keep them in the loop and aware of everything going on. They need to be able to talk to clients, to the sales team, to the customer support team. They must be a highly versatile communicator, able to juggle a large variety of audiences and contexts.

Team impact.#

They are responsible for the continued growth and development of the engineering team. Senior engineers need hard challenges in the form of ambiguous but critical problems and leadership opportunities. Junior engineers need mentorship and challenging work appropriate to their level. The leader should be able to balance a team with the appropriate seniority mix so that everyone has opportunities to learn and grow. They are responsible for delivering feedback as well as for letting people go when things are not working out.
Hire, hire, hire. If the engineering team is supposed to double or triple over the next 12-24 months, they need to be spending a substantial chunk of their time sourcing, interviewing and closing great talent. They should be able to figure out the org structure 12 and 24 months into the future and then hire toward that. They should be able to build up their bench by grooming or hiring the next set of leaders that can take their place.
They must be able to lead the team when times are good and bad. Keeping morale up when things are rough, keeping the team focused on the highest priority deliverables, helping people feel good about doing some of the most challenging, hard, and impactful work they have done - this all falls upon the VPE.

An important note about Bias and Diversity#

If you haven’t read Rachel Thomas’s excellent 3 part series on the diversity problem in tech and practical ideas on what to do about it, you really should. Diversity is a big topic for our industry and your company. The rest of this article can wait, please read the series.

The design of the interview process plays a crucial role in increasing the diversity of your company. Essential things to keep in mind:

You are biased. I am biased. Everyone is. If you believe otherwise, take the Race IAT. Then take the Managing Bias class to get a little better.
Without a corrective process, we tend to hire people like ourselves, we hire for confidence, not competence, and we are likely to tilt the balance in favor of candidates with pedigree.
Sponsoring and attending Grace Hopper does not solve your diversity problem. Treating women and under-represented minorities at your company well, and fixing your hiring and promotion processes can build diverse teams.

I recommend a Diverse Slate Approach for leadership hires: implement the Rooney Rule for your company. In short, for any leadership position, you must consider at least one woman and one underrepresented minority candidate.

DSA is hard work. In a difficult hiring environment, this can be especially challenging. Adopting this is a first step, but an important one. It signals that you are willing to do what it takes to build a diverse workforce and are committed to increasing workplace diversity, even with some short term pain.

The Basics: The Candidate Experience#

Treat every candidate with respect and gratitude for their time. They could be a current or future client. Even if they are not the right fit, you want them to walk away with a great interview experience. I have personally been on the receiving end of poor interview experiences (hostile interviewers, no food or drink, no time to stretch or reset my brain between interviews), and it was not fun. Make sure candidates stay hydrated (they are going to be talking a ton), are not hungry, and are offered frequent breaks to stretch or use the facilities. One interviewer even offered a walking interview (it was awesome!).

Process Overview#

The process comprises of one or two screens to make sure that the candidates pass the sniff test and are sufficiently excited about your company to bring them in for a full loop, followed by a day or two of interviewing, following by sells. Reference checks are essential - I recommend asking candidates for references as soon as you have some reasonable signal and interest so that you can get moving on talking to their references as quickly as possible. I prefer asking candidates for their references and using those rather than doing back-channel references, a common valley practice that is rife with challenges.

Screen
Onsite 2. Technical 1. Code comprehension. Lightweight coding. 2. Software design. 3. Domain-specific deep dive. Optional. 4. Management 5. People Management. 6. Strategy, Vision, and Execution. 7. XFN Collaboration and Communication. 8. Values
References
Sell

Screen#

Key questions include digging into experience managing teams at various stages of growth, familiarity with your domain, with your company’s tech stack. Don’t forget to spend a significant amount of time selling your company and your vision and getting them excited about coming in. Keep an open mind at this stage and don’t weed out candidates too early. If a candidate has a greater than 25% chance of passing your onsite interview, bring them in for a full loop.

Technical Interviews#

The chances are that your company already has a process for technical interviews. I recommend tweaking the senior engineer interview process and using that for the VPE. If you need some help setting up a technical loop, please drop me a line (Twitter @radoshi.

For leadership candidates, I recommend focusing on:

Code comprehension over code writing. They need to read and understand the code that the team is writing. They are unlikely to be coding the Next Great Feature.
Good software design instincts over specific technologies. They should understand why you need a cache but needn’t go into the finer details about Memcached vs. Redis.
Domain-specific deep dive. Product, ML, Infrastructure, DevOps, AdTech – each leader grew up in some background. You’re looking for depth of understanding in their field.

Leadership Interviews#

I recommend 3 interviews to understand the candidate’s leadership style.

People management.
Strategy, Vision, and Execution.
XFN Collaboration and Communication.

I found open-ended questions, such as “what is your management philosophy,” to be less useful in distinguishing great answers from mediocre ones. An Situation-Complication-Resolution style, as applied to interviews, is an effective alternative. Setting up specific examples, or asking for specific examples, gives you much firmer ground to dive into details. Continuing with the preceding example, to find out where they are on the directive / supportive spectrum, we could ask the following: “We have an engineer Alex, he’s been here 6 months. We hired him straight from university.” (situation) “He works hard, but writes buggy code and people have to clean up his mess frequently.” (complication). “What would you do?” (resolution). This setup gives you ample fodder to deep dive into how the candidate approaches these sorts of problems and where they lean philosophically.

People Management Interview#

The goal of this interview is to determine if the candidate is an exceptional people manager. Can the candidate provide the team with career guidance, help them work through obstacles, and push them forward? Does the candidate have sufficient experience dealing with problems, or are they going to be learning it on the fly? Is the candidate going to be able to hire and scale the team? Are they going to be able to scale themselves as the company grows?

Sample questions#

Basic data#

Management history.
Historical and current team sizes.
Growth curves - did they build the teams or inherit them?

Management mechanics#

How do they keep in touch with their team? 1-1 - what is the purpose, structure, and cadence?

Engineer growth#

How do they assess an engineer’s strengths and areas of growth?
What do they focus on, and what is the process behind making that decision?
How do they grow junior engineers? Senior engineers? TLs?
How do you give feedback? Give an example where you had to deliver hard feedback to someone?
Give an example of someone that wasn’t doing well and what you did to help them improve.
Have you ever had to let anyone go? What was the thinking and process behind it?

Manager growth#

When does a team need a manager?
When is someone ready to be a manager? How do they assess this readiness?
How is managing a manager different from managing ICs?
How is managing Directors / VPs different from managing managers?
How do you know when a manager isn’t doing well? How do you debug? How do you help?

Organizational leadership#

When do they go from an amalgam of engineers to a formal team?
What is their ideal team size?
How do they think of organizational structure? Is it necessary? At what stage? When does one re-organize the team?
What are the mechanics of a reorg? Have they executed a non-trivial one?

Team culture#

How do they build engineering culture?
What is their weekly eng cadence?
Do they do architecture reviews (why or why not)? Do they do regular engineering all hands (why or why not?)
How do people in the team learn from one another and grow?

Strategy, Vision and Execution Interview#

The goal of this interview is to determine if a candidate can set the engineering vision, determine a strategy, and execute on it. They need to understand company strategy and translate that into an engineering strategy. They need to be able to decompose complex, ambiguous directions into concrete projects. Every candidate ha a set of tools and processes they use to keep the trains running, to stay on top of things, and communicate upward, downward, and laterally. At a leadership level, communication is often the primary job, and you want the candidate to be exceptional in this regard. All good projects hit bumps at some point. What happens when things go wrong? How do they detect what’s wrong, debug it, and fix it? The ideal candidate is one that makes bold bets, has seen some of them fail, learned, and did something better next time.

Sample Questions#

History#

Describe a complex project that you ran, how much time did it take, how many people, was it successful, what went well, and what did you learn? (This is intended to be a warm-up, don’t spend the entire interview diving into this).

Inception#

Construct an ambiguous scenario where a new project is needed. Pretend you’re the CEO and let the candidate ask you questions. Can the candidate quickly get into the heart of the problem you’re asking them to solve (comprehension)?
Can they come up with a sufficient first approximation of a solution? Can they decompose the problem into different tracks?
How would they set up the engineering team to tackle such a project? How do you manage ongoing maintenance and bugs while tackling projects?
How do you deal with morale impact of thrashing the team?

Goals#

Goals discussion is significant enough to be pulled out into its section.
Does the candidate ask about a project’s goals?
Can they create reasonable metrics that track success? The discussion can get pretty nuanced but is incredibly important. Without clear goals and metrics, execution becomes an order of magnitude harder.
Pick a project they did that was successful. Dive into the project’s goals, measurements of success, relationships to company success, decomposition into smaller projects, and execution of the entire thing.

Project lifetime#

How do they track progress?
How do they communicate progress?
How do they know if things are going well or not going so well?
If they aren’t going well, what sort of corrective actions would they take?
What would they do if they incorrectly estimated time, complexity, or resources?

Conclusion#

How do they transition a team from one project to another?
What do they do if an idea fails?
What do they do if the project didn’t complete on time and requires further investment?

XFN Collaboration and Communication Interview#

Thanks to Connor Hayes for providing this section.

In this interview, we are trying to assess whether the candidate can communicate in a clear, concise, and structured way. Do they have the right attitude toward working with non-engineering functions? How do they maximize their work through other people? We’re also looking for grit and scrappiness - can they push through tough situations? Can they unblock themselves and get stuff done? Finally, we want someone that is going to be a thought partner for the company’s senior leadership: do they grasp abstract concepts quickly? Will they contribute to strategy and decision making?

Sample Questions#

Communication#

This is observed through the structure of answers to the other topical questions.

Collaboration#

What is the ideal working model for collaboration amongst various functions like Product, Data Science, and Design?
Describe a lousy collaboration situation. How did you handle that? What did you do?
Let’s say you were working with a PM who made a decision with which you didn’t agree. What would you do?
What if the situation didn’t change, and you still disagreed? How would you escalate?
What do you do when a product person gives you an unrealistic deadline for a project?

Grit and scrappiness#

How did you get to where you are today? Tell me about the times in your life where you had to work the hardest and overcome obstacles.
What is the most unfair thing that has happened to you in your career? What did you do about it?
Tell me about a time when you were blocked on a product decision and weren’t getting help from Product. How did you push through?
What’s the most challenging project you’ve worked on, and how did it go?

Thought partnership#

Describe a strategic problem that the company is facing. Ask for their input and see how they reason through things.
Let’s pretend we are 3 years into the future, and our company has gone out of business. What do you think are some possible causes?
How could we account for these risks with what we build today?

Values#

Your company likely has a values interview, done by senior leaders, to assess whether the candidate has a similar set of values as the company. This interview typically probes into a candidates work ethic, their motivations, their excitement about the company, about the problem it is solving, and whether they would help take the company to greater heights.

References#

Reference checking is an incredibly important part of the interview process. I recommend letting recruiting solicit references from the candidate, but having the hiring manager (CEO) do the reference calls.

The critical thing during reference calls is to elicit details about the candidates working style, strengths, and areas of development.

Conclusion#

The goal of this post is to provide a VPE interview process that you can duplicate, customize, and implement to hire the best engineering leader for your company or organization. This post is also available as a structured coda doc.

I would love to hear from you if you use this process, or if you have suggestions on making it better. I’d love to hear about things that I have missed, but have proven incredibly useful for you. Leave a comment or DM @radoshi on Twitter.

{% include about.md %}

Software Quality, Bugs, and SLAs

Rushabh Doshi — Fri, 18 Oct 2019 00:00:00 +0000

This post is also available as a live Coda doc. You can copy it, connect it to your JIRA, and customize it to suit your needs.

Your app feels buggy. There are lots of small quality issues, and it crashes now and then. The engineering team buried in bugs: they don’t know which ones to address first. What do you do? How do you get the product feeling great, the engineering team feeling productive and proud of delivering a high-quality product, pumping out features while keeping the bugs down?

Image from cantunsee – think you’ve got a good design eye? Take the quiz.

Software development is an inexact science. We write code and build software products full of defects. If we are lucky, we are aware of the defects and can catch them before they go to our customers. Engineering teams often feel they are in an epic struggle to build the perfect product while dealing with ever-changing requirements and product specs, framework changes, and evolving infrastructure.

An engineering team’s objective is to make forward progress on building their product while keeping their product quality “good enough.” The definition of “good enough” is defined by company values and the context surrounding the product. A hardware device with a slow update cycle has a significantly higher bar than an iPhone app that can be updated quickly. Different companies have different quality bars as well - Instagram pays a significant amount of attention to small visual design problems, resulting in a product that feels extremely polished and well crafted. This variability in the definition of a quality bar, from company to company or product to product can be overwhelming. People give up, resulting in the complete lack of a defined bar, or a quality bar where every bug is a blocker.

A typical scenario goes something like this:

Everyone’s grumbling about software quality.
Designers don’t feel like the product is polished enough - after all, their work is front and center. “Low quality” often becomes “poorly designed” through no fault of theirs.
Engineering’s unhappy - they have dealt with a constant slew of ever-changing specs, people changing their minds about interactions and details at the last minute, and a constant pressure to ship features instead of taking the time to do things right.
Product Management thinks the quality is terrible, and to make matters worse, everyone is moving too slowly.

Typical responses to this include bug bashes, quality weeks, bug triage meetings every day, burndown lists of launch-blocking bugs. Carried out over sufficient lengths of time, these lead to morale issues, finger-pointing, burnout, attrition, and, importantly, no sustained quality improvement. The herculean effort might result in the feature getting out, but you know in 4 weeks, we’re going to be here again.

Below, I lay out a pragmatic framework for dealing with software quality issues. It is not detailed enough where you can copy-paste it and put it into action. However, we can decompose a gnarly problem into different measurable axes, and forge a path toward maintaining a high quality standard and paying down long term bug debt.

1. Defining Quality#

There is much academic literature on this subject, generalized to everything that software engineering covers. For engineering teams building modern apps and consumer-facing software, I tend to use the following axes:

Correctness. Does the software behave as it was intended to? Does it solve the problem (at a detailed level) that it was meant to solve? Eg. A user tries to upload a video to YouTube using the app. They press the upload button, and nothing happens.
Performance. Is the app fast enough? Typical performance problems include app start performance (cold and warm start), data loading performance for key interactions (loading the home page, loading the profile page), interaction performance (scroll speed, animation speed). Eg. Google has famously invested much energy in speeding up Search because latency matters - a decrease of 400ms caused a 0.2-0.6% drop in searches, which compounded over time (users in the slow variants used Google less immediately, and even less over time).
Reliability. Does the app reliably accomplish its purpose? App crashes are an egregious example, especially on diverse platforms like Android. Another example: upload failures when trying to post photos or videos. Reliability issues are important because they cause a drag on retention as users churn off the app in frustration.
Craft. Do all the tiny details add up to create a genuinely polished experience? These bugs are rarely the most important one to fix or “launch blockers,” but they are essential to get right if you want the product as a whole to feel perfect. Want to find out how good your design craft eyes are? Take this quiz.

At this point, you’re thinking: hang on. What about: engineering architecture, testability, efficiency, and reusability? These aspects are critical to software development and can slow down engineering development, or contribute to fragility and a game of bug whack-a-mole. Isn’t it important to document those and burn those down as well?

These are essential engineering projects that a good team is working on at any given time. However, they do not directly manifest themselves as user-facing problems. Separating the two, treating “quality” as strictly user or customer-facing problems, helps engineering teams speak the same language as their product and design counterparts and bring pure engineering problems in-house. Longer-term infrastructure improvements can be decomposed and folded into the same priority system; it is merely out of scope for this conversation.

2. Prioritization#

Most engineering teams have a prioritization framework that provides an estimate on how quickly they a bug is addressed. A defect might take a long time to get fixed, even though it got looked at immediately, usually because it is a gnarly, hard to reproduce the problem or because the engineering team is so overloaded digging themselves out of a debt hole, that things are just going to take time.

A priority framework provides common terminology and an agreement about how urgently the team needs to look at a bug. In other words, it is an SLA that the engineering team agrees to uphold. Here is my general priority framework:

Priority 0. Look at this NOW. This priority should typically be tied together with an on-call rotation so that someone gets alerted. Services going down, significant parts of functionality going unavailable, major security problems - these sorts of things should result in a P0 bug. P0 bugs should be rare enough to warrant a formal review process around what caused the problem, its discovery, fixes applied, and lessons learned.
Priority 1. Fix within 24 hours. Examples include significant flaws in the product, bugs that affect a subset of the user base, bugs that impact the brand in a significant way.
Priority 2. Fix within a week. Minor problem, but it would be good to get fixed before then next major release of the feature goes out.
Priority 3. Fix within the next sprint (typically 2 weeks, but as long as a month.)
Priority 4. Catchall priority for things that should go into the next sprint. Expect these bugs never to be looked at unless they get upgraded to P2s. Different teams prefer to have these lying around because they are a documentation of everything wrong or delete them because they are mostly useless.

A meaningful and functional SLA process requires commitment and discipline from the engineering team to uphold their side of the bargain and look at (and fix) bugs in the allotted time. It also requires the product, design, QA, and other teams to not abuse the system by needlessly escalating every bug or arguing with the eng team about the priority. When this trust breaks down, the system goes from being a helpful framework to something that makes people feel trapped.

3. Continuous Improvement#

There is a constant tension between pushing features and improving quality. Determining when to pull the “emergency stop” brakes and get quality under control is hard to do. In the ideal world, all bugs would be perfectly prioritized, and there would be no SLA violating bugs. Good teams can reach and maintain this state. However, when a team is just starting to institute this process, the bug load can seem daunting. One approach to pay down the debt is to institute a process that limits the number of bugs that are allowed to be out of SLA to allow the team to continue to do feature development. Over time, this overage number ratchets down, going to zero over the space of a few months (or sooner depending on the team and how much bug debt they have).

The key metric to look at is “Number of SLA violating bugs in a priority bucket.”

Set a realistic target of how many SLA violations you’re willing to allow. As an example:

Priority 0: No outstanding bugs. If you have a P0 bug, stop all feature development until you have fixed that bug.
Priority 1: A small number, say 0.25 / engineer.
Priority 2: You should expect a fair number of these bugs and give teams room to get these under control - say 1 / engineer.
Priority 3: A larger number. 5 / engineer.
Priority 4: Unbounded.

At this point, the team can take an actual Goal to stay under the max-violations limit and continue to ratchet things down over time.

4. Accountability#

Processes are tools and like any tools, are subject to abuse. They are not meant to prevent adversarial plays - if the process becomes too cumbersome, teams stop adopting it, and it fails to achieve its purpose. A fundamental assumption of a Bug SLA process is that there is broad adoption and accountability at the highest level.

At Facebook, when we instituted such a process, there was a bi-weekly meeting where all the senior eng managers would sit in a room with the head of the Facebook app, with the person running the SLA process (TPM) presenting who was doing well and who wasn’t. There was no yelling or blame; it was a rather simple “What’s going on, and how can we help?”. But you did not want to be the person answering that. The accountability was a social construct, and it worked exceedingly well.

Making sure that the leadership is bought in, at the highest level, as well as peer functions (product and design), is essential to making the process a success.

Conclusion#

A systematic approach to product quality can help produce a product that is a joy to use, a product that feels polished and fast, one that customers keep returning to over and over again. Regardless of how bad things are today, you can institute a framework to pay off your bug debt, while building and pushing features. An SLA system has built-in checks and balances to slow down feature development when things get serious and provides natural incentives for teams to pay attention to the quality of their software, leading to more productive and happier engineering teams in the long run.

{% include about.md %}

Strategy and Tactics

Rushabh Doshi — Sat, 12 Oct 2019 00:00:00 +0000

A star product manager on your team comes to you one day, upset about the lack of coherent strategy. You’re confused: you have goals, a beautiful mission statement, an objective and measurable key results (or KPIs, depending on your parlance). The team has a good roadmap with specific things they are building. What is possibly wrong?

“We’re doing a bunch of tactical stuff but we don’t have a strategy.”

If you hear this, again and again, it is likely because the distinction between strategy and tactics is unclear. Sometimes one legitimately does not have a strategy — you’re in the problem definition phase, or your strategy is to out-execute the competition. More often, you haven’t outlined the strategy or communicated it to your team in a way that feels coherent.

A strong, written exposition of your strategy, including rejected options, accelerates the product development process. Teams that understand the central strategy are empowered to come up with their tactics and execute without centralized decision making. They don’t have to come to you for every last detail. This article captures my take on strategy and tactics as applied to building software products, based on nearly a decade of trial and error during my career at YouTube and Facebook.

What is Strategy?#

“Strategy is the creation of a unique and valuable position involving a different set of activities. It requires you to make trade-offs — choosing what not to do. Strategy involves creating a “fit” among the companies activities.”

— Michael Porter, What Is Strategy?, On Strategy

Strategy is easier to understand in the context of military theory or multi-player games, with clear goals and win/loss scenarios. Chess has been played and analyzed for centuries and is an excellent lens through which we can try to understand strategy. A player’s goal in chess is simple: checkmate your opponent. However, when the game starts, the board is wide open.

Here are three (amongst many) different things a white player could choose to do:

Focus on attacking quickly and going for the quick win. White aggressively develops his queen and attacks rapidly. Black must pay attention to traps while focusing on developing their pieces.

Control the center and developing your pieces. White opens into the center and then fights to maintain that control. This is a popular opening for a good reason: it is less risky than extremely rapid development and leaves a lot of options for future attacks.

Defend first, giving up the center. White has castled and has nicely defended the king but, in doing so, has lost control of the center. The result is a more aggressive black game while white plays passively and waits for a black mistake.

Strategy characteristics#

No particular strategy leads to an immediate, measurable advantage. Neither player captures or loses pieces immediately by following any of these strategies. However, they lead to wildly different games.
Choosing one strategy means not choosing the rest. One cannot defend the king and simultaneously control the center, using the same set of moves. One cannot be aggressive and attack with the queen while also developing the rest of the pieces. The strategic pick must fit with the player. Aggressive players picking a defensive strategy will result in neither sharp offense nor solid defense. A player must know who they are (or trying to become) and pick a plan that works well with them.

What are Tactics?#

Tactics are a set of actions that result in concrete, measurable gain.

Continuing with our chess analogy, let’s look at the following sequence of events:

In two moves (set of actions), by sacrificing the queen, white has checkmated the black king and won the game (concrete, measurable gain).

Tactics can involve any number of moves — but the result should be clear and unambiguous. In the example above, the gain is winning the game — the biggest prize of all. Not all tactical play will result in such a significant benefit; often, the increase is a single piece like a pawn or positional advantage of occupying a square in the center. The critical difference is that a tactical move quickly results in a definite advantage, while a strategic position is an opinionated way of making progress toward the ultimate goal.

From Chess to Products#

Chess has fairly clear rules and an unambiguous goal. Rules and goals may be a lot fuzzier when building products, creating ambiguity. How do we apply the learnings above to product development?

Start with a clear goal. Without this, the discussion about strategy and tactics falls apart. Regardless of the phase of product development, the goal should be clear, even though it may not be easily measurable. For example, in the very early stages, the goal is to determine 1) does the customer have the problem we think they have? 2) does our product or feature solve the customer’s need?. Post product/market fit, the goals are often to increase revenue, users or engagement, and thus grow the business.

Develop strategic positions toward the goal. A “strategic position” is a choice to focus on some part of the overall problem. The approach is to segment the problem space and then choose a combination of those segments to over-serve while ignoring or under-serving the rest. Porter lists three approaches to segmentation,

Segment an industry’s products and focus on building a particular subset.
Segment based on customers’ needs.
Segment based on customer access and how products reach them.

It is important to note that these are not MECE (mutually exclusive and completely exhaustive). Combinations of these sources will lead to different strategic positions that one could take.

Smaller, Better, Faster → Tactics. Improving the performance, reliability, and efficiency of software is an excellent thing, but it is not a strategy. These are operational indicators and will generate a definite tactical advantage, but they do not signify a unique position that your product could occupy.

Winning with Strategy and Tactics#

Achieving your goals involves excellent strategy and artful execution. One without the other is likely to result in unnecessary friction, if not outright failure. One must have a solid strategy, choose to communicate it often and follow it up with solid execution. Execution includes things like operational effectiveness — building products that are fast, reliable, and efficient. Another critical element of execution is excellent design — the product needs to be coherent and understandable, and communicate its purpose. Details matter and small issues and bugs diminish the impression of a well-built product. However, none of these things — A+ engineering, beautiful design, perfect understanding of data — constitute a strategy. One can win based on tactics alone, through fast, solid execution. But this advantage is not as sustainable as one achieved through a combination of strategy and tactics.

{% include about.md %}

Rushabh Doshi

Ship week at Hiro: MCP, Portfolio, and Import 2.0

MCP Server#

Customized cashflow dashboard#

Position solar system#

Iran impact#

What will you build?#

Portfolio tab#

Import flow part 2#

Bonus#

Cognitive Debt

Your brain on ChatGPT (arXiv)#

Computer systems vs. computer programs#

The way out#

An unsatisfying conclusion#

Stop counting tokens

Multi-Clauding Like a Boss

Setup and Install#

Claude subscription#

Claude code#

Terminal#

Code editor#

Commands#

Repository initialization#

Monorep setup#

Commands, plugins, subagents#

Code checkouts#

Workflow#

Plan, Write, Review#

Sidebar: Permission management#

Sidebar: Shared .claude directories#

Sidebar: Subagents#

Sidebar: Commands#

Think, Test, Write, Review#

Sidebar: MCP#

Multi-Claude like a boss#

Important note#

References#

My tmux.conf file#

Our commit-and-push command#

Coding for Managers

Setup#

Languages and frameworks#

Taste still matters#

Be public and consistent#

A note on hiring#

On Learning Faster

1. ChatGPT with branching chats#

2. Use NotebookLM#

3. ChatGPT voice mode#

Learning is hard#

About Hiro#

Your stack#

Rushabh Doshi

Vibe Coding Safely

A Return to Writing

LLM Leverage: Where to Build and Where to Bet

Introduction#

TL;DR#

Top-5 industries by Delta#

Top-5 industries by Output-Delta#

Methods#

Data#

Example#

Analysis#

Top Supergroups#

Top Groups#

Deep Dives#

Finance and insurance#

Professional, scientific, and technical services#

Information#

Healthcare#

Conclusion#

What about Foundational Models or Orchestration?#

Reference#

Appendix#

Software#

Detailed classification prompt#

Two books

Ministry for the Future#

Sidebar: Shared `.claude` directories#

`llm-code`#

What is `llm-code`?#

Live Demo #

3. Ban `@here` and `@channel`#

1. Use `/mute` liberally#

2. Use `/remind` or `Save` to keep your GTD flow#

4. Use `Notify me` for threaded conversations#

5. Quickly read through `All Unreads` and `Threads`#