Grading on a Scale – I Mean, a Curve
Miguel de Icaza tweeted this today:
Performance Review Season at Novell is easy when your entire team is made of 100% super-awesome hackers.
It's a very nice thing for Miguel to say about his team, something I have no reason to doubt. I don't know Miguel too well, but I have been in a handful of meetings with him back in my Novell days. He doesn't strike me as the type of person to be disingenuous, and regardless of how you feel about .NET on non-Windows platforms, you have to be impressed with the impact and effectiveness of Mono, so I figure his team probably is pretty awesome.
I guess when you are among the privileged at Novell, you can do things your way. I know when I was at Novell, having a team full of awesome people didn't make performance reviews any more straightforward. Novell is by no means unique in this; I can personally attest to this fact.
At most big companies I've been at, how performance is measured depends on what time of year it is. If you are measuring performance during any other time of the year, you do this by comparing a person's efforts against a published definition of their job description and level. For example, on your company's intranet somewhere there is a document that describes the expected level of performance for a mid-level software engineer. If you are a mid-level software engineer, you can compare your performance against the published standard and determine if your performance is currently on track or ahead of or lagging expectations.
This is true at any time of year except during performance review time. During performance review time, suddenly your performance is measured by how you compare to your peers. So now, if you are a mid-level software engineer, your performance is based on how your performance and contributions compare to the other mid-level software engineers in your organization.
For some reason this seems to be a pretty common way to measure employee performance, and yet it makes absolutely no sense to me. In casual conversations with peers (and, frankly, also with a number of managers who will remain unnamed here), I think I can confidently say that it makes no sense to most other people either.
Imagine you are attending college and you sign up to take a calculus class. On the first day of class, the instructor hands out a syllabus which clearly explains how you will be graded, which boils down to this: If you get 95% of all the questions right on homework and tests, you will get an A, and the scale breaks down from there. As the semester proceeds, you monitor your performance on homework and tests, studying to meet that magical 95% mark. On the last day of class, you enter confidently knowing you have a 96%-97% average which gives you a little bit of a buffer on the final. You leave the final feeling pleased with yourself, sure that you got at least 19 of the 20 questions right.
Then the grades come out, and you have a B-.
You go and ask the professor, and he says, "Well, I changed the grading from a scale to a curve. See, everyone was getting As, and we can't have that. So I graded on a curve instead. You got a B-."
This is effectively how most employee performance measurement techniques work. Your performance is graded on an objective, defined scale all year long. Then comes performance review time, and suddenly you are now being graded on a curve, not a scale. Before you knew where you stood; now, you have no idea.
A normal distribution curve makes sense when applied to a statistically average sample. The question is, does your employee population comprise a statistically average sample? In order to do that, you'd pretty much have to be willing to hire anyone who walks in the door asking for a job. You probably don't do that; you are probably very selective in who you hire and keep. You probably try to hire only the best talent you can find. You probably take the approach of preferring to sometimes pass on talented people that you aren't sure of, rather than give the benefit of doubt, and so you've probably rejected some talented candidates.
That means your employee population is not statistically average. So why is it that during performance review time you try to make this anomalous sample conform to a normal distribution curve?
I've worked for companies where they say that, organization-wide, some percentage of the population of any given job function MUST be underperforming, some percentage MUST be overperforming, and the rest MUST be performing at around average, within about a standard deviation or so. This isn't just a belief. It is enforced through policy. Essentially the policy says it is not possible to have an organization with 100 employees that are all high performers. Instead, you might mandate through policy that 10 of them must be underperforming, 10 must be overperforming, and the remaining 80 are average or near-average. Yet when you hire, you insist that you hire only the best.
This doesn't make any sense, and in my opinion it is a dangerous practice.
First off, you aren't fooling anyone, especially not the intelligent, highly-educated type of thought worker that tends to be evaluated this way. You wanted smart people, you hired smart people, and now you have to pay the consequences: You can't implement obviously flawed policy like this. They will pick up on it and it will cause them to lose faith in your organization.
Second, it encourages exactly the wrong type of behavior. Once they realize that their actual performance measurement has less to do with their evaluation against an objective metric than it does against their peers, employees will be incentivized to game the system. This may manifest itself in many ways: refusing to help teammates or to work well as a team; manipulating the assignments given to them and others to maximize their relative performance; taking opportunities to point out to you (the manager) how excellent they are in comparison to others or how rotten the others are in comparison to themselves; attempting to make others look poorly or to make themselves look good in public, e.g. meetings; trying to hire people who they know to be less competent than they are; avoiding promotions (and thus responsibility) for the sake of cherrypicking in a job function that they can excel in; etc. None of these helps your organization to become better.
Third, it encourages employees to leave. Good employees. Employees that passed your interview process. Employees that are exceeding the objective metrics, but don't rank at the right of the curve. They know they can choose a different company with lower hiring standards and at least perform at the top. If you are really hiring only the best, then this person is among the best, and they will be hard to replace.
Fourth, this system encourages sucking up. By definition, it is an entirely subjective measurement. As a manager, you are likely to be persuaded by those who have perfected the art of sucking up and give them better evaluations. You might do this without even realizing it.
So, what should you do about it?
Well, if you are empowered to do something about it, I think you should avoid doing it altogether. Evaluate your employees based on an objective scale. Period. Hire great people, expect greatness from them, and then stop doing dumb things that encourage them to not be great. Just get out of the way and revel in the greatness.
The main argument I've heard in favor of the curve is budgetary. "We budget 10% of employee salary for annual bonus. High performers get 18%-20%, low performers get 0%-2%, and the rest get from 5%-15%. The average is 10% so we meet budget." The corollary argument is that the bonuses should not be spread evenly across the team. More money for the high performers.
That's fine, but didn't you try to hire ALL high performers? If so, wouldn't you say you expect ALL of them to be in the 18%-20% range? Why not instead budget your bonus pool for the high performance amount across the board? You should still evaluate each employee's performance, but now you aren't being constrained by budget, so you can do it objectively and fairly against a published metric, the one you've been using in one-on-one's all year long. If an employee underperforms, you can still give a 2% bonus or none at all. Take the money you saved and use it next year or something.
If you aren't really empowered to do anything about it, my suggestion is to press onward anyway. Be excellent anyway. Don't let the system cause you to be non-excellent. Be a good team player. Try to surround yourself with the best people. Take the assignments you are given and do your best with them. Discuss with your manager any concerns you have about whether you are getting the opportunity to show what you've got, but move forward and do the job right even if the system doesn't reward you. I believe good things come about from good works, sooner or later. If that doesn't work, I guess you can always go work for someone else who's a bit more reasonable.
You can also blog about it.
How To Ruin Your Own Vacation
You know the feeling. The feeling you get as you are heading home from work on the last day before your vacation. That feeling of the weight and stress and pressure of delivering excellent software on time just floating away off your shoulders. Nothing to look forward to for the next two weeks but fun and relaxation.
Right?
Or are you worried you are going to get a call?
It's the nature of the beast. We plan to escape, but sometimes that call just can't be avoided.
Sometimes it can, though. Sometimes you get called because you did something boneheaded, something you could have avoided that would have made your vacation more like, uh, a real vacation.
Some time ago I was working through a bug that was assigned to me at work. After a bit of digging, I realized the bug was being caused because two methods in a related class had recently been deleted. Putting those methods back in the class would fix my bug, but was that the right thing to do?
I assumed not. I work with smart people, so I assumed that there was a good reason for removing those methods. So I dug into the source code control system to determine which checkin had removed those methods, who had performed the checkin, and why.
I found the commit in question, noticed who had done it, and looked into the commit notes. Here's essentially what I saw:
Description of problem: Code was broken. Description of fix: Solved problem.
Meat Loaf says "two out of three ain't bad," but that doesn't apply in this case. Note that our checkin system automatically adds the "Description of problem" and "Description of fix" tags so the commit notes weren't nearly so verbose as it appears.
Obviously this was of no help to me in determining why the change was made, and without a reference to a bug database record I couldn't look up the bug to find out about it either. So even though the developer in question was on vacation that day, I had to call him: I couldn't wait for him to get back for the answer.
So, there you have it: One way to ruin your own vacation is to avoid taking any time at all to provide useful commit notes.
Notice that even typing the notes that were put into this commit was an utter waste of time. I don't know about you, but I don't make a habit of checking in fixes to code that isn't broken. So the statement "Code was broken" is a statement of the obvious. Of course the code is broken; why else would you be checking in a fix? Saying the code was broken is redundant and unnecessary. Likewise for "Solved problem"; why would you be checking in if you didn't solve the problem?
The single most effective change that could be made here would be to simply reference the bug database record. Even if the problem description were simply changed to say "Bug #3165150" that would be enough. Annoying, but enough. With that I could at least look up the bug in the bug database and maybe figure out why the change was required. But seriously, how long does it really take to add a few notes about what changed and, more importantly, why?
I've also seen commit notes like this:
- removed GetLastWidgetStatus() method
- modified signature for UpdateWidget()
- added null check to SyncWidget()
Gosh, really? You needed to tell me that? I thought that's what diff tools were for.
Here's my suggestion for creating Awesome and Useful commit notes:
- Include a description of the problem. Use the title from the bug record as a guideline. You might want to be a little more verbose, but often it isn't necessary, as long as you...
- Include a reference to the bug record. A clickable link is best, if possible; if not, at least a record ID so the bug can easily be found.
- At this point some of you may be saying, "What if there is no associated bug record?" In that case, the answer is, "WHY ON EARTH ARE YOU DOING WORK WITHOUT AN ASSOCIATED BUG RECORD?!?"
- I mean, seriously. If for no other reason, you need a record so your boss knows what you are doing!
- Horrible Example: "Code was broken."
- Bad Example: "WidgetManager class not working."
- Good Example" "WidgetManager not correctly updating widgets on DisplayChanged event; see bug #5150316."
- At this point some of you may be saying, "What if there is no associated bug record?" In that case, the answer is, "WHY ON EARTH ARE YOU DOING WORK WITHOUT AN ASSOCIATED BUG RECORD?!?"
- Include a description of why you made the changes you made, not just THAT you made changes or a simple enumeration of the changes.
- Horrible Example: "Fixed code."
- Bad Example: "removed GetLastWidgetStatus(), modified signature for UpdateWidget(), added null check to SyncWidget()"
- Good Example: "The UpdateWidget() method had no knowledge of the event causing the call so it wasn't handling the DisplayChanged event properly; I added a parameter to UpdateWidget() to provide this information which allows us to handle the event correctly. Also added a null check to SyncWidget() which would have caught the bug. Removed GetLastWidgetStatus(); nobody is calling it anymore."
See? That's not so bad. Just a few minutes out of your life to show your love and caring to your team members when you are gone. Just a few minutes out of your life so you can really enjoy that vacation without worrying whether the phone is going to ring.
Incentive Programs and Professional Motivation
I recall some years ago having read this Joel Spolsky post entitled "Incentive Pay Considered Harmful." In typical Spolsky fashion, he made a very compelling case that I had a very hard time coming to grips with. At the time I'd just recently made a series of contributions to Novell, outside of the scope of my job description, that I felt had an obvious positive impact on the company, but I had not received any sort of recognition at all for this.
Joel's article strongly disagreed with me. I felt I should have been recognized for my efforts; why else would I take the personal risk that comes with extending one's self in that fashion? Instead, nothing had been done.
I've been pretty interested in this topic and so I've been observing and evaluating ever since. What I've seen is that, for the most part, incentive programs tend to have the opposite effect of their purpose; namely, they tend to demotivate rather than motivate.
Outlined below are some reasons why.
People Get Rewarded Just For Doing Their Job
Maybe you've seen this too: You attend a quarterly all-hands meeting when an individual you know and work with is asked to come to the front. They announce, "Tom did a really great job running our beta program last release, and so we are giving him the quarterly Mega-Smiley-Face award, with a free dinner for two and a new iPod Touch." You sit there and stare, clapping insincerely. Sure, Tom did a great job running the beta program. But Tom is the beta program coordinator! It is his job to run the beta program! You did great work with your assigned responsibilities last quarter also. Why aren't you getting an award?
One problem with this scenario is that the rewards program now seems arbitrary. Tom was rewarded this quarter for doing his job. Who will be next? Who knows? Everyone is doing their job, but not everyone gets rewarded. The rest of the people feel discouraged. The recognition seems to be something worth earning, but they don't know how to earn it.
Another problem with this scenario is that it sends the wrong message to the team. Nobody should be rewarded just for completing their assignment — or everybody should be. Tom gets paid a salary for him to do his job. He's expected to do it well. If he gets rewarded just for doing his job, it tells the rest of the team that some jobs matter more than others. How do I get an assignment that matters? they wonder. And why am I wasting my time doing work that isn't important?
Luck and Timing Often Play a Bigger Role Than Excellence
In my nearly 15-year career I've delivered a lot of software to customers. I've invented things, learned difficult technology, solved complex problems, led and motivated teams, and proposed key strategic decisions that have helped the companies I've worked with become more successful. How many trophies do I have on my shelf? One. It is a beautiful blue piece of plastic that I got last summer when the Microsoft division we belong to eclipsed the $1 billion revenue mark for the first time. I'd been with Microsoft about six weeks at the time.
Earning that award (using that word very loosely, by the way) required nothing of me other than accepting a job at Microsoft at the right time. I hadn't written a line of production code yet, and certainly did not contribute to that revenue. I figure it replaces many others I should have earned but didn't.
I'm proud of the award, but I'm the first to admit that I earned it primarily due to fortuitous timing.
Positive Effects Are Usually Short-Lived
I remember one time being on a software team when we realized one morning that we had a serious bug. A problem had been reported by our released product that was having a significant customer impact. One member of the team stepped up to the task of digging into the issue and finding the problem. When he found and fixed the problem later in the day, he was genuinely thanked and given a gift certificate as a show of gratitude from the company.
The following day, our team was in a team meeting when the topic of the previous day's bug came up. We were being asked to identify what we were going to do differently in the future to avoid this problem. The same team member that had saved the team the day before took this as a personal offense. He muttered something to himself under his breath, slammed the lid of his laptop shut, picked it up and stormed out of the conference room.
Clearly, the special effort the company had made to thank him had been completely forgotten in less than 24 hours.
Abuse Makes The Programs Become Meaningless
I've frequently seen well-intentioned incentive programs become so abused that they become a laughing stock. One is almost embarrassed to truly earn an award because so many others have been similarly recognized for awards they acquired through dubious means.
Novell's Employee of the Year kind of took on this sentiment for me after I first saw one person get the award after taking full credit for something I knew full well he did not do (because I did it). Later I saw a colleague, the proven highest contributor on his team, get passed over for this award when his manager instead gave it to the lowest contributor in an effort to motivate him.
Probably the strangest of all of these though is for a company I've heard of with a very generous patent award program. One of their employees has learned to double his annual compensation by filing patent after patent through the company's patent program. Of course, filing all of these patents doesn't leave much time for him to do his regular job, and the patents being filed may not have anything at all to do with the company's products or strategy. But he's so innovative and valuable to their inconsistent patent portfolio that they've had no choice but to promote him over and over.
They Encourage Individualism Over Teamwork
Through a trusted source I heard about a support department for a software company years ago. They had instituted an incentive program called "55 Stay Alive". This great program with such a catchy name was very simple to understand: Each employee had to close 55 incidents each month or they were fired.
Awesome, huh.
One employee made a pitch to management, explaining that the incentive program was very negative and instead suggested they rearrange things to have a single goal that the whole team could work toward together. Management was hesitant but the team was willing to put it to a test.
So they did, with amazing results. Every month the team closed far more incidents working as a group than they ever had working as individuals before.
Logically, management was very unhappy about this insubordinate behavior so they fired the guy.
This might be an extreme example, but you've probably seen lots of examples of this. Many incentive programs reward people for behavior that leads them to focus on individual achievement at the expense of team achievement, which would be much better for the company.
So What Does Work Then?
One of the best incentive programs I've ever participated in was at Novell, working for a manager who ironically was really not that great of a manager. Novell being a Linux company, he had been given a small 6-inch figurine in the likeness of a Linux penguin which we called the Tuxen.
Every so often, in a team meeting, he would recognize someone for a specific assignment they'd done well. Then they would be awarded the Tuxen for a week. Each winner was expected to proudly display the Tuxen and take it places and publish photos of their adventures on the team wiki.
One week I was awarded the Tuxen for work I had done to help start the Eclipse Linux Tools project. I proudly took the Tuxen home with me, took photographs of the Tuxen with my family, and brought it along on a trip with my son to the Las Vegas Supercross. I gave it back a week later, but it was one of the most rewarding recognitions I'd ever had.
When I first read Joel's blog post, I immediately e-mailed him with my concerned, contrary opinion. "So how DO you reward people and make sure they know they've done well?" I asked.
"Just thank them," Joel said.
Maybe you're wondering, could it really be that simple? Instead I'd ask, why does it have to be more complicated?
Trusting Teams
We all know how it feels to be trusted. That's a big part of our profession as software engineers. Our job is not just to solve a problem, but to do it in a way that is highly performant, easily maintained, efficient, extensible, secure, resilient, and correct. We have to understand what customers need, not what they say they need. And before we even solve the problem, we have to estimate how long it will take us to solve it, and make and keep commitments for when we will deliver.
It is a tough job, but we're pretty good at it. Like Michael Jordan in the NBA Finals, we know we can step up and perform. We enjoy that feeling of coming through and delivering.
Because we are good at what we do, we like feeling like our employers trust us. We don't want to work for employers that don't. It bugs us when our boss tries to tell us how to do our job or doesn't believe us when we declare that we will be done on time. The satisfaction that comes from keeping your word is diminished if someone is breathing down your neck every step of the way.
This is what true delegation is about, and I'm a big fan. I'm a Covey disciple and not afraid to admit it. In fact, when I once interviewed for a job, I was pushed on the concept of delegation, and I declared, "Why do they work for you if you don't believe they will do what they say they will do?" There's a great amount of job satisfaction that comes from filling a stewardship that is truly yours.
The problem is, it isn't quite like that in teams — at least, not good ones.
We had a rather interesting team status meeting this past week at UDC. I won't go into details, but as I thought about it over the rest of the week, I could see a clear issue near the heart: People not feeling trusted in their stewardships.
After thinking about this a lot, discussing it with team members, and re-reading "The Five Dysfunctions of a Team" by Patrick Lencioni, I hit upon a bit of an epiphony for myself that helped me understand the issue better.
The most basic team dysfunction he identifies is an absence of trust. Toward the end of Lencioni's book, he explains this further:
In the context of building a team, trust is the confidence among team members that their peers' intentions are good, and that there is no reason to be protective or careful around the group. In essence, teammates must get comfortable being vulnerable with one another. This description stands in contrast to a more standard definition of trust, one that centers around the ability to predict a person's behavior based on past experience.... As desirable as this may be, it is not enough to represent the kind of trust that is characteristic of a great team. It requires team members to make themselves vulnerable to one another, and be confident that their respective vulnerabilities will not be used against them.
Upon reading this, I realized instantly how this related to our team.
Suppose you are the lead of a team of six engineers. Your team is working in a one-month milestone, and you've got about six person-months of work to be done in that milestone. How do you go about getting it done?
Well, you could try running the whole show by yourself. After all, you are the lead. So you could go among the team members all day long, telling them what to do. You could tell them what class hierarchy to use, what data structure to use, what design pattern to use. You could critique their naming conventions or even tell them what character to type when.
You might make your milestone, if those engineers are still working with you at the end of a month. But even if they stay, you've lost the opportunity to get them to buy in.
Of course, we all know that strategy is flawed, so you try delegating instead. You trust your team. If they say they will do it, they will. So you give them each assignments, or you let them choose their assignments. They each have their own area of stewardship, and they each commit to be done at the end of the milestone. So, I guess you can just go golfing for the rest of the month, right? Then come back at the end of the month and everything should be done.
They said they would be done, so they'll be done, right? That's what it means to trust. Unless something came up, of course. But even if they all did make it, they missed out on a great opportunity: The opportunity to be a team.
See, teams do trust each other. They rely on each other and believe in each other. But they work together anyway. They meet together. They hold each other accountable. They win and lose together.
In the previous scenario, if five of your six engineers had completed their work, does that mean success or failure? Does that mean you were 83% successful? Do you just round up and say you were generally on target?
Great teams don't do that. They succeed and fail together. In a great team, five out of six people being successful is a failure on two fronts. First, they failed because they didn't accomplish the team goal. Second, they failed to pull together as a team.
So let's go back a month. How does that team pull together?
The team has to care about each member. It has to hold itself accountable for a common goal, and commit to win together or not at all. Because of this, the team must ask each other hard questions.
When this happens to you, as a member of that team, you might feel like you aren't being trusted. This is when you have to do what Lencioni suggests: You have to trust your team enough that they can ask. You have to trust them enough that you believe they have the right intentions at heart. You have to trust that when they dig on you and question you, they won't hold it against you if they find a flaw in you. You have to trust that when you question them, they won't take it personally either.
The questioning is part of the deal if you want people who are passionate and care and want to win. What do you do when someone you care about comes home? You ask them questions. You ask about their day, who they saw, how things went, what they learned, how did their presentation go, etc.
You don't ask these questions because you don't trust them (er, usually, anyway). You ask because you care. Asking questions is how we express interest, concern, and buy-in.
When you work for a company that cares, questions are part of the deal. Microsoft cares. We are passionate about technology. We want to win. We want our teams to win. We want to create great products and delight customers. Everyone in my management chain has this passion. So, I can expect this kind of questioning from everyone in my management chain.
I've worked for other companies that don't have this passion. It shows because nobody asks, about anything.
The same holds true for teams. Teams ask each other. They are honest with each other. They are open with each other. They rely and depend upon each other. And they win and lose together.
I've been on teams like this. They are AWESOME. As great as it feels individually to fulfill a stewardship, it feels even better to be a part of a team that works together and helps each other and wins. The cost of entry? Humility. Vulnerability. You have to be willing to let your team ask you about your approach and technique and decisions, and be willing to let them dig and dig. They do this because, like you, they want to win. They want to win WITH you, not OVER you. And they depend on you to question them so you can win with them also.
When your team does this, when you really trust each other and open up to each other, you'll find weaknesses, problems, and concerns. Each time you do this, it is an opportunity to pull together as a team and win together. Your strengths compensate for my weaknesses. Instead of six people individually trying to deliver six distinct units of work of about one person-month of effort each, you have six people working together to deliver six person-months of work. My weakness may have been my downfall when I worked alone, but now that we work together your strength can compensate for my weakness, and mine can compensate for yours. We can do more together.
Kum-ba-yah. Even though I believe this, I think I'm going to be sick with all this syrupy feel-goodness if I don't stop soon. But that doesn't mean it isn't real. Be passionate enough about success as a team that you are willing to learn from each other and challenge each other to grow together. Care enough about your team to respectfully question your teammates, and trust that they care about the team when they question you. If you do this, you'll find that working on that kind of a team is an experience you'll remember your whole career.
