Grading on a Scale – I Mean, a Curve
Miguel de Icaza tweeted this today:
Performance Review Season at Novell is easy when your entire team is made of 100% super-awesome hackers.
It's a very nice thing for Miguel to say about his team, something I have no reason to doubt. I don't know Miguel too well, but I have been in a handful of meetings with him back in my Novell days. He doesn't strike me as the type of person to be disingenuous, and regardless of how you feel about .NET on non-Windows platforms, you have to be impressed with the impact and effectiveness of Mono, so I figure his team probably is pretty awesome.
I guess when you are among the privileged at Novell, you can do things your way. I know when I was at Novell, having a team full of awesome people didn't make performance reviews any more straightforward. Novell is by no means unique in this; I can personally attest to this fact.
At most big companies I've been at, how performance is measured depends on what time of year it is. If you are measuring performance during any other time of the year, you do this by comparing a person's efforts against a published definition of their job description and level. For example, on your company's intranet somewhere there is a document that describes the expected level of performance for a mid-level software engineer. If you are a mid-level software engineer, you can compare your performance against the published standard and determine if your performance is currently on track or ahead of or lagging expectations.
This is true at any time of year except during performance review time. During performance review time, suddenly your performance is measured by how you compare to your peers. So now, if you are a mid-level software engineer, your performance is based on how your performance and contributions compare to the other mid-level software engineers in your organization.
For some reason this seems to be a pretty common way to measure employee performance, and yet it makes absolutely no sense to me. In casual conversations with peers (and, frankly, also with a number of managers who will remain unnamed here), I think I can confidently say that it makes no sense to most other people either.
Imagine you are attending college and you sign up to take a calculus class. On the first day of class, the instructor hands out a syllabus which clearly explains how you will be graded, which boils down to this: If you get 95% of all the questions right on homework and tests, you will get an A, and the scale breaks down from there. As the semester proceeds, you monitor your performance on homework and tests, studying to meet that magical 95% mark. On the last day of class, you enter confidently knowing you have a 96%-97% average which gives you a little bit of a buffer on the final. You leave the final feeling pleased with yourself, sure that you got at least 19 of the 20 questions right.
Then the grades come out, and you have a B-.
You go and ask the professor, and he says, "Well, I changed the grading from a scale to a curve. See, everyone was getting As, and we can't have that. So I graded on a curve instead. You got a B-."
This is effectively how most employee performance measurement techniques work. Your performance is graded on an objective, defined scale all year long. Then comes performance review time, and suddenly you are now being graded on a curve, not a scale. Before you knew where you stood; now, you have no idea.
A normal distribution curve makes sense when applied to a statistically average sample. The question is, does your employee population comprise a statistically average sample? In order to do that, you'd pretty much have to be willing to hire anyone who walks in the door asking for a job. You probably don't do that; you are probably very selective in who you hire and keep. You probably try to hire only the best talent you can find. You probably take the approach of preferring to sometimes pass on talented people that you aren't sure of, rather than give the benefit of doubt, and so you've probably rejected some talented candidates.
That means your employee population is not statistically average. So why is it that during performance review time you try to make this anomalous sample conform to a normal distribution curve?
I've worked for companies where they say that, organization-wide, some percentage of the population of any given job function MUST be underperforming, some percentage MUST be overperforming, and the rest MUST be performing at around average, within about a standard deviation or so. This isn't just a belief. It is enforced through policy. Essentially the policy says it is not possible to have an organization with 100 employees that are all high performers. Instead, you might mandate through policy that 10 of them must be underperforming, 10 must be overperforming, and the remaining 80 are average or near-average. Yet when you hire, you insist that you hire only the best.
This doesn't make any sense, and in my opinion it is a dangerous practice.
First off, you aren't fooling anyone, especially not the intelligent, highly-educated type of thought worker that tends to be evaluated this way. You wanted smart people, you hired smart people, and now you have to pay the consequences: You can't implement obviously flawed policy like this. They will pick up on it and it will cause them to lose faith in your organization.
Second, it encourages exactly the wrong type of behavior. Once they realize that their actual performance measurement has less to do with their evaluation against an objective metric than it does against their peers, employees will be incentivized to game the system. This may manifest itself in many ways: refusing to help teammates or to work well as a team; manipulating the assignments given to them and others to maximize their relative performance; taking opportunities to point out to you (the manager) how excellent they are in comparison to others or how rotten the others are in comparison to themselves; attempting to make others look poorly or to make themselves look good in public, e.g. meetings; trying to hire people who they know to be less competent than they are; avoiding promotions (and thus responsibility) for the sake of cherrypicking in a job function that they can excel in; etc. None of these helps your organization to become better.
Third, it encourages employees to leave. Good employees. Employees that passed your interview process. Employees that are exceeding the objective metrics, but don't rank at the right of the curve. They know they can choose a different company with lower hiring standards and at least perform at the top. If you are really hiring only the best, then this person is among the best, and they will be hard to replace.
Fourth, this system encourages sucking up. By definition, it is an entirely subjective measurement. As a manager, you are likely to be persuaded by those who have perfected the art of sucking up and give them better evaluations. You might do this without even realizing it.
So, what should you do about it?
Well, if you are empowered to do something about it, I think you should avoid doing it altogether. Evaluate your employees based on an objective scale. Period. Hire great people, expect greatness from them, and then stop doing dumb things that encourage them to not be great. Just get out of the way and revel in the greatness.
The main argument I've heard in favor of the curve is budgetary. "We budget 10% of employee salary for annual bonus. High performers get 18%-20%, low performers get 0%-2%, and the rest get from 5%-15%. The average is 10% so we meet budget." The corollary argument is that the bonuses should not be spread evenly across the team. More money for the high performers.
That's fine, but didn't you try to hire ALL high performers? If so, wouldn't you say you expect ALL of them to be in the 18%-20% range? Why not instead budget your bonus pool for the high performance amount across the board? You should still evaluate each employee's performance, but now you aren't being constrained by budget, so you can do it objectively and fairly against a published metric, the one you've been using in one-on-one's all year long. If an employee underperforms, you can still give a 2% bonus or none at all. Take the money you saved and use it next year or something.
If you aren't really empowered to do anything about it, my suggestion is to press onward anyway. Be excellent anyway. Don't let the system cause you to be non-excellent. Be a good team player. Try to surround yourself with the best people. Take the assignments you are given and do your best with them. Discuss with your manager any concerns you have about whether you are getting the opportunity to show what you've got, but move forward and do the job right even if the system doesn't reward you. I believe good things come about from good works, sooner or later. If that doesn't work, I guess you can always go work for someone else who's a bit more reasonable.
You can also blog about it.
Incentive Programs and Professional Motivation
I recall some years ago having read this Joel Spolsky post entitled "Incentive Pay Considered Harmful." In typical Spolsky fashion, he made a very compelling case that I had a very hard time coming to grips with. At the time I'd just recently made a series of contributions to Novell, outside of the scope of my job description, that I felt had an obvious positive impact on the company, but I had not received any sort of recognition at all for this.
Joel's article strongly disagreed with me. I felt I should have been recognized for my efforts; why else would I take the personal risk that comes with extending one's self in that fashion? Instead, nothing had been done.
I've been pretty interested in this topic and so I've been observing and evaluating ever since. What I've seen is that, for the most part, incentive programs tend to have the opposite effect of their purpose; namely, they tend to demotivate rather than motivate.
Outlined below are some reasons why.
People Get Rewarded Just For Doing Their Job
Maybe you've seen this too: You attend a quarterly all-hands meeting when an individual you know and work with is asked to come to the front. They announce, "Tom did a really great job running our beta program last release, and so we are giving him the quarterly Mega-Smiley-Face award, with a free dinner for two and a new iPod Touch." You sit there and stare, clapping insincerely. Sure, Tom did a great job running the beta program. But Tom is the beta program coordinator! It is his job to run the beta program! You did great work with your assigned responsibilities last quarter also. Why aren't you getting an award?
One problem with this scenario is that the rewards program now seems arbitrary. Tom was rewarded this quarter for doing his job. Who will be next? Who knows? Everyone is doing their job, but not everyone gets rewarded. The rest of the people feel discouraged. The recognition seems to be something worth earning, but they don't know how to earn it.
Another problem with this scenario is that it sends the wrong message to the team. Nobody should be rewarded just for completing their assignment — or everybody should be. Tom gets paid a salary for him to do his job. He's expected to do it well. If he gets rewarded just for doing his job, it tells the rest of the team that some jobs matter more than others. How do I get an assignment that matters? they wonder. And why am I wasting my time doing work that isn't important?
Luck and Timing Often Play a Bigger Role Than Excellence
In my nearly 15-year career I've delivered a lot of software to customers. I've invented things, learned difficult technology, solved complex problems, led and motivated teams, and proposed key strategic decisions that have helped the companies I've worked with become more successful. How many trophies do I have on my shelf? One. It is a beautiful blue piece of plastic that I got last summer when the Microsoft division we belong to eclipsed the $1 billion revenue mark for the first time. I'd been with Microsoft about six weeks at the time.
Earning that award (using that word very loosely, by the way) required nothing of me other than accepting a job at Microsoft at the right time. I hadn't written a line of production code yet, and certainly did not contribute to that revenue. I figure it replaces many others I should have earned but didn't.
I'm proud of the award, but I'm the first to admit that I earned it primarily due to fortuitous timing.
Positive Effects Are Usually Short-Lived
I remember one time being on a software team when we realized one morning that we had a serious bug. A problem had been reported by our released product that was having a significant customer impact. One member of the team stepped up to the task of digging into the issue and finding the problem. When he found and fixed the problem later in the day, he was genuinely thanked and given a gift certificate as a show of gratitude from the company.
The following day, our team was in a team meeting when the topic of the previous day's bug came up. We were being asked to identify what we were going to do differently in the future to avoid this problem. The same team member that had saved the team the day before took this as a personal offense. He muttered something to himself under his breath, slammed the lid of his laptop shut, picked it up and stormed out of the conference room.
Clearly, the special effort the company had made to thank him had been completely forgotten in less than 24 hours.
Abuse Makes The Programs Become Meaningless
I've frequently seen well-intentioned incentive programs become so abused that they become a laughing stock. One is almost embarrassed to truly earn an award because so many others have been similarly recognized for awards they acquired through dubious means.
Novell's Employee of the Year kind of took on this sentiment for me after I first saw one person get the award after taking full credit for something I knew full well he did not do (because I did it). Later I saw a colleague, the proven highest contributor on his team, get passed over for this award when his manager instead gave it to the lowest contributor in an effort to motivate him.
Probably the strangest of all of these though is for a company I've heard of with a very generous patent award program. One of their employees has learned to double his annual compensation by filing patent after patent through the company's patent program. Of course, filing all of these patents doesn't leave much time for him to do his regular job, and the patents being filed may not have anything at all to do with the company's products or strategy. But he's so innovative and valuable to their inconsistent patent portfolio that they've had no choice but to promote him over and over.
They Encourage Individualism Over Teamwork
Through a trusted source I heard about a support department for a software company years ago. They had instituted an incentive program called "55 Stay Alive". This great program with such a catchy name was very simple to understand: Each employee had to close 55 incidents each month or they were fired.
Awesome, huh.
One employee made a pitch to management, explaining that the incentive program was very negative and instead suggested they rearrange things to have a single goal that the whole team could work toward together. Management was hesitant but the team was willing to put it to a test.
So they did, with amazing results. Every month the team closed far more incidents working as a group than they ever had working as individuals before.
Logically, management was very unhappy about this insubordinate behavior so they fired the guy.
This might be an extreme example, but you've probably seen lots of examples of this. Many incentive programs reward people for behavior that leads them to focus on individual achievement at the expense of team achievement, which would be much better for the company.
So What Does Work Then?
One of the best incentive programs I've ever participated in was at Novell, working for a manager who ironically was really not that great of a manager. Novell being a Linux company, he had been given a small 6-inch figurine in the likeness of a Linux penguin which we called the Tuxen.
Every so often, in a team meeting, he would recognize someone for a specific assignment they'd done well. Then they would be awarded the Tuxen for a week. Each winner was expected to proudly display the Tuxen and take it places and publish photos of their adventures on the team wiki.
One week I was awarded the Tuxen for work I had done to help start the Eclipse Linux Tools project. I proudly took the Tuxen home with me, took photographs of the Tuxen with my family, and brought it along on a trip with my son to the Las Vegas Supercross. I gave it back a week later, but it was one of the most rewarding recognitions I'd ever had.
When I first read Joel's blog post, I immediately e-mailed him with my concerned, contrary opinion. "So how DO you reward people and make sure they know they've done well?" I asked.
"Just thank them," Joel said.
Maybe you're wondering, could it really be that simple? Instead I'd ask, why does it have to be more complicated?
