Hard Coded Breakpoints
This little line of code totally saved me this week:
__asm int 3;
I'm writing a plugin to an existing Windows application and I could not, for the life of me, figure out if my plugin was even being loaded or if it was executing properly. All I knew was that I wasn't seeing the behavior from my plugin I expected. I needed to step through my plugin with a debugger, but since it loads as part of another app I didn't know how to get the debugger to break. Setting breakpoints in Visual Studio wasn't working.
Fortunately for me I have smart teammates, and one of them taught me this little trick to programmatically insert a hard coded breakpoint into your code. If I add this line, when my code runs the breakpoint is triggered and the application stops executing, with a dialog giving me the option to debug it in Visual Studio. When I get in there, behold: The cursor is pointing at that line of code and I can step through my plugin.
Since then I've done a bit more research and found out that, in truth,
__asm int 3;
is really only meant for x86 architectures. A more portable way to do this for Windows is:
__debugbreak();
which invokes the correct assembly interrupt for the current architecture.
On Linux, apparently you can do this:
__asm__("int $0x03");
although I haven't tried it. It looks enough like the x86-only version for Windows that it makes me wonder if it would work on x64 architectures; I'd love to hear back if anyone knows for sure.
Also, interested to know how to do this on OS X. Comment or mail if you know.
Incentive Programs and Professional Motivation
I recall some years ago having read this Joel Spolsky post entitled "Incentive Pay Considered Harmful." In typical Spolsky fashion, he made a very compelling case that I had a very hard time coming to grips with. At the time I'd just recently made a series of contributions to Novell, outside of the scope of my job description, that I felt had an obvious positive impact on the company, but I had not received any sort of recognition at all for this.
Joel's article strongly disagreed with me. I felt I should have been recognized for my efforts; why else would I take the personal risk that comes with extending one's self in that fashion? Instead, nothing had been done.
I've been pretty interested in this topic and so I've been observing and evaluating ever since. What I've seen is that, for the most part, incentive programs tend to have the opposite effect of their purpose; namely, they tend to demotivate rather than motivate.
Outlined below are some reasons why.
People Get Rewarded Just For Doing Their Job
Maybe you've seen this too: You attend a quarterly all-hands meeting when an individual you know and work with is asked to come to the front. They announce, "Tom did a really great job running our beta program last release, and so we are giving him the quarterly Mega-Smiley-Face award, with a free dinner for two and a new iPod Touch." You sit there and stare, clapping insincerely. Sure, Tom did a great job running the beta program. But Tom is the beta program coordinator! It is his job to run the beta program! You did great work with your assigned responsibilities last quarter also. Why aren't you getting an award?
One problem with this scenario is that the rewards program now seems arbitrary. Tom was rewarded this quarter for doing his job. Who will be next? Who knows? Everyone is doing their job, but not everyone gets rewarded. The rest of the people feel discouraged. The recognition seems to be something worth earning, but they don't know how to earn it.
Another problem with this scenario is that it sends the wrong message to the team. Nobody should be rewarded just for completing their assignment — or everybody should be. Tom gets paid a salary for him to do his job. He's expected to do it well. If he gets rewarded just for doing his job, it tells the rest of the team that some jobs matter more than others. How do I get an assignment that matters? they wonder. And why am I wasting my time doing work that isn't important?
Luck and Timing Often Play a Bigger Role Than Excellence
In my nearly 15-year career I've delivered a lot of software to customers. I've invented things, learned difficult technology, solved complex problems, led and motivated teams, and proposed key strategic decisions that have helped the companies I've worked with become more successful. How many trophies do I have on my shelf? One. It is a beautiful blue piece of plastic that I got last summer when the Microsoft division we belong to eclipsed the $1 billion revenue mark for the first time. I'd been with Microsoft about six weeks at the time.
Earning that award (using that word very loosely, by the way) required nothing of me other than accepting a job at Microsoft at the right time. I hadn't written a line of production code yet, and certainly did not contribute to that revenue. I figure it replaces many others I should have earned but didn't.
I'm proud of the award, but I'm the first to admit that I earned it primarily due to fortuitous timing.
Positive Effects Are Usually Short-Lived
I remember one time being on a software team when we realized one morning that we had a serious bug. A problem had been reported by our released product that was having a significant customer impact. One member of the team stepped up to the task of digging into the issue and finding the problem. When he found and fixed the problem later in the day, he was genuinely thanked and given a gift certificate as a show of gratitude from the company.
The following day, our team was in a team meeting when the topic of the previous day's bug came up. We were being asked to identify what we were going to do differently in the future to avoid this problem. The same team member that had saved the team the day before took this as a personal offense. He muttered something to himself under his breath, slammed the lid of his laptop shut, picked it up and stormed out of the conference room.
Clearly, the special effort the company had made to thank him had been completely forgotten in less than 24 hours.
Abuse Makes The Programs Become Meaningless
I've frequently seen well-intentioned incentive programs become so abused that they become a laughing stock. One is almost embarrassed to truly earn an award because so many others have been similarly recognized for awards they acquired through dubious means.
Novell's Employee of the Year kind of took on this sentiment for me after I first saw one person get the award after taking full credit for something I knew full well he did not do (because I did it). Later I saw a colleague, the proven highest contributor on his team, get passed over for this award when his manager instead gave it to the lowest contributor in an effort to motivate him.
Probably the strangest of all of these though is for a company I've heard of with a very generous patent award program. One of their employees has learned to double his annual compensation by filing patent after patent through the company's patent program. Of course, filing all of these patents doesn't leave much time for him to do his regular job, and the patents being filed may not have anything at all to do with the company's products or strategy. But he's so innovative and valuable to their inconsistent patent portfolio that they've had no choice but to promote him over and over.
They Encourage Individualism Over Teamwork
Through a trusted source I heard about a support department for a software company years ago. They had instituted an incentive program called "55 Stay Alive". This great program with such a catchy name was very simple to understand: Each employee had to close 55 incidents each month or they were fired.
Awesome, huh.
One employee made a pitch to management, explaining that the incentive program was very negative and instead suggested they rearrange things to have a single goal that the whole team could work toward together. Management was hesitant but the team was willing to put it to a test.
So they did, with amazing results. Every month the team closed far more incidents working as a group than they ever had working as individuals before.
Logically, management was very unhappy about this insubordinate behavior so they fired the guy.
This might be an extreme example, but you've probably seen lots of examples of this. Many incentive programs reward people for behavior that leads them to focus on individual achievement at the expense of team achievement, which would be much better for the company.
So What Does Work Then?
One of the best incentive programs I've ever participated in was at Novell, working for a manager who ironically was really not that great of a manager. Novell being a Linux company, he had been given a small 6-inch figurine in the likeness of a Linux penguin which we called the Tuxen.
Every so often, in a team meeting, he would recognize someone for a specific assignment they'd done well. Then they would be awarded the Tuxen for a week. Each winner was expected to proudly display the Tuxen and take it places and publish photos of their adventures on the team wiki.
One week I was awarded the Tuxen for work I had done to help start the Eclipse Linux Tools project. I proudly took the Tuxen home with me, took photographs of the Tuxen with my family, and brought it along on a trip with my son to the Las Vegas Supercross. I gave it back a week later, but it was one of the most rewarding recognitions I'd ever had.
When I first read Joel's blog post, I immediately e-mailed him with my concerned, contrary opinion. "So how DO you reward people and make sure they know they've done well?" I asked.
"Just thank them," Joel said.
Maybe you're wondering, could it really be that simple? Instead I'd ask, why does it have to be more complicated?
The Truth About Novell Forge
I got an interesting e-mail the other day from Novell:
Please Note: You have been sent this email because you are listed as an administrator of one or more Novell Forge projects.
When Novell Forge was first launched Novell recognized the need for a site dedicated to providing hosting services to a growing number of software development projects, many supporting our open source initiatives. Novell Forge quickly grew and was soon providing these service to nearly 1000 such projects. Demand for new projects has all but disappeared during the past two years while a number of additional project hosting options have begun that can provide a similar set of services to those of Novell Forge.
Now that there are many other options, Novell can turn its focus to other areas and pass the project hosting responsibilities to these other dedicated hosting sites. Novell will be decommissioning the Novell Forge system on December 15, 2009.
This is interesting to me because it is not entirely true. I should know, because without me there would never have been a Novell Forge.
It's a bold statement, I know. It's one I'm happy to explain.
I came to Novell from IBM in 2000. It didn't take long to realize that Novell's developer story and strategy, or rather the complete lack thereof, was (and still is) a significant weakness in their overall execution. People buy a computer operating system in large part because of the applications that they can run on it; if a business wants to run a CRM system, they'll want to be sure that whatever platform they buy will run a CRM suite that is acceptable to them. This is why having a strong developer strategy is crucial to platform providers, and almost everyone seems to understand this. Novell certainly should; NetWare owned the x86 server market in the 80's and early 90's until Microsoft entered that market. Initially, the Microsoft offering was not necessarily better than NetWare in terms of stability or performance, but Microsoft definitely outgunned Novell when it came to applications. It was so much easier to create applications for Microsoft's platform that their supported portfolio dwarfed Novell's, and that was a significant key to dethroning Novell's dominant position in the x86 server market in the mid 90's.
Anyway, when I came to Novell and learned this, I thought that probably Novell's Developer Services organization just didn't know what to do (a mistaken analysis, I later learned) and if I worked there I could probably fix everything. I was pretty young, arrogant, and naive then. But in 2002 I was presented an opportunity to work in Developer Services and I took it.
One of the first things I was asked to do was to provide support to customers programming to eDirectory. I decided to try to learn more about how to do this the same way our third-party developers would, by using the resources that were available online. I found what appeared to be our authoritative how-to-program-to-eDirectory tutorial, got most of the way through my sample app, and got stuck. Finally I started asking questions. I quickly learned that everything I'd been doing was wrong; the authoritative documentation was incorrect. It used an out-of-date and deprecated API and was no longer considered best practice. It was some two or three years out of date, but hadn't been changed yet because changing the documentation was just too painful.
I felt this situation was unacceptable. We needed the freedom to create an abundance of rich and helpful developer content and to have it published and updated freely and frequently. We needed to be able to do this without going through drawn-out and tedious approval processes and staging phases for even minor edits. We needed to be able to continuously deliver not only whitepapers but tutorials and sample applications. I felt that what was needed was a complete overhaul of Novell's developer site, converting it into a web application where administrators (Novell Developer Services employees) could update the content and have complete control over what information was being provided to our developer community.
I discussed this with a colleague and my manager, and then we called a formal meeting to discuss this proposal. I think there were four Developer Services employees in the room. As we discussed the reasons to do this, other advantages surfaced. A key issue was that, in Novell's then-existing developer forums, many Novell developers were already contributing to solving each other's problems, including answering each other's questions and even sharing code, from small snippets to complete applications. We realized that instead of top-down support flowing from company to customer, what our customers really preferred was community support with Novell as an active participant. As we discussed this, one of my colleagues suggested that instead of writing the web app I suggested, we should do a project hosting site, like SourceForge. Such a site would allow us to participate as a community with our users to exchange sample code, documentation, tutorials, and other content. Novell Forge was born.
As we began to socialize the idea, we found out that a separate group within Novell had been tasked with creating a project hosting site for internal company use. When we both became aware of each other's goals, the synergies were obvious and it seemed apparent that we should try to coordinate our efforts. Interestingly, we had human resources to give to the project but lacked funding for capital expenses; the other group had capital expense budget but lacked human resources. Ultimately we agreed that, as my team developed the Novell Forge solution, we would also develop an internal-use version of the site to meet the goals of this team; in exchange, they would help us to get the hardware we needed to host Novell Forge.
Around the time Novell Forge was launched and completed, a number of people involved directly or indirectly from that team claimed credit for having launched Novell Forge. Some of them were quite handsomely rewarded by the company, presumably at least in part due to their claimed credit for the site. Others still claim in public that they are responsible for the site even though they had absolutely nothing to do with the conceptualization, proposal, approval, or implementation.
Meanwhile, those of us who did come up with the idea, who did make the business case and get the approval and deliver the site, well, we pretty much had to settle for a brief pat on the back from Novell. Or did we even get that? Anyway.
Novell Forge, despite its pretty lame name and humble beginnings, was actually quite well received by the press. It earned kudos for Novell from Dave Kearns of NetworkWorld, which was not exactly easy to come by. And as Novell tried to reinvent itself with an open source focus, purchasing such open source companies as Ximian and SUSE Linux, the existence of Novell Forge was frequently cited as evidence that Novell was serious about an open source strategy (example). Interest in the site grew quickly and it soon hosted over 1000 external projects, as stated in the e-mail I quoted above. My team was excited about the traction the site was gaining. We had many, many ideas for how to grow the site and make it an even more useful tool for software developers. We had more work to do than time to do it, and it was neat to feel like what we were doing had an impact to Novell.
Even though Novell didn't seem to care about it.
Oddly, in spite of what my team thought was a pretty obvious success, we could not get approval for funding to continue to promote the site. The team was gradually reduced in size, again and again. When people would leave, their vacancies would languish unfilled until that position was eventually lost. The team was instructed to not develop the site but instead to work on undefined new work in other undefined areas, wasting many person-years of development effort. The community could sense Novell's lack of investment and they lost interest. Novell Forge became a laughing stock. It was used as evidence of what a company does when they "just don't get" open source, when it was ironically used as evidence of Novell's good faith not too long before.
Things finally came to the point where there was only one employee assigned to maintain the site, along with other unrelated duties (I, and the rest of the team, had by now been reassigned to different projects). Novell Forge was completely unsupported by Novell's IT group, leaving instead the support of the site to this one individual. I recall an occasion where the site went down over the weekend and was out for a couple of days. It was obvious that the site was in demand, because users made Novell aware of the outage quite quickly. However, Novell was not willing to pay for 24/7 support for the site, so instead of being brought back online right away, the site was down for the entire weekend until that resource came in to work the next Monday. My manager brought this to the attention of our team with the insistence that we address it. He stated that from that point on, that one employee would be the primary off-hours maintenance person for the site, and I would be the backup.
I then asked if Novell was going to start reimbursing me for my cell phone bill. He said no. I asked if they were going to buy me an additional cell phone, pay that bill, and also pay me extra to carry that additional phone. He said no. He said they would just list my personal cell number in the emergency contact list, and would call it if there were an emergency. I stated that in that case I maintained the right to not answer. He stated that I would have to answer, that it was my assignment. I claimed that Novell could not require me to answer my personal cell phone if I'm the one paying the bill. I then reminded him that in Novell's support organization, at least at that time, people that were expected to respond 24/7 had their cell phone bill paid by Novell, were paid an additional amount to be on call, and were paid an additional amount if they actually took a call and worked that call during off hours. I said, "If the site is important to Novell, that is what Novell should do. If the site is important, it should be important enough that Novell is willing to pay in order to maintain uptime and keep our customers satisfied."
Novell was not willing to pay.
I shortly moved on to a different team within Novell, and the other guy left the company altogether. I'm not sure who has been maintaining the site since then.
What Novell chooses to do with their money and their human resources is their business. This isn't meant as a criticism; I don't claim to have the right experience to criticize their decision to strangle Novell Forge to death. This is simply meant as a statement of fact, and the facts are pretty clear:
- You get what you pay for.
- Novell did not pay for Novell Forge by giving due reward and recognition to those who truly brought this idea to the company.
- Novell did not pay for Novell Forge by feeding its success with additional funding, promotion, and development.
- Novell did not pay for Novell Forge by giving it the kind of support and maintenance that its customers expected.
- The customers of Novell Forge were initially enthusiastic, but grew to sense the lack of commitment by the company and thus stopped participating.
- Novell Forge died as a result.
Novell Forge may be planned for decommission this December, but it died years ago. And don't think you can fool me, Novell. Novell Forge did not die because of lack of interest by the user community. Novell Forge died because you did not care about it. Whether that was a good decision or not is not for me to decide, but please, Novell, at least be honest with your community. We did not kill Novell Forge — you did.
UPDATE: Dan Reese, a member of my team back then, corroborated this in his blog.
