Software Development Methodology Profile – Novell Forge
It was the fall of 2002 and I hadn't been in Novell's Developer Services group for more than a couple of months when I realized that the way Novell was maintaining and offering their developer content was flat-out broken. We needed a new process for delivering developer content that was a lot more dynamic and interactive. So I proposed we create a new web site with a web application that would allow us to publish more freely. We soon realized that we wanted to publish code too, not just documents, and that many of our developers wanted to also host and share their own code. The idea behind Novell Forge was born.
We were starting with an open source CMS, Xoops, and a fairly good-sized Xoops plugin named XoopsForge, but there was still a lot of software development needed to deliver Novell Forge from these foundational pieces. We had a smallish development team of four or five engineers and a project manager. I was somewhat unofficially designated as the lead. Having recently read a bit about Extreme Programming and being interested in trying some of the techniques out, I somehow convinced the project manager that we should try out a new way of developing software.
This methodology was so amazingly lightweight and simple, and yet it worked so remarkably well. It's shaped a lot of my thinking about software development ever since.
I wrote a very simple web application in PHP to facilitate the process.
- There was a story card entry page, where you could add a new story to the list of stories that comprised the product roadmap, or edit a story. Each story had only a couple of fields: a name, a description, and exit criteria. The exit criteria outlined a number of objective criteria that, if true, meant the story was complete. A developer could also assign himself or someone else to work on the story.
- There was a participant entry page, where you could add a new developer to the project, or edit one. This was pretty simple also, maybe just the name.
- There was a story prioritization list page.
- There was a dashboard where you could easily see who was on the team and what they were working on currently.
That's pretty much it. It took me about a day or so.
There were two keys to this. The first was that the stories were prioritized from 1 to N where N is the total number of stories remaining to be completed. This required a leap of faith on the part of the project manager, who had spent her entire career prioritizing features as P1 (mandatory), P2 (important), and P3 (desirable). Maybe your project does things this way too. Most companies I've worked for use this same technique.
Here's the problem with the P1-P2-P3 prioritization. Suppose the product manager, or whoever, puts together a requirements document. He lists all of the features and prioritizes each one P1, P2, or P3. Any guess how many of the features are P1? Based on my experience, probably about 60%-70%. Another 20%-30% are P2, with maybe only 10% or less prioritized as P3. What if the schedule can only allow us to do half of the requirements? This prioritization doesn't help us at all. When a developer finishes one assigned task, which should she work on next? If they are all prioritized P1 it is hard to differentiate.
Sorting all the stories from 1 to N forces the people managing the requirements to think about what really matters first. If you deliver each story completely and meet the exit criteria, the product should be "functional" with the new story fully integrated by the time it is done. Note that this doesn't necessarily mean the product is shippable after the first story. With Novell Forge, we knew we needed to deliver to about story 30 or 40 before we had a product we felt was worth releasing, but still we numbered every story from 1 to N.
A great benefit that comes from this is that now the developers can self-manage. Is my work done? This is easily determined by evaluating the progress against the exit criteria. What should I work on next? Just assign to yourself the next item at the top of the list.
The second key is trust. A lot of more complex methodologies add a lot more process to replace the need to address incorrect behavior by your developers. My web app didn't have any access controls, didn't have any roles or responsibilities. Anyone could add, delete, reprioritize, or assign stories, but the key is that they DIDN"T unless they were supposed to. Developers closed issues when they were done and assigned themselves more work when they needed stuff to do. They kept themselves busy and continually marching toward the goal. My point is this: If your team can't follow very simple processes, you might have a personnel problem, not a process problem. So don't try to fix your personnel problem by adding process complexity.
What were the results of this simple process? Well, for one thing, it put the team and management at ease. At any given point in the cycle, we had a functioning product with the most important features already delivered and the next most important features in progress. We could ship whenever we felt like it. We could decide to delay shipment until specific stories were done, or align our shipment with a specific event or announcement and go with what we had at the time. Great flexibility here.
But another, arguably better, feature of this methodology was that the developers loved it. They wanted to follow it because it was so lightweight, required so little of them, and assumed such a high level of trust from them. It helped them to focus on just doing their work, and it was easy to see us progressing toward shipment.
Ultimately, Novell Forge was delivered on schedule with the full feature set, but having the flexibility along the way made it a lot less stressful. Really, does the process have to be more complicated than that?
Incentive Programs and Professional Motivation
I recall some years ago having read this Joel Spolsky post entitled "Incentive Pay Considered Harmful." In typical Spolsky fashion, he made a very compelling case that I had a very hard time coming to grips with. At the time I'd just recently made a series of contributions to Novell, outside of the scope of my job description, that I felt had an obvious positive impact on the company, but I had not received any sort of recognition at all for this.
Joel's article strongly disagreed with me. I felt I should have been recognized for my efforts; why else would I take the personal risk that comes with extending one's self in that fashion? Instead, nothing had been done.
I've been pretty interested in this topic and so I've been observing and evaluating ever since. What I've seen is that, for the most part, incentive programs tend to have the opposite effect of their purpose; namely, they tend to demotivate rather than motivate.
Outlined below are some reasons why.
People Get Rewarded Just For Doing Their Job
Maybe you've seen this too: You attend a quarterly all-hands meeting when an individual you know and work with is asked to come to the front. They announce, "Tom did a really great job running our beta program last release, and so we are giving him the quarterly Mega-Smiley-Face award, with a free dinner for two and a new iPod Touch." You sit there and stare, clapping insincerely. Sure, Tom did a great job running the beta program. But Tom is the beta program coordinator! It is his job to run the beta program! You did great work with your assigned responsibilities last quarter also. Why aren't you getting an award?
One problem with this scenario is that the rewards program now seems arbitrary. Tom was rewarded this quarter for doing his job. Who will be next? Who knows? Everyone is doing their job, but not everyone gets rewarded. The rest of the people feel discouraged. The recognition seems to be something worth earning, but they don't know how to earn it.
Another problem with this scenario is that it sends the wrong message to the team. Nobody should be rewarded just for completing their assignment — or everybody should be. Tom gets paid a salary for him to do his job. He's expected to do it well. If he gets rewarded just for doing his job, it tells the rest of the team that some jobs matter more than others. How do I get an assignment that matters? they wonder. And why am I wasting my time doing work that isn't important?
Luck and Timing Often Play a Bigger Role Than Excellence
In my nearly 15-year career I've delivered a lot of software to customers. I've invented things, learned difficult technology, solved complex problems, led and motivated teams, and proposed key strategic decisions that have helped the companies I've worked with become more successful. How many trophies do I have on my shelf? One. It is a beautiful blue piece of plastic that I got last summer when the Microsoft division we belong to eclipsed the $1 billion revenue mark for the first time. I'd been with Microsoft about six weeks at the time.
Earning that award (using that word very loosely, by the way) required nothing of me other than accepting a job at Microsoft at the right time. I hadn't written a line of production code yet, and certainly did not contribute to that revenue. I figure it replaces many others I should have earned but didn't.
I'm proud of the award, but I'm the first to admit that I earned it primarily due to fortuitous timing.
Positive Effects Are Usually Short-Lived
I remember one time being on a software team when we realized one morning that we had a serious bug. A problem had been reported by our released product that was having a significant customer impact. One member of the team stepped up to the task of digging into the issue and finding the problem. When he found and fixed the problem later in the day, he was genuinely thanked and given a gift certificate as a show of gratitude from the company.
The following day, our team was in a team meeting when the topic of the previous day's bug came up. We were being asked to identify what we were going to do differently in the future to avoid this problem. The same team member that had saved the team the day before took this as a personal offense. He muttered something to himself under his breath, slammed the lid of his laptop shut, picked it up and stormed out of the conference room.
Clearly, the special effort the company had made to thank him had been completely forgotten in less than 24 hours.
Abuse Makes The Programs Become Meaningless
I've frequently seen well-intentioned incentive programs become so abused that they become a laughing stock. One is almost embarrassed to truly earn an award because so many others have been similarly recognized for awards they acquired through dubious means.
Novell's Employee of the Year kind of took on this sentiment for me after I first saw one person get the award after taking full credit for something I knew full well he did not do (because I did it). Later I saw a colleague, the proven highest contributor on his team, get passed over for this award when his manager instead gave it to the lowest contributor in an effort to motivate him.
Probably the strangest of all of these though is for a company I've heard of with a very generous patent award program. One of their employees has learned to double his annual compensation by filing patent after patent through the company's patent program. Of course, filing all of these patents doesn't leave much time for him to do his regular job, and the patents being filed may not have anything at all to do with the company's products or strategy. But he's so innovative and valuable to their inconsistent patent portfolio that they've had no choice but to promote him over and over.
They Encourage Individualism Over Teamwork
Through a trusted source I heard about a support department for a software company years ago. They had instituted an incentive program called "55 Stay Alive". This great program with such a catchy name was very simple to understand: Each employee had to close 55 incidents each month or they were fired.
Awesome, huh.
One employee made a pitch to management, explaining that the incentive program was very negative and instead suggested they rearrange things to have a single goal that the whole team could work toward together. Management was hesitant but the team was willing to put it to a test.
So they did, with amazing results. Every month the team closed far more incidents working as a group than they ever had working as individuals before.
Logically, management was very unhappy about this insubordinate behavior so they fired the guy.
This might be an extreme example, but you've probably seen lots of examples of this. Many incentive programs reward people for behavior that leads them to focus on individual achievement at the expense of team achievement, which would be much better for the company.
So What Does Work Then?
One of the best incentive programs I've ever participated in was at Novell, working for a manager who ironically was really not that great of a manager. Novell being a Linux company, he had been given a small 6-inch figurine in the likeness of a Linux penguin which we called the Tuxen.
Every so often, in a team meeting, he would recognize someone for a specific assignment they'd done well. Then they would be awarded the Tuxen for a week. Each winner was expected to proudly display the Tuxen and take it places and publish photos of their adventures on the team wiki.
One week I was awarded the Tuxen for work I had done to help start the Eclipse Linux Tools project. I proudly took the Tuxen home with me, took photographs of the Tuxen with my family, and brought it along on a trip with my son to the Las Vegas Supercross. I gave it back a week later, but it was one of the most rewarding recognitions I'd ever had.
When I first read Joel's blog post, I immediately e-mailed him with my concerned, contrary opinion. "So how DO you reward people and make sure they know they've done well?" I asked.
"Just thank them," Joel said.
Maybe you're wondering, could it really be that simple? Instead I'd ask, why does it have to be more complicated?
The Truth About Novell Forge
I got an interesting e-mail the other day from Novell:
Please Note: You have been sent this email because you are listed as an administrator of one or more Novell Forge projects.
When Novell Forge was first launched Novell recognized the need for a site dedicated to providing hosting services to a growing number of software development projects, many supporting our open source initiatives. Novell Forge quickly grew and was soon providing these service to nearly 1000 such projects. Demand for new projects has all but disappeared during the past two years while a number of additional project hosting options have begun that can provide a similar set of services to those of Novell Forge.
Now that there are many other options, Novell can turn its focus to other areas and pass the project hosting responsibilities to these other dedicated hosting sites. Novell will be decommissioning the Novell Forge system on December 15, 2009.
This is interesting to me because it is not entirely true. I should know, because without me there would never have been a Novell Forge.
It's a bold statement, I know. It's one I'm happy to explain.
I came to Novell from IBM in 2000. It didn't take long to realize that Novell's developer story and strategy, or rather the complete lack thereof, was (and still is) a significant weakness in their overall execution. People buy a computer operating system in large part because of the applications that they can run on it; if a business wants to run a CRM system, they'll want to be sure that whatever platform they buy will run a CRM suite that is acceptable to them. This is why having a strong developer strategy is crucial to platform providers, and almost everyone seems to understand this. Novell certainly should; NetWare owned the x86 server market in the 80's and early 90's until Microsoft entered that market. Initially, the Microsoft offering was not necessarily better than NetWare in terms of stability or performance, but Microsoft definitely outgunned Novell when it came to applications. It was so much easier to create applications for Microsoft's platform that their supported portfolio dwarfed Novell's, and that was a significant key to dethroning Novell's dominant position in the x86 server market in the mid 90's.
Anyway, when I came to Novell and learned this, I thought that probably Novell's Developer Services organization just didn't know what to do (a mistaken analysis, I later learned) and if I worked there I could probably fix everything. I was pretty young, arrogant, and naive then. But in 2002 I was presented an opportunity to work in Developer Services and I took it.
One of the first things I was asked to do was to provide support to customers programming to eDirectory. I decided to try to learn more about how to do this the same way our third-party developers would, by using the resources that were available online. I found what appeared to be our authoritative how-to-program-to-eDirectory tutorial, got most of the way through my sample app, and got stuck. Finally I started asking questions. I quickly learned that everything I'd been doing was wrong; the authoritative documentation was incorrect. It used an out-of-date and deprecated API and was no longer considered best practice. It was some two or three years out of date, but hadn't been changed yet because changing the documentation was just too painful.
I felt this situation was unacceptable. We needed the freedom to create an abundance of rich and helpful developer content and to have it published and updated freely and frequently. We needed to be able to do this without going through drawn-out and tedious approval processes and staging phases for even minor edits. We needed to be able to continuously deliver not only whitepapers but tutorials and sample applications. I felt that what was needed was a complete overhaul of Novell's developer site, converting it into a web application where administrators (Novell Developer Services employees) could update the content and have complete control over what information was being provided to our developer community.
I discussed this with a colleague and my manager, and then we called a formal meeting to discuss this proposal. I think there were four Developer Services employees in the room. As we discussed the reasons to do this, other advantages surfaced. A key issue was that, in Novell's then-existing developer forums, many Novell developers were already contributing to solving each other's problems, including answering each other's questions and even sharing code, from small snippets to complete applications. We realized that instead of top-down support flowing from company to customer, what our customers really preferred was community support with Novell as an active participant. As we discussed this, one of my colleagues suggested that instead of writing the web app I suggested, we should do a project hosting site, like SourceForge. Such a site would allow us to participate as a community with our users to exchange sample code, documentation, tutorials, and other content. Novell Forge was born.
As we began to socialize the idea, we found out that a separate group within Novell had been tasked with creating a project hosting site for internal company use. When we both became aware of each other's goals, the synergies were obvious and it seemed apparent that we should try to coordinate our efforts. Interestingly, we had human resources to give to the project but lacked funding for capital expenses; the other group had capital expense budget but lacked human resources. Ultimately we agreed that, as my team developed the Novell Forge solution, we would also develop an internal-use version of the site to meet the goals of this team; in exchange, they would help us to get the hardware we needed to host Novell Forge.
Around the time Novell Forge was launched and completed, a number of people involved directly or indirectly from that team claimed credit for having launched Novell Forge. Some of them were quite handsomely rewarded by the company, presumably at least in part due to their claimed credit for the site. Others still claim in public that they are responsible for the site even though they had absolutely nothing to do with the conceptualization, proposal, approval, or implementation.
Meanwhile, those of us who did come up with the idea, who did make the business case and get the approval and deliver the site, well, we pretty much had to settle for a brief pat on the back from Novell. Or did we even get that? Anyway.
Novell Forge, despite its pretty lame name and humble beginnings, was actually quite well received by the press. It earned kudos for Novell from Dave Kearns of NetworkWorld, which was not exactly easy to come by. And as Novell tried to reinvent itself with an open source focus, purchasing such open source companies as Ximian and SUSE Linux, the existence of Novell Forge was frequently cited as evidence that Novell was serious about an open source strategy (example). Interest in the site grew quickly and it soon hosted over 1000 external projects, as stated in the e-mail I quoted above. My team was excited about the traction the site was gaining. We had many, many ideas for how to grow the site and make it an even more useful tool for software developers. We had more work to do than time to do it, and it was neat to feel like what we were doing had an impact to Novell.
Even though Novell didn't seem to care about it.
Oddly, in spite of what my team thought was a pretty obvious success, we could not get approval for funding to continue to promote the site. The team was gradually reduced in size, again and again. When people would leave, their vacancies would languish unfilled until that position was eventually lost. The team was instructed to not develop the site but instead to work on undefined new work in other undefined areas, wasting many person-years of development effort. The community could sense Novell's lack of investment and they lost interest. Novell Forge became a laughing stock. It was used as evidence of what a company does when they "just don't get" open source, when it was ironically used as evidence of Novell's good faith not too long before.
Things finally came to the point where there was only one employee assigned to maintain the site, along with other unrelated duties (I, and the rest of the team, had by now been reassigned to different projects). Novell Forge was completely unsupported by Novell's IT group, leaving instead the support of the site to this one individual. I recall an occasion where the site went down over the weekend and was out for a couple of days. It was obvious that the site was in demand, because users made Novell aware of the outage quite quickly. However, Novell was not willing to pay for 24/7 support for the site, so instead of being brought back online right away, the site was down for the entire weekend until that resource came in to work the next Monday. My manager brought this to the attention of our team with the insistence that we address it. He stated that from that point on, that one employee would be the primary off-hours maintenance person for the site, and I would be the backup.
I then asked if Novell was going to start reimbursing me for my cell phone bill. He said no. I asked if they were going to buy me an additional cell phone, pay that bill, and also pay me extra to carry that additional phone. He said no. He said they would just list my personal cell number in the emergency contact list, and would call it if there were an emergency. I stated that in that case I maintained the right to not answer. He stated that I would have to answer, that it was my assignment. I claimed that Novell could not require me to answer my personal cell phone if I'm the one paying the bill. I then reminded him that in Novell's support organization, at least at that time, people that were expected to respond 24/7 had their cell phone bill paid by Novell, were paid an additional amount to be on call, and were paid an additional amount if they actually took a call and worked that call during off hours. I said, "If the site is important to Novell, that is what Novell should do. If the site is important, it should be important enough that Novell is willing to pay in order to maintain uptime and keep our customers satisfied."
Novell was not willing to pay.
I shortly moved on to a different team within Novell, and the other guy left the company altogether. I'm not sure who has been maintaining the site since then.
What Novell chooses to do with their money and their human resources is their business. This isn't meant as a criticism; I don't claim to have the right experience to criticize their decision to strangle Novell Forge to death. This is simply meant as a statement of fact, and the facts are pretty clear:
- You get what you pay for.
- Novell did not pay for Novell Forge by giving due reward and recognition to those who truly brought this idea to the company.
- Novell did not pay for Novell Forge by feeding its success with additional funding, promotion, and development.
- Novell did not pay for Novell Forge by giving it the kind of support and maintenance that its customers expected.
- The customers of Novell Forge were initially enthusiastic, but grew to sense the lack of commitment by the company and thus stopped participating.
- Novell Forge died as a result.
Novell Forge may be planned for decommission this December, but it died years ago. And don't think you can fool me, Novell. Novell Forge did not die because of lack of interest by the user community. Novell Forge died because you did not care about it. Whether that was a good decision or not is not for me to decide, but please, Novell, at least be honest with your community. We did not kill Novell Forge — you did.
UPDATE: Dan Reese, a member of my team back then, corroborated this in his blog.
Actions Speak Louder Than Code
It took me a while, but I finally settled into my routine and got to where I'm reading my RSS feeds most days again. I was going through the posts of the past month or so, since the job change, and ran across this article on the "Making Good Software" blog about things that keep someone from being a good software engineer, outside of (and often in spite of) an ability to engineer software.
I'll summarize here. It isn't my intent to plagiarize; if you are remotely interested go read the article. Here are the things:
- Lack of discipline
- Big ego
- Poor communication
- Forgetting the customer
- Lack of proper work prioritization
I have known many of these people during my career. Indeed, I was one of them. I remember coming to Novell from IBM almost ten years ago. I thought I was pretty hot stuff and I made sure my team knew it. In fact, I actually said (this is embarrassing to admit) on more than one occasion, "There are people who know C++ better than I do, but I haven't met any of them." My ego surely made me hard to work with. It definitely was a cause of friction between myself and my management chain, and ended up being a (deserved) source of frustration and difficulty for me, until I recognized the problem and started working to address it.
I'm pretty ashamed of having behaved that way back then. I hope I'm better than that today. I guess recognizing the weakness is a good first step. Fortunately for me, back then I was on a really great team with a lot of very capable, patient, and talented engineers that waited for me to learn from my mistakes and to grant them the mutual respect they deserved. I consider myself pretty fortunate to have been able to learn from them what real software engineering is about.
Over my career I've had to work with people like this from time to time, software engineers that manifest one or more of these traits. Sometimes these guys are pretty talented technically. I've felt sorry for them as I've observed, realizing that these weaknesses are going to hold their career back until they recognize them and work to overcome them. No amount of programming prowess will compensate for it. And what's even worse is, often because these people have the personality issues they have, you don't get anywhere by trying to bring these weaknesses to their attention; they are often unreceptive to this type of feedback. Like I said, you just have to wait until they recognize it themselves.
I can imagine being in a performance review with someone like this, having them explain to me all the technical awesome they did, and me replying, "Your poor soft skills are shouting so loudly that I cannot hear your technical awesomeness." Or, as I said in the title, actions speak louder than code.
I really believe this is true. To write software professionally, of course you must have technical ability; however, this is a necessary but not sufficient condition for greatness. The best software engineers I've had the fortune to work with in my career, past and present, not only had awesome technical ability but did not exhibit weakness in these areas. And I'll tell you what: Those teams are wonderful teams to be a part of. Those teams create strong. uplifting work environments and are able to deliver great products that meet customer demand.
Another way to say this is, in order to be a good software engineer, you must first be a good employee.
In fact, I'll tell you how important I think this is. The ability to mitigate or eliminate these defects from a software engineer's persona is so important to me that, if I had my own company and were making the hiring decisions, I would not hire a candidate that I knew had these problems, no matter how incredible their technical ability.
A person with these weaknesses is really only suited to be set to the side to work on a special side R&D project where interaction with other employees is limited, and they don't have to interact with customers at all. Problem is, those kind of projects are either a) strategically important to the long-term future of the company, or b) of little to no real value, or c) a combination, often high potential value but with a lot of inherent risk that causes the real value to be low. If the project is strategically important or of high value, do you really want to reward the biggest jerk in your company by giving him the highest profile assignment, leaving your best engineers to maintain the legacy project? Wouldn't you want to have someone working on that high profile assignment that knows how to collaborate with others and assemble all the best ideas to solve the problem the best way, even if that solution isn't his/her own? Contrariwise, if the project is of little real value or has so much risk that it offsets the real value, why even do it at all?
Nope. In my company, if I were ever to have one, I wouldn't hire or keep an employee who had these weaknesses and was not committed to addressing them. I've seen the difference, both in morale and productivity, between teams where they don't have these problems and teams that do.
