I came across a post yesterday that really made me think. Giles Bowkett posted a take-down of scrum (a popular form of agile software development) and in fact of agile processes in general. The post made it into my twitter feed (I forget how) and I read it, ironically enough, right before I headed into a sprint review and planning meeting.
What got me thinking was how much of it I agree with, even though I don’t mind scrum as a process and in fact think there are aspects of the agile approach that scientific projects could really benefit from adopting. This is obviously a bit of a disconnect, and I’ve been thinking about how I can both agree with large parts of a post that eviscerates scrum and like scrum enough that there are aspects I’d take and transplant into a completely different sort of project.
I think it comes down to how I view process. I am almost completely process agnostic as a project manager. I have used agile methods, waterfall methods, weird homegrown methods, and even weirder amalgamations of all three. Even when I’m part of a team that is technically using scrum (for instance), we are not using The One True Scrum Process. We always tweak it. To me, the best process is the one that makes the team happy and productive, and since each team is unique, I guess it doesn’t surprise me that each team eventually settles into its own unique process. One of my most important roles as project manager is to help the team find its best process.
So I read Giles Bowkett’s post and agreed that all of the issues he describes with scrum are real issues. I have also seen teams fall into similar pathologies. I don’t really place the blame for those issues on the process, though. I think those teams would have some manifestation of the same issues no matter what process they used. Sure, the manifestations would be different- there probably wouldn’t be any long, obnoxious meetings in which only the manager gets to sit down, for instance- but there would be some manifestation of the issue of a manager too full of his or her place in the corporate hierarchy and too lacking in respect for the team. There would be no farcical version of planning poker, but there would still be an overbearing dominant team member who shut down dissent from other people. There wouldn’t be meaningless velocity charts, but there would still be managers looking for a concrete measure of progress in the time between asking for a feature and delivery, instead of just taking the team at its word for whether or not they are on track.
In short, I view those issues as problems with management and/or culture, not with the process. The process is just determining how the problems are displayed.
The thing is, doing a good job at management is hard. Fixing a dysfunctional corporate culture is harder still. People want magic solutions. I have yet to find any. I recently had someone ask me for my “top tips for becoming a better manager.” I was a bit flummoxed. Everything I came up with sounds like an empty platitude. Listen to your team. Respect your team. Protect them from corporate politics as much as you can. Facilitate good communication. Try to understand their motivations and understand and respect how events will make them feel, even if your own reaction would be quite different. Don’t agree to impossible timelines. And so on.
Everything I came up with sounds like simple common sense, and in a way, it is just common sense. It is putting these ideas into practice that is the hard part. How do you know if a timeline is impossible? How do you protect the team from politics when one of the team members is embroiled in the latest politics-driven crisis? How do you understand the motivations of someone who responds to everything in a way that truly makes no sense to you?
I have no easy answers for those questions, in part because the best answer will change from company to company, and even from team to team within one company. But if you can’t do these things at least somewhat well it won’t matter what process you pick to manage your projects. No process will hide your failings. The best you can hope for from a process is that it will help you identify your weaknesses and your team’s weaknesses, so that you can work to make them better. The great thing about agile processes is that they all include a mechanism for feedback and a chance to improve. The terrible thing is that most teams ignore these mechanisms, and very few managers or teams want to take the feedback that does make it through and improve. No process in the world can solve that.
An aside that is funny only in retrospect: I put the word “usually” in the title because there was one time when a large part of my role was to help protect project teams from a ridiculous and overbearing process- basically, I ran interference with the requirements of the process so that the team could actually do their work. In that case, the process really was a large part of the problem.
Long time reader, first time commenter!
I was recently promoted (read, thrown into) a project management role in a very very complex/challenging (er, and dysfunctional) software dev project…. so I’ve been poring over your blog. My PM manager is asking me to come up with “concrete measure of progress”… which is how I stumbled on this comment in your post:
There wouldn’t be meaningless velocity charts, but there would still be managers looking for a concrete measure of progress in the time between asking for a feature and delivery, instead of just taking the team at its word for whether or not they are on track.
So what’s wrong with asking for that?
Here’s my situation:
My team has a big, overbearing regulatory process to deal with (lots of documentation!) and lots of dependencies on other teams. The dev manager is a bit fly-by-the-seat of his pants, but has good instincts about feature and implementation priorities IMO. In the 1.5 years I’ve observed them (not always as a PM), it’s been a huge struggle to forecast their schedule, in part because of the external dependencies (IE, they receive code that they need to ‘commercialize’, and that can balloon out based on the quality of the received code), but also in part because there’s very little tracking or planning of any sort, waterfall or agile. They are consistently way behind schedule, and right now, we are doing a replan and I’m very nervous they are once again overcommitting themselves.
I was asked to serve as an (inexperienced) agile coach, and pretty much failed to introduce many scrum practices, in part because of the preexisting overbearing process, in part because of lack of upper management support, and no doubt in part due to my own experience.
So we have a team where I have very little visibility into who’s overburdened, who’s got it light, how a given “sprint” is progressing (for example– no definition of ‘done done’, and people often fail to close issues after implementing them because of the documentation overhead associated with closing issues), or whether the time estimates people give me for a given ‘commericalization’ feature are realistic. At this point I’m just not sure how to effect change; I have a dev exec offering to groom + size the issues to reflect the remainder of the release, but I’m not sure of the top-down approach. I was almost thinking kaban might be one approach to just get a grip on what the team is doing at an individual level, since they are overwhelmed by scrum. Did you have any thoughts?
Sorry, I expect that this comment is waaaay too long (and there’s even more context I’m leaving out), but maybe just to narrow it all down again — I’m being asked for concrete measures. If that’s a bad ask, then how to increase transparency of progress and get some harder data so that I can push back against overcommitting?
Your comment isn’t too long! I’m not sure I can help, but I will try.
First, there is nothing wrong with wanting concrete measures of progress, as long as those measures are appropriate to the team, and the team buys in to them. I.e., if they are real measures not meaningless charts. This is one reason I like milestones, because they are a pretty natural way to check if you’re tracking to your plan. They aren’t perfect, either- you can meet three milestones, no problem, and then completely derail on the next one. (For that matter, there’s nothing wrong with a velocity chart, if you have a team for which it is meaningful. They just get used a lot with teams that aren’t anywhere near mature enough in their use of scrum techniques for the velocity chart to be meaningful.) Managers will always need concrete measures of progress. Part of the art of being a project manager is figuring out how to get meaningful measures you are willing to commit to!
Now, to your specific issue. It doesn’t sound like your team has a process that is able to give you meaningful concrete measures. So I’d instead try to just get an accurate reading of a communal gut instinct about whether the plan is reasonable and if the team is on track, and focus your effort on improving their process to the point where meaningful concrete measures might be possible. Maybe something as simple as asking everyone to rate the project as being green (on track), yellow (at risk), or red (in trouble) and then feeding that into your own global rating.
From what you’ve put in this comment, my first priority would be to work on being able to call things “done done.” I can’t tell from your comment why this is not possible right now, but it is a fundamental issue, i.e., if you can’t reliably declare some piece of work done, no other process improvement is really going to stick. So I’d start by trying to understand why you can’t reliably declare something done. Is there not enough testing in the process? Are the stories too big? You don’t want a story that says something like “commercialize external product X.” That’s too big. You want to break it down into smaller bits of work. For instance, it could start with “analyze product X and produce a list of issues” and then the list of issues would be your stories. Someone other than the developer who works on each of those stories should test the result before closing the story. If that sounds impossible because you can’t test in pieces, I’d try to brainstorm a test plan with the engineers. Try to introduce the concept that once a piece of work is declared “done” it doesn’t get re-opened. Instead, bugs get filed against it. Even this little bit of discipline will help move the team towards a better process that give more visibility into the true status of the project.
As for what framework to use: I don’t think it matters. It sounds like the team will have a learning curve in any process, so there is no process you can apply that will fix the issues “out of the box.” Maybe call the team together and describe the options you see, and ask what feels more natural to them, or what they most want to be able to say they know how to do (being able to put “scrum” on their resume might be a selling point). Whatever framework you all pick, just start working in it and try to get better at using it.
Does that help? If not, ask a follow up question and I’ll try again! (Also, FYI: I edited your first sentence slightly for privacy reasons- email if you want to know more.)
Thank you so much for the thoughtful reply!
So not getting successfully to done-done — thank you for highlighting that as a major roadblock, it jives with what I’m hearing from my more experience PM colleague/mentor (who, despite being absolutely tip-top and having bucketloads of experience, has very much struggled to get done-done defined and adhered to with her project, and has also failed to make happen two integration milestones she proposed….which helps me to take my own failures to get things done less personally. it’s just a very, very challenging environment).
I would say we do have a definition of done-done defined by our mandatory regulatory process — it’s just a huge challenge to get people to adhere to it. yesterday I sat down and combed through all our issues and came up with some metrics to expose our lack of issue management (stuff like –we close 10 issues a month in the team, but are now proposing closing 100 issues in the next 2 months! Pretty sure the 10/issues a month doesn’t capture all the work actually done). I also now feel like I have dev manager support to start enforcing issue management, including the regulatory process that it takes to get each issue to done-done (formerly, I didn’t feel like I had that support, so all my efforts to bug the team to create and close issues seemed to fizzle out). I’m going to hold some training sessions on issue management and lean heavily on the dev manager to enforce. I know this doesn’t match the collaborative approach you were suggesting … but since it’s a regulated, mandated process, I see no other way that ‘hey, guys, we really need to stick to these rules’. We’ll see what comes of it!
That does sound like a tough environment. You could try visualizing the workflow, including the regulatory step, and perhaps even somehow highlighting issues that made it to the regulatory step and then got sent back (maybe with a different color?) both so that you can start to highlight the bottlenecks (and maybe get the team to start thinking about how to avoid getting stuck there so much) and so that you can get a feel for what your team’s throughput is minus that regulatory check. Exactly how I’d manage the regulatory process depends on why it is there. I’ve worked places where there was a heavy process on my project just because *some* projects required it and all projects had to use that process. In those cases, I insulated the team from the process as much as I could, and jumped through the paperwork hoops for them. In other cases, the regulatory process is there for a damn good reason, and even if it is painful, the checks it provides are worth the pain. In those cases, I tell the story of therac 25 (as an example of a tragedy that could have been avoided by better software processes) and try to get them to buy in to the process and to think about how to work with it instead of resenting it. Good luck! Improving process is definitely hard.