There was an episode of the Simpsons in which Homer was the union negotiator dealing with Mr. Burns. Homer, sitting at home, receives a knock on the front door. When he asked who is there, the answer was "Hired goons!" I always found that really funny for some reason... maybe it took me back to the Saturday Night Live glory days, and the Land Shark.
What isn't so funny, is that those Hired Goons are knocking at the door on a project at one of my current clients right now. A recent thread in the XP Yahoo group about Technical Debt had me thinking about how as a team we've maxed our high-limit, high-interest credit card. We've tried to pay it down by refactoring, but the interest payments (just keeping the existing code running) are too high. In fact we aren't paying down the debt - it's increasing. The Hired Goons are at the door, and they've got baseball bats.
So, to avoid being beaten savagely by the Goons, what can be done? Well, it's time for bankruptcy and debt restructuring. I don't mean the wonderful Chapter 11 kind where life goes on and debts are magically written off, but the personal kind where you can conceivably lose your home, must sell off possessions, and have a note on your credit report about the bankruptcy for 6 years (in Canada). Let's apply that analogy to software development: the system is in ruins, the team breaks up, and you all bear the stigma of a failed project.
In our case, the system isn't quite in ruins. There are places where there is reasonably good test coverage. Not surprisingly, very few defects are found in that code. The problem is that a good 2/3 of the code has little or no tests. This includes the GUI itself, and the controller logic behind the GUI. As I mentioned in People Are Funny, the system started as a single application, and other lines of business have been folded in over time. The first application was a success, the second worked quite well (though it hasn't been used all that much in production), and the third line of business was a great success. In all of these cases, the overall system had remained small enough that the manual testing efforts were manageable. However, by the time the third line of business was completed, it was becoming more and more clear that we were incurring a significant amount of technical debt that would have to be paid off.
What has happened since, not unlike the credit card analogy, is that a number of factors have conspired to prevent us from performing refactoring work that we know needs to be done. First, a certain level of detachment from the customer has occurred because the main person was needed on a "problem child" project. Second, a rather aggressive schedule for implementing new lines of business was established. Third, the developers charged madly ahead with the new lines of business, without applying new techniques or changes to the older code. Finally, our beleaguered testers were still way behind the developers, so they weren't testing what we were working on.
When the first new line of business was just about completed, some of the team started on the second. They did perform some refactoring in order to include the next LoB. The problem is, that refactoring wasn't really supported by any tests and broke a whole whack (and this is Canada, so it's a metric whack!) of the functionality we had just completed in the first new line of business. The decision made was to fix the code, and not to roll it back to before the changes. Hindsight being a clear as it is... well, suffice to say that wasn't the best decision.
So, 8 months later we have just endured a lengthy testing process that uncovered numerous defects that simple unit tests should have caught. The deployment was challenged, and our credibility as a team took a pretty severe hit. Ain't life grand.
"Enough whining!! Give me something concrete!!"
I hear you. When I joined this particular project about 7 months after it started, I noticed (and voiced my concern about) 2 things right away - the use or hard classes rather than interfaces in most cases, and the lack of tests in the GUI and controller layers. In the 22 months since I started, some baby steps have been made towards using interfaces, though there are only a few more GUI and controller tests.
One of the developers feels very strongly that we should start using the Spring framework. I think that's a great idea, but I also feel that it wouldn't really make life much easier considering our current code. Our model layer isn't bad, but does rely too heavily on inheritance, IMO. That's OK, though, we can work with it.
One big problem is that we have classes to represent our Code Tables, and these classes are hard-coupled to the entire code table structure all the way to the EJB and database. It's a huge pain in the ass, and I'd love to rip as much of it out as possible. This is a place where even a marker interface could provde the type safety that necessitates the use of different types for each code table.
For example, why not have:
Our code could still refer to a Gender type, but the class that actually implements Gender is completely hidden. It could come from the database, XML, flat file, Mars, I don't care!! It would also mean that testing would be considerably easier. (I'm sure that the Smalltalk crowd is all shaking their heads about now. Please bear with me...) Anyway, this is a simple step that we could easily perform with the refactoring tools built into our IDE of choice.
This isn't the only place where we're in need of interfaces. We don't have any interfaces in our model layer, although that isn't hurting too much at the moment. Where it is a problem is in the GUI and controller logic in both our Swing and web applications. The Swing application has some nasty hard-coded references in many places, in addition to embedded business logic. Testing these classes is difficult because of the dependencies on other classes (which have dependencies on other classes, etc.). We can use jMock with classes to a certain extent, but even that can't be done in all cases. So, the introduction of interfaces for some of the classes would simplify the testing. So, this is definitely a refactoring target, but we're leaving it alone until we have stories that require changes to the Swing app.
The web application, however, is where virtually all of the development is occurring at the moment. It's using the Struts framework, which is another place in desperate need of interfaces. Testing the Struts actions isn't easy, although using jMock has helped somewhat. However, these tests were written for precious few Actions (which, not surprisingly, have had few if any defects reported). Older code was not retrofitted with tests as we learned how to mock out dependencies (sounds like debt to me).
Then, just to make things really interesting, a couple of developers decided that we needed to add a new abstraction on top of the HTTP Session. This isn't a bad thing in an of itself, but I didn't believe that introducing it late in the release supported by limited tests was such a great idea. I lost that battle, their code was checked in, and all sorts of things that once worked began to break. Unfortunately, the debt we had accrued by not having tests in the controller layer (Actions) was coming back to bite us at this point. The developers making the change should have been able to push a button and see if anything had broken. So, the problems were found in a very slow, sequential process of finding one bug, fixing it, finding the next, fixing it, etc. Even more debt was accrued when the developers decided that it was too time-consuming and/or too difficult to always write tests to expose the bugs. Sigh.
Since we have to start adding more functionality to the web application, we essentially do have stories that support the refactorings that are necessary. Put simply, if we don't add tests and refactor properly, it's going to take even longer to develop the next lines of business.
One issue that has arisen several times is that we are trying to perform too many large refactorings. During the worst of these, the application was dead for 6 weeks. That means that we couldn't run the application, let alone test it, for 6 weeks. That's 30 working days. This was during a large database refactoring that wouldn't have been possible without the test we do have in our EJB and data layer. This was an extreme case, but our team history is rife with examples of multi-day refactorings. Put simply, we need to make all (and I mean ALL) of our refactorings in tiny, incremental steps. I have been shouting that message from the hilltops for months, and I believe that some people are now beginning to listen. Big, long refactorings bad... small, incremental refactorings good. It isn't rocket science (unless you're Bil Kleb).
So, to get to the Promised Land of defect-free code, what do we need to do right now? Here are my ideas:
What isn't so funny, is that those Hired Goons are knocking at the door on a project at one of my current clients right now. A recent thread in the XP Yahoo group about Technical Debt had me thinking about how as a team we've maxed our high-limit, high-interest credit card. We've tried to pay it down by refactoring, but the interest payments (just keeping the existing code running) are too high. In fact we aren't paying down the debt - it's increasing. The Hired Goons are at the door, and they've got baseball bats.
So, to avoid being beaten savagely by the Goons, what can be done? Well, it's time for bankruptcy and debt restructuring. I don't mean the wonderful Chapter 11 kind where life goes on and debts are magically written off, but the personal kind where you can conceivably lose your home, must sell off possessions, and have a note on your credit report about the bankruptcy for 6 years (in Canada). Let's apply that analogy to software development: the system is in ruins, the team breaks up, and you all bear the stigma of a failed project.
In our case, the system isn't quite in ruins. There are places where there is reasonably good test coverage. Not surprisingly, very few defects are found in that code. The problem is that a good 2/3 of the code has little or no tests. This includes the GUI itself, and the controller logic behind the GUI. As I mentioned in People Are Funny, the system started as a single application, and other lines of business have been folded in over time. The first application was a success, the second worked quite well (though it hasn't been used all that much in production), and the third line of business was a great success. In all of these cases, the overall system had remained small enough that the manual testing efforts were manageable. However, by the time the third line of business was completed, it was becoming more and more clear that we were incurring a significant amount of technical debt that would have to be paid off.
What has happened since, not unlike the credit card analogy, is that a number of factors have conspired to prevent us from performing refactoring work that we know needs to be done. First, a certain level of detachment from the customer has occurred because the main person was needed on a "problem child" project. Second, a rather aggressive schedule for implementing new lines of business was established. Third, the developers charged madly ahead with the new lines of business, without applying new techniques or changes to the older code. Finally, our beleaguered testers were still way behind the developers, so they weren't testing what we were working on.
When the first new line of business was just about completed, some of the team started on the second. They did perform some refactoring in order to include the next LoB. The problem is, that refactoring wasn't really supported by any tests and broke a whole whack (and this is Canada, so it's a metric whack!) of the functionality we had just completed in the first new line of business. The decision made was to fix the code, and not to roll it back to before the changes. Hindsight being a clear as it is... well, suffice to say that wasn't the best decision.
So, 8 months later we have just endured a lengthy testing process that uncovered numerous defects that simple unit tests should have caught. The deployment was challenged, and our credibility as a team took a pretty severe hit. Ain't life grand.
"Enough whining!! Give me something concrete!!"
I hear you. When I joined this particular project about 7 months after it started, I noticed (and voiced my concern about) 2 things right away - the use or hard classes rather than interfaces in most cases, and the lack of tests in the GUI and controller layers. In the 22 months since I started, some baby steps have been made towards using interfaces, though there are only a few more GUI and controller tests.
One of the developers feels very strongly that we should start using the Spring framework. I think that's a great idea, but I also feel that it wouldn't really make life much easier considering our current code. Our model layer isn't bad, but does rely too heavily on inheritance, IMO. That's OK, though, we can work with it.
One big problem is that we have classes to represent our Code Tables, and these classes are hard-coupled to the entire code table structure all the way to the EJB and database. It's a huge pain in the ass, and I'd love to rip as much of it out as possible. This is a place where even a marker interface could provde the type safety that necessitates the use of different types for each code table.
For example, why not have:
public interface Gender {}
public class TestGender implements Gender
{
:
:
}
public class CodeTableBasedGender implements Gender
{
:
:
}Our code could still refer to a Gender type, but the class that actually implements Gender is completely hidden. It could come from the database, XML, flat file, Mars, I don't care!! It would also mean that testing would be considerably easier. (I'm sure that the Smalltalk crowd is all shaking their heads about now. Please bear with me...) Anyway, this is a simple step that we could easily perform with the refactoring tools built into our IDE of choice.
This isn't the only place where we're in need of interfaces. We don't have any interfaces in our model layer, although that isn't hurting too much at the moment. Where it is a problem is in the GUI and controller logic in both our Swing and web applications. The Swing application has some nasty hard-coded references in many places, in addition to embedded business logic. Testing these classes is difficult because of the dependencies on other classes (which have dependencies on other classes, etc.). We can use jMock with classes to a certain extent, but even that can't be done in all cases. So, the introduction of interfaces for some of the classes would simplify the testing. So, this is definitely a refactoring target, but we're leaving it alone until we have stories that require changes to the Swing app.
The web application, however, is where virtually all of the development is occurring at the moment. It's using the Struts framework, which is another place in desperate need of interfaces. Testing the Struts actions isn't easy, although using jMock has helped somewhat. However, these tests were written for precious few Actions (which, not surprisingly, have had few if any defects reported). Older code was not retrofitted with tests as we learned how to mock out dependencies (sounds like debt to me).
Then, just to make things really interesting, a couple of developers decided that we needed to add a new abstraction on top of the HTTP Session. This isn't a bad thing in an of itself, but I didn't believe that introducing it late in the release supported by limited tests was such a great idea. I lost that battle, their code was checked in, and all sorts of things that once worked began to break. Unfortunately, the debt we had accrued by not having tests in the controller layer (Actions) was coming back to bite us at this point. The developers making the change should have been able to push a button and see if anything had broken. So, the problems were found in a very slow, sequential process of finding one bug, fixing it, finding the next, fixing it, etc. Even more debt was accrued when the developers decided that it was too time-consuming and/or too difficult to always write tests to expose the bugs. Sigh.
Since we have to start adding more functionality to the web application, we essentially do have stories that support the refactorings that are necessary. Put simply, if we don't add tests and refactor properly, it's going to take even longer to develop the next lines of business.
One issue that has arisen several times is that we are trying to perform too many large refactorings. During the worst of these, the application was dead for 6 weeks. That means that we couldn't run the application, let alone test it, for 6 weeks. That's 30 working days. This was during a large database refactoring that wouldn't have been possible without the test we do have in our EJB and data layer. This was an extreme case, but our team history is rife with examples of multi-day refactorings. Put simply, we need to make all (and I mean ALL) of our refactorings in tiny, incremental steps. I have been shouting that message from the hilltops for months, and I believe that some people are now beginning to listen. Big, long refactorings bad... small, incremental refactorings good. It isn't rocket science (unless you're Bil Kleb).
So, to get to the Promised Land of defect-free code, what do we need to do right now? Here are my ideas:
- Get at the very least some interaction level tests in place in the web application using a tool such as Selenium, etc. This isn't perfect, but provides a baseline from which we can measure
- We do need to implement the abstraction of the HTTP Session, but we need tests surrounding everywhere that refactoring is performed. The tests also need to be written and tested first in order to prove that the change hasn't broken the code.
- We're going to need to apply the new testing and coding techniques to older code that isn't using them. That's a wad of debt that's going to bite us if we don't address it.
- Finally, we need to communicate all this to our customer. It might not be as specific, but more along the lines of, "In order to add new functionality, we need to fix some plumbing as we go. Therefore, it's going to take us longer for this release.